Package: reedsolo
A pythonicuniversal errors-and-erasures Reed-Solomon Codecto protect your data from errors and bitrot. It includes a pure python implementation and an optional speed-optimized Cython/C extension.This is a burst-type implementation, so that it supports any Galois field higher than 2^3, but not binary streams. Burst errors are non-random errors that more often happen on data storage mediums such as hard drives, hence this library is better suited for data storage protection, and less for streams noise correction, although it also works for this purpose but with a bit of overhead (since it works with bytes only, instead of bits).Based on the wonderful tutorial atWikiversity, written by “Bobmath” and “LRQ3000”. If you are just starting with Reed-Solomon error correction codes, the Wikiversity article is a good beginner’s introduction.Table of contentsInstallationUsageBasic usage with high-level RSCodec classLow-level usage via direct access to math functionsExtended descriptionEdge casesRecommended readingAuthorsLicenseInstallationFor the latest stable release, install with:pipinstall--upgradereedsoloFor the latest development release (do not use in production!), use:pipinstall--upgradegit+https://github.com/tomerfiliba/reedsolomonIf you have some issues installing through pip, maybe this command may help:pipinstallreedsolo--no-binary={reedsolo}By default, only a pure-python implementation is installed. If you have Cython and a C++ compiler, a faster cythonized binary can be optionally built with:pipinstall--upgradereedsolo--install-option="--cythonize"--verboseor locally with:pythonsetup.pyinstall--cythonizeThe setup.py will then try to build the Cython optimized modulecreedsolo.pyxif Cython is installed, which can then be imported asimport creedsoloinstead ofimport reedsolo, with the same features between both modules.As an alternative, usecondato install a compiled version for various platforms:condainstall-cconda-forgereedsoloUsageBasic usage with high-level RSCodec class# Initialization>>>fromreedsoloimportRSCodec,ReedSolomonError>>>rsc=RSCodec(10)# 10 ecc symbols# Encoding# just a list of numbers/symbols:>>>rsc.encode([1,2,3,4])b'\x01\x02\x03\x04,\x9d\x1c+=\xf8h\xfa\x98M'# bytearrays are accepted and the output will be matched:>>>rsc.encode(bytearray([1,2,3,4]))bytearray(b'\x01\x02\x03\x04,\x9d\x1c+=\xf8h\xfa\x98M')# encoding a byte string is as easy:>>>rsc.encode(b'hello world')b'hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa'# Note: strings of any length, even if longer than the Galois field, will be encoded as well using transparent chunking.# Decoding (repairing)>>>rsc.decode(b'hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa')[0]# originalb'hello world'>>>rsc.decode(b'heXlo worXd\xed%T\xc4\xfdX\x89\xf3\xa8\xaa')[0]# 3 errorsb'hello world'>>>rsc.decode(b'hXXXo worXd\xed%T\xc4\xfdX\x89\xf3\xa8\xaa')[0]# 5 errorsb'hello world'>>>rsc.decode(b'hXXXo worXd\xed%T\xc4\xfdXX\xf3\xa8\xaa')[0]# 6 errors - failTraceback(mostrecentcalllast):...reedsolo.ReedSolomonError:Toomany(orfew)errorsfoundbyChienSearchfortheerratalocatorpolynomial!Important upgrade notice for pre-1.0 users:Note thatRSCodec.decode()returns 3 variables:the decoded (corrected) messagethe decoded message and error correction code (which is itself also corrected)and the list of positions of the errata (errors and erasures)Here is how to use these outputs:>>>tampered_msg=b'heXlo worXd\xed%T\xc4\xfdX\x89\xf3\xa8\xaa'>>>decoded_msg,decoded_msgecc,errata_pos=rsc.decode(tampered_msg)>>>print(decoded_msg)# decoded/corrected messagebytearray(b'hello world')>>>print(decoded_msgecc)# decoded/corrected message and ecc symbolsbytearray(b'hello world\xed%T\xc4\xfd\xfd\x89\xf3\xa8\xaa')>>>print(errata_pos)# errata_pos is returned as a bytearray, hardly intelligiblebytearray(b'\x10\t\x02')>>>print(list(errata_pos))# convert to a list to get the errata positions as integer indices[16,9,2]Since we failed to decode with 6 errors with a codec set with 10 error correction code (ecc) symbols, let’s try to use a bigger codec, with 12 ecc symbols.>>>rsc=RSCodec(12)# using 2 more ecc symbols (to correct max 6 errors or 12 erasures)>>>rsc.encode(b'hello world')b'hello world?Ay\xb2\xbc\xdc\x01q\xb9\xe3\xe2='>>>rsc.decode(b'hello worXXXXy\xb2XX\x01q\xb9\xe3\xe2=')[0]# 6 errors - ok, but any more would failb'hello world'>>>rsc.decode(b'helXXXXXXXXXXy\xb2XX\x01q\xb9\xe3\xe2=',erase_pos=[3,4,5,6,7,8,9,10,11,12,15,16])[0]# 12 erasures - OKb'hello world'This shows that we can decode twice as many erasures (where we provide the location of errors ourselves) than errors (with unknown locations). This is the cost of error correction compared to erasure correction.To get the maximum number of errorsorerasures that can be independently corrected (ie, not simultaneously):>>>maxerrors,maxerasures=rsc.maxerrata(verbose=True)Thiscodeccancorrectupto6errorsand12erasuresindependently>>>print(maxerrors,maxerasures)612To get the maximum number of errorsanderasures that can be simultaneously corrected, you need to specify the number of errors or erasures you expect:>>>maxerrors,maxerasures=rsc.maxerrata(erasures=6,verbose=True)# we know the number of erasures, will calculate how many errors we can affordThiscodeccancorrectupto3errorsand6erasuressimultaneously>>>print(maxerrors,maxerasures)36>>>maxerrors,maxerasures=rsc.maxerrata(errors=5,verbose=True)# we know the number of errors, will calculate how many erasures we can affordThiscodeccancorrectupto5errorsand2erasuressimultaneously>>>print(maxerrors,maxerasures)52Note that if a chunk has more errors and erasures than the Singleton Bound as calculated by themaxerrata()method, the codec will try to raise aReedSolomonErrorexception,
but may very well not detect any error either (this is a theoretical limitation of error correction codes). In other words, error correction codes are unreliable to detect if a chunk of a message
is corrupted beyond the Singleton Bound. If you want more reliability in errata detection, use a checksum or hash such as SHA or MD5 on your message, these are much more reliable and have no bounds
on the number of errata (the only potential issue is with collision but the probability is very very low).Note: to catch aReedSolomonErrorexception, do not forget to import it first with:from reedsolo import ReedSolomonErrorTo check if a message is tampered given its error correction symbols, without decoding, use thecheck()method:# Checking>>rsc.check(b'hello worXXXXy\xb2XX\x01q\xb9\xe3\xe2=')# Tampered message will return False[False]>>rmes,rmesecc,errata_pos=rsc.decode(b'hello worXXXXy\xb2XX\x01q\xb9\xe3\xe2=')>>rsc.check(rmesecc)# Corrected or untampered message will return True[True]>>print('Number of detected errors and erasures:%i, their positions:%s'%(len(errata_pos),list(errata_pos)))Numberofdetectederrorsanderasures:6,theirpositions:[16,15,12,11,10,9]By default, most Reed-Solomon codecs are limited to characters that can be encoded in 256 bits and with a length of maximum 256 characters. But this codec is universal, you can reduce or increase the length and maximum character value by increasing the Galois Field:# To use longer chunks or bigger values than 255 (may be very slow)>>rsc=RSCodec(12,nsize=4095)# always use a power of 2 minus 1>>rsc=RSCodec(12,c_exp=12)# alternative way to set nsize=4095>>mes='a'*(4095-12)>>mesecc=rsc.encode(mes)>>mesecc[2]=1>>mesecc[-1]=1>>rmes,rmesecc,errata_pos=rsc.decode(mesecc)>>rsc.check(mesecc)[False]>>rsc.check(rmesecc)[True]Note that theRSCodecclass supports transparent chunking, so you don’t need to increase the Galois Field to support longer messages, but characters will still be limited to 256 bits (or
whatever field you set withc_exp).Low-level usage via direct access to math functionsIf you want full control, you can skip the API and directly use the library as-is. Here’s how:First you need to init the precomputed tables:>>importreedsoloasrs>>rs.init_tables(0x11d)Pro tip: if you get the error: ValueError: byte must be in range(0, 256), please check that your prime polynomial is correct for your field.
Pro tip2: by default, you can only encode messages of max length and max symbol value = 256. If you want to encode bigger messages,
please use the following (where c_exp is the exponent of your Galois Field, eg, 12 = max length 2^12 = 4096):>>prim=rs.find_prime_polys(c_exp=12,fast_primes=True,single=True)>>rs.init_tables(c_exp=12,prim=prim)Let’s define our RS message and ecc size:>>n=255# length of total message+ecc>>nsym=12# length of ecc>>mes="a"*(n-nsym)# generate a sample messageTo optimize, you can precompute the generator polynomial:>>gen=rs.rs_generator_poly_all(n)Then to encode:>>mesecc=rs.rs_encode_msg(mes,nsym,gen=gen[nsym])Let’s tamper our message:>>mesecc[1]=0To decode:>>rmes,recc,errata_pos=rs.rs_correct_msg(mesecc,nsym,erase_pos=erase_pos)Note that both the message and the ecc are corrected (if possible of course).
Pro tip: if you know a few erasures positions, you can specify them in a listerase_posto double the repair power. But you can also just specify an empty list.You can check how many errors and/or erasures were corrected, which can be useful to design adaptive bitrate algorithms:>>print('A total of%ierrata were corrected over all chunks of this message.'%len(errata_pos))If the decoding fails, it will normally automatically check and raise a ReedSolomonError exception that you can handle.
However if you want to manually check if the repaired message is correct, you can do so:>>rs.rs_check(rmes+recc,nsym)Note: if you want to use multiple reedsolomon with different parameters, you need to backup the globals and restore them before calling reedsolo functions:>>rs.init_tables()>>globalgf_log,gf_exp,field_charac>>bak_gf_log,bak_gf_exp,bak_field_charac=gf_log,gf_exp,field_characThen at anytime, you can do:>>globalgf_log,gf_exp,field_charac>>gf_log,gf_exp,field_charac=bak_gf_log,bak_gf_exp,bak_field_charac>>mesecc=rs.rs_encode_msg(mes,nsym)>>rmes,recc,errata_pos=rs.rs_correct_msg(mesecc,nsym)The globals backup is not necessary if you use RSCodec, it will be automatically managed.Read the sourcecode’s comments for more info about how it works, and for the various parameters you can setup if
you need to interface with other RS codecs.Extended descriptionThe code of wikiversity is here consolidated into a nice API with exceptions handling.
The algorithm can correct up to 2*e+v <= nsym, where e is the number of errors,
v the number of erasures and nsym = n-k = the number of ECC (error correction code) symbols.
This means that you can either correct exactly floor(nsym/2) errors, or nsym erasures
(errors where you know the position), and a combination of both errors and erasures.
This is called the Singleton Bound, and is the maximum/optimal theoretical number
of erasures and errors any error correction algorithm can correct (although there
are experimental approaches to go a bit further, named list decoding, not implemented
here, but feel free to do pull request!).
The code should work on pretty much any reasonable version of python (2.4-3.7),
but I’m only testing on 2.7 and 3.7. Python 3.8 should work except for Cython which is
currently incompatible with this version.The codec has quite reasonable performances if you either use PyPy on the pure-python
implementation (reedsolo.py) or either if you compile the Cython extension creedsolo.pyx
(which is about 2x faster than PyPy). You can expect encoding rates of several MB/s.This library is also thoroughly unit tested so that nearly any encoding/decoding case should be covered.The codec is universal, meaning that it can decode any message encoded by another RS encoder
as long as you provide the correct parameters.
Note however that if you use higher fields (ie, bigger c_exp), the algorithms will be slower, first because
we cannot then use the optimized bytearray() structure but only array.array(‘i’, …), and also because
Reed-Solomon’s complexity is quadratic (both in encoding and decoding), so this means that the longer
your messages, the longer it will take to encode/decode (quadratically!).The algorithm itself can handle messages of a length up to (2^c_exp)-1 symbols per message (or chunk), including the ECC symbols,
and each symbol can have a value of up to (2^c_exp)-1 (indeed, both the message length and the maximum
value for one character is constrained by the same mathematical reason). By default, we use the field GF(2^8),
which means that you are limited to values between 0 and 255 (perfect to represent a single hexadecimal
symbol on computers, so you can encode any binary stream) and limited to messages+ecc of maximum
length 255. However, you can “chunk” longer messages to fit them into the message length limit.
TheRSCodecclass will automatically apply chunking, by splitting longer messages into chunks and
encode/decode them separately; it shouldn’t make a difference from an API perspective (ie, from your POV).To use the Cython implementation, you need topip install cythonand to install a C++ compiler (Microsoft Visual C++ 14.x for Windows and Python 3.10+), read the up-to-date instructions in theofficial wiki. Then you can simply cd to the root of the folder where creedsolo.pyx is, and typepython setup.py build_ext--inplace--cythonize. Alternatively, you can generate just the C++ code by typingcython -3 creedsolo.pyx. When building a distributable egg or installing the module from source, the Cython module can be transpiled and compiled if both Cython and a C compiler are installed and the--cythonizeflag is supplied to the setup.py, otherwise by default only the pure-python implementation and the.pyxcython source code will be included, but the binary won’t be in the wheel.Then, useimport RSCodec from creedsoloinstead of importing from thereedsolomodule, and finally only feedbytearray()objects to theRSCodecobject. Exclusively using bytearrays is one of the reasons creedsolo is faster than reedsolo. You can convert any string by specifying the encoding:bytearray(“Hello World”, “UTF-8”).Note that there is an inherent limitation of the C implementation which cannot work with higher galois fields than 8 (= characters of max 255 value) because the C implementation only works with bytearrays, and bytearrays only support characters up to 255. If you want to use higher galois fields, you need to use the pure python version, which includes a fake_bytearrayfunction that overloads the standard bytearray in case galois fields higher than 8 are used toinit_tables(), or rewrite the C implementation to use lists instead of bytearrays (which will be MUCH slower so this defeats the purpose and you are better off simply using the pure python version under PyPy - an older version of the C implementation was doing just that, and without bytearrays, all performance gains were lost, hence why the bytearrays were kept despite the limitations).Edge casesAlthough sanity checks are implemented whenever possible and when they are not too much resource consuming, there are a few cases where messages will not be decoded correctly without raising an exception:If an incorrect erasure location is provided, the decoding algorithm will just trust the provided locations and create a syndrome that will be wrong, resulting in an incorrect decoded message. In case reliability is critical, always use the check() method after decoding to check the decoding did not go wrong.Reed-Solomon algorithm is limited by the Singleton Bound, which limits not only its capacity to correct errors and erasures relatively to the number of error correction symbols, but also its ability to check if the message can be decoded or not. Indeed, if the number of errors and erasures are greater than the Singleton Bound, the decoder has no way to mathematically know for sure whether there is an error at all, it may very well be a valid message (although not the message you expect, but mathematically valid nevertheless). Hence, when the message is tampered beyond the Singleton Bound, the decoder may raise an exception, but it may also return a mathematically valid but still tampered message. Using the check() method cannot fix that either. To work around this issue, a solution is to use parity or hashing functions in parallel to the Reed-Solomon codec: use the Reed-Solomon codec to repair messages, use the parity or hashing function to check if there is any error. Due to how parity and hashing functions work, they are much less likely to produce a false negative than the Reed-Solomon algorithm. This is a general rule: error correction codes are efficient at correcting messages but not at detecting errors, hashing and parity functions are the adequate tool for this purpose.Recommended reading“Reed-Solomon codes for coders”, free practical beginner’s tutorial with Python code examples on WikiVersity. Partially written by one of the authors of the present software.“Algebraic codes for data transmission”, Blahut, Richard E., 2003, Cambridge university press.Readable online on Google Books. This book was pivotal in helping to understand the intricacies of the universal Berlekamp-Massey algorithm (see figures 7.5 and 7.10).AuthorsThis module was conceived and developed by Tomer Filiba in 2012.It was further extended and is currently maintained by Stephen Karl Larroque since 2015.And several other contributors helped improve and make it more robust:For a list of all contributors, please seethe GitHub Contributors graphand thecommits history.LicenseThis software is released under your choice of the Unlicense or the MIT-0 (MIT No Attribution) License. Both licenses arepublic-domain-equivalent licenses, as intended by the original author Tomer Filiba.