Package: smaz-py3
smaz-py3Small string compression usingsmazcompression
algorithm.This library wraps the original C code, so it should be quite fast. It also has a
testsuite that useshypothesisbased
property testing - a fancy way of saying that the tests are run with randomly
generated strings using most of unicode, to better guard against edge cases.Why do I need this?You are working with tons of short strings (text messages, urls,...) and want to save
space.According to the original code and notes, it achieves the best compression with english
strings (up to 50%) that do not contain a ton of numbers. However, any other language
might just work as well (allegedly still up to 30%).Note that in certain cases, it is possible that the compression increases the size.
Keep that in mind and maybe first run some tests. Measuring size is explained in the
example below as well.How do I use this?Let's install:$pipinstallsmaz-py3Note: the-py3is important. There is an original release, kudos to Benjamin
Sergeant, but it does not work with Python 3+.Now, a usage example.importsmaz# First we compress our example sentence.compressed=smaz.compress("The quick brown fox jumps over the lazy dog.")# The output is raw bytes. As can be seen in the decompress() call below.# Now, we decompress these raw bytes again. This should return our example sentence.decompressed=smaz.decompress(b'H\x00\xfeq&\x83\xfek^sA)\xdc\xfa\x00\xfej&-<\x95\xe7\r\x0b\x89\xdbG\x18\x06;n')#  This does not fail, which means we have successfully compressed and decompressed#  without damaging anything.assertdecompressed=="The quick brown fox jumps over the lazy dog."How much did we compress?# First, we get the actual byte size of our example string.original_size=len("The quick brown fox jumps over the lazy dog.".encode("utf-8"))# 44 bytes# As `compressed` is already raw bytes, we can also call len() on thiscompressed_size=len(compressed)# 31 bytescompression_ratio=1-compressed_size/original_size# 0.295So we saved about 30% (0.295 * 100 and some rounding 😉).If the compression ratio would be below 0, we would have actually increased the
string. Yes, this can happen. Again, smaz works best onsmallstrings.A small note about NULL bytesCurrently,smaz-py3does not support strings with NULL bytes (\x00) in compression:>>>importsmaz>>>smaz.compress("The quick brown fox\x00jumps over the lazy dog.")Traceback(mostrecentcalllast):File"<stdin>",line1,in<module>ValueError:embeddednullcharacterMy reasoning behind this is that in most scenarios you want to clean that away
beforehand anyways. If you think this is wrong, please open up anissue on github. I am happy for further input!Migrating from Python 2smazIf you have been using thePython 2smazlibrary,
this Python 3 version exposes the same API, so it is a drop-in replacement.Important: While developing this extension, I think I found a bug in the original
library. Using Python 2.7.16:>>>importsmaz>>>smaz.compress("The quick brown fox jumps over the lazy dog.")'H'# this is wrong.>>>small=smaz.compress("The quick brown fox jumps over the lazy dog.")>>>smaz.decompress(small)'The'# information lost.So, if you are actually upgrading from this, please make sure that you are not
affected by this.smaz-py3is not prone to this bug.Behind the scenes, smaz uses NULL bytes in compression. However, when converting from
C back to a Python string object, NULL is used to mark the end of the string. The
above sentence, compressed, has the NULL byte right after theH(H\x00\xfeq…).
That's why it stops right then and there. Again,smaz-py3is not affected by this,
mostly because I got lucky in choosing this example sentence.CreditsCredit where credit is due. First toantirez's SMAZ compressionand to theoriginal python 2 wrapperby Benjamin
Sergeant.