Package: unishox2-py3
unishox2-py3This package enables Python projects to easily use Unishox2 fromsiara-cc/Unishox2, which is a C library for compressing short strings. Unishox2 has many potential applications, and this package can enable developers to make use of it for several of those:✅ Unicode-native text compression. Unishox2 isNOTEnglish- or ASCII-only!✅ Bandwidth and storage cost reduction for databases and/or cloud services.⚠️ Byte columns can result in a faster retrieval speed when used as join keys in RDBMSes.Author's note: haven't tested this, but I'd trust the claim generally⛔️ Compression for low memory devices such as Arduino and ESP8266.Author's note: just use the C bindings for this, they're very approachableWant to learn more about Unishox2? Read the source paperhere.How to UseThis package is availableon PyPI, and can be installed withpip3 install unishox2-py3. Please note that this packageonlysupports Python3, and does not work for Python2 or below. You can see its CI status and testing matrix in this repository'sActions tab.Getting started with unishox2-py3 is easy. If you want to give it a try via the command line, you can usedemo.pyto compress some sample strings or try one of your own.If you're looking to integrate, unishox2-py3 currently provides two APIs that pass data to Unishox2's correspondingsimpleAPIs - accepting the default optimization preset, which is good for most data. These are:unishox2.compress(str)Arguments:str- This requires a string as input (generally, Unicode-encoded).Returns a tuple:bytes- The compressed data.int- The original length of the string.unishox2.decompress(bytes, int)Takes two arguments:bytes- The compressed data.int- The original length of the string.Returns:str- A string, the original data.Taken together, this looks like:importunishox2# the string we want to compressoriginal_data="What the developers know:\n1. Whole codebase is spaghetti\n2. Also, spaghetti is delicious."# drop that in as-is, nothing else is needed for compressioncompressed_data,original_size=unishox2.compress(original_data)# compressed_data now holds bytes, such as b'\x87\xbfi\x85\x1d\x9a\xe9\xfd ...'# original_size now holds an integer, such as 89# to get the original string back, we need compressed_data AND original_sizedecompressed_data=unishox2.decompress(compressed_data,original_size)# decompressed_data now holds a string, such as "What the developers know:\n..."Important NotesFirst, you have to have theoriginal_size, or know what themaximumoriginal_sizecan be for your data, as Unishox2 does not dynamically allocate memory for the resultant string when decompressing. If you need to track the exact size (ex. if some documents are KB, where others are GB), and you are saving the Unishox2-compressed data to a database, youmuststore theoriginal_sizevalue as well.As mentioned before, any reasonable maximum for the resultant data also works. So if you are storing usernames that must be 3-20 characters in length, you can skip saving theoriginal_sizeand use 20 as theoriginal_sizefor all values during decompression.Conversely, if you give anoriginal_sizevalue that is too small, too little memory will be allocated. This means that Unishox2 will write past the memory boundary (as it's C under the hood, which is just bound to Python via a module), and your Python program will crash.OS/Architecture SupportSince unishox2-py3 includes a C library, it must be built specifically for each CPU architecture and OS in use. Wheels are built and tested for many platforms viacibuildwheel, allowing for unishox2-py3 to be used with the following platforms out-of-the-box:Linux: x86_64, i686, aarch64, ppc64le, s390xMacOS: x86_64, arm64, universal2Windows: AMD64, x86If you need another OS/architecture supported, please filean issueand I'm glad to look into supporting it.PerformanceWhile Unishox doesn't provideguaranteedcompression forallshort strings (see the test cases for some examples where the output is larger than the input), it tends to provide better compression than many competitors in real-world usecases for short string compression. In addition, as unishox2-py3 is using a C module instead of reimplementing Unishox2 in Python, there is acceptable performance loss across most applications.When tested on Reddit data (technical subreddits, mostly English-oriented, 3.3m entries), the average number of bytes required for storing each post's title was:Original: 60.34zlib(1): 61.83 (+2.47%)zlib(9): 61.80 (+2.42%)smaz: 43.46 (-27.98%)Unishox2: 40.08 (-33.58%)And the average number of bytes required for storing each text post's body was:Original: 561.07zlib(1): 319.93 (-42.98%)zlib(9): 312.87 (-44.23%)smaz: 369.04 (-34.23%)Unishox2: 310.56 (-44.65%)And the average number of bytes required for storing the URL that any link posts pointed to:Original: 25.72zlib(1): 30.08 (+16.96%)zlib(9): 30.08 (+16.96%)smaz: 20.78 (-19.21%)Unishox2: 19.76 (-23.16%)Unishox2 shows clear benefits over traditional compressors when compressing short strings, and maintains comparable performance even to moderate-length documents. Unishox2 would be expected to pull farther ahead of smaz for non-English posts as well, though I don't have data to test that. I welcome a PR with additional performance tests.Integration TestsThe original test suite fromtest_unishox2.chas been copied.Tests were added to ensure the Python-to-C binding is type safe:Ensuringcompress()only takes stringsEnsuringdecompress()only takes bytes and an integerThough it will still take a negative integer, and instantly crash, so... don't do thatEnsuringcompress()won't take a Unicode surrogate e.g.\ud800Tests were also added to check certain edge cases:Ensuring Unishox2 allocates enough memory for rare, very-high-entropy strings which are enlarged when encoded by Unishox2Ensuring Unishox2 works with ASCII inputsAnd finally,hypothesisbased property testing was added to generate random inputs which:Build confidence that as long as aminimumlength is known for strings, decompression will always work, and original_size doesn't need to be savedThis runs 25,000 tests which allocate 44 bytes <= x <= 1 gigabyte for decompressionBuild confidence that Unishox2 is resilient and functional when presented with arbitrary Unicode inputThis runs 25,000 tests which generate valid Unicode of any length and compositionNote: 25k property tests per test type are only used for testing when compiled to x86, tests of each built package (ex. for aarch64) only run 100 property tests per test type per package. This alongside other tests provides assurance that unishox2-py3 is working as intended, but notas muchassurance as x86.CreditsFirst and foremost, thank you toArunofSiara Logicsfor the incredible compression library - Unishox2 is lean, fast, and versatile. I am looking forward to using this in my projects and hope it benefits others as well.In addition, this package is largely based on work fromoriginell/smaz-py3, and would not have been as quickly-developed or strongly-tested without Luis' lead.Finally, I would like to thankJosh Bickingfor his debugging insights and pragmatic thoughts on C as a whole.