Package: libstreamvbyte
Explore Usage »Report Bug·Request FeatureTable of ContentsAbout The ProjectGetting StartedInstallationUsageExampleBenchmarkRoadmapContributingLicenseReferenceContactAbout The Projectlibstreamvbyteis aC++implementation ofStreamVByte, withPythonbindings usingpybind11.StreamVByteis an integer compression technique that use SIMD instructions (vectorization) to improve performance. The library is optimized withSSSE3intrinsics, which are supported by mostx86_64processors. It usessse2neonto translateSSSE3intrinsics toNEONintrinsics forARMprocessors. The library can also be used with other 32-bit architectures, although it will fall back to scalar implementations in those cases.Withlibstreamvbyte, you can quickly and efficiently compress integer sequences, reducing the amount of storage space and network bandwidth required. The library is easy to use and integrates seamlessly withPythonviapybind11bindings. Whether you're working with large datasets or building a distributed computing system,libstreamvbytecan help you improve performance and reduce the resources needed to handle your data.Currently supportsPython 3.8+on Windows, Linux (manylinux_2_17,musllinux_1_1) and macOS (universal2).(back to top)Getting StartedInstallationForPythonInstall fromPyPIusingpip.pipinstalllibstreamvbyteOr install from.whlfile.pipinstall"path/to/your/downloaded/whl"To find appropriate.whlfile, please visitreleases.ForC++You must haveCMakeinstalled on your system.# clone the repogitclonehttps://github.com/wst24365888/libstreamvbytecdlibstreamvbyte# build and installcmake.
make
sudomakeinstallUsageForPythonImportlibstreamvbytefirst.importlibstreamvbyteassvbAnd here are the APIs.# Encode an array of unsigned integers into a byte array.encode(in_uint32:numpy.ndarray[numpy.uint32])->numpy.ndarray[numpy.uint8]# Decode a byte array into an array of unsigned integers.decode(in_uint8:numpy.ndarray[numpy.uint8],size:int)->numpy.ndarray[numpy.uint32]# Encode an array of signed integers into an array of unsigned integers.zigzag_encode(in_int32:numpy.ndarray[numpy.int32])->numpy.ndarray[numpy.uint32]# Decode an array of unsigned integers into an array of signed integers.zigzag_decode(in_uint32:numpy.ndarray[numpy.uint32])->numpy.ndarray[numpy.int32]# Check if the current wheel is a vectorized version.is_vectorized_version()->boolForC++Includestreamvbyte.hfirst.#include"streamvbyte.h"For the APIs, please refer toinclude/streamvbyte.h.ExampleForPythonimportlibstreamvbyteassvbN=2**20+2# type(original_data) == np.ndarray# original_data.dtype == np.int32original_data=np.random.randint(-2**31,2**31,N,dtype=np.int32)# type(compressed_bytes) == np.ndarray# compressed_bytes.dtype == np.uint8compressed_bytes=svb.encode(svb.zigzag_encode(original_data))# type(recovered_data) == np.ndarray# recovered_data.dtype == np.int32recovered_data=svb.zigzag_decode(svb.decode(compressed_bytes,N))ForC++#include"streamvbyte.h"intmain(){std::size_tN=(1<<20)+2;std::vector<int32_t>original_data(N);for(std::size_ti=0;i<N;++i){original_data[i]=rand()-rand();}std::vector<uint8_t>compressed_bytes=streamvbyte::encode(streamvbyte::zigzag_encode(original_data));std::vector<int32_t>recovered_data=streamvbyte::zigzag_decode(streamvbyte::decode(compressed_bytes,N));return0;}Compile it with linking tolibstreamvbyte.g++-oexampleexample.cpp-lstreamvbyte(back to top)BenchmarkOS:Linux5.15.79.1-microsoft-standard-WSL2x86_64
CPU:AMDRyzen536006-CoreProcessor(12)@3.600GHz

Runon(12X3593.26MHzCPUs)CPUCaches:L1Data32KiB(x6)L1Instruction32KiB(x6)L2Unified512KiB(x6)L3Unified16384KiB(x1)LoadAverage:0.81,0.85,0.69
-----------------------------------------------------------------------------------
BenchmarkTimeCPUIterationsThroughput
-----------------------------------------------------------------------------------
BM_memcpy/4096149ns149ns468853113.7122G/s
BM_memcpy/8192548ns548ns12758037.46783G/s
BM_memcpy/163841139ns1138ns6408357.19553G/s
BM_memcpy/327682185ns2185ns3208407.49932G/s
BM_memcpy/655364921ns4921ns1427036.65895G/s
BM_memcpy/13107210968ns10968ns635025.97511G/s
BM_memcpy/26214422465ns22465ns311345.83457G/s
BM_memcpy/52428845101ns45100ns155415.81245G/s
BM_memcpy/104857691131ns91131ns76395.75314G/s
BM_streamvbyte_encode/40961222ns1222ns5808551.67556G/s
BM_streamvbyte_encode/81922470ns2467ns2823491.66064G/s
BM_streamvbyte_encode/163844945ns4945ns1396711.65662G/s
BM_streamvbyte_encode/327689990ns9989ns704971.64017G/s
BM_streamvbyte_encode/6553619853ns19853ns309631.65051G/s
BM_streamvbyte_encode/13107239933ns39932ns174011.64118G/s
BM_streamvbyte_encode/26214480563ns80562ns81931.62697G/s
BM_streamvbyte_encode/524288160716ns160716ns42841.6311G/s
BM_streamvbyte_encode/1048576319253ns319253ns19421.64223G/s
BM_streamvbyte_decode/4096691ns691ns10404622.96191G/s
BM_streamvbyte_decode/81921341ns1341ns5169793.05539G/s
BM_streamvbyte_decode/163842683ns2683ns2612083.05359G/s
BM_streamvbyte_decode/327685348ns5348ns1303193.06353G/s
BM_streamvbyte_decode/6553610817ns10817ns644273.02936G/s
BM_streamvbyte_decode/13107223207ns23207ns315462.824G/s
BM_streamvbyte_decode/26214445746ns45746ns112912.86519G/s
BM_streamvbyte_decode/52428888660ns88660ns79472.95673G/s
BM_streamvbyte_decode/1048576178497ns178497ns39072.93724G/s
BM_zigzag_encode/4096810ns810ns8540762.52829G/s
BM_zigzag_encode/81921611ns1608ns4331542.548G/s
BM_zigzag_encode/163843174ns3174ns2191652.58084G/s
BM_zigzag_encode/327686457ns6457ns1084152.53754G/s
BM_zigzag_encode/6553612582ns12582ns547472.60432G/s
BM_zigzag_encode/13107225243ns25243ns278022.59617G/s
BM_zigzag_encode/26214450278ns50278ns139522.60693G/s
BM_zigzag_encode/524288100563ns100562ns69322.60678G/s
BM_zigzag_encode/1048576211846ns211845ns32222.47487G/s
BM_zigzag_decode/4096675ns675ns10410443.03263G/s
BM_zigzag_decode/81921342ns1342ns5235533.05196G/s
BM_zigzag_decode/163842643ns2643ns2654973.09905G/s
BM_zigzag_decode/327685383ns5383ns1309763.04377G/s
BM_zigzag_decode/6553611474ns11474ns608172.85588G/s
BM_zigzag_decode/13107221777ns21777ns323453.00944G/s
BM_zigzag_decode/26214443477ns43478ns143873.0147G/s
BM_zigzag_decode/52428886120ns86120ns81453.04393G/s
BM_zigzag_decode/1048576173095ns173093ns40283.02894G/sThe unit ofThroughputisGB/s(Giga Bytes per second).Build Benchmarks from Sourcecmake.\-DCMAKE_BUILD_TYPE=Release\-DBUILD_SHARED_LIBS=OFF\-DBUILD_PYBIND11=OFF\-DPRINT_BENCHMARK=OFF\-DBUILD_TESTS=ON\-DBUILD_BENCHMARKS=ON
makelibstreamvbyte_benchmarks
./libstreamvbyte_benchmarks--benchmark_counters_tabular=true(back to top)RoadmapZigzag encoding/decoding.Support ARM processors withNEONintrinsics.Differential coding (delta encoding/decoding).See theopen issuesfor a full list of proposed features (and known issues).(back to top)ContributingContributions are what make the open source community such an amazing place to
learn, inspire, and create. Any contributions you make aregreatly
appreciated.If you have a suggestion that would make this better, please fork the repo and
create a pull request. You can also simply open an issue with the tag
"enhancement". Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feat/amazing-feature)Commit your Changes withConventional CommitsPush to the Branch (git push origin feat/amazing-feature)Open a Pull Request(back to top)LicenseDistributed under the MIT License. SeeLICENSEfor more information.(back to top)ReferenceDaniel Lemire, Nathan Kurz, Christoph Rupp,Stream VByte: Faster Byte-Oriented Integer Compression, Information Processing Letters 130, 2018.(back to top)ContactAuthorHSING-HAN, WU (Xyphuz)Mail me:xyphuzwu@gmail.comAbout me:https://www.xyphuz.comGitHub:https://github.com/wst24365888Project Linkhttps://github.com/wst24365888/libstreamvbyte(back to top)