Package: compressio
CompressioCompressio provides lossless in-memory compression of pandas DataFrames and Series powered by thevisionstype system. Use up to 10x less RAM with the same data!Getting started is as easy asfromcompressioimportCompresscompress=Compress()compressed_data=compress.it(data)The FrameworkCompressio is a general framework for automated data compression and representation management not limited to any specific compression algorithm or implementation.
You have complete control, define your own types, write your own compression algorithms, or get started with the large library of types provided byvisionsand the suite of powerful algorithms included in compressio by default.These algorithms can be subdivided around three basic optimization strategies:Prefer smaller dtypes where possible without loss of informationConsider (more efficient) data representations*Compress data using more efficient data structures* this is where things get messy without visions1. Smaller dtypesUnder the hood, pandas leverages numpy arrays to store data.
Each numpy array has an associateddtypespecifying a physical, on disk, representation of the data.
For instance, a sequence of integers might be stored as 64-bit integers (int64), 8-bit unsigned integers (uint8) or even 32-bit floating point number (float32).
An overview of the numpy type system can be foundhere.These type differences have numerous computational implications, for example, where an 8 bit integer can represent numbers between 0 and 255, the range of a 64 bit integer is  between -9,223,372,036,854,775,808 and 9,223,372,036,854,775,807 at the cost of an 8x larger memory footprint.
There can also be computational performance implications for different sizes.importnumpyasnparray_int_64=np.ones((1000,1000),dtype=np.int64)print(array_int_64.nbytes)8000000array_int_8=np.ones((1000,1000),dtype=np.int8)print(array_int_8.nbytes)1000000As you can see, the 8-bit integer array decreases the memory usage by 87.5%.2. Appropriate machine representationCompressio uses visions to infer the semantic type of data and coerce it into alternative computational representations which minimize memory impact while maintaining it's semantic meaning.For instance, although pandas can use the generic object dtype to store boolean sequences, it comes at the cost of a 4x memory footprint.
Visions can automatically handle these circumstances to find an appropriate representation for your data.>>>>importpandasaspd>>>># dtype: object>>>>series=pd.Series([True,False,None,None,None,None,True,False]*1000)>>>>print(series.nbytes)64000>>>># dtype: boolean (pandas' nullable boolean)>>>>new_series=series.astype("boolean")>>>>print(new_series.nbytes)16000Further background information is available in thevisions documentation,github repositoryandJOSS publication.3. Efficient data structuresWithout additional instructions, pandas represents your data asdensearrays. This is a good all-round choice.When your data is not randomly distributed, it can be compressed (Theory).Low cardinality data can often be more efficiently stored usingsparse data structures, which are provided by pandas by default.
These structures offer efficiency by storing the predominant values only once and instead keeping indices for all other values.This notebookshows how to use compressio with sparse data structures.Data structure optimization is not limited to sparse arrays but instead include numerous domain specific opportunities such asrun-length encoding (RLE)which can be applied to compress sequential data.
We note that a pandas-specific third-party implementation is currently under development:RLEArray.UsageInstallationYou can easily install compressio with pip:pip install compressioOr, alternatively, install from source.git clone https://github.com/dylan-profiler/compressio.gitExamples.There is a collection of example notebooks to play with in theexamples directorywith a quick start notebook availablehere.Optimizing strings in pandasPandas allows for multiple ways of storing strings: as string objects or aspandas.Category. Recent version of pandas have apandas.Stringtype.How you store strings in pandas can significantly impact the RAM required.The key insights from this analysis are:The Category is more memory efficient when values are recurring and the String representation the percentage of distinct values.The size of the Series isnotdecisive for the string representation choice.You can find the full analysishere.Gotcha'sCompressing DataFrames can be helpful in many situations, but not all.
Be mindful of how to apply it in the following cases:Overflow: compression by dropping precision can lead to overflows if the array is manipulated afterwards.
This can be an issue for instance fornumpy integers. In case this is a problem for your application, you can explicitly choose a precision.Compatibility: other libraries may make different decisions to how to handle your compressed data.
One example where code needs to be adjusted to the compressed data is when the sparse data structure is used in combination with.groupby. (observedmust be set toTrue).
Thisarticleprovides another example of scikit-image, which for some functions immediately converts a given array to a float64 dtype.