Package: dtype-diet
dtype_dietAttempt to shrink Pandasdtypeswithout losing data so you have more RAM (and maybe more speed)This file will become your README and also the index of your documentation.Installpip install dtype_dietDocumentationhttps://noklam.github.io/dtype_diet/How to useThis is a fork ofhttps://github.com/ianozsvald/dtype_dietto continue supoprt and develop the library with approval from the original author @ianozsvald.This tool checks each column to see if larger dtypes (e.g. 8 bytefloat64andint64) could be shrunk to smallerdtypeswithout causing any data loss.
Dropping an 8 byte type to a 4 (or 2 or 1 byte) type will keep halving the RAM requirement for that column.  Categoricals are proposed forobjectcolumns which can bring significant speed and RAM benefits.Here's an minimal example with 3 lines of code running on a Kaggle dataset showing a reduction of 957 -> 85MB, you can find the notebook in therepository:#slow# sell_prices.csv.zip# Source data: https://www.kaggle.com/c/m5-forecasting-uncertainty/importpandasaspdfromdtype_dietimportreport_on_dataframe,optimize_dtypesdf=pd.read_csv('data/sell_prices.csv')proposed_df=report_on_dataframe(df,unit="MB")new_df=optimize_dtypes(df,proposed_df)print(f'Original df memory:{df.memory_usage(deep=True).sum()/1024/1024}MB')print(f'Propsed df memory:{new_df.memory_usage(deep=True).sum()/1024/1024}MB')Original df memory: 957.5197134017944 MB
Propsed df memory: 85.09655094146729 MB#slowproposed_df<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}</style>Current dtypeProposed dtypeCurrent Memory (MB)Proposed Memory (MB)Ram Usage Improvement (MB)Ram Usage Improvement (%)Columnstore_idobjectcategory203763.9204103340.907715200423.01269598.360403item_idobjectcategory233039.9775396824.677734226215.29980597.071456wm_yr_wkint64int1626723.1914066680.84472720042.34668074.999825sell_pricefloat64None26723.191406NaNNaNNaNRecommendations:Runreport_on_dataframe(your_df)to get recommendationsRunoptimize_dtypes(df, proposed_df)to convert to recommeded dtypes.Consider if Categoricals will save you RAM (see Caveats below)Consider if f32 or f16 will be useful (see Caveats - f32 isprobablya reasonable choice unless you have huge ranges of floats)Consider if int32, int16, int8 will be useful (see Caveats - overflow may be an issue)Look athttps://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.convert_dtypes.htmlwhich recommends Pandas nullable dtype alternatives (e.g. to avoid promoting an int64 with NaN items to float64, instead you get Int64 with NaNs and no data loss)Look at Extension arrays likehttps://github.com/JDASoftwareGroup/rle-array(thanks @repererumfor the tweet)Look atreport_on_dataframe(your_df)to get a printed report - no changes are made to your dataframe.Caveatsreduced numeric ranges might lead to overflow (TODO document)category dtype can have unexpected effects e.g. need for observed=True in groupby (TODO document)f16 is likely to be simulated on modern hardware so calculations will be 2-3* slower than on f32 or f64we could do with a link that explains binary representation of float & int for those wanting to learn moreDevelopmentContributorsAntony Milbournehttps://github.com/amilbourneManihttps://github.com/neomatrix369Local Setup$ conda create -n dtype_diet python=3.8 pandas jupyter pyarrow pytest
$ conda activate dtype_dietReleasemake releaseContributingThe repository is developed withnbdev, a system for developing library with notebook.Make sure you run this if you want to contribute to the library. For details, please refer to nbdev documentation (https://github.com/fastai/nbdev)nbdev_install_git_hooksSome other useful commandsnbdev_build_docs
nbdev_build_lib
nbdev_test_nbs