Package: pandas-categorical
The package contains just a few features that make using pandas categorical types easier to use.
The main purpose of using categorical types is to reduce RAM consumption when working with large datasets. Experience shows that on average, there is a decrease of 2 times (for datasets of several GB, this is very significant). The full justification of the reasons and examples will be given below.Quickstartpip install pandas-categoricalimportpandasaspdimportpandas-categoricalaspdcdf.astype('category')->pdc.cat_astype(df,...)pd.concat()->pdc.concat_categorical()pd.merge()->pdc.merge_categorical()df.groupby(...)->df.groupby(...,observed=True)cat_astypedf=pd.read_csv("path_to_dataframe.csv")SUB_DTYPES={'cat_col_with_int_values':int,'cat_col_with_string_values':'string','ordered_cat_col_with_bool_values':bool,}pdc.cat_astype(data=df,cat_cols=SUB_DTYPES.keys(),sub_dtypes=SUB_DTYPES,ordered_cols=['ordered_cat_col_with_bool_values'])concat_categoricaldf_1=...# dataset with some categorical columnsdf_2=...# dataset with some categorical columns (categories are not equals)df_res=pdc.concat_categorical((df_1,df_2),axis=0,ignore_index=True)merge_categoricaldf_1=...# dataset with some categorical columnsdf_2=...# dataset with some categorical columns (categories are not equals)df_res=pdc.merge_categorical(df_1,df_2,on=['cat_col_1','cat_col_2'])A bit of theoryThe advantages are discussed in detail in the articleshere,hereandhere.The categorical type implies the presence of a certain set of unique values in this column, which are often repeated. By reducing the copying of identical values, it is possible to reduce the size of the column (the larger the dataset, the more likely repetitions are). By default, categories (unique values) have no order. That is, they are not comparable to each other. It is possible to make them ordered.Pandas already has everything for this (for example,.astype(’category’)). However, standard methods, in my opinion, require a high entry threshold and therefore are rarely used.Let's try to outline a number of problems and ways to solve them.1. Categorical types are easy to loseSuppose you want to connect two datasets into one usingpd.concat(..., axis=0). Datasets contain columns with categorical types.
If the column categories of the source datasets are different, thenpandasit does not combine multiple values, but simply resets their default type (for example,object,int, ...).
In other words,
$$\textcolor{red}{category1} + \textcolor{green}{category2} = object$$
$$\textcolor{red}{category1} + \textcolor{red}{category1} = \textcolor{red}{category1}$$
But we would like to observe a different behavior:
$$\textcolor{red}{category1} + \textcolor{green}{category2} = \textcolor{blue}{category3}$$
$$(\textcolor{blue}{category3} = \textcolor{red}{category1} \cup \textcolor{green}{category2})$$
As a result, you need to monitor the reduction of categories before actions such asmergeorconcat.2.  Categories type controlWhen you do a type conversiondf['col_name']=df['col_name'].astype('category')the type of categories is equal to the type of the source column.
But, if you want to change the type of categories, you probably want to write something likedf['col_name']=df['col_name'].astype('some_new_type').astype('category')That is, you will temporarily lose the categorical type (respectively, and the advantages of memory).
By the way, the usual way of controldf.dtypesdoes not display information about the type of categories themselves. You will only see onlycategorynext to the desired column.3. Unused valuesSuppose you have filtered the dataset. At the same time, the actual set of values of categorical columns could decrease, but the data type will not be changed.
This can negatively affect, for example, when working withgroupbyon such a column. As a result, grouping will also occur by unused categories. To prevent this from happening, you need to specify theobserved=Trueparameter.
For example,df.groupby(['cat_col_1','cat_col_2'],observed=True).agg('mean')4. Ordered categoriesThere is a understandable instruction for converting a column type to a categorical (unordered) onedf[col]=df[col].astype('category')But there is no similar command to convert to an ordered categorical type.
There are two non-obvious ways:df[col]=df[col].astype('category').cat.as_ordered()Ordf[col]=df[col].astype(pd.CategoricalDtype(ordered=True))5. Minimum copyingTo process large datasets, you need to minimize the creation of copies of even its parts. Therefore, the functions from this package do the transformations in place.6. Data storage in parquet formatWhen usingpd.to_parquet(path, engine='pyarrow')andpd.read_parque(path, engine='pyarrow')categorical types of some columns can be reset to normal. To solve this problem, you can useengine='fastparquet'.Note 1:fastparquetusually runs a little slower thanpyarrow.Note 2:pyarrowandfastparquetcannot be used together (for example, save by one and read by the other). This can lead to data loss.importpandasaspddf=pd.DataFrame({"Date":pd.date_range('2023-01-01',periods=10),"Object":["a"]*5+["b"]+["c"]*4,"Int":[1,1,1,2,3,1,2,4,3,2],"Float":[1.1]*5+[2.2]*5,})print(df.dtypes)df=df.astype('category')print(df.dtypes)df.to_parquet('test.parquet',engine='pyarrow')df=pd.read_parquet('test.parquet',engine='pyarrow')print(df.dtypes)Output:Date      datetime64[ns]
Object    object
Int       int64
Float     float64
dtype: object

Date      category
Object    category
Int       category
Float     category
dtype: object

Date      datetime64[ns]
Object    category
Int       int64
Float     float64
dtype: objectExamplesJupiter notebook with examplesof problems is posted on kaggle. A copy can be found in theexamples/folder.Jupiter notebook with solutionof problems. A copy can be found in theexamples/folder.Also, usage examples can be found in the tests folder.RemarksProcessing of categorical indexes has not yet been implemented.In the future, the functionpdc.join_categorical()will appear.Thecat_astypefunction was designed so that the type information could be redundant (for example, it is specified for all possible column names in the project at once). In the future, it will be possible to set default values for this function.LinksOfficial pandas documentation.https://towardsdatascience.com/staying-sane-while-adopting-pandas-categorical-datatypes-78dbd19dcd8ahttps://towardsdatascience.com/pandas-groupby-aggregate-transform-filter-c95ba3444bbbThe source of the ideathat I wanted to develop.