Package: pandas-save-profiler
Pandas Save Profilerpandas_save_profilerhelps you evaluating and comparing the performance of different pandas read and write methods.Installpip install pandas-save-profilerUsageLoad pandas and a dataframe you want to save.import pandas as pd
data = pd.util.testing.makeMissingDataframe()Loadpandas_save_profilerand use it to evaluate pandas performance saving apicklefile:import pandas_save_profiler
data.save_profiler('to_pickle')The output is a pandas series:format                                                 pickle
writer                                              to_pickle
reader                                            read_pickle
writer_args                      {'path': '/tmp/tmppk7nkivk'}
reader_args        {'filepath_or_buffer': '/tmp/tmppk7nkivk'}
writer_time                                         0.0798338
reader_time                                         0.0294895
writer_memory                                     1.09087e+08
reader_memory                                     1.09118e+08
df_memory                                                 288
file_size                                                1122
writer_memory_h                                      109.1 MB
reader_memory_h                                      109.1 MB
df_memory_h                                         288 Bytes
file_size_h                                            1.1 kB
repeats                                                     5
reads_the_same                                           True
dtype: objectValues in the series indicate:Theformatused to persist the dataframe and the writing and reading options.Writing and readingtimesin seconds.Writing and readingmemoryincrement.Size of the dataframe in memory.Sizeof the saved file.Memory values are inbytesbut a "humanized" version is also reported.
The saving and reloading process isrepeated5 times and average values are returned.
The flagreads_the_sameindicates whether the reloaded file is exactly the same as the original one or has some differences.Tocompare several writing optionsyou can use thesave_profilerfunction on each of them
and combine the results into a results dataframe:pd.DataFrame([
    data.save_profiler('to_csv'),
    data.save_profiler('to_pickle'),
    data.save_profiler('to_parquet'),
])returns:format      writer        reader                          writer_args  \
0      csv      to_csv      read_csv  {'path_or_buf': '/tmp/tmpsedehjob'}   
1   pickle   to_pickle   read_pickle         {'path': '/tmp/tmp_vhue2q7'}   
2  parquet  to_parquet  read_parquet         {'path': '/tmp/tmp0zn8qsnk'}   

                                  reader_args  writer_time  reader_time  \
0  {'filepath_or_buffer': '/tmp/tmpsedehjob'}     0.031842     0.039830   
1  {'filepath_or_buffer': '/tmp/tmp_vhue2q7'}     0.025705     0.028469   
2                {'path': '/tmp/tmp0zn8qsnk'}     0.039009     0.052447   

   writer_memory  reader_memory  df_memory  file_size writer_memory_h  \
0    110149632.0    110599372.8        288        139        110.1 MB   
1    110813184.0    110813184.0        288       1122        110.8 MB   
2    116892467.2    118014771.2        288       3449        116.9 MB   

  reader_memory_h df_memory_h file_size_h  repeats  reads_the_same  
0        110.6 MB   288 Bytes   139 Bytes        5           False  
1        110.8 MB   288 Bytes      1.1 kB        5            True  
2        118.0 MB   288 Bytes      3.4 kB        5            True