The Power of Pickletools. The data is increasing. More the data... | by Amarpreet Singh | Towards Data Science

The volume of data is increasing. The more the data, the more we can use it for solving different problems. Suppose you train a machine learning model for some solution and want to save it for later predictions. Here come some serialization-deserialization methods: Pickling (Pickle, cPickle, Joblib, JsonPickle, Dill, Mlflow), saving to PMML format as pipeline (sklearn2pmml), saving as JSON format (sklearn_json, sklearn_export, msgpack, JsonPickle).
You may also use m2cgen library to export your model to python/java/c++ code or write your own code to serialize and de-serialize your model.
When you build a machine learning model to solve some problem by using huge data, the model is gonna be huge too! So the problem arises when you try to save this ginormous encoding to your system (I am going to talk here about the large RandomForestClassifier model/pickle file I tried to take care of).
More data implies a large-sized model. Try to reduce your data as much as possible without losing your valuable information in the data. Removing duplicates or near-duplicates, Stratified Under-Sampling can save you here.
Try to bring down your model size by tweaking the model parameters without compromising much accuracy. Hyper-parameter tuning will help you in this case. My base RandomForestClassifier model was of whooping 60 GB but hyper-parameter tuning managed to get it lower to 8 GB. Still large? Let’s check out the next cure.
You’ve now done everything to make your model lighter without trading off much of its predictive power and still getting 8 GB of pickle file. Oh, Dear, Life is so unfair.
Hold on, still, some hope is left. Pickletools comes to rescue!
The library helps you to reduce the size of the pickle file as well as makes loading the pickle file back as an RF object easy and faster.
Although Joblib and using compression methods such as zlib, gzip made my pickle file to shrink down to 1 GB, but loading back that file as a random forest classifier object was a headache. It would take more than 16 GB RAM to load my pickle (1 GB in size) and de-serialize the RF object which would result in MemoryError.
Using Pickletools solves both problems of shrinking the size and faster loading of pickle file back without shooting the RAM more than your system can handle as follows.
To serialize:
clf.fit(X_train, y_train) #your classifier/regressor modelimport gzip, pickle, pickletoolsfilepath = "random_forest.pkl"with gzip.open(filepath, "wb") as f:    pickled = pickle.dumps(clf)    optimized_pickle = pickletools.optimize(pickled)    f.write(optimized_pickle)
To deserialize/loading back:
with gzip.open(filepath, 'rb') as f:    p = pickle.Unpickler(f)    clf = p.load()
Woohoo!! While gzip helps to reduce the pickle size (to 1 GB), pickletools helps the pickle file to load faster without consuming much memory (took 7 GB RAM this time)
You may also set pickle attribute fast = True for this purpose but it may be deprecated in future as the documentation says so:
import picklefilepath = "random_forest.pkl"with open(filepath, 'wb') as f:    p = pickle.Pickler(f)    p.fast = True    p.dump(clf)with open(filepath, 'rb') as f:    p = pickle.Unpickler(f)    clf = p.load()
If you can in any way parallelize the whole process by making each core of your CPU work at the same time, it may rescue you too up-to some extent!
Okay, your turn to try it yourself. May the force be with ya!
Further reading and References: