Multiprocessing and Pickle, How to Easily fix that? | by Salma El Shahawy | Towards Data Science

As a data scientist, you may sometimes require to send complex object hierarchies over a network or save your objects’ internal state to a disk or database for later use. To achieve this, you can use a process called serialization, which is entirely supported by the standard library. This article will discuss the following:
How to serialize an object using both pickle and dill packages.When to use dill over pickleThe right class initialization for the pickling classWhat is multiprocessing and how to use it in the context of serialization?
How to serialize an object using both pickle and dill packages.
When to use dill over pickle
The right class initialization for the pickling class
What is multiprocessing and how to use it in the context of serialization?
Python 3 installed in your computer — check this quick guide for a proper installation.Familiar with the object-oriented programming concept — check this one.Code editor — VScode, for example.Familiar with the lambda function syntax.
Python 3 installed in your computer — check this quick guide for a proper installation.
Familiar with the object-oriented programming concept — check this one.
Code editor — VScode, for example.
Familiar with the lambda function syntax.
This tutorial doesn’t include serialization using jypter notebooks or anaconda configurations.
Pickling or Serialization transforms from object state into a series of bits — the object could be methods, data, class, API end-points, etc. Serialization is an effective way to share big objects easily without losing information. The original object could be retrieved through the object Deserialization process. As usual, every great thing comes with a tradeoff; You need to be vigilant when downloading a pickle file from an unknown source, where it could have malware.
Suppose you have the following class named Process, and you have some helpful methods that you frequently use in your data preprocessing step. To save you some time, you’ve decided to pickle this class using the pickle module.
If you need to pickle an object that has a database connection, for instance, then you’re in for a hard time because it’s an unserializable object even for dill. The solution is to freeze the object out from the serialization process. Then, reinitializing the connection after the object is deserialized utilizing setter and getter methods within the class definition.
At the end of the class Process, create a new method called main() which is responsible for initializing the class Process as an object.
At the end of the class Process, create a new method called main() which is responsible for initializing the class Process as an object.
def main():  # initializing the class as an object    a = Process()   # open the file as binary    with open('test_pickle.pkl', 'wb') as f:     # use .dump() to serialize it     pickle.dump(process, f, -1)  # close the file    f.close()
2. Save the file and run it through python process.py in the terminal.
The test_pickle.pkl supposed to appear on the left-hand side of the code editor with no raised errors in the running terminal.
Now, you can easily reuse that pickle file anytime within any project. You can open it using the open() within main() method and initialize that class and start using the methods within that class.
As demonstrated in the screenshot above, __dict__ has only one key args, and fun parameter was excluded when opening the test_pickle.py file.
If you didn’t exclude the lambda initialization in the __getstate__() , the pickling would fail because lambda cannot pickle as we mentioned before.
To unpickle the file and retrieve all lambda functions that you excluded in the pickling process, all you need to do is to update the state attribute inside the __setstate__() like the following — redefine the method:
We retrieved the fun attribute. Awesome 😎
In real life situation, you might need high computational power to execute some tasks. A great example is to train the machine learning model or neural network, which are intensive and time-consuming processes.
If you are interested to read more about multiprocessing, Brendan Fortuner wrote a great article about threads and processes in Python. You can check it out here
One of the routes you might consider is distributing the training task over several processes utilizing the pathos fork from python’s multiprocessing module. However, the multiprocess tasks can’t be pickled; it would raise an error failing to pickle. That’s because when dividing a single task over multiprocess, these might need to share data; however, it doesn’t share memory space.
In this situation, the dill package comes in handy, where it can serialize many types of objects that aren’t pickleable. It can serialize database connections, lambda functions, running threads, and more.
dill is slower typically, but that's the penalty you pay for more robust serialization. If you are serializing a lot of classes and functions, then you might want to try one of the dill variants in dill.settings . If you use byref=True then dill will pickle several objects by reference (which is faster then the default). Other settings trade off picklibility for speed in selected objects. — Mike McKerns (dill author answer on Stackoverflow dill vs. cPickle)
To make sense of it, let’s have an example.
>>> import dill>>> from pathos.multiprocessing import ProcessingPool# set the processing pool to 4 core>>> pool = ProcessingPool(nodes=4)# do some processing>>> result = pool.map(lambda x: x**2, range(10))# save the method as pickle using dill >>> dill.dump(lambda x: x**2, open('use_dill', 'wb'))# save the session as pickle file >>> dill.dump_session('dill_session.pkl')
The dump_session() used to serialize the entire session of the interpreter.
Now, restart the python terminal and investigate the global environment:
>>> globals().items()
It should return something similar to the following, which is the interpreter initial state
dict_items([('__name__', '__main__'), ('__doc__', None), ('__package__', None), ('__loader__', <class '_frozen_importlib.BuiltinImporter'>), ('__spec__', None), ('__annotations__', {}), ('__builtins__', <module 'builtins' (built-in)>)])
Next, try to reload the dill session you saved earlier by:
>>> import dill >>> dill.load_session('dill_session.pkl')>>> globals().items()
The output should be:
dict_items([('__name__', '__main__'), ('__doc__', None), ('__package__', None), ('__loader__', <class '_frozen_importlib.BuiltinImporter'>), ('__spec__', None), ('__annotations__', {}), ('__builtins__', <module 'builtins' (built-in)>), ('dill', <module 'dill' from '/Users/salmaelshahawy/.pyenv/versions/3.8.6/envs/venv38/lib/python3.8/site-packages/dill/__init__.py'>), ('ProcessingPool', <class 'pathos.multiprocessing.ProcessPool'>), ('pool', <pool ProcessPool(ncpus=4)>), ('result', [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]), ('__warningregistry__', {'version': 0})])
Pickle module can serialize most of the python’s objects except for a few types, including lambda expressions, multiprocessing, threading, database connections, etc.Dill module might work as a great alternative to serialize the unpickable objects. It is more robust; however, it is slower than pickle — the tradeoff.Pickling is not secure, which means don’t arbitrarily open untrusted pickle files.It is a good practice to initialize the class with the setter and getter methods to make control which attributes to include in your pickle file.
Pickle module can serialize most of the python’s objects except for a few types, including lambda expressions, multiprocessing, threading, database connections, etc.
Dill module might work as a great alternative to serialize the unpickable objects. It is more robust; however, it is slower than pickle — the tradeoff.
Pickling is not secure, which means don’t arbitrarily open untrusted pickle files.
It is a good practice to initialize the class with the setter and getter methods to make control which attributes to include in your pickle file.
Finally, I hope this tutorial gave you a good idea about serialization. If you have any questions or suggestions to improve the article, please write them in the comment section, or if you have any issues following along, please feel free to contact me over Linkedin; I would be more than happy to help. Happy learning!👍