Package: cloudpickle
cloudpicklecloudpicklemakes it possible to serialize Python constructs not supported
by the defaultpicklemodule from the Python standard library.cloudpickleis especially useful forcluster computingwhere Python
code is shipped over the network to execute on remote hosts, possibly close
to the data.Among other things,cloudpicklesupports pickling forlambda functionsalong withfunctions and classes defined interactivelyin the__main__module (for instance in a script, a shell or a Jupyter notebook).Cloudpickle can only be used to send objects between theexact same version
of Python.Usingcloudpickleforlong-term object storage is not supported and
strongly discouraged.Security notice: one shouldonly load pickle data from trusted sourcesas
otherwisepickle.loadcan lead to arbitrary code execution resulting in a critical
security vulnerability.InstallationThe latest release ofcloudpickleis available frompypi:pip install cloudpickleExamplesPickling a lambda expression:>>>importcloudpickle>>>squared=lambdax:x**2>>>pickled_lambda=cloudpickle.dumps(squared)>>>importpickle>>>new_squared=pickle.loads(pickled_lambda)>>>new_squared(2)4Pickling a function interactively defined in a Python shell session
(in the__main__module):>>>CONSTANT=42>>>defmy_function(data:int)->int:...returndata+CONSTANT...>>>pickled_function=cloudpickle.dumps(my_function)>>>depickled_function=pickle.loads(pickled_function)>>>depickled_function<function__main__.my_function(data:int)->int>>>>depickled_function(43)85Overriding pickle's serialization mechanism for importable constructs:An important difference betweencloudpickleandpickleis thatcloudpicklecan serialize a function or classby value, whereaspicklecan only serialize itby reference. Serialization by reference treats
functions and classes as attributes of modules, and pickles them through
instructions that trigger the import of their module at load time.
Serialization by reference is thus limited in that it assumes that the module
containing the function or class is available/importable in the unpickling
environment. This assumption breaks when pickling constructs defined in an
interactive session, a case that is automatically detected bycloudpickle,
that pickles such constructsby value.Another case where the importability assumption is expected to break is when
developing a module in a distributed execution environment: the worker
processes may not have access to the said module, for example if they live on a
different machine than the process in which the module is being developed. By
itself,cloudpicklecannot detect such "locally importable" modules and
switch to serialization by value; instead, it relies on its default mode, which
is serialization by reference. However, sincecloudpickle 2.0.0, one can
explicitly specify modules for which serialization by value should be used,
using theregister_pickle_by_value(module)//unregister_pickle_by_value(module)API:>>>importcloudpickle>>>importmy_module>>>cloudpickle.register_pickle_by_value(my_module)>>>cloudpickle.dumps(my_module.my_function)# my_function is pickled by value>>>cloudpickle.unregister_pickle_by_value(my_module)>>>cloudpickle.dumps(my_module.my_function)# my_function is pickled by referenceUsing this API, there is no need to re-install the new version of the module on
all the worker nodes nor to restart the workers: restarting the client Python
process with the new source code is enough.Note that this feature is stillexperimental, and may fail in the following
situations:If the body of a function/class pickled by value contains animportstatement:>>>deff():>>>...fromanother_moduleimportg>>>...# calling f in the unpickling environment may fail if another_module>>>...# is unavailable>>>...returng()+1If a function pickled by reference uses a function pickled by value during its execution.Running the testsWithtox, to test run the tests for all the supported versions of
Python and PyPy:pip install tox
toxor alternatively for a specific environment:tox -e py312Withpytestto only run the tests for your current version of
Python:pip install -r dev-requirements.txt
PYTHONPATH='.:tests' pytestHistorycloudpicklewas initially developed bypicloud.comand shipped as part of
the client SDK.A copy ofcloudpickle.pywas included as part of PySpark, the Python
interface toApache Spark. Davies Liu, Josh
Rosen, Thom Neale and other Apache Spark developers improved it significantly,
most notably to add support for PyPy and Python 3.The aim of thecloudpickleproject is to make that work available to a wider
audience outside of the Spark ecosystem and to make it easier to improve it
further notably with the help of a dedicated non-regression test suite.