Package: antipickle
image source. 
        antipickle keeps your heterogeneous data freshantipicklewhen you want to use pickle, but you shouldn'tWhy? Because pickle isn't the right way to persist or share data, andwe all know that.When it comes to practice, it takes time and effort to substitute pickle.'Hmm, I can use json here' — I thought on many occasions, and usually was wrong.Something small but annoying was in the way:datetimethat can't be stored ornp.arraythat serializers don't know how to deal with.
Or evenbytes! And many smaller things.At this point I either had to give up and pickle itOR allocate time on figuring out 'how do I make this right'.antipicklesolves this for me.antipickleis arestrictedformat forsafe, persistent, and platform-independentstorage.Also, it is very convenient:importantipickleantipickle.dump(data,'data.antipickle')antipickle.dump(data,'s3://mybucket/data.antipickle')# stores in s3antipickle.dump(data,'s3://mybucket/data.antipickle.gz')# will additionally gziploaded_date=antipickle.load('s3://mybucket/data.antipickle.gz')# or local fileTo download/upload from s3 you need an additional dependency:pip install s3fs.Batteries included:Here is a simple example of what antipickle can save/load:data={'constants':[3.1415,2.718,True,False,42],'with nones':[1,None,0],b'bytes':b"can be stored too!",'nested lists and tuples':[[1,[2]],(1,2,None),{'nested':'dict'}],('tuple','as','key'):{'is_ok':True},'numpy nd':np.zeros([3,4],dtype='uint32'),}antipickle.dump(data,'data.antipickle')More formally,antipicklesupports python pieces commonly used for computations:bytes,str,int,float,complex,bool, andNonelist,tuple,set(all of them are stored as different entities)dict(including integer keys andtuplekeys)numpyarrays (native.npyformat used;dtype=Onot supported)pandasseries and dataframe (using parquet serialization viapyarrow)polarsseries and dataframe (using parquet)Any tree-formed structure of the above (no loops allowed)Configurable support:dataclassesandpydanticclasses.For reference, other non-pythonic formats (json and its binary relatives) have problems with native types
(not making difference between list and tuple) or encodings (not storing bytes)
or collections (not allowing integers, bytes and tuples in dict keys).Antipickle is python-centric and has it solved.Installationpip install antipickleWhat is it forLet's set the expectation bar.antipickle isnot fast, but isn't slow eithernot super-compact, but quite okrestricted: it wasn't designed to serialize just anything,
it focuses on common python types and cases for data folksAt the same time,antipickle issafepersistentvery convenientmodular and easy to extendand thus suitable for data sharing and data preservation.When to (not) usepicklepickleis designed for interprocess communication or as a temporary storage.picklehas a good tradeoff of space- and time- efficiency and can serialize almost anything, including graphs with cycles.Namepicklesuggests you could use it for long-term preservation of data, but that's not true:pickle's serialization is tied to an internal object representation, which is not guaranteed
to be preserved in the next release (or even on a different OS).
Developers of some packages (notablyscikit-learn) provide some guarantees about being able to parse models
that were saved with previous 1-2 minor package releases, but that's an exception not a rule.Second,pickleis insecure. And unreadable. And pickles can be large.
During unpickling they can do anything python can, i.e. anything at all.
So python docssay it clear:Only unpickle data you trust!.That said,pickleis extremely convenient and simple to use, and works as a short-term solution for many cases,
so we all (python data folks) kinda doing that wrong pickling thing from time to time, because of convenience.
And because very few of us are ready to spend time on figuring out proper serialization.All comments above apply topickle-like libs likejoblib,dill,cloudpickle.Licenseantipickleis distributed under the terms of theMITlicense.Otherantipicklebuilds uponmsgpack-python(the only dependency).antipicklesupports all maintained python versions (Python 3.7+)