Package: json-tricks
JSON tricks (python)The [pyjson-tricks] package brings several pieces of
functionality to python handling of json files:Store and load numpy arraysin human-readable format.Store and load class instancesboth generic and customized.Store and load date/timesas a dictionary (including timezone).Preserve map order{}usingOrderedDict.Allow for commentsin json files by starting lines with#.Sets, complex numbers, Decimal, Fraction, enums, compression,
duplicate keys, pathlib Paths, bytes ...As well as compression and disallowing duplicate keys.Code:https://github.com/mverleg/pyjson_tricksDocumentation:http://json-tricks.readthedocs.org/en/latest/PIP:https://pypi.python.org/pypi/json_tricksSeveral keys of the format__keyname__have special meanings, and more
might be added in future releases.If you're considering JSON-but-with-comments as a config file format,
have a look atHJSON, it might be
more appropriate. For other purposes, keep reading!Thanks for all the Github stars⭐!Installation and useYou can install usingpipinstalljson-tricksDecoding of some data types needs the corresponding package to be
installed, e.g.numpyfor arrays,pandasfor dataframes andpytzfor timezone-aware datetimes.You can import the usual json functions dump(s) and load(s), as well as
a separate comment removal function, as follows:fromjson_tricksimportdump,dumps,load,loads,strip_commentsThe exact signatures of these and other functions are in thedocumentation.Quite some older versions of Python are supported. For an up-to-date  list seethe automated tests.FeaturesNumpy arraysWhen not compressed, the array is encoded in sort-of-readable and very
flexible and portable format, like so:arr=arange(0,10,1,dtype=uint8).reshape((2,5))print(dumps({'mydata':arr}))this yields:{"mydata":{"dtype":"uint8","shape":[2,5],"Corder":true,"__ndarray__":[[0,1,2,3,4],[5,6,7,8,9]]}}which will be converted back to a numpy array when usingjson_tricks.loads. Note that the memory order (Corder) is only
stored in v3.1 and later and for arrays with at least 2 dimensions.As you see, this uses the magic key__ndarray__. Don't use__ndarray__as a dictionary key unless you're trying to make a numpy
array (and know what you're doing).Numpy scalars are also serialized (v3.5+). They are represented by the
closest python primitive type. A special representation was not
feasible, because Python's json implementation serializes some numpy
types as primitives, without consulting custom encoders. If you want to
preserve the exact numpy type, useencode_scalars_inplace.There is also a compressed format (thanksclaydugofor fix). From
the next major release, this will be default when using compression.
For now, you can use it as:dumps(data,compression=True,properties={'ndarray_compact':True})This compressed format encodes the array data in base64, with gzip
compression for the array, unless 1) compression has little effect for
that array, or 2) the whole file is already compressed. If you only want
compact format for large arrays, pass the number of elements tondarray_compact.Example:data=[linspace(0,10,9),array([pi,exp(1)])]dumps(data,compression=False,properties={'ndarray_compact':8})[{"__ndarray__":"b64.gz:H4sIAAAAAAAC/2NgQAZf7CE0iwOE5oPSIlBaEkrLQegGRShfxQEAz7QFikgAAAA=","dtype":"float64","shape":[9]},{"__ndarray__":[3.141592653589793,2.718281828459045],"dtype":"float64","shape":[2]}]Class instancesjson_trickscan serialize class instances.If the class behaves normally (not generated dynamic, no__new__or__metaclass__magic, etc)andall it's attributes are serializable,
then this should work by default.# json_tricks/test_class.pyclassMyTestCls:def__init__(self,**kwargs):fork,vinkwargs.items():setattr(self,k,v)cls_instance=MyTestCls(s='ub',dct={'7':7})json=dumps(cls_instance,indent=4)cls_instance_again=loads(json)You'll get your instance back. Here the json looks like this:{"__instance_type__":["json_tricks.test_class","MyTestCls"],"attributes":{"s":"ub","dct":{"7":7}}}As you can see, this stores the module and class name. The class must be
importable from the same module when decoding (and should not have
changed). If it isn't, you have to manually provide a dictionary tocls_lookup_mapwhen loading in which the class name can be looked up.
Note that if the class is imported, thenglobals()is such a
dictionary (so tryloads(json, cls_lookup_map=glboals())). Also note
that if the class is defined in the 'top' script (that you're calling
directly), then this isn't a module and the import part cannot be
extracted. Only the class name will be stored; it can then only be
deserialized in the same script, or if you providecls_lookup_map.Note that this also works withslotswithout having to do anything
(thanks tokoffieanddominicdoty), which encodes like this (custom
indentation):{"__instance_type__":["module.path","ClassName"],"slots":{"slotattr":37},"attributes":{"dictattr":42}}If the instance doesn't serialize automatically, or if you want custom
behaviour, then you can implement__json__encode__(self)and__json_decode__(self, **attributes)methods, like so:classCustomEncodeCls:def__init__(self):self.relevant=42self.irrelevant=37def__json_encode__(self):# should return primitive, serializable types like dict, list, int, string, float...return{'relevant':self.relevant}def__json_decode__(self,**attrs):# should initialize all properties; note that __init__ is not called implicitlyself.relevant=attrs['relevant']self.irrelevant=12As you've seen, this uses the magic key__instance_type__. Don't use__instance_type__as a dictionary key unless you know what you're
doing.Date, time, datetime and timedeltaDate, time, datetime and timedelta objects are stored as dictionaries of
"day", "hour", "millisecond" etc keys, for each nonzero property.Timezone name is also stored in case it is set, as is DST (thankseumir).
You'll need to havepytzinstalled to use timezone-aware date/times,
it's not needed for naive date/times.{"__datetime__":null,"year":1988,"month":3,"day":15,"hour":8,"minute":3,"second":59,"microsecond":7,"tzinfo":"Europe/Amsterdam"}This approach was chosen over timestamps for readability and consistency
between date and time, and over a single string to prevent parsing
problems and reduce dependencies. Note that ifprimitives=True,
date/times are encoded as ISO 8601, but they won't be restored
automatically.Don't use__date__,__time__,__datetime__,__timedelta__or__tzinfo__as dictionary keys unless you know what you're doing, as
they have special meaning.OrderGiven an ordered dictionary like this (see the tests for a longer one):ordered=OrderedDict((('elephant',None),('chicken',None),('tortoise',None),))Converting to json and back will preserve the order:fromjson_tricksimportdumps,loadsjson=dumps(ordered)ordered=loads(json,preserve_order=True)wherepreserve_order=Trueis added for emphasis; it can be left out
since it's the default.As a note onperformance,
both dicts and OrderedDicts have the same scaling for getting and
setting items (O(1)). In Python versions before 3.5, OrderedDicts were
implemented in Python rather than C, so were somewhat slower; since
Python 3.5 both are implemented in C. In summary, you should have no
scaling problems and probably no performance problems at all, especially
in Python 3. Python 3.6+ preserves order of dictionaries by default
making this redundant, but this is an implementation detail that should
not be relied on.CommentsWarning: in the next major version, comment parsing will be opt-in, not
default anymore (for performance reasons). Update your code now to passignore_comments=Trueexplicitly if you want comment parsing.This package uses#and//for comments, which seem to be the most
common conventions, though only the latter is valid javascript.For example, you could callloadson the following string:{ # "comment 1
"hello": "Wor#d", "Bye": ""M#rk"", "yes\"": 5,# comment" 2
"quote": ""th#t's" what she said", // comment "3"
"list": [1, 1, "#", """, "", 8], "dict": {"q": 7} #" comment 4 with quotes
}
// comment 5And it would return the de-commented version:{"hello":"Wor#d","Bye":""M#rk"","yes\\"": 5,"quote": ""th#t's"whatshesaid","list": [1, 1, "#", ""","\", 8], "dict": {"q":7}}Since comments aren't stored in the Python representation of the data,
loading and then saving a json file will remove the comments (it also
likely changes the indentation).The implementation of comments is a bit crude, which means that there are
some exceptional cases that aren't handled correctly (#57).It is also not very fast. For that reason, ifignore_commentswasn't
explicitly set to True, then json-tricks first tries to parge without
ignoring comments. If that fails, then it will automatically re-try
with comment handling. This makes the no-comment case faster at the cost
of the comment case, so if you are expecting comments make sure to setignore_commentsto True.Other featuresSpecial floats likeNaN,Infinityand-0using theallow_nan=Trueargument
(non-standardjson, may not decode in other implementations).Sets are serializable and can be loaded. By default the set json
representation is sorted, to have a consistent representation.Save and load complex numbers (py3) with1+2jserializing as{'__complex__': [1, 2]}.Save and loadDecimalandFraction(including NaN, infinity, -0
for Decimal).Save and loadEnum(thanks toJenselme), either built-in in
python3.4+, or with theenum34package in earlier versions.IntEnumneedsencode_intenums_inplace.json_tricksallows for gzip compression using thecompression=Trueargument (off by default).json_trickscan check for duplicate keys in maps by settingallow_duplicatesto False. These arekind of
allowed,
but are handled inconsistently between json implementations. In
Python, fordictandOrderedDict, duplicate keys are silently
overwritten.Save and loadpathlib.Pathobjects (e.g., the current path,Path('.'), serializes as{"__pathlib__": "."})
(thanks tobburan).Save and load bytes (python 3+ only), which will be encoded as utf8 if
that is valid, or as base64 otherwise. Base64 is always used if
primitives are requested. Serialized as[{"__bytes_b64__": "aGVsbG8="}]vs[{"__bytes_utf8__": "hello"}].Save and load slices (thanks toclaydugo).Preserve type vs use primitiveBy default, types are encoded such that they can be restored to their
original type when loaded withjson-tricks. Example encodings in this
documentation refer to that format.You can also choose to store things as their closest primitive type
(e.g. arrays and sets as lists, decimals as floats). This may be
desirable if you don't care about the exact type, or you are loading
the json in another language (which doesn't restore python types).
It's also smaller.To forego meta data and store primitives instead, passprimitivestodump(s). This is available in version3.8and later. Example:data=[arange(0,10,1,dtype=int).reshape((2,5)),datetime(year=2017,month=1,day=19,hour=23,minute=00,second=00),1+2j,Decimal(42),Fraction(1,3),MyTestCls(s='ub',dct={'7':7}),# see laterset(range(7)),]# Encode with metadata to preserve types when decodingprint(dumps(data))// (comments added and indenting changed)[// numpy array{"__ndarray__":[[0,1,2,3,4],[5,6,7,8,9]],"dtype":"int64","shape":[2,5],"Corder":true},// datetime (naive){"__datetime__":null,"year":2017,"month":1,"day":19,"hour":23},// complex number{"__complex__":[1.0,2.0]},// decimal & fraction{"__decimal__":"42"},{"__fraction__":true"numerator":1,"denominator":3,},// class instance{"__instance_type__":["tests.test_class","MyTestCls"],"attributes":{"s":"ub","dct":{"7":7}}},// set{"__set__":[0,1,2,3,4,5,6]}]# Encode as primitive types; more simple but loses type informationprint(dumps(data,primitives=True))// (comments added and indentation changed)[// numpy array[[0,1,2,3,4],[5,6,7,8,9]],// datetime (naive)"2017-01-19T23:00:00",// complex number[1.0,2.0],// decimal & fraction42.0,0.3333333333333333,// class instance{"s":"ub","dct":{"7":7}},// set[0,1,2,3,4,5,6]]Note that valid json is produced either way:json-tricksstores meta data as normal json, but other packages probably won't interpret it.Note that valid json is produced either way:json-tricksstores meta
data as normal json, but other packages probably won't interpret it.Usage & contributionsCode is underRevised BSD Licenseso you can use it for most purposes including commercially.Contributions are very welcome! Bug reports, feature suggestions and
code contributions help this project become more useful for everyone!
There is a shortcontribution
guide.Contributors not yet mentioned:janLo(performance boost).TestsTests are run automatically for commits to the repository for all
supported versions. This is the status:To run the tests manually for your version, seethis guide.