Package: xorjson
xorjsonxorjson is a fast, correct JSON library for Python. Itbenchmarksas the fastest Python
library for JSON and is more correct than the standard json library or other
third-party libraries. It serializesdataclass,datetime,numpy, andUUIDinstances natively.Its features and drawbacks compared to other Python JSON libraries:serializesdataclassinstances 40-50x as fast as other librariesserializesdatetime,date, andtimeinstances to RFC 3339 format,
e.g., "1970-01-01T00:00:00+00:00"serializesnumpy.ndarrayinstances 4-12x as fast with 0.3x the memory
usage of other librariespretty prints 10x to 20x as fast as the standard libraryserializes tobytesrather thanstr, i.e., is not a drop-in replacementserializesstrwithout escaping unicode to ASCII, e.g., "好" rather than
"\\u597d"serializesfloat10x as fast and deserializes twice as fast as other
librariesserializes subclasses ofstr,int,list, anddictnatively,
requiringdefaultto specify how to serialize othersserializes arbitrary types using adefaulthookhas strict UTF-8 conformance, more correct than the standard libraryhas strict JSON conformance in not supporting Nan/Infinity/-Infinityhas an option for strict JSON conformance on 53-bit integers with default
support for 64-bitdoes not provideload()ordump()functions for reading from/writing to
file-like objectsxorjson supports CPython 3.8, 3.9, 3.10, 3.11, and 3.12. It distributes
amd64/x86_64, aarch64/armv8, arm7, POWER/ppc64le, and s390x wheels for Linux,
amd64 and aarch64 wheels for macOS, and amd64 and i686/x86 wheels for Windows.
xorjson does not and will not support PyPy. xorjson does not and will not
support PEP 554 subinterpreters. Releases follow semantic versioning and
serializing a new object type without an opt-in flag is considered a
breaking change.xorjson is licensed under both the Apache 2.0 and MIT licenses. The
repository and issue tracker isgithub.com/timkpaine/xorjson, and patches may be
submitted there. There is aCHANGELOGavailable in the repository.UsageInstallQuickstartMigratingSerializedefaultoptionFragmentDeserializeTypesdataclassdatetimeenumfloatintnumpystruuidTestingPerformanceLatencyMemoryReproducingQuestionsPackagingLicenseUsageInstallTo install a wheel from PyPI:pipinstall--upgrade"pip>=20.3"# manylinux_x_y, universal2 wheel supportpipinstall--upgradexorjsonTo build a wheel, seepackaging.QuickstartThis is an example of serializing, with options specified, and deserializing:>>>importxorjson,datetime,numpy>>>data={"type":"job","created_at":datetime.datetime(1970,1,1),"status":"🆗","payload":numpy.array([[1,2],[3,4]]),}>>>xorjson.dumps(data,option=xorjson.OPT_NAIVE_UTC|xorjson.OPT_SERIALIZE_NUMPY)b'{"type":"job","created_at":"1970-01-01T00:00:00+00:00","status":"\xf0\x9f\x86\x97","payload":[[1,2],[3,4]]}'>>>xorjson.loads(_){'type':'job','created_at':'1970-01-01T00:00:00+00:00','status':'🆗','payload':[[1,2],[3,4]]}Migratingxorjson version 3 serializes more types than version 2. Subclasses ofstr,int,dict, andlistare now serialized. This is faster and more similar
to the standard library. It can be disabled withxorjson.OPT_PASSTHROUGH_SUBCLASS.dataclasses.dataclassinstances
are now serialized by default and cannot be customized in adefaultfunction unlessoption=xorjson.OPT_PASSTHROUGH_DATACLASSis
specified.uuid.UUIDinstances are serialized by default.
For any type that is now serialized,
implementations in adefaultfunction and options enabling them can be
removed but do not need to be. There was no change in deserialization.To migrate from the standard library, the largest difference is thatxorjson.dumpsreturnsbytesandjson.dumpsreturns astr. Users withdictobjects using non-strkeys should specifyoption=xorjson.OPT_NON_STR_KEYS.sort_keysis replaced byoption=xorjson.OPT_SORT_KEYS.indentis replaced byoption=xorjson.OPT_INDENT_2and other levels of indentation are not
supported.Serializedefdumps(__obj:Any,default:Optional[Callable[[Any],Any]]=...,option:Optional[int]=...,)->bytes:...dumps()serializes Python objects to JSON.It natively serializesstr,dict,list,tuple,int,float,bool,None,dataclasses.dataclass,typing.TypedDict,datetime.datetime,datetime.date,datetime.time,uuid.UUID,numpy.ndarray, andxorjson.Fragmentinstances. It supports arbitrary types throughdefault. It
serializes subclasses ofstr,int,dict,list,dataclasses.dataclass, andenum.Enum. It does not serialize subclasses
oftupleto avoid serializingnamedtupleobjects as arrays. To avoid
serializing subclasses, specify the optionxorjson.OPT_PASSTHROUGH_SUBCLASS.The output is abytesobject containing UTF-8.The global interpreter lock (GIL) is held for the duration of the call.It raisesJSONEncodeErroron an unsupported type. This exception message
describes the invalid object with the error messageType is not JSON serializable: .... To fix this, specifydefault.It raisesJSONEncodeErroron astrthat contains invalid UTF-8.It raisesJSONEncodeErroron an integer that exceeds 64 bits by default or,
withOPT_STRICT_INTEGER, 53 bits.It raisesJSONEncodeErrorif adicthas a key of a type other thanstr,
unlessOPT_NON_STR_KEYSis specified.It raisesJSONEncodeErrorif the output ofdefaultrecurses to handling bydefaultmore than 254 levels deep.It raisesJSONEncodeErroron circular references.It raisesJSONEncodeErrorif atzinfoon a datetime object is
unsupported.JSONEncodeErroris a subclass ofTypeError. This is for compatibility
with the standard library.If the failure was caused by an exception indefaultthenJSONEncodeErrorchains the original exception as__cause__.defaultTo serialize a subclass or arbitrary types, specifydefaultas a
callable that returns a supported type.defaultmay be a function,
lambda, or callable class instance. To specify that a type was not
handled bydefault, raise an exception such asTypeError.>>>importxorjson,decimal>>>defdefault(obj):ifisinstance(obj,decimal.Decimal):returnstr(obj)raiseTypeError>>>xorjson.dumps(decimal.Decimal("0.0842389659712649442845"))JSONEncodeError:TypeisnotJSONserializable:decimal.Decimal>>>xorjson.dumps(decimal.Decimal("0.0842389659712649442845"),default=default)b'"0.0842389659712649442845"'>>>xorjson.dumps({1,2},default=default)xorjson.JSONEncodeError:TypeisnotJSONserializable:setThedefaultcallable may return an object that itself
must be handled bydefaultup to 254 times before an exception
is raised.It is important thatdefaultraise an exception if a type cannot be handled.
Python otherwise implicitly returnsNone, which appears to the caller
like a legitimate value and is serialized:>>>importxorjson,json,rapidjson>>>defdefault(obj):ifisinstance(obj,decimal.Decimal):returnstr(obj)>>>xorjson.dumps({"set":{1,2}},default=default)b'{"set":null}'>>>json.dumps({"set":{1,2}},default=default)'{"set":null}'>>>rapidjson.dumps({"set":{1,2}},default=default)'{"set":null}'optionTo modify how data is serialized, specifyoption. Eachoptionis an integer
constant inxorjson. To specify multiple options, mask them together, e.g.,option=xorjson.OPT_STRICT_INTEGER | xorjson.OPT_NAIVE_UTC.OPT_APPEND_NEWLINEAppend\nto the output. This is a convenience and optimization for the
pattern ofdumps(...) + "\n".bytesobjects are immutable and this
pattern copies the original contents.>>>importxorjson>>>xorjson.dumps([])b"[]">>>xorjson.dumps([],option=xorjson.OPT_APPEND_NEWLINE)b"[]\n"OPT_INDENT_2Pretty-print output with an indent of two spaces. This is equivalent toindent=2in the standard library. Pretty printing is slower and the output
larger. xorjson is the fastest compared library at pretty printing and has
much less of a slowdown to pretty print than the standard library does. This
option is compatible with all other options.>>>importxorjson>>>xorjson.dumps({"a":"b","c":{"d":True},"e":[1,2]})b'{"a":"b","c":{"d":true},"e":[1,2]}'>>>xorjson.dumps({"a":"b","c":{"d":True},"e":[1,2]},option=xorjson.OPT_INDENT_2)b'{\n"a": "b",\n"c": {\n"d": true\n},\n"e": [\n1,\n2\n]\n}'If displayed, the indentation and linebreaks appear like this:{"a":"b","c":{"d":true},"e":[1,2]}This measures serializing the github.json fixture as compact (52KiB) or
pretty (64KiB):Librarycompact (ms)pretty (ms)vs. xorjsonxorjson0.030.041ujson0.180.194.6rapidjson0.10.122.9simplejson0.250.8921.4json0.180.7117This measures serializing the citm_catalog.json fixture, more of a worst
case due to the amount of nesting and newlines, as compact (489KiB) or
pretty (1.1MiB):Librarycompact (ms)pretty (ms)vs. xorjsonxorjson0.590.711ujson2.93.595rapidjson1.812.83.9simplejson10.4342.1359.1json4.1633.4246.9This can be reproduced using thepyindentscript.OPT_NAIVE_UTCSerializedatetime.datetimeobjects without atzinfoas UTC. This
has no effect ondatetime.datetimeobjects that havetzinfoset.>>>importxorjson,datetime>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0),)b'"1970-01-01T00:00:00"'>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0),option=xorjson.OPT_NAIVE_UTC,)b'"1970-01-01T00:00:00+00:00"'OPT_NON_STR_KEYSSerializedictkeys of type other thanstr. This allowsdictkeys
to be one ofstr,int,float,bool,None,datetime.datetime,datetime.date,datetime.time,enum.Enum, anduuid.UUID. For comparison,
the standard library serializesstr,int,float,boolorNoneby
default. xorjson benchmarks as being faster at serializing non-strkeys
than other libraries. This option is slower forstrkeys than the default.>>>importxorjson,datetime,uuid>>>xorjson.dumps({uuid.UUID("7202d115-7ff3-4c81-a7c1-2a1f067b1ece"):[1,2,3]},option=xorjson.OPT_NON_STR_KEYS,)b'{"7202d115-7ff3-4c81-a7c1-2a1f067b1ece":[1,2,3]}'>>>xorjson.dumps({datetime.datetime(1970,1,1,0,0,0):[1,2,3]},option=xorjson.OPT_NON_STR_KEYS|xorjson.OPT_NAIVE_UTC,)b'{"1970-01-01T00:00:00+00:00":[1,2,3]}'These types are generally serialized how they would be as
values, e.g.,datetime.datetimeis still an RFC 3339 string and respects
options affecting it. The exception is thatintserialization does not
respectOPT_STRICT_INTEGER.This option has the risk of creating duplicate keys. This is because non-strobjects may serialize to the samestras an existing key, e.g.,{"1": true, 1: false}. The last key to be inserted to thedictwill be
serialized last and a JSON deserializer will presumably take the last
occurrence of a key (in the above,false). The first value will be lost.This option is compatible withxorjson.OPT_SORT_KEYS. If sorting is used,
note the sort is unstable and will be unpredictable for duplicate keys.>>>importxorjson,datetime>>>xorjson.dumps({"other":1,datetime.date(1970,1,5):2,datetime.date(1970,1,3):3},option=xorjson.OPT_NON_STR_KEYS|xorjson.OPT_SORT_KEYS)b'{"1970-01-03":3,"1970-01-05":2,"other":1}'This measures serializing 589KiB of JSON comprising alistof 100dictin which eachdicthas both 365 randomly-sortedintkeys representing epoch
timestamps as well as onestrkey and the value for each key is a
single integer. In "str keys", the keys were converted tostrbefore
serialization, and xorjson still specifesoption=xorjson.OPT_NON_STR_KEYS(which is always somewhat slower).Librarystr keys (ms)int keys (ms)int keys sorted (ms)xorjson1.532.164.29ujson3.075.65rapidjson4.29simplejson11.2414.5021.86json7.178.49ujson is blank for sorting because it segfaults. json is blank because it
raisesTypeErroron attempting to sort before converting all keys tostr.
rapidjson is blank because it does not support non-strkeys. This can
be reproduced using thepynonstrscript.OPT_OMIT_MICROSECONDSDo not serialize themicrosecondfield ondatetime.datetimeanddatetime.timeinstances.>>>importxorjson,datetime>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0,1),)b'"1970-01-01T00:00:00.000001"'>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0,1),option=xorjson.OPT_OMIT_MICROSECONDS,)b'"1970-01-01T00:00:00"'OPT_PASSTHROUGH_DATACLASSPassthroughdataclasses.dataclassinstances todefault. This allows
customizing their output but is much slower.>>>importxorjson,dataclasses>>>@dataclasses.dataclassclassUser:id:strname:strpassword:strdefdefault(obj):ifisinstance(obj,User):return{"id":obj.id,"name":obj.name}raiseTypeError>>>xorjson.dumps(User("3b1","asd","zxc"))b'{"id":"3b1","name":"asd","password":"zxc"}'>>>xorjson.dumps(User("3b1","asd","zxc"),option=xorjson.OPT_PASSTHROUGH_DATACLASS)TypeError:TypeisnotJSONserializable:User>>>xorjson.dumps(User("3b1","asd","zxc"),option=xorjson.OPT_PASSTHROUGH_DATACLASS,default=default,)b'{"id":"3b1","name":"asd"}'OPT_PASSTHROUGH_DATETIMEPassthroughdatetime.datetime,datetime.date, anddatetime.timeinstances
todefault. This allows serializing datetimes to a custom format, e.g.,
HTTP dates:>>>importxorjson,datetime>>>defdefault(obj):ifisinstance(obj,datetime.datetime):returnobj.strftime("%a,%d%b %Y %H:%M:%S GMT")raiseTypeError>>>xorjson.dumps({"created_at":datetime.datetime(1970,1,1)})b'{"created_at":"1970-01-01T00:00:00"}'>>>xorjson.dumps({"created_at":datetime.datetime(1970,1,1)},option=xorjson.OPT_PASSTHROUGH_DATETIME)TypeError:TypeisnotJSONserializable:datetime.datetime>>>xorjson.dumps({"created_at":datetime.datetime(1970,1,1)},option=xorjson.OPT_PASSTHROUGH_DATETIME,default=default,)b'{"created_at":"Thu, 01 Jan 1970 00:00:00 GMT"}'This does not affect datetimes indictkeys if using OPT_NON_STR_KEYS.OPT_PASSTHROUGH_SUBCLASSPassthrough subclasses of builtin types todefault.>>>importxorjson>>>classSecret(str):passdefdefault(obj):ifisinstance(obj,Secret):return"******"raiseTypeError>>>xorjson.dumps(Secret("zxc"))b'"zxc"'>>>xorjson.dumps(Secret("zxc"),option=xorjson.OPT_PASSTHROUGH_SUBCLASS)TypeError:TypeisnotJSONserializable:Secret>>>xorjson.dumps(Secret("zxc"),option=xorjson.OPT_PASSTHROUGH_SUBCLASS,default=default)b'"******"'This does not affect serializing subclasses asdictkeys if using
OPT_NON_STR_KEYS.OPT_SERIALIZE_DATACLASSThis is deprecated and has no effect in version 3. In version 2 this was
required to serializedataclasses.dataclassinstances. For more, seedataclass.OPT_SERIALIZE_NUMPYSerializenumpy.ndarrayinstances. For more, seenumpy.OPT_SERIALIZE_UUIDThis is deprecated and has no effect in version 3. In version 2 this was
required to serializeuuid.UUIDinstances. For more, seeUUID.OPT_SORT_KEYSSerializedictkeys in sorted order. The default is to serialize in an
unspecified order. This is equivalent tosort_keys=Truein the standard
library.This can be used to ensure the order is deterministic for hashing or tests.
It has a substantial performance penalty and is not recommended in general.>>>importxorjson>>>xorjson.dumps({"b":1,"c":2,"a":3})b'{"b":1,"c":2,"a":3}'>>>xorjson.dumps({"b":1,"c":2,"a":3},option=xorjson.OPT_SORT_KEYS)b'{"a":3,"b":1,"c":2}'This measures serializing the twitter.json fixture unsorted and sorted:Libraryunsorted (ms)sorted (ms)vs. xorjsonxorjson0.320.541ujson1.62.073.8rapidjson1.121.653.1simplejson2.253.135.8json1.782.324.3The benchmark can be reproduced using thepysortscript.The sorting is not collation/locale-aware:>>>importxorjson>>>xorjson.dumps({"a":1,"ä":2,"A":3},option=xorjson.OPT_SORT_KEYS)b'{"A":3,"a":1,"\xc3\xa4":2}'This is the same sorting behavior as the standard library, rapidjson,
simplejson, and ujson.dataclassalso serialize as maps but this has no effect on them.OPT_STRICT_INTEGEREnforce 53-bit limit on integers. The limit is otherwise 64 bits, the same as
the Python standard library. For more, seeint.OPT_UTC_ZSerialize a UTC timezone ondatetime.datetimeinstances asZinstead
of+00:00.>>>importxorjson,datetime,zoneinfo>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0,tzinfo=zoneinfo.ZoneInfo("UTC")),)b'"1970-01-01T00:00:00+00:00"'>>>xorjson.dumps(datetime.datetime(1970,1,1,0,0,0,tzinfo=zoneinfo.ZoneInfo("UTC")),option=xorjson.OPT_UTC_Z)b'"1970-01-01T00:00:00Z"'Fragmentxorjson.Fragmentincludes already-serialized JSON in a document. This is an
efficient way to include JSON blobs from a cache, JSONB field, or separately
serialized object without first deserializing to Python objects vialoads().>>>importxorjson>>>xorjson.dumps({"key":"zxc","data":xorjson.Fragment(b'{"a": "b", "c": 1}')})b'{"key":"zxc","data":{"a": "b", "c": 1}}'It does no reformatting:xorjson.OPT_INDENT_2will not affect a
compact blob nor will a pretty-printed JSON blob be rewritten as compact.The input must bebytesorstrand given as a positional argument.This raisesxorjson.JSONEncodeErrorif astris given and the input is
not valid UTF-8. It otherwise does no validation and it is possible to
write invalid JSON. This does not escape characters. The implementation is
tested to not crash if given invalid strings or invalid JSON.This is similar toRawJSONin rapidjson.Deserializedefloads(__obj:Union[bytes,bytearray,memoryview,str])->Any:...loads()deserializes JSON to Python objects. It deserializes todict,list,int,float,str,bool, andNoneobjects.bytes,bytearray,memoryview, andstrinput are accepted. If the input
exists as amemoryview,bytearray, orbytesobject, it is recommended to
pass these directly rather than creating an unnecessarystrobject. That is,xorjson.loads(b"{}")instead ofxorjson.loads(b"{}".decode("utf-8")). This
has lower memory usage and lower latency.The input must be valid UTF-8.xorjson maintains a cache of map keys for the duration of the process. This
causes a net reduction in memory usage by avoiding duplicate strings. The
keys must be at most 64 bytes to be cached and 1024 entries are stored.The global interpreter lock (GIL) is held for the duration of the call.It raisesJSONDecodeErrorif given an invalid type or invalid
JSON. This includes if the input containsNaN,Infinity, or-Infinity,
which the standard library allows, but is not valid JSON.JSONDecodeErroris a subclass ofjson.JSONDecodeErrorandValueError.
This is for compatibility with the standard library.Typesdataclassxorjson serializes instances ofdataclasses.dataclassnatively. It serializes
instances 40-50x as fast as other libraries and avoids a severe slowdown seen
in other libraries compared to serializingdict.It is supported to pass all variants of dataclasses, including dataclasses
using__slots__, frozen dataclasses, those with optional or default
attributes, and subclasses. There is a performance benefit to not
using__slots__.Librarydict (ms)dataclass (ms)vs. xorjsonxorjson1.401.601ujsonrapidjson3.6468.4842simplejson14.2192.1857json13.2894.9059This measures serializing 555KiB of JSON, xorjson natively and other libraries
usingdefaultto serialize the output ofdataclasses.asdict(). This can be
reproduced using thepydataclassscript.Dataclasses are serialized as maps, with every attribute serialized and in
the order given on class definition:>>>importdataclasses,xorjson,typing@dataclasses.dataclassclassMember:id:intactive:bool=dataclasses.field(default=False)@dataclasses.dataclassclassObject:id:intname:strmembers:typing.List[Member]>>>xorjson.dumps(Object(1,"a",[Member(1,True),Member(2)]))b'{"id":1,"name":"a","members":[{"id":1,"active":true},{"id":2,"active":false}]}'datetimexorjson serializesdatetime.datetimeobjects toRFC 3339format,
e.g., "1970-01-01T00:00:00+00:00". This is a subset of ISO 8601 and is
compatible withisoformat()in the standard library.>>>importxorjson,datetime,zoneinfo>>>xorjson.dumps(datetime.datetime(2018,12,1,2,3,4,9,tzinfo=zoneinfo.ZoneInfo("Australia/Adelaide")))b'"2018-12-01T02:03:04.000009+10:30"'>>>xorjson.dumps(datetime.datetime(2100,9,1,21,55,2).replace(tzinfo=zoneinfo.ZoneInfo("UTC")))b'"2100-09-01T21:55:02+00:00"'>>>xorjson.dumps(datetime.datetime(2100,9,1,21,55,2))b'"2100-09-01T21:55:02"'datetime.datetimesupports instances with atzinfothat isNone,datetime.timezone.utc, a timezone instance from the python3.9+zoneinfomodule, or a timezone instance from the third-partypendulum,pytz, ordateutil/arrowlibraries.It is fastest to use the standard library'szoneinfo.ZoneInfofor timezones.datetime.timeobjects must not have atzinfo.>>>importxorjson,datetime>>>xorjson.dumps(datetime.time(12,0,15,290))b'"12:00:15.000290"'datetime.dateobjects will always serialize.>>>importxorjson,datetime>>>xorjson.dumps(datetime.date(1900,1,2))b'"1900-01-02"'Errors withtzinforesult inJSONEncodeErrorbeing raised.To disable serialization ofdatetimeobjects specify the optionxorjson.OPT_PASSTHROUGH_DATETIME.To use "Z" suffix instead of "+00:00" to indicate UTC ("Zulu") time, use the optionxorjson.OPT_UTC_Z.To assume datetimes without timezone are UTC, use the optionxorjson.OPT_NAIVE_UTC.enumxorjson serializes enums natively. Options apply to their values.>>>importenum,datetime,xorjson>>>classDatetimeEnum(enum.Enum):EPOCH=datetime.datetime(1970,1,1,0,0,0)>>>xorjson.dumps(DatetimeEnum.EPOCH)b'"1970-01-01T00:00:00"'>>>xorjson.dumps(DatetimeEnum.EPOCH,option=xorjson.OPT_NAIVE_UTC)b'"1970-01-01T00:00:00+00:00"'Enums with members that are not supported types can be serialized usingdefault:>>>importenum,xorjson>>>classCustom:def__init__(self,val):self.val=valdefdefault(obj):ifisinstance(obj,Custom):returnobj.valraiseTypeErrorclassCustomEnum(enum.Enum):ONE=Custom(1)>>>xorjson.dumps(CustomEnum.ONE,default=default)b'1'floatxorjson serializes and deserializes double precision floats with no loss of
precision and consistent rounding.xorjson.dumps()serializes Nan, Infinity, and -Infinity, which are not
compliant JSON, asnull:>>>importxorjson,ujson,rapidjson,json>>>xorjson.dumps([float("NaN"),float("Infinity"),float("-Infinity")])b'[null,null,null]'>>>ujson.dumps([float("NaN"),float("Infinity"),float("-Infinity")])OverflowError:InvalidInfvaluewhenencodingdouble>>>rapidjson.dumps([float("NaN"),float("Infinity"),float("-Infinity")])'[NaN,Infinity,-Infinity]'>>>json.dumps([float("NaN"),float("Infinity"),float("-Infinity")])'[NaN, Infinity, -Infinity]'intxorjson serializes and deserializes 64-bit integers by default. The range
supported is a signed 64-bit integer's minimum (-9223372036854775807) to
an unsigned 64-bit integer's maximum (18446744073709551615). This
is widely compatible, but there are implementations
that only support 53-bits for integers, e.g.,
web browsers. For those implementations,dumps()can be configured to
raise aJSONEncodeErroron values exceeding the 53-bit range.>>>importxorjson>>>xorjson.dumps(9007199254740992)b'9007199254740992'>>>xorjson.dumps(9007199254740992,option=xorjson.OPT_STRICT_INTEGER)JSONEncodeError:Integerexceeds53-bitrange>>>xorjson.dumps(-9007199254740992,option=xorjson.OPT_STRICT_INTEGER)JSONEncodeError:Integerexceeds53-bitrangenumpyxorjson natively serializesnumpy.ndarrayand individualnumpy.float64,numpy.float32,numpy.int64,numpy.int32,numpy.int16,numpy.int8,numpy.uint64,numpy.uint32,numpy.uint16,numpy.uint8,numpy.uintp,numpy.intp,numpy.datetime64, andnumpy.boolinstances.xorjson is faster than all compared libraries at serializing
numpy instances. Serializing numpy data requires specifyingoption=xorjson.OPT_SERIALIZE_NUMPY.>>>importxorjson,numpy>>>xorjson.dumps(numpy.array([[1,2,3],[4,5,6]]),option=xorjson.OPT_SERIALIZE_NUMPY,)b'[[1,2,3],[4,5,6]]'The array must be a contiguous C array (C_CONTIGUOUS) and one of the
supported datatypes.Note a difference between serializingnumpy.float32usingndarray.tolist()orxorjson.dumps(..., option=xorjson.OPT_SERIALIZE_NUMPY):tolist()converts
to adoublebefore serializing and xorjson's native path does not. This
can result in different rounding.numpy.datetime64instances are serialized as RFC 3339 strings and
datetime options affect them.>>>importxorjson,numpy>>>xorjson.dumps(numpy.datetime64("2021-01-01T00:00:00.172"),option=xorjson.OPT_SERIALIZE_NUMPY,)b'"2021-01-01T00:00:00.172000"'>>>xorjson.dumps(numpy.datetime64("2021-01-01T00:00:00.172"),option=(xorjson.OPT_SERIALIZE_NUMPY|xorjson.OPT_NAIVE_UTC|xorjson.OPT_OMIT_MICROSECONDS),)b'"2021-01-01T00:00:00+00:00"'If an array is not a contiguous C array, contains an unsupported datatype,
or contains anumpy.datetime64using an unsupported representation
(e.g., picoseconds), xorjson falls through todefault. Indefault,obj.tolist()can be specified. If an array is malformed, which
is not expected,xorjson.JSONEncodeErroris raised.This measures serializing 92MiB of JSON from annumpy.ndarraywith
dimensions of(50000, 100)andnumpy.float64values:LibraryLatency (ms)RSS diff (MiB)vs. xorjsonxorjson194991.0ujsonrapidjson3,04830915.7simplejson3,02329715.6json3,13329716.1This measures serializing 100MiB of JSON from annumpy.ndarraywith
dimensions of(100000, 100)andnumpy.int32values:LibraryLatency (ms)RSS diff (MiB)vs. xorjsonxorjson1781151.0ujsonrapidjson1,5125518.5simplejson1,6065049.0json1,5065038.4This measures serializing 105MiB of JSON from annumpy.ndarraywith
dimensions of(100000, 200)andnumpy.boolvalues:LibraryLatency (ms)RSS diff (MiB)vs. xorjsonxorjson1571201.0ujsonrapidjson7103274.5simplejson9313985.9json9964006.3In these benchmarks, xorjson serializes natively, ujson is blank because it
does not support adefaultparameter, and the other libraries serializendarray.tolist()viadefault. The RSS column measures peak memory
usage during serialization. This can be reproduced using thepynumpyscript.xorjson does not have an installation or compilation dependency on numpy. The
implementation is independent, readingnumpy.ndarrayusingPyArrayInterface.strxorjson is strict about UTF-8 conformance. This is stricter than the standard
library's json module, which will serialize and deserialize UTF-16 surrogates,
e.g., "\ud800", that are invalid UTF-8.Ifxorjson.dumps()is given astrthat does not contain valid UTF-8,xorjson.JSONEncodeErroris raised. Ifloads()receives invalid UTF-8,xorjson.JSONDecodeErroris raised.xorjson and rapidjson are the only compared JSON libraries to consistently
error on bad input.>>>importxorjson,ujson,rapidjson,json>>>xorjson.dumps('\ud800')JSONEncodeError:strisnotvalidUTF-8:surrogatesnotallowed>>>ujson.dumps('\ud800')UnicodeEncodeError:'utf-8'codec...>>>rapidjson.dumps('\ud800')UnicodeEncodeError:'utf-8'codec...>>>json.dumps('\ud800')'"\\ud800"'>>>xorjson.loads('"\\ud800"')JSONDecodeError:unexpectedendofhexescapeatline1column8:line1column1(char0)>>>ujson.loads('"\\ud800"')''>>>rapidjson.loads('"\\ud800"')ValueError:Parseerroratoffset1:Thesurrogatepairinstringisinvalid.>>>json.loads('"\\ud800"')'\ud800'To make a best effort at deserializing bad input, first decodebytesusing
thereplaceorlossyargument forerrors:>>>importxorjson>>>xorjson.loads(b'"\xed\xa0\x80"')JSONDecodeError:strisnotvalidUTF-8:surrogatesnotallowed>>>xorjson.loads(b'"\xed\xa0\x80"'.decode("utf-8","replace"))'���'uuidxorjson serializesuuid.UUIDinstances toRFC 4122format, e.g.,
"f81d4fae-7dec-11d0-a765-00a0c91e6bf6".>>>importxorjson,uuid>>>xorjson.dumps(uuid.UUID('f81d4fae-7dec-11d0-a765-00a0c91e6bf6'))b'"f81d4fae-7dec-11d0-a765-00a0c91e6bf6"'>>>xorjson.dumps(uuid.uuid5(uuid.NAMESPACE_DNS,"python.org"))b'"886313e1-3b8a-5372-9b90-0c9aee199e5d"'TestingThe library has comprehensive tests. There are tests against fixtures in theJSONTestSuiteandnativejson-benchmarkrepositories. It is tested to not crash against theBig List of Naughty Strings.
It is tested to not leak memory. It is tested to not crash
against and not accept invalid UTF-8. There are integration tests
exercising the library's use in web servers (gunicorn using multiprocess/forked
workers) and when
multithreaded. It also uses some tests from the ultrajson library.xorjson is the most correct of the compared libraries. This graph shows how each
library handles a combined 342 JSON fixtures from theJSONTestSuiteandnativejson-benchmarktests:LibraryInvalid JSON documents not rejectedValid JSON documents not deserializedxorjson00ujson310rapidjson60simplejson100json170This shows that all libraries deserialize valid JSON but only xorjson
correctly rejects the given invalid JSON fixtures. Errors are largely due to
accepting invalid strings and numbers.The graph above can be reproduced using thepycorrectnessscript.PerformanceSerialization and deserialization performance of xorjson is better than
ultrajson, rapidjson, simplejson, or json. The benchmarks are done on
fixtures of real data:twitter.json, 631.5KiB, results of a search on Twitter for "一", containing
CJK strings, dictionaries of strings and arrays of dictionaries, indented.github.json, 55.8KiB, a GitHub activity feed, containing dictionaries of
strings and arrays of dictionaries, not indented.citm_catalog.json, 1.7MiB, concert data, containing nested dictionaries of
strings and arrays of integers, indented.canada.json, 2.2MiB, coordinates of the Canadian border in GeoJSON
format, containing floats and arrays, indented.Latencytwitter.json serializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson0.335601ujson2.14737.5rapidjson1.75965.9simplejson3.132410.8json2.53978.9twitter.json deserializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson1.28111ujson2.93472.3rapidjson5.11974.1simplejson2.83522.3json3.32992.7github.json serializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson0399161ujson0.249698rapidjson0.257546.9simplejson0.3291613.7json0.3391610.3github.json deserializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson0.198791ujson0.240592.3rapidjson0.337722.6simplejson0.250921.9json0.249442citm_catalog.json serializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson0.616011ujson2.93404.8rapidjson2.34293.8simplejson12.57920.3json5.71769.2citm_catalog.json deserializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson2.93411ujson52021.7rapidjson8.31192.8simplejson6.61512.2json71412.4canada.json serializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson5.31861ujson17.2573.2rapidjson45.3228.5simplejson70.91413.3json49.7209.3canada.json deserializationLibraryMedian latency (milliseconds)Operations per secondRelative (latency)xorjson6.71491ujson15.2662.3rapidjson30.1334.5simplejson29.9324.5json30.4324.5Memoryxorjson as of 3.7.0 has higher baseline memory usage than other libraries
due to a persistent buffer used for parsing. Incremental memory usage when
deserializing is similar to the standard library and other third-party
libraries.This measures, in the first column, RSS after importing a library and reading
the fixture, and in the second column, increases in RSS after repeatedly
callingloads()on the fixture.twitter.jsonLibraryimport, read() RSS (MiB)loads() increase in RSS (MiB)xorjson15.73.4ujson16.43.4rapidjson16.64.4simplejson14.51.8json13.91.8github.jsonLibraryimport, read() RSS (MiB)loads() increase in RSS (MiB)xorjson15.20.4ujson15.40.4rapidjson15.70.5simplejson13.70.2json13.30.1citm_catalog.jsonLibraryimport, read() RSS (MiB)loads() increase in RSS (MiB)xorjson16.810.1ujson17.310.2rapidjson17.628.7simplejson15.830.1json14.820.5canada.jsonLibraryimport, read() RSS (MiB)loads() increase in RSS (MiB)xorjson17.222.1ujson17.418.3rapidjson1823.5simplejson15.721.4json15.420.4ReproducingThe above was measured using Python 3.11.6 on Linux (amd64) with
xorjson 3.9.11, ujson 5.9.0, python-rapidson 1.14, and simplejson 3.19.2.The latency results can be reproduced using thepybenchandgraphscripts. The memory results can be reproduced using thepymemscript.QuestionsWhy can't I install it from PyPI?Probablypipneeds to be upgraded to version 20.3 or later to support
the latest manylinux_x_y or universal2 wheel formats."Cargo, the Rust package manager, is not installed or is not on PATH."This happens when there are no binary wheels (like manylinux) for your
platform on PyPI. You can installRustthroughrustupor a package manager and then it will compile.Will it deserialize to dataclasses, UUIDs, decimals, etc or support object_hook?No. This requires a schema specifying what types are expected and how to
handle errors etc. This is addressed by data validation libraries a
level above this.Will it serialize tostr?No.bytesis the correct type for a serialized blob.PackagingTo package xorjson requires at leastRust1.65
and thematurinbuild tool. The recommended
build command is:maturinbuild--release--stripIt benefits from also having a C build environment to compile a faster
deserialization backend. See this project'smanylinux_2_28builds for an
example using clang and LTO.The project's own CI tests againstnightly-2024-02-03and stable 1.65. It
is prudent to pin the nightly version because that channel can introduce
breaking changes.xorjson is tested for amd64, aarch64, arm7, ppc64le, and s390x on Linux. It
is tested for amd64 on macOS and cross-compiles for aarch64. For Windows
it is tested on amd64 and i686.There are no runtime dependencies other than libc.The source distribution on PyPI contains all dependencies' source and can be
built without network access. The file can be downloaded fromhttps://files.pythonhosted.org/packages/source/o/xorjson/xorjson-${version}.tar.gz.xorjson's tests are included in the source distribution on PyPI. The
requirements to run the tests are specified intest/requirements.txt. The
tests should be run as part of the build. It can be run withpytest -q test.LicenseThe original orjson was written by ijl <ijl@mailbox.org>, copyright 2018 - 2024, available
to you under either the Apache 2 license or MIT license at your choice.