Package: serde-numpy
serde-numpyserde-numpy is a library for efficient deserializing of various file formats directly into numpy arrays.See how it works for:imagesjson formatsInstallationCurrently only available for linux, python >= 3.7pipinstall--upgradepip
pipinstallserde-numpyImage formatsExample usage>>>fromserde_numpyimportdecode_jpeg,read_jpeg,decode_png,read_png>>>>>>img=read_jpeg("test.jpg")>>>imgarray([[[75,29,82],[96,56,133],[72,47,168],[63,56,179]],[[216,176,203],[173,139,190],[111,93,188],[129,128,225]],[[75,46,21],[73,51,48],[81,73,115],[157,167,209]],[[165,142,99],[181,165,144],[169,169,188],[185,203,222]]],dtype=uint8)>>>>>>byte_array=open("test.png","rb").read()>>>img=decode_png(byte_array)>>>imgarray([[[33,47,146],[206,19,120],[185,8,55],[33,54,176]],[[252,156,169],[169,139,100],[24,128,222],[136,146,213]],[[28,24,192],[184,51,58],[39,61,252],[237,165,113]],[[239,111,72],[30,242,38],[165,161,223],[91,246,217]]],dtype=uint8)BenchmarksAll benchmarks were performed on an AMD Ryzen 9 3950X (Python 3.8.12, numpy 1.23.2, orjson 3.6.4). We compare serde_numpy'sdecode_pnganddecode_jpegversus pillow'sImage.open+np.asarray(which is the de facto standard for libraries than do a lot of image loading e.g. pytorch'storchvision).JPEGJPEG decoding for square images:PNGPNG decoding for square images:JSON FormatsMotivationIf you've ever done something like this in your code:data=json.load(open("data.json"))arr=np.array(data["x"])then this library does it faster by using minimal array allocations and less python.Speed ups are 1.5x - 8x times faster, depending on array sizes (and CPU), when compared to orjson + numpy.UsageThe user specifies the numpy dtypes within astructurecorresponding to the data that they want to deserialize.N-dimensional arrayA subset of the json's (or msgpack) keys are specified in thestructurewhich is used to initialize theNumpyDeserializerand then that subset of keys are deserialized accordingly:>>>fromserde_numpyimportNumpyDeserializer>>>>>>json_str=b"""... {...     "name": "coordinates",...     "version": "0.1.0",...     "arr": [[1.254439975231648, -0.6893827594332794],...             [-0.2922560025562806, 0.5204819306523419]]... }... """>>>>>>structure={...'name':str,...'arr':np.float32...}>>>>>>deserializer=NumpyDeserializer.from_dict(structure)>>>>>>deserializer.deserialize_json(json_str){'arr':array([[1.25444,-0.68938273],[-0.292256,0.52048194]],dtype=float32),'name':'coordinates'}Transposed arraysSometimes people store data in jsons in a row-wise fashion as opposed to column-wise. Therefore each row can contain multiple dtypes. serde-numpy allows you to specify the types of each row and then deserializes into columns. To tell the numpy deserializer that you want to transpose the columns put square brackets outside either a dictionary[{key: Type, ...}]like this example:>>>json_str=b"""... {...     "df": [{"a": 3, "b": 4.23},...            {"a": 4, "b": 5.12}]... }... """>>>>>>structure={"df":[{"a":np.uint16,"b":np.float64}]}>>>>>>deserializer=NumpyDeserializer.from_dict(structure)>>>>>>deserializer.deserialize_json(json_str){'df':{'b':array([4.23,5.12]),'a':array([3,4],dtype=uint16)}}orput square brackets outside a list[[Type, ...]]of types:>>>json_str=b"""... {...     "df": [["i", true],...            ["j", false],...            ["k", true]]... }... """>>>>>>structure={"df":[[str,np.bool_]]}>>>>>>deserializer=NumpyDeserializer.from_dict(structure)>>>>>>deserializer.deserialize_json(json_str){'df':[['i','j','k'],array([True,False,True])]}Currently supported data formats:JSON::NumpyDeserializer.deserialize_jsonMessagePack::NumpyDeserializer.deserialize_msgpackCurrently supported types:Numpy types:np.int8np.int16np.int32np.int64np.uint8np.uint16np.uint32np.uint64np.float32np.float64np.bool_Python types:intfloatstrdictlistBenchmarksAll benchmarks were performed on an AMD Ryzen 9 3950X (Python 3.8.12, numpy 1.23.2, orjson 3.6.4). Orjson was selected as the comparison as it is the fastest on python json benchmarks and we have also found it to be fastest in practice.2D Array deserializationTwo tests are performed. The number of rows are kept constant at 10 while varying the number of columnsandthe number of columns are kept constant at 10 while varying the number of rows. We compare againstorjson.loads+np.arraywith the desired data type. Results are presented below for deserializing arrays of various data types:Transposed arrays deserializationFor this test we test the speed of deserializing multiple data types which have been serialized in a row-wise fashion and converting it to column-wise arrays during deserializition.