Package: cerializer
CerializerCerializer is an Avro de/serialization library that aims at providing an even faster alternative to FastAvro and Avro standard library.This speed increase does not come without a cost. Cerializer will work only with predefined set of schemata for which it will generate tailor made Cython code. This way, the overhead caused by the universality of other serialization libraries will be avoided.Special credit needs to be given toFastAvrolibrary, by which is this project heavily inspired.Example of a schema and the corresponding codeSCHEMA{'name':'array_schema','doc':'Array schema','namespace':'cerializer','type':'record','fields':[{'name':'order_id','doc':'Id of order','type':'string'},{'name':'trades','type':{'type':'array','items':['string','int']}}]}CORRESPONDING CODEdefserialize(data,output):cdefbytearraybuffer=bytearray()cdefdictdatumcdefstrtype_0write.write_string(buffer,data['order_id'])iflen(data['trades'])>0:write.write_long(buffer,len(data['trades']))forval_0indata['trades']:iftype(val_0)istuple:type_0,val_1=val_0iftype_0=='string':write.write_long(buffer,0)write.write_string(buffer,val_1)eliftype_0=='int':write.write_long(buffer,1)write.write_int(buffer,val_1)else:iftype(val_0)isstr:write.write_long(buffer,0)write.write_string(buffer,val_0)eliftype(val_0)isint:write.write_long(buffer,1)write.write_int(buffer,val_0)write.write_long(buffer,0)output.write(buffer)defdeserialize(fo):cdeflonglongi_0cdeflonglongi_1cdeflongi_2data={}data['order_id']=read.read_string(fo)data['trades']=[]i_1=read.read_long(fo)whilei_1!=0:ifi_1<0:i_1=-i_1read.read_long(fo)fori_0inrange(i_1):i_2=read.read_int(fo)ifi_2==0:val_2=read.read_string(fo)ifi_2==1:val_2=read.read_int(fo)data['trades'].append(val_2)i_1=read.read_long(fo)returndataUsage ExampleCreate an instance of CerializerSchemata
For initializing CerializerSchemata it is necessary to supply a list of tuples in form of (schema_identifier, schema)
where schema_identifier is a string and schema is a dict representing the Avro schema.
schema tuple = (namespace.schema_name, schema). eg.:importcerializer.schema_handlerimportosimportyamldeflist_schemata():# iterates through all your schemata and yields schema_identifier and path to schema folderraiseNotImplementeddefschemata()->cerializer.schema_handler.CerializerSchemata:schemata=[]forschema_identifier,schema_rootinlist_schemata():schema_tuple=schema_identifier,yaml.unsafe_load(# type: ignoreopen(os.path.join(schema_root,'schema.yaml')))schemata.append(schema_tuple)returncerializer.schema_handler.CerializerSchemata(schemata)Create an instance of Cerializer for each of your schemata by callingcerializer_handler.Cerializer.
eg.cerializer_instance = cerializer_handler.Cerializer(cerializer_schemata, schema_namespace, schema_name)This will create an instance of Cerializer that can serialize and deserialize data in the particular schema format.Use the instance accordingly.
eg.:data_record={'order_id':'aaaa','trades':[123,456,765]}cerializer_instance=cerializer.cerializer_handler.Cerializer(cerializer_schemata,'school','student')serialized_data=cerializer_instance.serialize(data_record)print(serialized_data)Serialized datab'\x08aaaa\x06\x02\xf6\x01\x02\x90\x07\x02\xfa\x0b\x00'You can also useserialize_intoif you already have an IO buffer:output=io.BytesIO()cerializer_instance.serialize_into(output,data_record)print(output.getvalue())Benchmarkcerializer.default_schema:3            2.5661 times faster,   0.0209s : 0.0082s
cerializer.fixed_decimal_schema:1      1.2795 times faster,   0.1588s : 0.1241s
cerializer.int_date_schema:1           2.8285 times faster,   0.0273s : 0.0097s
cerializer.plain_int:1                 2.2334 times faster,   0.0146s : 0.0065s
cerializer.timestamp_schema_micros:1   2.3759 times faster,   0.0577s : 0.0243s
cerializer.default_schema:2            2.8129 times faster,   0.0240s : 0.0085s
cerializer.array_schema:3              1.2177 times faster,   0.3088s : 0.2536s
cerializer.timestamp_schema:1          2.5928 times faster,   0.0577s : 0.0223s
cerializer.array_schema:2              1.4756 times faster,   0.6542s : 0.4434s
cerializer.union_schema:1              3.0796 times faster,   0.0284s : 0.0092s
cerializer.bytes_decimal_schema:1      1.8449 times faster,   0.0490s : 0.0266s
cerializer.array_schema:1              2.1771 times faster,   0.0344s : 0.0158s
cerializer.string_uuid_schema:1        1.8887 times faster,   0.0494s : 0.0262s
cerializer.map_schema:2                2.0896 times faster,   0.0331s : 0.0158s
cerializer.fixed_schema:1              3.4042 times faster,   0.0213s : 0.0062s
cerializer.long_time_micros_schema:1   2.3747 times faster,   0.0352s : 0.0148s
cerializer.array_schema:4              2.8779 times faster,   0.0591s : 0.0205s
cerializer.default_schema:1            2.0182 times faster,   0.0393s : 0.0195s
cerializer.map_schema:1                3.4610 times faster,   0.0597s : 0.0172s
cerializer.string_schema:1             2.2048 times faster,   0.0352s : 0.0159s
cerializer.reference_schema:1          2.9309 times faster,   0.1525s : 0.0520s
cerializer.enum_schema:1               3.0065 times faster,   0.0217s : 0.0072s
cerializer.tree_schema:1               4.0494 times faster,   0.0869s : 0.0215s
cerializer.huge_schema:1               2.8161 times faster,   0.1453s : 0.0516s
AVERAGE: 1.7814 times fasterMeasured against Fastavro using the benchmark in Cerializer/tests.Device: ASUS ZenBook 14 UM425QA, AMD Ryzen 7 5800H, 16 GB 2133 MHz LPDDR4X