Package: serializor
Serialize common python/numpy/pandas objects.Created to be used in a project, this package is published to github
for ease of management and installation across different modules.FeaturesThis package is designed to serialize common python objects, and
deserializes them back to their original (or compatiable) python
data type.Supported data types includes:boolean:bool&numpy.bool_-> deserialize toboolinteger:int&numpy.int&numpy.uint-> deserialize tointfloat:float&numpy.float_-> deserialize tofloatdecimal:decimal.Decimal-> deserialize todecimal.Decimalstring:str-> deserialize tostrbytes:bytes-> deserialize tobytesdate:datetime.date-> deserialize todatetime.datetime:datetime.time-> deserialize todatetime.timedatetime:datetime.datetime&pandas.Timestamp-> deserialize todatetime.datetimedatetime64*:numpy.datetime64&time.struct_time-> deserialize todatetime.datetimetimedelta:datetime.timedelta&pandas.Timedelta-> deserialize todatetime.timedeltatimedelta64:numpy.timedelta64-> deserialize todatetime.timedeltaNone:None&numpy.nan-> deserialize toNonelist:listof above supported data types -> deserialize tolisttuple:tupleof above supported data types -> deserialize tolistset:setof above supported data types -> deserialize tolistfrozenset:frozensetof above supported data types -> deserialize tolistdict:dictof above supported data types -> deserialize todictnumpy.record:numpy.recordof above supported data types -> deserialize tolistnumpy.ndarray:numpy.ndarrayof above supported data types -> deserialize tonp.ndarraypandas.Series:pandas.Seriesof above supported data types -> deserialize topandas.Seriespandas.DataFrame:pandas.DataFrameof above supported data types -> deserialize topandas.DataFrameThe core serialization is based onorjson, which is a fast, superb JSON
library for Python. As a result, thedumpsresult is inbytesformat
(just asorjsondoes). However, in order to serialize non-JSON objects,
andloadsthe serialized data back to its original python date type,dumpsresult can not be simply treated as serialized JSON. Instead, requires theloadsfunction in this package to deserialize it properly.Also, in order to support data types such asDecimal,pandas.Series, etc.,
The serialization performance will be slower thanorjsonwhen non-standard
data types are involved. The deserialization performance will always be slower
thanorjsondue to the process of validation and reconstruction of the
serialized data to its original (or compatible) python data type.InstallationInstall fromPyPipipinstallserializorInstall fromgithubpipinstallgit+https://github.com/AresJef/Serializor.gitCompatibilityOnly support for python 3.10 and above.BooleanAll supported boolean data types will be deserialized to python
nativebooltype.importserializorassr,numpyasnp## Python native booleanobj=Trues=sr.dumps(obj)d=sr.loads(s)# True <class 'bool'>## Numpy booleanobj=np.bool_(True)s=sr.dumps(obj)d=sr.loads(s)# True <class 'bool'>IntegerAll supported integer data types will be deserialized to python
nativeinttype.importserializorassr,numpyasnp## Python native integerobj=1s=sr.dumps(obj)d=sr.loads(s)# 1 <class 'int'>## Numpy integerobj=np.int64(1)s=sr.dumps(obj)d=sr.loads(s)# 1 <class 'int'>## Numpy unsigned integerobj=np.uint64(1)s=sr.dumps(obj)d=sr.loads(s)# 1 <class 'int'>FloatAll supported float data types will be deserialized to python
nativefloattype.importserializorassr,numpyasnp## Python native floatobj=1.0s=sr.dumps(obj)d=sr.loads(s)# 1.0 <class 'float'>## Numpy floatobj=np.float64(1.0)s=sr.dumps(obj)d=sr.loads(s)# 1.0 <class 'float'>DecimalDecimal data type will be deserialized back todecimal.Decimaltype.importserializorassrfromdecimalimportDecimalobj=Decimal('1.0')s=sr.dumps(obj)d=sr.loads(s)# 1.0 <class 'decimal.Decimal'>StringString data type will be deserialized back tostrtype.importserializorassrobj='STRING's=sr.dumps(obj)d=sr.loads(s)# 'STRING' <class 'str'>BytesBytes data type will be deserialized back tobytestype.importserializorassrobj=b'BYTES's=sr.dumps(obj)d=sr.loads(s)# b'BYTES' <class 'bytes'>DateDate data type will be deserialized back todatetime.datetype.importserializorassr,datetimeobj=datetime.date(2023,8,27)s=sr.dumps(obj)d=sr.loads(s)# 2023-08-27 <class 'datetime.date'>TimeTime data type will be deserialized back todatetime.timetype.importserializorassr,datetime# Time (timezone-naive)obj=datetime.time(21,12,31,354654)s=sr.dumps(obj)d=sr.loads(s)# 21:12:31.354654 <class 'datetime.time'># Time (timezone-aware)tzinfo=datetime.timezone(datetime.timedelta(hours=8),"CST")obj=datetime.time(21,12,31,354654,tzinfo=tzinfo)s=sr.dumps(obj)d=sr.loads(s)# 21:12:31.354654+08:00 <class 'datetime.time'>DatetimeAll supported datetime data types will be deserialized to python
nativedatetime.datetimetype.importserializorassr,datetime,numpyasnp,time# Datetime (timezone-naive) / pandas.Timestampobj=datetime.datetime(2023,8,27,21,16,4,780600)s=sr.dumps(obj)d=sr.loads(s)# 2023-08-27 21:16:04.780600 <class 'datetime.datetime'># Datetime (timezone-aware) / pandas.Timestamptzinfo=datetime.timezone(datetime.timedelta(hours=8),"CST")obj=datetime.datetime(2023,8,27,21,16,4,780600,tzinfo=tzinfo)s=sr.dumps(obj)d=sr.loads(s)# 2023-08-27 21:16:04.780600+08:00 <class 'datetime.datetime'># Numpy datetime64obj=np.datetime64('2023-08-27T21:16:04.780600')s=sr.dumps(obj)d=sr.loads(s)# 2023-08-27 21:16:04.780600 <class 'datetime.datetime'># Time struct_timeobj=time.struct_time((2023,8,27,21,16,4,0,0,0))s=sr.dumps(obj)d=sr.loads(s)# 2023-08-27 21:16:04 <class 'datetime.datetime'>TimedeltaAll supported timedelta data types will be deserialized to python
nativedatetime.timedeltatype.importserializorassr,datetime,numpyasnp# Timedelta / pandas.Timedeltaobj=datetime.timedelta(days=1)s=sr.dumps(obj)d=sr.loads(s)# 1 day, 0:00:00 <class 'datetime.timedelta'># Numpy timedelta64obj=np.timedelta64(1,'D')s=sr.dumps(obj)d=sr.loads(s)# 1 day, 0:00:00 <class 'datetime.timedelta'>Sequence (list, tuple, set, frozenset, np.record)All supported sequence data types will be deserialized to python
nativelisttype.importserializorassr,datetime,numpyasnp,pandasaspd## List / Tuple / Set / Frozenset / Numpy recordtzinfo=datetime.timezone(datetime.timedelta(hours=8),'CST')obj=[True,np.bool_(False),1,np.int8(-2),np.int16(-3),np.int32(-4),np.int64(-5),np.uint16(6),np.uint32(7),np.uint64(8),1.1,np.float16(2.2),np.float32(3.3),np.float64(4.4),Decimal('3.3'),'STRING',b'BYTES',datetime.date(2023,8,27),datetime.datetime(2023,1,1,1,1,1,1),datetime.datetime(2023,1,1,1,1,1,1,tzinfo=tzinfo),pd.Timestamp('2023-01-01 01:01:01.000001'),pd.Timestamp('2023-01-01 01:01:01.000001+0800'),np.datetime64('2023-01-01T01:01:01.000001'),datetime.time(1,1,1,1),datetime.time(1,1,1,1,tzinfo=tzinfo),datetime.timedelta(days=1),pd.Timedelta(2,unit='D'),np.timedelta64(3,'D'),None,]s=sr.dumps(obj)d=sr.loads(s)# [#   True, False,#   1, -2, -3, -4, -5, 6, 7, 8,#   1.1, 2.19921875, 3.299999952316284, 4.4,#   Decimal('3.3'),#   'STRING',#   b'BYTES',#   datetime.date(2023, 8, 27),#   datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   datetime.datetime(2023, 1, 1, 1, 1, 1, 1, tzinfo=...),#   datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   datetime.datetime(2023, 1, 1, 1, 1, 1, 1, tzinfo=...),#   datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   datetime.time(1, 1, 1, 1),#   datetime.time(1, 1, 1, 1, tzinfo=...),#   datetime.timedelta(days=1), datetime.timedelta(days=2), datetime.timedelta(days=3),#   None# ]DictionaryDictionary data type will be deserialized to python nativedicttype.
Notice, the key of the dictionary must be ofstrtype only.importserializorassr,datetime,numpyasnp,pandasaspd## Dictionarytzinfo=datetime.timezone(datetime.timedelta(hours=8),'CST')obj={"bool":True,"np_bool":np.bool_(False),"int":1,"np_int8":np.int8(-2),"np_int16":np.int16(-3),"np_int32":np.int32(-4),"np_int64":np.int64(-5),"np_uint16":np.uint16(6),"np_uint32":np.uint32(7),"np_uint64":np.uint64(8),"float":1.1,"np_float16":np.float16(2.2),"np_float32":np.float32(3.3),"np_float64":np.float64(4.4),"decimal":Decimal('3.3'),"str":'STRING',"bytes":b'BYTES',"date":datetime.date(2023,8,27),"datetime":datetime.datetime(2023,1,1,1,1,1,1),"datetime_tz":datetime.datetime(2023,1,1,1,1,1,1,tzinfo=tzinfo),"pd_timestamp":pd.Timestamp('2023-01-01 01:01:01.000001'),"pd_timestamp_tz":pd.Timestamp('2023-01-01 01:01:01.000001+0800'),"np_datetime64":np.datetime64('2023-01-01T01:01:01.000001'),"time":datetime.time(1,1,1,1),"time_tz":datetime.time(1,1,1,1,tzinfo=tzinfo),"timedelta":datetime.timedelta(days=1),"pd_timedelta":pd.Timedelta(2,unit='D'),"np_timedelta64":np.timedelta64(3,'D'),"none":None,}s=sr.dumps(obj)d=sr.loads(s)# {#   'bool': True, 'np_bool': False,#   'int': 1, 'np_int8': -2, 'np_int16': -3, 'np_int32': -4, 'np_int64': -5,#   'np_uint16': 6, 'np_uint32': 7, 'np_uint64': 8,#   'float': 1.1, 'np_float16': 2.19921875, 'np_float32': 3.299999952316284, 'np_float64': 4.4,#   'decimal': Decimal('3.3'),#   'str': 'STRING',#   'bytes': b'BYTES',#   'date': datetime.date(2023, 8, 27),#   'datetime': datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   'datetime_tz': datetime.datetime(2023, 1, 1, 1, 1, 1, 1, tzinfo=...),#   'pd_timestamp': datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   'pd_timestamp_tz': datetime.datetime(2023, 1, 1, 1, 1, 1, 1, tzinfo=...),#   'np_datetime64': datetime.datetime(2023, 1, 1, 1, 1, 1, 1),#   'time': datetime.time(1, 1, 1, 1),#   'time_tz': datetime.time(1, 1, 1, 1, tzinfo=...),#   'timedelta': datetime.timedelta(days=1),#   'pd_timedelta': datetime.timedelta(days=2),#   'np_timedelta64': datetime.timedelta(days=3),#   'none': None# }Numpy ndarrayNumpy ndarray data type will be deserialized tonumpy.ndarraytype.importserializorassr,numpyasnpobj=np.array([[1,2,3],[4,5,6]])s=sr.dumps(obj)d=sr.loads(s)# [[1 2 3], [4 5 6]] <class 'numpy.ndarray'>Pandas SeriesPandas Series data type will be deserialized topandas.Seriestype.importserializorassr,pandasaspd# Series of integersobj=pd.Series([1,2,3],name='Integers')s=sr.dumps(obj)d=sr.loads(s)# <pandas.Series># 0    1# 1    2# 2    3# Name: Integers, dtype: int64# Series of Timestampsobj=pd.Series(pd.date_range("2023-01-01",periods=3,freq="D"),name="Dates")s=sr.dumps(obj)d=sr.loads(s)# <pandas.Series># 0   2023-01-01# 1   2023-01-02# 2   2023-01-03# Name: Dates, dtype: datetime64[ns]Pandas DataFramePandas DataFrame data type will be deserialized topandas.DataFrametype.importserializorassr,pandasaspd,numpyasnp,datetimedt=datetime.datetime(2023,8,1,12,34,56,789)val={"bool":True,"int":-1,"float":1.1,"decimal":Decimal("3.3"),"str":"STRING","bytes":b"BYTES","datetime":dt,"time":dt.time(),"timedelta":datetime.timedelta(1),}obj=pd.DataFrame([val]*3)s=sr.dumps(obj)d=sr.loads(s)# <pandas.DataFrame># bool int float decimal str bytes datetime time timedelta# 0 True -1 1.1 3.3 STRING b'BYTES' 2023-08-01 12:34:56.000789 12:34:56.000789 1 days# 1 True -1 1.1 3.3 STRING b'BYTES' 2023-08-01 12:34:56.000789 12:34:56.000789 1 days# 2 True -1 1.1 3.3 STRING b'BYTES' 2023-08-01 12:34:56.000789 12:34:56.000789 1 daysAcknowledgementsserializor is based on several open-source repositories.cytimesnumpyorjsonpandas