Package: scityping
Scityping: type hints and serializers for scientific typesA collection of type annotations specifiers for Python types common in the fields of scientific computing, data science and machine learning. Most types come with a pair of JSON serializer/deserializer functions, which can be used both to archive and transfer data.
In contrast to pickling, the resulting JSON data is both future safe and human readable, which makes especially useful forreproducible research.Some supported data types (seethe documentationfor a full list):rangeNumPy arrays[^1]NumPy random number generatorsSciPy statistical distributionsPure functionsPyTorch tensorsValues with units (Pint)More importantly,scitypingis built upon asimple yet flexible mechanismfor defining custom JSON serializers and associating them to preexisting types. This means that any Python type which is be serializable, is also serializable in practice. So you can take an existing analysis pipeline, and make the input or output of any intermediate step exportable to JSON – without ever touching the pipeline itself ![^1]: Large arrays are compressed, but saved alongside a human-readable snippet to allow inspection.Compatibility with PydanticTypes were designed and tested with Pydantic in mind, so they can be used as types in PydanticBaseModels. To support serialization, usefromscityping.pydanticimportBaseModel(Don’t worry: this is not a rewrite ofBaseModel. It is just a small wrapper class which adds a hook for our extensible type machinery.)Example usage:fromscityping.numpyimportArrayfromscityping.pydanticimportBaseModelclassMyModel(BaseModel):data:Array[float,1]# 1D array of floatsmodel=MyModel(data=[1,2,3])print(model.data)# array([1., 2., 3.,], dtype=float)json_data=model.json()print(json_data)# {"data": ["scityping.numpy._ArrayType", {"data": {"data": [1.0, 2.0, 3.0], "dtype": ["scityping.numpy.DType", {"desc": "float64"}]}}]}model2=MyModel.parse_raw(json_data)print(model2.data)# array([1., 2., 3.,], dtype=float)Usage without PydanticSerializers are class methods, so they can also be used by accessing them directly.importjsonimportnumpyasnpfromscitypingimportSerializablefromscityping.jsonimportscityping_encoderx=np.array([1,1,2,3,5])reduced_x=Serializable.reduce(x)print(reduced_x)# ('scityping.numpy._ArrayType', _ArrayType.Data(data=ListArrayData(data=[1, 1, 2, 3, 5], dtype=dtype('int64'))))json_x=json.dumps(x,default=scityping_encoder)# "['scityping.numpy._ArrayType', {'data': {'data': [1, 1, 2, 3, 5], 'dtype': ['scityping.numpy.DType', {'desc': 'int64'}]}}]"x2=Serializable.validate(reduced_x)x2=Serializable.validate(json.loads(json_x))# Equivalentprint(x2)# array([1, 1, 2, 3, 5], dtype=int)Note here that we use the.reduce(),.deep_reduce()and.validate()methods attached to the base class: it is not necessary to know before hand the type ofx.Origin and motivationThis package is myfifthiteration,overthreeyears, of a simple system for defining Python types that package their own serialization/deserialization routines. The motivatinguse caseis to support fully self-contained “units of work” for research computations; these are used both to ensure reproducibility by archiving work, and to transfer units of work between processes or machines. (For example between simulation and analysis scripts, or between laptop and server.) This use case imposed the following requirements:The serialized format must be portable and suitable for long-term storage (which precludes solutions like pickle or dill).The format should be plain text and, to the extent possible, human-readable. This further increases long-term reliability, since in the worst case, the file can be opened in a text editor and interpreted by a human.It must be possible to associate a pair of serialization/deserialization routines to a type.These routines must be recognized byPydantic, since this is the serialization library I currently use.It should be possible for imported librairies to use custom serializers, without worrying about the order of imports. Note that this is difficult with vanilla Pydantic, because the list ofjson_encodersis set when the class is defined. One canwork aroundthis by defining a global dictionary of JSON encoders, but the behaviour is still dependent on the order of imports, and is thus fragile.It should be possible to associate custom serializers to already existing types. (E.g. add a serializer forcomplex.)It should be possible to use generic base classes in a model definition, while subclasses are preserved after deserialization. For example, in the snippet below this isnotthe case:classSignal:...classAMSignal(Signal):...classFMSignal(Signal):...classMyModel(BaseModel):signal:Signalsig=AMSignal(...)model=MyModel(signal=sig)type(model.signal)# AMSignalmodel2=MyModel.parse_raw(model.json())type(model2.signal)# Signal -> Serialization round trip does not preserve type.Since it is often the cases that the same computational pipeline can be applied to many different types, the ability to define classes in terms of only generic base types is extremely useful.The solution provided by this package satisfies all of these requirements.An extensible serialization hierarchyThe providedSerializableclass can be used to make almost any class serializable.
A standardized pattern is used to specify what needs to be serialized, andscitypingtakes care of the rest.Note that types defined withSerializabletake precedence over those already supported by Pydantic; this allows to override builtin Pydantic serializers and deserializers for specific types.Example: Define a serializableSignalclass.# mysignal.pyfromscitypingimportSerializablefromscityping.numpyimportArrayfromscityping.pydanticimportdataclassclassSignal(Serializable):@dataclassclassData:# The `Data` class stores the minimalvalues:Array[float,2]# set of information required totimes:Array[int,1]# recreate the `Signal` class.freq:float@staticmethoddefencode(signal:"Signal"):# The `encode` method defines how to extract valuesreturn(signal.values,signal.times,signal.freq)# for the fields in `Data`def__init__(values,times,freq):...## Analysis methods (mean, window, fft, etc.) go here ##Now the class can be used as a type hint in a Pydantic model:fromscityping.pydanticimportBaseModelfromtypingimportListfrommysignalimportSignal# Type defined in the code block aboveclassRecorder(BaseModel):signals:List[Signal]Moreover,subclassesofSignalcan also be used and will be recognized correctly.
They only need to define their ownDatacontainer (which may reuse the one from the base class, as we do below).classAMSignal(Signal):classData(Signal.Data):passsignal=Signal(...)amsignal=AMSignal(...)rec=Recorder(signals=[signal,amsignal])rec2=Recorder.parse_raw(rec.json())asserttype(rec2.signals[0])isSignal# Succeedsasserttype(rec2.signals[1])isAMSignal# Also succeeds(With standard Pydantic types, in the example above, values would all be coerced to theSignaltype after deserialization.)The default decoder will pass the attributes of theDataobject to the class’__init__method as keyword arguments. For cases where this is unsatisfactory, one can define a customdecodemethod. The example below uses this to return a truerangeobject, rather than theRangewrapper:classRange(Serializable,range):@dataclassclassData:start:intstop:Optional[int]=Nonestep:Optional[int]=Nonedefencode(self,r):returnr.start,r.stop,r.stepdefdecode(data):returnrange(data.start,data.stop,data.step)Note that while thedecodemechanism can help solve corner cases, in many situations the benefits are cosmetic.Finally, extending anexistingtype with a serializer is just a matter of defining a subclass. For example, theComplextype provided byscitypingis implemented as follows:classComplex(Serializable,complex):@dataclassclassData:real:floatimag:floatdefencode(z):returnz.real,z.imagdefdecode(data):returncomplex(data.real,data.imag)(When the subclass uses thesame name(case-insensitive), a bit of magic is applied to associate it to the base type. Otherwise an extraComplex.register(complex)is needed.)
It can then be used to allow serializing all complex numbers (not just instance ofComplex:classFoo(BaseModel):z:Complex# Serializes even the original data typeFoo(z=3+4j).json()# '{"z": ["__main__.Complex", {"real": 3.0, "imag": 4.0}]}'Note that this works even ifComplexis defined afterFoo, without any patching ofFoo’s list of__json_encoders__.[^post-type-def]HintIt is recommended that the nestedDataprovide type-aware deserialization logic. The easiest way to do this is to use eitherscityping.pydantic.BaseModelorscityping.pydantic.dataclass. We do add basic deserialization support for the builtindataclasses.dataclass, but this is intentionally limited to a few basic types (int,float,str) andSerializablesubclasses. If your data are very simple, this may be sufficient – see the docstring ofscityping.Dataclassfor what exactly is supported.[^post-type-def]: The corollary of this is that it makes it easier for modules to arbitrarily modify how types are serialized byalready importedmodules. Thus by adding a new serializer, a new package may break a previously working one. Also, while the hooks for extending type suport don't increase the attack surface vis-à-vis a malicious actor (imported Python modules are already allowed to inject code wherever they please), they might make things easier for them.PerformanceIn usage with PydanticFor types serialized with Pydantic, this adds a singleisinstancecheck for each serialization call, which should be negligeable.[^negligeable-comp-cost]Serialization withSerializableis not as aggressively tested with regards to performance and is expected to be a slower.Deserialization ofSerializabletypes in general will be slower, although this should only be noticeable if deserialization is a big part of your application. In some cases it may even be faster: Pydantic fields withUniontypes execute a try-catch for each possible type, keeping the first successful result. SinceSerializableincludes the target type in serialized data, the correct data type is generally attempted first.In general, within scientific applications these performance considerations should not matter: data is typically serialized/deserialized only at the beginning/end of a computation, which is not where the speed bottlenecks usually are.[^negligeable-comp-cost]: Unlessa lotof builtin types are extended with serializers.Running testsTo avoid unnecessarily downloading the massive CUDA dependencies just to run the PyTorch tests, consider installing the test dependencies with the following:pip install --index-url https://download.pytorch.org/whl/cpu torch
pip install -e .[test](The CPU-only torch package is 200MB instead of 620MB.)