Package: upandup
Up-and-up :rocket:upandupis a simple schema versioning system for Python dataclasses.Why?In Python,dataclassesare a great way to define data schemas. However, when the schema changes, you need to be able to update the old data to the latest version, or risk breaking the ability to load old data from JSON, YAML, or other formats.upandupprovides a simple way to define how to update between different versions of a schema, and then load the latest version of the schema from old data.Let's say you have adataclasslike this:@dataclassclassDataSchemaV1:x:intUsers might end up serializing this to JSON:{"x":1}Later, you decide to add a new fieldy:@dataclassclassDataSchema:x:intx_str:strNow, users can no longer load the old data, because the schema has changed. You need to define how to update the old data to the new schema.upandupprovides a way to do this.importupandupasupupupdate=lambdacls_start,cls_end,obj_start:cls_end(x=obj_start.x,x_str="the value is:%d"%obj_start.x)# Register the updateupup.register_updates("DataSchema",DataSchemaV1,DataSchema,fn_update=update)In the end,upandupexposes aloadmethod that users can call to load the latest version of the schema from old dataevery time.data={x:1}obj=upup.load("DataSchema",data)print(obj.x_str)# the value is: 1Thisloadmethod can also be exposed as an anonymous function such as:# In your package:load_data_schema=upup.make_load_fn("DataSchema")# In scripts using your packagedata={x:1}obj=load_data_schema(data)Serialization formats supportedDictionary - defineto_dictandfrom_dictmethods on your dataclasses.JSON - defineto_jsonandfrom_jsonmethods on your dataclasses.YAML - defineto_yamlandfrom_yamlmethods on your dataclasses.TOML - defineto_tomlandfrom_tomlmethods on your dataclasses.ExampleFirst, define some dataclasses. Let's say you have aDataSchemadataclass, which is the latest version, but also 2 older versions calledDataSchemaV1andDataSchemaV2. These classes haveto_jsonandfrom_jsonmethods defined via themashumaropackage by inheriting fromDataClassDictMixin(they could also be defined manually).fromdataclassesimportdataclassfrommashumaroimportDataClassDictMixin@dataclassclassDataSchemaV1(DataClassDictMixin):x:int@dataclassclassDataSchemaV2(DataClassDictMixin):x:inty:intz:int=0@dataclassclassDataSchema(DataClassDictMixin):x:intname:strHere, the first versionDataSchemaV1has only one fieldx, and the second versionDataSchemaV2has 3 fieldsx,y, andz. The fieldyhas no default, so we will have to define how to update it. The fieldzalready has a default in the definition. In the final version, the fieldsyandzhave been removed again, and a new fieldnamehas been added.Now, we can define how to update between the versions. We can use theupanduppackage to do this.importupandupasupup# Define the functions to update between the versions# The functions take the start and end classes, and the object to update# The functions should return an object of the end class# The functions can be lambdas or regular functions# For the first update, we need to add a default value for the new field `y` (`z` already has a default).update_1_to_2=lambdacls_start,cls_end,obj_start:cls_end(x=obj_start.x,y=0)# For the second update, we need to exclude the fields `y` and `z`, and add the new field `name` with a default value.update_2_to_latest=lambdacls_start,cls_end,obj_start:cls_end(x=obj_start.x,name="default")# Register the update under the label `DataSchema`upup.register_updates("DataSchema",DataSchemaV1,DataSchemaV2,fn_update=update_1_to_2)upup.register_updates("DataSchema",DataSchemaV2,DataSchema,fn_update=update_2_to_latest)# Expose a helper function to load the latest version of the schema# This makes a thin wrapper around upup.loadload_data_schema=upup.make_load_fn("DataSchema")Finally, we can test the update.# Test the updatedata={"x":1}obj=load_data_schema(data,options=upup.LoadOptions())print("Result:")print(f"Loaded object:{obj}of type{type(obj)}")# Loaded object: DataSchema(x=1, name='default') of type DataSchemaAdvancedWrite intermediate versionsBy default, the intermediate versions from updating to the latest are not written to the output. If you want to write them, you can set thewrite_intermediateoption toTrue.data={"x":1}options=upup.LoadOptions(write_versions=True,write_version_prefix="version",write_versions_dir=".")obj=upup.load("DataSchema",data,options=options)This will write the files:version_DataSchema.json
version_DataSchemaV1.json
version_DataSchemaV2.jsonExample in a packageWe can organize the same example above to demonstrate how to use it in a package.Create the following files:setup.py
mypackage/
    __init__.py
    data_latest.py
    data_v1.py
    data_v2.py
    register_updates.py
run_example.pyThe data schemas are defined bydata_v1.py,data_v2.py, anddata_latest.py. The update functions between them are defined inregister_updates.py.The package is installed by thesetup.pyfile:fromsetuptoolsimportsetup,find_packagessetup(name='mypackage',version='0.1.0',description='An example package',packages=find_packages(),install_requires=["loguru","mashumaro","setuptools","upandup"],python_requires='>=3.11',)The contents ofdata_v1.pyare:frommashumaroimportDataClassDictMixinfromdataclassesimportdataclass@dataclassclassDataSchemaV1(DataClassDictMixin):x:intThe contents ofdata_v2.pyare:frommashumaroimportDataClassDictMixinfromdataclassesimportdataclass@dataclassclassDataSchemaV2(DataClassDictMixin):x:inty:intz:int=0The contents ofdata_latest.pyare:frommashumaroimportDataClassDictMixinfromdataclassesimportdataclass@dataclassclassDataSchema(DataClassDictMixin):x:intname:strThe__init__.pyexposes only the latest version of the schema:from.data_latestimportDataSchemafrom.register_updatesimportload_data_schema,OptionsTheregister_updates.pycontains the update functions:importupandupasupupfrommypackage.data_v1importDataSchemaV1frommypackage.data_v2importDataSchemaV2frommypackage.data_latestimportDataSchemaupdate_1_to_2=lambdacls_start,cls_end,obj_start:cls_end(x=obj_start.x,y=0)update_2_to_latest=lambdacls_start,cls_end,obj_start:cls_end(x=obj_start.x,name="default")# Register the updateupup.register_updates("DataSchema",DataSchemaV1,DataSchemaV2,fn_update=update_1_to_2)upup.register_updates("DataSchema",DataSchemaV2,DataSchema,fn_update=update_2_to_latest)# Expose the load function and options in a nicer wayload_data_schema=upup.make_load_fn("DataSchema")Options=upup.LoadOptionsAs noted in the__init__.py, we also expose theload_data_schemaandOptionsfromregister_updates.py. This lets users easily load the latest version of the schemaevery timefrom any old version.Finally, therun_example.pycontains the test code:importmypackageasmp# Test the updatedata={"x":1}obj=mp.load_data_schema(data,mp.Options())print("Result:")print(f"Loaded object:{obj}of type{type(obj).__name__}")# Loaded object: DataSchema(x=1, name='default') of type DataSchemaNote that theupanduppackage itself did not have to be called.TestsTests are included in thetestsdirectory and built onpytest- from the root directory, run:pytest