Package: versionedobj
versionedobjis an object serialization framework that allows you to create
complex python objects that can be serialized/deserialized to and from strings,
or dicts, or JSON files.versionedobjalso provides a versioning mechanism, to track changes in object
structure across time, and to migrate between different object versions.SeeAPI documentationInstallingInstallversionedobjusing pip:pip install versionedobjGetting startedObject definitionDefine objects by creating a new class that inherits fromVersionedObject,
and set class attributes to define your object attributes:fromversionedobjimportVersionedObjbectclassUserConfig(VersionedObject):version="v1.0.0"username="john smith"friend_list=["user1","user2","user3"]You can also nest VersionedObjects by simply assigning anotherVersionedObjectclass or instance object to a class attribute:fromversionedobjimportVersionedObjectclassDisplayConfig(VersionedObject):display_mode="windowed"resolution="1920x1080"volume=0.66# Populate class attributes to build your objectclassUserConfig(VersionedObject):version="v1.0.0"username="john smith"friend_list=["user1","user2","user3"]display_config=DisplayConfig()# VersionedObjects can be nested# Nested VersionedObjects can be a class object, or an instance of the# class, either way will behave the same# display_config = DisplayConfigCreating object instances and accessing object attributesThe values you set on the class attributes of aVersionedObjectserve as the default
values for that object. When you create an instance of yourVersionedObjectclass,
instance attributes will automatically be created to match the class attributes, and
the values of the class attributes will be copied over to the instance attributes:obj=UserConfig()print(obj.friend_list)# Output looks like this: ["user1", "user2", "user3"]print(obj.display_config.display_mode)# Output looks like this: "windowed"As well as regular dot notation, you can also treat an object instance like a dict,
and access individual attributes using their full dot name as the key:print(obj['friend_list'])# Output looks like this: ["user1", "user2", "user3"]print(obj['display_config.display_mode'])# Output looks like this: "windowed"# Change the value of an instance attributeobj['display_config.display_mode']="fullscreen"print(obj['display_config.display_mode'])# Output looks like this: "fullscreen"You can also treat aVersionedObjbectinstance as an iterable, to iterate
over all object attribute names, as you would with keys in a dict:forattr_nameinobj:print(f"{attr_name}:{obj[attr_name]}")# Output looks like this:## version: v1.0.0# username: john smith# friend_list: ["user1", "user2", "user3"]# display_config.display_mode: windowed# display_config.resolution: 1920x1080# display_config.volume: 0.66Serializing and de-serializingCreate an instance of theversionedobj.Serializerclass, and use theto_fileandfrom_filemethods to serialize/deserialize data to/from a JSON file:fromversionedobjimportVersionedObject,SerializerclassDisplayConfig(VersionedObject):display_mode="windowed"resolution="1920x1080"volume=0.66classUserConfig(VersionedObject):version="v1.0.0"username="john smith"friend_list=["user1","user2","user3"]display_config=DisplayConfig()# VersionedObjects can be nested# Create an instance of our VersionedObjectobj=UserConfig()# Create a serializer instanceserializer=Serializer(obj)# Save object instance to JSON fileserializer.to_file('user_config.json',indent=4)# Load JSON file and populate the same object instanceserializer.from_file('user_config.json')You can also save/load object data as a JSON string:# Save object instance to JSON stringobj_as_json=serializer.to_json(indent=4)# Load object instance from JSON stringserializer.from_json(obj_as_json)Or, as a dict:# Save object instance to dictobj_as_dict=serializer.to_dict()# Load object instance from dictserializer.from_dict(obj_as_dict)Using one Serializer instance with multiple object typesFor convenience, you can pass an object instance when you create aversionedobj.Serializer,
and this object will be used for all future serialization/deserialization operations,
so that you don’t have to pass in the object instance every time (as shown in previous
examples).However, this is not required, and you can optionally provide an object instance
for all serialization/deserialization methods, if you want to (for example) use
a singleversionedobj.Serializerinstance for multiple object types:fromversionedobjimportVersionedObject,SerializerclassObjectA(VersionedObject):name="john"age=44classObjectB(VersionedObject):last_login_time=12345678enabled=False# Create an instance of each objecta=ObjectA()b=ObjectB()serializer=Serializer()# Serialize both objects using the same serializera_jsonstr=serializer.to_json(a)b_jsonstr=serializer.to_json(b)# De-serialize both objects using the same serializerserializer.from_json(a_jsonstr,a)serializer.from_json(b_jsonstr,b)Filtering serialization/deserialization outputWhitelisting by field nameWhen serializing, if you only want to output certain fields, you can use the ‘only’
parameter to specify which fields should be output (effectively a whitelist by field name):serializer.to_file('user_config.json',only=['version','username','display_config.resolution'])# Output looks like this:## {#     "version": "v1.0.0",#     "username": "jane doe",#     "display_config": {#         "resolution": "1920x1080",#     }# }The same parameter can be used for de-serializing:serializer.from_file('user_config.json',only=['display_config.display_mode'])# Only the 'display_config.display_mode' field is loaded from the fileBlacklisting by field nameWhen serializing, if youdon’twant to output certain fields, you can use the ‘ignore’
parameter to specify which fields should be excluded from output (effectively a blacklist
by field name):serializer.to_file('user_config.json',ignore=['friend_list','display_config.volume'])# Output looks like this:## {#     "version": "v1.0.0",#     "username": "jane doe",#     "display_config": {#         "display_mode": "windowed",#         "resolution": "1920x1080"#     }# }The same parameter can be used for de-serializing:serializer.from_file('user_config.json',ignore=['friend_list'])# Every field except for the 'friend_list' field is loaded from the fileversionedobj.ListField: store a sequence of objects in a single fieldversionedobj.ListFieldis a list class that behaves exactly like a regular python list,
except for the following 2 differences:Only instances of a class which is a subclass of theVersionedObjectmay be added to lists
(ValueError is raised otherwise)Only instances of the same class may be added to a single list (ValueError is raised otherwise)You can assign aversionedobj.ListFieldinstance as the value for a field in your versioned object
class definition, and that field can then hold a sequence of multiple versioned objects. This
is useful if you need to store a variably-sized collection of objects that are created a runtime.fromversionedobjimportVersionedObject,Serializer,ListField# The list will contain objects of this type onlyclassUserData(VersionedObject):name="john"age=30# This object will contain a list of multiple usersclassAllUserData(VersionedObject):# a List may only contain instances of the same classusers=ListField(UserData)all_user_data=AllUserData()# Add some users to the listall_user_data.users.append(UserData(initial_values={'name':'sam','age':66}))all_user_data.users.append(UserData(initial_values={'name':'sally','age':28}))# Serialize object and print out JSON dataprint(Serializer(all_user_data).to_json(indent=4))# Output looks like this:## {#     "users": [#         {#             "name": "sam",#             "age": 66#         },#         {#             "name": "sally",#             "age": 28#         }#     ]# }Context manager for loading & editing saved object dataIf you want to load object data from a JSON file, make some changes to the data,
and save it back to the same JSON file, then you can use theFileLoadercontext
manager, which will load/create the file for you on entry, return a deserialized
object for you to modify, and then serializes your modified object back to the same
file on exit. This may be useful if you are worried about forgetting to re-serialize
the object when you are done.fromversionedobjimportVersionedObject,FileLoaderclassRecipe(VersionedObject):ingredient_1="onions"ingredient_2="tomatoes"ingredient_3="garlic"# Creates a new instance of the object, and loads data from# "recipe.json" if the file already existswithFileLoader(Recipe,"recipe.json")asobj:# Change somethingobj.ingredient_3="celery"# recipe.json now looks like this:## {#     "ingredient_1": "onions",#     "ingredient_2": "tomatoes",#     "ingredient_3": "celery",# }Migrations: making use of the version numberA VersionedObject object can have aversionattribute, which can be any object,
although it is typically a string (e.g."v1.2.3"). This version attribute can be
used to support migrations for older objects, in the event that you need to
change the format of your object.Example scenario, part 1: you have created a beautiful versioned objectLet’s take the same config file definition from the previous example:fromversionedobjimportVersionedObject# Nested config objectclassDisplayConfig(VersionedObject):display_mode="windowed"resolution="1920x1080"volume=0.66# Top-level config object with another nested config objectclassUserConfig(VersionedObject):version="v1.0.0"username="john smith"friend_list=["user1","user2","user3"]display_config=DisplayConfig()Imagine you’ve already released this code out into the world. People are already
using it, and they have JSON files generated by yourUserConfigclass sitting
on their computers.Example scenario, part 2: you update your software, modifying the versioned objectNow, imagine you are making a new release of your software, and some new features
require you to make the following changes to your versioned object:remove the theDisplayConfig.resolutionfield entirelychange the name ofDisplayConfig.volumetoDisplayConfig.volumeschange the value ofDisplayConfig.volumesfrom a float to a listfromversionedobjimportVersionedObject# Nested config objectclassDisplayConfig(VersionedObject):display_mode="windowed"# 'resolution' field is deletedvolumes=[0.66,0.1]# 'volume' is now called 'volumes', and is a list# Top-level config object with another nested config objectclassUserConfig(VersionedObject):version="v1.0.0"username="john smith"friend_list=["user1","user2","user3"]display_config=DisplayConfig()Uh-oh, you have a problem…Right now, if you send this updated UserConfig class to your existing users, it will fail
to load their existing JSON files with versionv1.0.0, since those files will contain
theDisplayConfig.resolutionfield that we deleted inv1.0.1, andDisplayConfig.volumewill similarly be gone, having been replaced withDisplayConfig.volumes. This situation is what migrations are for.Solution– migrations!The solution is to:Change the version number to something new, e.g.v1.0.0becomesv1.0.1Write a migration function to transformv1.0.0object data intov1.0.1object dataUse theversionedobj.migrationdecorator to register your migration functionfromversionedobjimportVersionedObject,migration# Nested config objectclassDisplayConfig(VersionedObject):display_mode="windowed"# 'resolution' field is deletedvolumes=[0.66,0.1]# 'volume' is now called 'volumes', and is a list# Top-level config object with another nested config objectclassUserConfig(VersionedObject):version="v1.0.1"# Version has been updated to 1.0.1username="john smith"friend_list=["user1","user2","user3"]display_config=DisplayConfig()# Create the migration function for v1.0.0 to v1.0.1@migration(UserConfig,"v1.0.0","v1.0.1")defmigrate_100_to_101(attrs):delattrs['display_config']['resolution']# Delete resolution fielddelattrs['display_config']['volume']# Delete volume fieldattrs['display_config']['volumes']=[0.66,0.1]# Add defaults for new volume valuesreturnattrs# Return modified data (important!)after you add the migration function and update the version tov1.0.1, JSON files
that are loaded and contain the versionv1.0.0will be automatically migrated to versionv1.0.1using the migration function you added.The downside to this approach, is that you have to manually udpate the version number,
and write a new migration function, anytime the structure of your config data changes.The upside, of course, is that you can relatively easily support migrating any older
version of your config file to the current version.If you don’t need the versioning/migration functionality, just never change your version
number, or don’t create aversionattribute on yourVersionedObjectclasses.Migrations: migrating an unversioned objectYou may run into a situation where you release an unversioned object, but then
later you need to make changes, and migrate an unversioned object to a versioned object.This can be handled simply by passing “None” to the “add_migration()” method, for the
“from_version” parameter. For example:fromversionedobjimportVersionedObj,migrationclassUserConfig(VersionedObject):version="v1.0.0"username=""friend_list=[]@migration(UserConfig,None,"v1.0.0")defmigrate_none_to_100(attrs);attrs['friend_list']=[]# Add new 'friend_list' fieldreturnattrsValidating input data without deserializingYou may want to validate some serialized object data without actually deserializing
and loading the object values. You can use theSerializer.validate_dictmethod for this.fromversionedobjimportVersionedObject,SerializerclassRecipe(VersionedObject):ingredient_1="onions"ingredient_2="tomatoes"ingredient_3="garlic"rcp=Recipe()serializer=Serializer(rcp)serializer.validate_dict({"ingredient_1":"celery","ingredient_2":"carrots"})# Raises versionedobj.exceptions.InputValidationError because 'ingredient_3' is missingserializer.validate_dict({"ingredient_1":"celery","ingredient_2":"carrots","ingredient_12":"cumin"})# Raises versionedobj.exceptions.InputValidationError because 'ingredient_12' is not a valid attributeResetting object instance to default valuesYou can use theSerializer.reset_to_defaultsmethod to set all instance attributes to
the default values defined in the matching class attributes.fromversionedobjimportVersionedObject,SerializerclassRecipe(VersionedObject):ingredient_1="onions"ingredient_2="tomatoes"ingredient_3="garlic"rcp=Recipe()serializer=Serializer(rcp)# Change a valuercp.ingredient_1="celery"print(serializer.to_dict())# {"ingredient_1": "celery", "ingredient_2": "tomatoes", "ingredient_3": "garlic"}# Reset object instance to defaultsserializer.reset_to_defaults()print(serializer.to_dict())# {"ingredient_1": "onions", "ingredient_2": "tomatoes", "ingredient_3": "garlic"}Testing object instance equalityYou can test whether twoVersionedObjectinstances are equal in both structure and
values, the same way in which you would check equality of any other two objects:fromversionedobjimportVersionedObjectclassRecipe(VersionedObject):ingredient_1="onions"ingredient_2="tomatoes"ingredient_3="garlic"rcp1=Recipe()rcp2=Recipe()print(rcp1==rcp2)# Truercp1.ingredient_3="ginger"print(rcp1==rcp2)# FalseIn order for twoVersionedObjectinstances to be considered equal, the following
conditions must be true:Both objects are instances of the same classBoth objects contain matching attribute names and valuesObject instance hashingObjects can be uniquely hashed based on their instance attribute values, using the builtinhash()function. This means, for example, that you can use object instances as dict keys:fromversionedobjimportVersionedObjectclassPerson(VersionedObject):name="sam"age=31p1=Person()p2=Person()# Change 1 value on p2 so the hash value is differentp2.age=32d={p1:"a",p2:"b"}print(d)# { Person({"name": "sam", "age": 31}): "a", Person({"name": "sam", "age": 32}): "b" }Testing whether object instances contain specific valuesYou can check whether an object instance contains a particular attribute value using theinkeyword:fromversionedobjimportVersionedObjectclassPerson(VersionedObject):name="sam"age=31p=Person()print("sam"inp)# Truep.name="sally"print("sam"inp)# Falseprint("sally"inp)# TruePerformance/stress test visualizationThe following image is generated by thetests/performance_tests/big_class_performance_test.pyscript,
which creates and serializes/deserializes multiple versioned objects of an incrementally increasing size,
and simultaneously having an increasing depth of contained nested objects.Each data point in the graph represents measurements taken for an object of a particular size.
The time taken to serialize the object to a dict, and also to deserialize the object data
from a dict, and also to create an instance of the object, is measured for each object size. It is
worth mentioning that measuring thefrom/to_jsonandfrom/to_filemethods is not very
useful in this case, since that would only be measuringto/from_dictwith additional JSON
parser or file I/O overhead. That is why this graph only measuresto/from_dict.This test was executed on a system with an Intel Core-i7 running Debian GNU/Linux 10 (buster)
with Linux debian 4.19.0-21-amd64.ContributionsContributions are welcome, please open a pull request athttps://github.com/eriknyquist/versionedobjand ensure that:All existing unit tests pass (run tests viapython setup.py test)New unit tests are added to cover any modified/new functionality (runpython code_coverage.pyto ensure that coverage is above 98%)You will need to install packages required for development, these are listed indev_requirements.txt:pip install -r dev_requirements.txtIf you have any questions about / need help with contributions or unit tests, please
contact Erik ateknyquist@gmail.com.