Package: config-state
config_stateThepythonlanguage is a flexible language often used as an interface to manipulate high performance libraries coded in less flexible native languages likeC/C++.ConfigStateis this idea applied on an higher level in the hierarchy, it provides a frame to bridge human-readableconfiguration languages(e.g.jsonoryaml) withpython.WithConfigStateone can configure a complex hierarchy of python classes and instantiate them using a single configuration file.  To avoid pitfalls and enhance the developer's experience,ConfigStateprovides a frame preventing inconsistencies and raising explicit explanation in failing situations. The performance is optimized for low runtime overhead, most of the logic is done during the class definition.The ConfigState classThe core component is the classConfigStatethat defines a pattern to represent python classes with two distinctive set of attributes: a set of immutableconfigurationvalues and a set of mutablestatevalues.The configuration is set upon initialization and is passed through the constructor. Once initialized, the configuration is frozen and cannot change.
The state variables constitute the mutable state of the instance and can be updated throughout its lifetime.
The configuration and state variables are meant to represent the necessary and sufficient information required to clone the object's instance. They can be used to save and restore the object from disk.The configuration fields are defined usingConfigFieldclass attributes. They can have typing constraints and be provided with a factory method for building complex types out of simpler/built-in ones.State variables are defined usingStateVarattributes within the constructor. They can alternatively be defined as class properties using@statepropertyif random logic execution is needed upon accession/modification.Implementing a class inheriting fromConfigStateas parent offers the following benefits:Provides clear semantic separation between the static configuration values and the mutable state variables.Configuration values and state variables are accessible through pythonic syntax and benefit from the IDE's type hinting feature.Using a configuration file, one can instantiate a complex hierarchy of python classes. A config field may be anotherConfigStateobject allowing to define tree-like structuredConfigStatehierarchies.A config field can be a reference to a nestedConfigStateobject's config field. This allows coupling between config fields. For example, configuration of a log folder path can be injected into the nestedConfigStateobjects through the configuration of the topmostConfigStateobject.ConfigStateobjects can be serialized/deserialized into/from a stream. They arepickleableand in some casesjsonable.Basic usagefrompathlibimportPathfromconfig_stateimportConfigFieldfromconfig_stateimportConfigStatefromconfig_stateimportStateVarimportnumpyasnpclassFoo(ConfigState):learning_rate:float=ConfigField(0.1,'The learning rate',force_type=True)license_key:str=ConfigField(None,'License key',required=True)log_dir:Path=ConfigField('./','Path to a folder',type=Path)def__init__(self,config=None):super().__init__(config=config)self.weights:np.ndarray=StateVar(np.random.random((10,10)),'The weights of the model')self.iteration:int=StateVar(0,'Training iterations')We can instantiate aConfigStatewith a dictionary (that may have been obtained from loading a json or yaml file):conf={'learning_rate':0.1,'license_key':'ID123','log_dir':'logs/'}foo=Foo(conf)The configuration offoocan be summarized:print(foo.config_summary())Output:learning_rate: 0.1
license_key: ID123
log_dir: logsValues are accessible with pythonic syntax (the IDE should be able to perform type hinting and code completion):assertisinstance(foo.learning_rate,float)assertfoo.learning_rate==0.1Config values are immutable:foo.learning_rate=0.2# Not OK, raises 'AttributeError: Updating a conf field is forbidden'But changing a state variable is ok:foo.iteration+=1# Ok, state variableMissing required fields raises an exception:conf={'learning_rate':0.1,'log_dir':'logs/'}foo=Foo(conf)# ConfigError: Configuring 'Foo': Those required fields have not been specified {'license_key'}Configuring invalid fields raise an exception:conf={'color':'red','license_key':'ID123'}foo=Foo(conf)# ConfigError: Configuring 'Foo': Trying to update the conf field 'color' which has not been definedConfiguring with an invalid type raise an exception:conf={'learning_rate':'0.1','license_key':'ID123'}foo=Foo(conf)# ConfigError: Configuring 'Foo': Value `0.1` of type `<class 'str'>` is not compatible with specified type `float`State propertyA state variable can be defined using properties with the@statepropertydecorator, this is convenient in case some logic need to be run while accessing or setting the variable.fromconfig_stateimportConfigStatefromconfig_stateimportstatepropertyimportnumpyasnpclassModel(ConfigState):def__init__(self,config):super().__init__(config)self._weights:np.ndarray=np.random.random((10,10))@statepropertydefweights(self)->np.ndarray:'''Weights of the model'''returnself._weights@weights.setterdefweights(self,val):self._weights=valSerializationConfigStateobjects are serializable if their config and state variables are serializable too. The state of an object is considered to be entirely encapsulated within the config values and the state variables. The state can be obtained withfoo.get_state()which returns anObjectStateinstance. Those objects represent the serialized information of aConfigStateobject.importpicklepickle.dump(foo,open('foo.pkl','wb'))foo2=pickle.load(open('foo.pkl','rb'))In some cases,ConfigStateobjects arejson serializable:fromconfig_state.serializersimportJsonclassJsonableFoo(ConfigState):log_dir:str=ConfigField('log_dir/','Path to output folder')learning_rate:float=ConfigField(0.1,'The learning rate')def__init__(self,config=None):super().__init__(config=config)self.iteration=StateVar(0,'Training iterations')foo=JsonableFoo()# savingJson().save(foo,'foo.json')# loadingfoo=Json().load('foo.json')Content offoo.json:{"type":"__main__.JsonableFoo","config":{"__VERSION__":{"value":1.0,"doc":"ConfigState protocol's version","type":"builtins.float"},"log_dir":{"value":"log_dir/","doc":"Path to output folder.","type":"builtins.str"},"learning_rate":{"value":0.1,"doc":"The learning rate","type":"builtins.float"}},"internal_state":{"iteration":{"value":0,"doc":"Training iterations","type":"builtins.int"}}}PickleandJsonserializers are available as plugin:serializer=Serializer({'class':'Pickle'})serializer.save(foo,'foo.pkl')Config field factoryImplicit factoryIf aConfigFieldhas a specifiedtypebut the type of the providedvalueis different,typeis used as an implicit factory by callingtype(value). This is useful for nestedConfigStateobjects:classNestedFoo(ConfigState):license_key:str=ConfigField(type=str,required=True)foo:Foo=ConfigField(type=Foo,doc='A ConfigState as config field',required=True)conf={'license_key':'4321','foo':{'learning_rate':0.1,'license_key':'ID123','log_dir':'logs/'}}nested_foo=NestedFoo(conf)# Ok, nested_foo.foo is instantiated using conf['foo']isinstance(nested_foo.foo,Foo)# TrueExplicit factoryA factory can be explicitly provided through acallable:fromdatetimeimportdatetimedefdate_factory(str_date):returndatetime.strptime(str_date,'%Y-%m-%d%H:%M:%S')classDateFoo(ConfigState):date:datetime=ConfigField(value='2019-01-01 00:00:00',type=datetime,doc='some date',factory=date_factory)date_foo=DateFoo({'date':'2021-04-28 00:00:00'})print(type(date_foo.date))# <class 'datetime.datetime'>Deferred config fieldsIt may happen that the full configuration of an object is not known at the time of its instantiation. In such case it is possible to defer their specification at a later time usingEllipsis:foo=Foo({'license_key':...})foo.license_keyisEllipsis# Truefoo.license_key=1337# ok, we can update an Ellipsisfoo.license_key=42# Not OK, raises 'AttributeError: Updating a conf field is forbidden'# Note: For convenience with configs defined within json or yaml files, strings '...' are interpreted as Ellipsis:foo=Foo({'license_key':str('...')})foo.license_keyisEllipsis# TrueReference fieldsAConfigFieldcan be references to fields in nestedConfigStatefields simplifying the configuration of complex hierarchies:classFooWithRef(ConfigState):foo:Foo=ConfigField(type=Foo)# a nested ConfigStatelicense_key=ConfigField(foo.license_key)# Reference to a nested field# FooWithRef.license_key is a reference to FooWithRef.foo.license_key# allowing to simplify the configuration, instead of:FooWithRef({'foo':{'license_key':'ABC123'}})# we can do:foo_with_ref=FooWithRef({'license_key':'ABC123'})foo_with_ref.license_key=='ABC123'# Truefoo_with_ref.foo.license_key=='ABC123'# Truefoo_with_ref.foo.license_keyisfoo_with_ref.license_key# TrueA reference can point to another reference:classFooWithRef2(ConfigState):foo_with_ref:FooWithRef=ConfigField(type=FooWithRef)license_key=ConfigField(foo_with_ref.license_key)# It is a reference to another referencefoo=FooWithRef2({'license_key':'ABC123'})foo.foo_with_ref.license_keyisfoo.license_key# Truefoo.foo_with_ref.foo.license_keyisfoo.license_key# Truefoo.license_key=='ABC123'# TrueA reference can point to multiple fields using list or tuples:classSubFooWithMultiRef(ConfigState):foo1:Foo=ConfigField(type=Foo)foo2:Foo=ConfigField(type=Foo)license_key=ConfigField([foo1.license_key,foo2.license_key])# Reference to foo1 and foo2's license_key field# Now instead of:conf={'foo1':{'license_key':'ABC123'},'foo2':{'license_key':'ABC123'}}SubFooWithMultiRef(conf)# One can simply do:foo=SubFooWithMultiRef({'license_key':'ABC123'})foo.license_key=='ABC123'# Truefoo.foo1.license_keyisfoo.license_key# Truefoo.foo2.license_keyisfoo.license_key# TruePlugins managementAConfigStateclass can be decorated with@builder, this registers the class as abuilder, this allows its sub classes to be decorated with@register, that registers them as plugins and enable their instantiation using the builder parent.fromconfig_stateimportbuilderfromconfig_stateimportregister@builderclassColoredFoo(ConfigState):color:str=ConfigField(None,"Color",static=True)value:int=ConfigField(type=int,doc="Value")@registerclassRedFoo(ColoredFoo):color:str=ConfigField("Red","Color",static=True)@registerclassBlueFoo(ColoredFoo):color:str=ConfigField("Blue","Color",static=True)colored_foo=ColoredFoo({'class':'BlueFoo','value':1})print(type(colored_foo))# <class '__main__.BlueFoo'>print(colored_foo.color)# Blueprint(colored_foo.value)# 1Builders can be defined in a hierarchy. For instance, we can define amaster builderfrom which everybuildercan inherit. Building an object is made by specifying the hierarchy path:@builderclassMasterBuilder(ConfigState):pass@builder@registerclassColoredFoo(MasterBuilder):passcolored_foo=MasterBuilder({'class':'ColoredFoo.BlueFoo','value':1})