Package: tree-config
tree-configConfiguration of objects that are nested in a tree-like fashion.For more information:https://matham.github.io/tree-config/index.htmlInstallationtree-configcan be installed with:pipinstalltree-configConfiguration usagetree-configcan dump all the configurable properties of all your classes to
a yaml file and then load the the file and restore/apply the values to the
properties. E.g.:classApp:_config_props_=('name',)_config_children_={'app panel':'panel'}def__init__(self):self.name='Desk'self.panel=AppPanel()classAppPanel:_config_props_=('color',)color='A4FF67'will automatically configurenameandcolor.Following is a guide by example of the multiples ways to control the configuration.
See theconfiguration API, including
theConfigurationclass for complete details.See the examples directory in the repo for complete runnable code of the following
examples.Basic propertiesThis example has an app class that contains two panels that are configurable._config_props_lists the configurable properties for the class, while_config_children_constructs the tree of objects that are configurable.classApp:_config_props_=('size','name')_config_children_={'app panel':'panel1','home panel':'panel2'}def__init__(self):self.size=12self.name='Desk'self.panel1=AppPanel()self.panel2=HomePanel()classAppPanel:_config_props_=('color',)color='A4FF67'classHomePanel:_config_props_=('shape',)shape='circle'Then, running:fromtree_configimportdump_config,read_config_from_objectapp=App()dump_config('basic_example.yaml',read_config_from_object(app))print(f'Shape is:{app.panel2.shape}')creates abasic_example.yamlfile with the following contents:app panel:{color:A4FF67}home panel:{shape:circle}name:Desksize:12and it printsShape is: circle. If we want to load a previous yaml file,
where say the shape was"square"and apply it to the instance, we do:fromtree_configimportload_config,apply_configapp=App()apply_config(app,load_config(app,'basic_example.yaml'))print(f'Shape is:{app.panel2.shape}')This in turn printsShape is: square.Hooking property discovery_config_props_and_config_children_are defined on a class, not on
instances. Whentree-configuses them, it will walk the whole class
hierarchy and accumulate their values from all super classes because a
sub-class does not overwrite them, but rather adds to them.If_config_propsand/or_config_childrenis defined on a
class or instance, tree-config will use that value directly, instead of
walking_config_props_and/or_config_children_, respectively.E.g. the following code:fromtree_configimportdump_config,read_config_from_objectclassApp:_config_children_={'app panel':'panel1','home panel':'panel2'}def__init__(self):self.panel1=AppPanel()self.panel2=HomePanel()classRootPanel:_config_props_=('size','name')size=12name='Desk'classAppPanel(RootPanel):_config_props_=('color',)color='A4FF67'classHomePanel(AppPanel):_config_props_=('shape',)shape='circle'group='window'_config_props=('group','size')when run with:app=App()# now get and save config to yaml filedump_config('hook_properties.yaml',read_config_from_object(app))will generate this yaml file:app panel:color:A4FF67name:Desksize:12home panel:group:windowsize:12Notice howapp panelcontains the properties
of bothRootPanelandAppPanel, whilehome panelonly has the
properties listed in_config_props._config_childrenbehaves
similarly.Custom values hooksWe may wish to hook the property getting/setting process to
change the value before it is saved or before it is applied again.E.g. consider that we have a property that stores a namedtuple that we need
to dump as a list (because yaml doesn’t understand named tuple) and create
a named tuple again when restoring.get_config_propertyandapply_config_propertyare the needed hook methods, that are
automatically used if present in the class:fromcollectionsimportnamedtuplefromtree_configimportdump_config,load_config,apply_config,\read_config_from_objectPoint=namedtuple('Point',['x','y'])classApp:_config_props_=('point','name')point=Point(11,34)name=''defget_config_property(self,name):ifname=='point':returntuple(self.point)returngetattr(self,name)defapply_config_property(self,name,value):ifname=='point':self.point=Point(*value)else:setattr(self,name,value)Then, running:fromtree_configimportdump_config,read_config_from_objectapp=App()dump_config('custom_value_example.yaml',read_config_from_object(app))print(f'point is:{app.point}')creates acustom_value_example.yamlfile with the following contents:name:''point:[11,34]and it printspoint is: Point(x=11, y=34). If we want to load and apply the
yaml file again, we do:fromtree_configimportload_config,apply_configapp=App()apply_config(app,load_config(app,'custom_value_example.yaml'))print(f'point is:{app.point}')This in turn prints againpoint is: Point(x=11, y=34).See alsoapply_config_childfor similarly hooking into applying the children
objects. The default, when not provided is to useapply_config, so if
overriding, that should probably also be used for the base case.Custom tags (pickling)Yaml offers support for representing arbitrary objects using custom tags in the
file. This enables global support for the objects, without having to useget_config_property/apply_config_propertywherever they are used.Using the point example above:fromcollectionsimportnamedtuplefromtree_configimportdump_config,load_config,apply_config,\read_config_from_objectfromruamel.yamlimportBaseConstructor,BaseRepresenterPoint=namedtuple('Point',['x','y'])yaml_tag='!tree_config_example_point'# encoderdef_represent_point(representer:BaseRepresenter,val):returnrepresenter.represent_sequence(yaml_tag,tuple(val))# decoderdef_construct_point(constructor:BaseConstructor,tag,node):returnPoint(*constructor.construct_sequence(node))# tell yaml how to represent a Pointdefregister_point_yaml_support()->None:BaseRepresenter.add_multi_representer(Point,_represent_point)BaseConstructor.add_multi_constructor(yaml_tag,_construct_point)classApp:_config_props_=('point','name')point=Point(11,34)name=''Now, call:register_point_yaml_support()before running the tree-config dumping/loading code from the last section and
it will generate a yaml file with contents:name:''point:!tree_config_example_point[11,34]See alsoyaml_dumpsandyaml_loadsfor additional customization.
Most functions take ayaml_dump_str/yaml_load_strto allow further
customizing the yaml objects. See alsoregister_torch_yaml_supportintree_config.yamlfor a more complete example as well as some built-in
optional representers that can be registered directly.Post-applying dispatchAfter applying configuration to a object and its children objects,
tree-config will call thepost_config_appliedmethod of the object, if
the method exists. E.g.:fromtree_configimportdump_config,load_config,apply_config,\read_config_from_objectclassApp:_config_props_=('size','name')_config_children_={'app panel':'panel'}size=12name='Desk'def__init__(self):self.panel=Panel()defapply_config_property(self,name,value):print('applying',name)setattr(self,name,value)defpost_config_applied(self):print('done applying app')classPanel:_config_props_=('color',)color='A4FF67'defapply_config_property(self,name,value):print('applying',name)setattr(self,name,value)defpost_config_applied(self):print('done applying panel')Then, saving and again applying the yaml using:# create app and set propertiesapp=App()# now get and save config to yaml filedump_config('post_apply_dispatch.yaml',read_config_from_object(app))# load config and apply itapply_config(app,load_config(app,'post_apply_dispatch.yaml'))prints the following:applying color
done applying panel
applying name
applying size
done applying appConfigurable classThe above examples used a duck typing approach to these special configuration/hook
methods, and any/all of these methods were optional. tree-config also offers aConfigurablesuperclass that defines all these methods with appropriate
default values.There’s no benefit to inheriting fromConfigurable, but it does provide a
baseclass listing all the special configuration methods. Additionally,
it does cache the list of properties/config children for each class,
so once looked up, it does not need to walk the tree, unlike the duck
typing approach that re-computes at every save/apply.Auto docsIn addition to configuration, tree-config can also hook into the sphinx doc
generating build steps and generate docs listing all the properties that
can be configured by the application and show the doc string for each of them.
This is helpful to users who want to configure these properties using the
configuration yaml file.The example directory has a complete doc example.Given a root object (e.g. App in the examples), we can add callbacks inconf.pythat is called by sphinx as it encounters properties listed in_config_props_. The callback then saves the doc strings of these properties
into a yaml file.Subsequently, when the build is done, tree-config can go through all the
configurable properties and starting from the root object or class, extract
the doc strings from the yaml file, and create a rst file of those docstrings.E.g. starting with this code in :classApp:"""The app."""_config_props_=('size','name')_config_children_={'app panel':'panel1','home panel':'panel2'}size=55"""Some filename."""name=''"""Some name."""panel1:'AppPanel'=None"""The app panel."""panel2:'HomePanel'=None"""The home panel."""def__init__(self,size,name,color,shape):self.size=sizeself.name=nameself.panel1=AppPanel()self.panel1.color=colorself.panel2=HomePanel()self.panel2.shape=shapeclassAppPanel:"""The app panel."""_config_props_=('color',)color=''"""Color of the app."""classHomePanel:"""The home panel."""_config_props_=('shape',)shape=''"""Shape of the home."""then, we add the following to the top of theconf.pyfile:importosimportsysfromfunctoolsimportpartialsys.path.insert(0,os.path.abspath('../'))fromconfig_exampleimportAppfromtree_config.doc_genimportcreate_doc_listener,write_config_props_rstthe exact path added tosys.pathdepends on where the code is, or if it’s a python
package that is not needed because it’s already installed.We also need to add'sphinx.ext.autodoc'to the list of extensions. Finally,
at the end ofconf.pyadd:defsetup(app):# dump all config_example package/subpackages config docstrings to config_prop_docs.yamlcreate_doc_listener(app,'config_example','config_prop_docs.yaml')# then get docstrings from yaml file, walk all config properties from App and# dump formatted config docs to source/config.rstapp.connect('build-finished',partial(write_config_props_rst,App,'config_example',filename='config_prop_docs.yaml',rst_filename='source/config.rst'))Finally, to the sphinx generatedindex.rstwe addedconfig.rst(the filename
of the file that will be automatically created under source).
We also need to add somewhere in the index or files it references the auto-doc
references for all the classes, otherwise we won’t get the relevant docstrings.
We added it as:..toctree:::maxdepth:2:caption:Contents:config.rstAPI===..automodule::config_example:members:inindex.rst.Finally, we run:echo$'Config\n==========='>source/config.rstmakehtmlmakehtmlFirst we created a mostly empty config.rst file. Otherwise sphinx doesn’t
include it when it is generated. Next we ranmake htmltwice, the first
time it automatically generates the followingconfig_prop_docs.yamlfile:config_example.App:name:-Some name.-''size:-Some filename.-''config_example.AppPanel:color:-Color of the app.-''config_example.HomePanel:shape:-Shape of the home.-''The secondmake htmlextracts the docstrings from this yaml file and
uses that createconfig.rstwith the following contents:CONFIG_EXAMPLE Config=====================The following are the configuration options provided by the app. It can be configured by changing appropriate values in the``config.yaml``settings file. The options default to the default value of the classes for each of the options.`name`:Default value::''Some name.`size`:Default value::55Some filename.home panel----------`shape`:Default value::''Shape of the home.app panel---------`color`:Default value::''Color of the app.This rst is automatically rendered by sphinx to nice html with the rest of the docs and
it looks something like:CONFIG_EXAMPLE ConfigThe following are the configuration options provided by the app. It can be configured by changing appropriate values in theconfig.yamlsettings file. The options default to the default value of the classes for each of the options.name:Default value:''Some name.size:Default value:55Some filename.home panelshape:Default value:''Shape of the home.app panelcolor:Default value:''Color of the app.Class vs instanceThe configuration examples above save the config from the Appinstance.
One can also use the Appclassto dump the yaml. The major difference is that theapply_config_child,get_config_property,apply_config_property,
andpost_config_appliedmethods, which are instance methods, are skipped and
not used.Also, unlike for instances, where it would fail if_config_children_lists
a child property whose value is None, for the class it will fallback on the type
hint of the property, if one is defined.Using theAppclass, rather than aApp()instance is helpful during doc
building when it may not be possible to instantiate the full App
(see the docs example above that uses the class instance with type hints).Reusing other project docsBecause we rely on autodoc to generateconfig_prop_docs.yaml, tree-config
provides a mechanism to reuse the docstrings from other projects we depend on.E.g. imagine we depend onremote1andremote2projects who defines classes
that is configurable and our projects inherits and extends them with further
configurable properties.
Also assume these remote projects dumped their configurable docstrings toconfig_prop_docs.yamllike in the example and made it available in the
root of their sphinx generated docs e.g. on github-pages.Then, tree-config provides tools to merge those docstrings into ours to be able
to createconfig.rstfrom them as follows:echo$'Config\n==========='>source/config.rstpython-mtree_config.doc_gendownload\-u"https://user.github.io/remote1/config_prop_docs.yaml"-oconfig_prop_docs.yamlpython-mtree_config.doc_gendownload-fconfig_prop_docs.yaml\-u"https://matham.github.io/remote2/config_prop_docs.yaml"-oconfig_prop_docs.yamlmakehtmlmakehtmlThis downloads and merges the yaml files from our dependencies, adds to it our own
docs, and generates theconfig.rst.