Package: experiment-config
InstallationInstall with pip:pipinstallexperiment-configUsageexpfig.Configallows for straightforward hyperparameter selection and logging.It reads hyperparameters from YAML files, the command line, and user inputs and makes them
available as both attributes and keys.
It can be embedded in both a script or a class.Quick StartWe will build a simple version ofFizzBuzzthat allows custom replacement of the wordsFizzandBuzz.A simple solution of FizzBuzz looks like this:# examples/quick_start/fizz_buzz.pyclassSolution:n=15deffizzBuzz(self):out=[]forjinrange(1,self.n+1):val=''ifj%3==0:val='Fizz'ifj%5==0:val+='Buzz'elifnotval:val=str(j)out.append(val)print(out)returnoutSolution().fizzBuzz()Callingpython examples/quick_start/fizz_buzz.pyat the command line will print['1','2','Fizz','4','Buzz','Fizz','7','8','Fizz','Buzz','11','Fizz','13','14','FizzBuzz']We can use expfig.Config to quickly replace'Fizz','Buzz', and the integernat the command line.Let's define a filefizz_buzz_default_config.yaml:n:15words:buzz:Buzzfizz:Fizzand replace the values of'Fizz','Buzz', andnin our script with the corresponding
values in the config. We will also add a pretty-print of our config, just to keep track of what is going on:# examples/quick_start/fizz_buzz.pyfromexpfigimportConfigclassSolution:config=Config(default='fizz_buzz_default_config.yaml')deffizzBuzz(self):self.config.pprint()out=[]forjinrange(1,self.config.n+1):val=''ifj%3==0:val=self.config.words.fizzifj%5==0:val+=self.config.words.buzzelifnotval:val=str(j)out.append(val)print(out)returnoutSolution().fizzBuzz()Callingpython examples/quick_start/fizz_buzz.pyat the command line will now printconfig:n:15words:buzz:Buzzfizz:Fizz['1','2','Fizz','4','Buzz','Fizz','7','8','Fizz','Buzz','11','Fizz','13','14','FizzBuzz']which is, as expected, our config followed by the solution.We can now easily modify any combination of our values:$pythonexamples/quick_start/fizz_buzz.py--n10--words.buzzBuzzword

config:n:10words:buzz:Buzzwordfizz:Fizz['1','2','Fizz','4','Buzzword','Fizz','7','8','Fizz','Buzzword'].This example can be viewed in theexamples/quick_startdirectory.verbose--verboseis a special key thatexpfig.Configwill read. It accepts positive integer values and will print
the config is increasing verbosity depending on its value.--verbose 0: nothing is printed (this is the default).--verbose 1: the symmetric difference between the config and the default config is printed.--verbose 2: the entire config is printed.For example:$pythonexamples/quick_start/fizz_buzz.py--n10--words.buzzBuzzword--verbose1config:n:10words:buzz:Buzzword
config:n:10words:buzz:Buzzwordfizz:Fizz['1','2','Fizz','4','Buzzword','Fizz','7','8','Fizz','Buzzword']The first block is the difference between the config and the default config, while the second is the pretty-print of
the entire config.Saving aConfigexpfig.Configtakes advantage of YAML-serialization (and de-serialization) for
reproducibility.You can use bothexpfig.Config.serializeandexpfig.Config.serialize_to_dirfor serialization.expfig.Config.serializeperforms a simple YAML-dump of the underlying dictionary:# python examples/quick_start/serialize_fizz_buzz.pyfromfizz_buzzimportSolutionwithopen('simple_serialization.yaml','w')asf:Solution().config.serialize(f)expfig.Config.serialize_to_dirwill ensure that you are not overwriting any existing directories
(if desired), and can also handle serializing the default config and the difference:# python examples/quick_start/serialize_fizz_buzz.pyfromfizz_buzzimportSolution# Serialize the underlying dict. Makes sure it does not overwrite any existing `fizz_buzz_config` directory# by appending an integer on the end if one exists.Solution().config.serialize_to_dir('fizz_buzz_configs')# Same as the above, but also serialize the default config and the difference.Solution().config.serialize_to_dir('fizz_buzz_configs_with_default',with_default=True)You can then useexpfig.Config.deserializeto load your saved serialization and reproduce your settings, for example:fromexpfigimportConfigwithopen('simple_serialization.yaml','r')asf:config=Config.deserialize(f)Note that doing so effectively treatssimple_serialization.yamlas a default config;
you can use command-line arguments to update it upon loading.Additional methods of inputting custom hyperparametersThere are three other ways to define custom settings/hyperparameters:You can pass the--config path_to_a_config.yamlargument at the command line.path_to_a_config.yamlmay
contain any combination of values as defined in your default config file; they must be in the same format.
You may pass any number of config files this way:--configpath_to_a_config.yamlpath_to_another_config.yamlIf you pass multiple config files with conflicting values, the value from thelastconfig file
will be used.Note that any values passed this way will be overridden by explicit arguments or arguments passed by the below
two methods.You can pass a path to ayamlfile containing settings toexpfig.Config. You may pass any combination
of values as defined inconfig/default_config.yaml; they must be in the same format. This is equivalent to
the above except it is done within a script and not at the command line:fromexpfigimportConfigconfig=Config(config='path_to_a_config.yaml')You can pass a nested dictionary defining configuration settings toexpfig.Config.
For example:fromexpfigimportConfigconfig_dict={'microgrid':{'config':{'scenario':1}},'algo':{'sampler':{'type':'local','n_workers':4}},'context':{'verbose':True}}config=Config(config_dict)Hyperparameter Resolution OrderYou may encounter the situation where you pass the same key in different ways, with different values. For example, you may have a key in both your default config, a config passed via--config path_to_a_config.yaml,
and by direct argument at the command line:--key value.For any key set in your default config, the resolution order is as follows:Values passed directly toexpfig.Configupon object initialization. This includes values
defined within a config file passed toexpfig.Config:expfig.Config(config='path_to_a_config.yaml).Values passed explicitly at the command line.Values within a config file passed at the command line. If multiple config files are passed and the
key is contained in more than one of said files, the value from thelastfile will be used.Values within your default config.Variations between different methods of setting parameters1. Type castingValues passed at the command line are casted to the type of the default value as defined by the yaml-load of the default
value. For example, a default config containingvalue: 1will result in the expectation thatvalueis anint.
This is not true with values passed in python code or in separate config files.There are two exceptions to this:Values where the default value isNoneparse command line arguments to string.The stringnullpassed at the command line results in the valueNone.Additional ExamplesAn example of usingexpfig.Configto set hyperparameters for a machine learning problem
is available inexamples/knn. This example demonstrates a simple class to run a classification problem usingscikit-learn'sKNeighborsClassifier.