Package: configue-cli
Configue CLIAconfigueextension that adds the ability to dynamically configure your application via the command line.Configue CLI overlaps in functionality withHydrabut without all the unnecessary boilerplate and with the benefit of being compatible withconfigue.Table of contentsInstallationQuick startInspection of the configuration stateConfiguration from the command lineConfiguration with YAML filesExporting the final configurationUnstructured configurationConfiguring the loggingIntegration with SkypilotInstallationTo install the library, usepipinstallconfigue-cliTo develop locally, clone the repository and usepipinstall-rrequirements-dev.txtQuick startWithconfigue-cli, configurations are defined with structured and arbitrarily nested Python objects (both native dataclasses andattrdataclasses are supported and can be nested).importdataclassesimportattrs@dataclasses.dataclassclassDatasetConfig:name:strn_samples:int=10_000@dataclasses.dataclassclassOptimizerConfig:learning_rate:float=0.001weight_decay:float=1e-2@attrs.defineclassModelConfig:name:strbatch_size:int=12optimizer:OptimizerConfig=attrs.Factory(lambdaself:OptimizerConfig(learning_rate=0.001*self.batch_size),takes_self=True)@dataclasses.dataclassclassExperimentConfig:model:ModelConfigdataset:DatasetConfigThese objects are injected at configuration time in your application entrypoint by theinject_from_clidecorator. To useconfigue-cli, simply wrap aclickentrypoint with theconfigue_cli.click.inject_from_clidecorator and provide a target type to be injected.importclickfromconfigue_cli.clickimportinject_from_cli@click.command()@inject_from_cli(ExperimentConfig)defmain(config:ExperimentConfig)->None:print("Passed configuration: ",config)if__name__=="__main__":main()To display a help message, use the following:pythonmain.py--helpInspection of the configuration stateTo visually inspect your application configuration state, use the following command:$pythonmain.py--dry-run

╭─Configurationhelper────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:Missing│
│├──batch_size:12│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.012│
│└──weight_decay:0.01│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:Missing│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────╯This is useful to quickly identify which parameters are not yet defined (those marked with aMissing) and which values are used in the other parameters without inspecting the code.Configuration from the command lineParameters can be specified from the command line using dotted notation.$pythonmain.pymodel.name=camembert-basedataset.name=fquadmodel.batch_size=48╭─Configuration───────────────────────────────────────────────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:camembert-base│
│├──batch_size:48│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.048│
│└──weight_decay:0.01│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:fquad│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────────────────────────────────────────╯
Passedconfiguration:ExperimentConfig(model=ModelConfig(name='camembert-base',batch_size=48,optimizer=OptimizerConfig(learning_rate=0.048,weight_decay=0.01)),dataset=DatasetConfig(name='fquad',n_samples=10000))Any missing required parameter at configuration time will result in an exception:$pythonmain.pymodel.batch_size=3Traceback(mostrecentcalllast):...
configue_cli.core.exceptions.MissingMandatoryValueError:Missingmandatoryvalue:dataset.nameConfiguration with YAML filesAny parameter can be overridden using aconfiguecompliant YAML file. Suppose the model is configured in the followingmodel.ymlfile:model:():__main__.ModelConfigname:camembert-largebatch_size:72optimizer:():__main__.OptimizerConfiglearning_rate:0.01weight_decay:0.0This configuration file can be loaded from the CLI using the-cflag:$pythonmain.py-cmodel.yml--dry-run

╭─Configurationhelper────────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:camembert-large│
│├──batch_size:72│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.01│
│└──weight_decay:0.0│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:Missing│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────────╯Multiple configuration files can be used simultaneously, the final configuration is assembled by merging all files in the order they are provided. For instance, let's suppose we have the followinglarge_batch.ymlfile:model:batch_size:512This file can be merged into our previous configuration using the following:$pythonmain.py-cmodel.yml-clarge_batch.yml--dry-run╭─Configurationhelper────────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:camembert-large│
│├──batch_size:512│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.01│
│└──weight_decay:0.0│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:Missing│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────────╯Parameters specified with the command line take precedence over the ones specified in YAML files:$pythonmain.pymodel.batch_size=32-cmodel.yml-clarge_batch.yml--dry-run

╭─Configurationhelper────────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:camembert-large│
│├──batch_size:32│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.01│
│└──weight_decay:0.0│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:Missing│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────────╯This feature encourages a modular configuration pattern where different subparts of the application (the model and the dataset in this example) are configured in separate YAML files and are dynamically assembled at configuration time. Different variations of these subparts can easily be assembled. All arguments can be overridden using the command line without having to edit the config files.Exporting the final configurationTo ease reproducibility, the final configuration used for the run can be exported by using the-oflag and specifying an output YAML file:$pythonmain.pydataset.name=hello-world-cmodel.yml-clarge_batch.yml-ooutput.yml

╭─Configuration───────────────────────────────────────────╮
││
│model│
│├──():__main__.ModelConfig│
│├──name:camembert-large│
│├──batch_size:512│
│└──optimizer│
│├──():__main__.OptimizerConfig│
│├──learning_rate:0.01│
│└──weight_decay:0.0│
││
│dataset│
│├──():__main__.DatasetConfig│
│├──name:hello-world│
│└──n_samples:10000│
││
╰───────────────────────────────────────────────────────────╯
PassedconfigurationExperimentConfig(model=ModelConfig(name='camembert-large',batch_size=512,optimizer=OptimizerConfig(learning_rate=0.01,weight_decay=0.0)),dataset=DatasetConfig(name='hello-world',n_samples=10000))$catoutput.yml
model:():__main__.ModelConfigname:camembert-largebatch_size:512optimizer:():__main__.OptimizerConfiglearning_rate:0.01weight_decay:0.0
dataset:():__main__.DatasetConfigname:hello-worldn_samples:10000Unstructured configurationIt is possible to use theinject_from_clidecorator without specifying a target type:@click.command()@inject_from_cli()defmain(config:configue_cli.core.dict_config.DictConfig)->None:...In that case, the wrapped entrypoint will be passed aconfigue_cli.core.dict_config.DictConfigobject upon injection.Configuring the loggingTo load alogging configurationlocated under the"logging"key in your final configuration, use the following:@click.command()@inject_from_cli(ExperimentConfig,logging_config_path="logging")defmain(config:ExperimentConfig)->None:...Integration with SkypilotSkyPilotis a framework for easily running jobs on any cloud through a unified interface. Any function decorated withinject_from_clican easily be executed remotely by providing a Skypilot configuration.The following configuration defines a job to be executed in a SkyPilot cluster namedtest-cluster. The job is defined under thetaskkey, we refer to theSkyPilot YAML specificationfor more details on this section.The Python command and all its arguments are captured and interpolated inside theruncommand, respectively in a{command}and{parameters}placeholder.# skypilot.ymlskypilot:cluster-name:test-clustertask:resources:cloud:gcpaccelerators:K80:1workdir:.setup:|echo 'Setup the job...'run:|set -ecd ~/sky_workdir{command} {parameters}To load the SkyPilot configuration in your final configuration, use the following:@click.command()@inject_from_cli(ExperimentConfig,skypilot_config_path="skypilot")defmain(config:ExperimentConfig)->None:...As with the other arguments, all SkyPilot configuration arguments can be redefined on the fly:pythonmain.py-cskypilot.ymlskypilot.cluster-name=another-cluster