Package: draccus
Draccus - Slightly Less Simple Configuration with DataclassesDraccus: "A large herbivorous reptilian creature, known for their ability to breathe fire."Draccus is a fork of the excellentPyrallislibrary, but with
a few changes to make it more suitable for more complex use cases. The main changes are:Support for subtyping configs (that is, choosing between different configs based on a parameter)Support for including config files in config filesBetter support for containers of configs (e.g. a list of configs)I swear I didn't want to fork it, but the Pyrallis devs (understandably) didn't want to merge some of these.Whydraccus?We support everything in Pyrallis (see their examples), but also support subtyping and including config files within
config files. We try to maintain the original repository's simple, clean approach.Withdraccusyour configuration is linked directly to your pre-defineddataclass, allowing you to easily create
different configuration structures, including nested ones, using an object-oriented design. The parsed arguments are
used to initialize yourdataclass, inheriting the corresponding type hints and code completion features.My First Draccus Example(This example is the same as in Pyrallis. Draccus differs mainly in advanced features like subtyping.)Here's a simple example of how to usedraccusto parse arguments into adataclass:fromdataclassesimportdataclassimportdraccus@dataclassclassTrainConfig:"""Training Config for Machine Learning"""workers:int=8# The number of workers for trainingexp_name:str='default_exp'# The experiment name@draccus.wrap()defmain(cfg:TrainConfig):print(f"Training{cfg.exp_name}with{cfg.workers}workers...")The arguments can then be specified using command-line arguments, ayamlconfiguration file, or both.$pythontrain_model.py--config_path=some_config.yaml--exp_name=my_first_expTraining my_first_exp with 42 workers...Assuming the following configuration fileexp_name:my_yaml_expworkers:42model:type:bertnum_layers:24num_heads:24hidden_size:1024dropout:0.2Inclusion of Config Files(This is a difference from Pyrallis.)We support including config files from other config files viapyyaml-include.
This is useful for splitting up your config into multiple files, or for including a base config file in your config.It works like this:# model_config.yamltype:bertnum_layers:24num_heads:24hidden_size:1024dropout:0.2# train_config.yamlexp_name:my_yaml_expworkers:42model:!includemodel_config.yamlIncluding Configs at Top LevelPyYAML, upon which draccus is based, supports a common YAML extension<<for merging keys from multiple maps.
We can combine this with!includeto include a config file:# base_config.yamltype:bertlr:0.001# train_config.yaml<<:!includebase_config.yamlexp_name:my_yaml_exp(I don't love this syntax, but it's consistent with PyYAML.)More Flexible Configuration with Choice Types(This is a difference from Pyrallis.)Choice Types, aka "Sum Types" or "Tagged Unions", are a powerful way to define a choice of types that can be selected at
runtime. For instance, you might want to choose what kind of model to train, or what kind of optimizer to use.Draccus provides aChoiceRegistryclass that lets you define a choice of types that can be selected at runtime. You
can then use theregister_subclassdecorator to register a subclass of your choice type. Thetypefield of the
choice type is used to select the subclass.Here's a modified version of the example above, where we use aChoiceRegistryto define a choice of model types:fromdataclassesimportdataclassimportdraccus# Choice Registry lets you define a choice of implementations that can be selected at runtime@dataclassclassModelConfig(draccus.ChoiceRegistry):pass@ModelConfig.register_subclass('gpt')@dataclassclassGPTConfig(ModelConfig):"""GPT Model Config"""num_layers:int=12num_heads:int=12hidden_size:int=768@ModelConfig.register_subclass('bert')@dataclassclassBERTConfig(ModelConfig):"""BERT Model Config"""num_layers:int=12num_heads:int=12hidden_size:int=768dropout:float=0.1@dataclassclassTrainConfig:"""Training Config for Machine Learning"""workers:int=8# The number of workers for trainingexp_name:str='default_exp'# The experiment namemodel:ModelConfig=GPTConfig()# The model configuration@draccus.wrap()defmain(cfg:TrainConfig):print(f"Training{cfg.exp_name}with{cfg.workers}workers...")The arguments can then be specified using command-line arguments, ayamlconfiguration file, or both.$pythontrain_model.py--config_path=some_config.yaml--exp_name=my_first_expTraining my_first_exp with 42 workers...Assuming the following configuration fileexp_name:my_yaml_expworkers:42model:type:bertnum_layers:24num_heads:24hidden_size:1024dropout:0.2Everything below here is from Pyrallis. I'll update it eventually.(It all still applies, substitutingdraccusforpyrallis.)Key FeaturesBuilding on that designpyrallisoffers some really enjoyable features includingBuiltin IDE support for autocompletion and linting thanks to the structured config. 🤓Joint reading from command-line and a config file, with support for specifying a default config file. 😍Support for builtin dataclass features, such as__post_init__and@property😁Support for nesting and inheritance of dataclasses, nested arguments are automatically created! 😲A magical@pyrallis.wrap()decorator for wrapping your main class 🪄Easy extension to new types usingpyrallis.encode.registerandpyrallis.decode.register👽Easy loading and saving of existing configurations usingpyrallis.dumpandpyrallis.load💾Magical--helpcreation from dataclasses, taking into account the comments as well! 😎Support for multiple configuration formats (yaml,json,toml) usingpyrallis.set_config_type⚙️Getting to Know ThepyrallisAPI in 5 Simple Steps 🐲The best way to understand the fullpyrallisAPI is through examples, let's get started!🐲 1/5pyrallis.parsefordataclassParsing 🐲Creation of an argparse configuration is really simple, just usepyrallis.parseon your predefined dataclass.fromdataclassesimportdataclass,fieldimportdraccus@dataclassclassTrainConfig:""" Training config for Machine Learning """# The number of workers for trainingworkers:int=field(default=8)# The experiment nameexp_name:str=field(default='default_exp')defmain():cfg=draccus.parse(config_class=TrainConfig)print(f'Training{cfg.exp_name}with{cfg.workers}workers...')if__name__=='__main__':main()Not familiar withdataclasses? you should probably check thePython Tutorialand come back here.The config can then be parsed directly from command-line$pythontrain_model.py--exp_name=my_first_modelTraining my_first_model with 8 workers...Oh, andpyrallisalso generates an--helpstring automatically using the comments in your dataclass 🪄$pythontrain_model.py--helpusage: train_model.py [-h] [--config_path str] [--workers int] [--exp_name str]optional arguments:-h, --help      show this help message and exit--config_path str    Path for a config file to parse with pyrallis (default:None)TrainConfig:Training config for Machine Learning--workers int   The number of workers for training (default: 8)--exp_name str  The experiment name (default: default_exp)🐲 2/5 Thepyrallis.wrapDecorator 🐲Don't like thepyrallis.parsesyntax?defmain():cfg=pyrallis.parse(config_class=TrainConfig)print(f'Training{cfg.exp_name}with{cfg.workers}workers...')One can equivalently use thepyrallis.wrapsyntax 😎@pyrallis.wrap()defmain(cfg:TrainConfig):# The decorator automagically uses the type hint to parsers arguments into TrainConfigprint(f'Training{cfg.exp_name}with{cfg.workers}workers...')We will use this syntax for the rest of our tutorial.🐲 3/5 Better Configs Using InherentdataclassFeatures 🐲When using a dataclass we can add additional functionality using existingdataclassfeatures, such as thepost_initmechanism or@properties:grin:fromdataclassesimportdataclass,fieldfrompathlibimportPathfromtypingimportOptionalimportdraccus@dataclassclassTrainConfig:""" Training config for Machine Learning """# The number of workers for trainingworkers:int=field(default=8)# The number of workers for evaluationeval_workers:Optional[int]=field(default=None)# The experiment nameexp_name:str=field(default='default_exp')# The experiment root folder pathexp_root:Path=field(default=Path('/share/experiments'))def__post_init__(self):# A builtin method of dataclasses, used for post-processing our configuration.self.eval_workers=self.eval_workersorself.workers@propertydefexp_dir(self)->Path:# Properties are great for arguments that can be derived from existing onesreturnself.exp_root/self.exp_name@draccus.wrap()defmain(cfg:TrainConfig):print(f'Training{cfg.exp_name}...')print(f'\tUsing{cfg.workers}workers and{cfg.eval_workers}evaluation workers')print(f'\tSaving to{cfg.exp_dir}')$python-mtrain_model.py--exp_name=my_second_exp--workers=42Training my_second_exp...Using 42 workers and 42 evaluation workersSaving to /share/experiments/my_second_expNotice that in all examples we use the explicitdataclass.fieldsyntax. This isn't a requirement ofpyrallisbut rather a style choice. As some of your arguments will probably requiredataclass.field(mutable types for example) we find it cleaner to always use the same notation.🐲 4/5 Building Hierarchical Configurations 🐲Sometimes configs get too complex for a flat hierarchy 😕, luckilypyrallissupports nested dataclasses 💥@dataclassclassComputeConfig:""" Config for training resources """# The number of workers for trainingworkers:int=field(default=8)# The number of workers for evaluationeval_workers:Optional[int]=field(default=None)def__post_init__(self):# A builtin method of dataclasses, used for post-processing our configuration.self.eval_workers=self.eval_workersorself.workers@dataclassclassLogConfig:""" Config for logging arguments """# The experiment nameexp_name:str=field(default='default_exp')# The experiment root folder pathexp_root:Path=field(default=Path('/share/experiments'))@propertydefexp_dir(self)->Path:# Properties are great for arguments that can be derived from existing onesreturnself.exp_root/self.exp_name# TrainConfig will be our main configuration class.# Notice that default_factory is the standard way to initialize a class argument in dataclasses@dataclassclassTrainConfig:log:LogConfig=field(default_factory=LogConfig)compute:ComputeConfig=field(default_factory=ComputeConfig)@pyrallis.wrap()defmain(cfg:TrainConfig):print(f'Training{cfg.log.exp_name}...')print(f'\tUsing{cfg.compute.workers}workers and{cfg.compute.eval_workers}evaluation workers')print(f'\tSaving to{cfg.log.exp_dir}')The argument parse will be updated accordingly$pythontrain_model.py--log.exp_name=my_third_exp--compute.eval_workers=2Training my_third_exp...Using 8 workers and 2 evaluation workersSaving to /share/experiments/my_third_exp🐲 5/5 Easy Serialization withpyrallis.dump🐲As your config get longer you will probably want to start working with configuration files. Pyrallis supports encoding a dataclass configuration into ayamlfile 💾The commandpyrallis.dump(cfg, open('run_config.yaml','w'))will result in the followingyamlfilecompute:eval_workers:2workers:8log:exp_name:my_third_expexp_root:/share/experimentspyrallis.dumpextendsyaml.dumpand uses the same syntax.Configuration files can also be loaded back into a dataclass, and can even be used together with the command-line arguments.cfg=pyrallis.parse(config_class=TrainConfig,config_path='/share/configs/config.yaml')# or the decorator synrax@pyrallis.wrap(config_path='/share/configs/config.yaml')# or with the CONFIG argumentpythonmy_script.py--log.exp_name=readme_exp--config_path=/share/configs/config.yaml# Or if you just want to load from a .yaml without cmd parsingcfg=pyrallis.load(TrainConfig,'/share/configs/config.yaml')Command-line arguments have a higher priority and will override the configuration fileFinally, one can easily extend the serialization to support new types 🔥# For decoding from cmd/yamlpyrallis.decode.register(np.ndarray,np.asarray)# For encoding to yamlpyrallis.encode.register(np.ndarray,lambdax:x.tolist())# Or with the wrapper version instead@pyrallis.encode.registerdefencode_array(arr:np.ndarray)->str:returnarr.tolist()🐲 That's it you are now apyrallisexpert! 🐲Why Another Parsing Library?XKCD 927 - StandardsThe builtinargparsehas many great features but is somewhat outdated :older_man: with one its greatest weakness being the lack of typing. This has led to the development of many great libraries tackling different weaknesses ofargparse(shout out for all the great projects out there! You rock! :metal:).In our case, we were looking for a library that would  support the vanilladataclasswithout requiring dedicated classes, and would have a loading interface from both command-line and files. The closest candidates werehydraandsimple-parsing, but they weren't exactly what we were looking for. Below are the pros and cons from our perspective:HydraA framework for elegantly configuring complex applications from Facebook Research.Supports complex configuration from multiple files and allows for overriding them from command-line.Does not support non-standard types, does not play nicely withdatclass.__post_init__and requires aConfigStoreregistration.SimpleParsingA framework for simple, elegant and typed Argument Parsing by Fabrice NormandinStrong integration withargparse, support for nested configurations together with standard arguments.No support for joint loading from command-line and files, dataclasses are still wrapped by a Namespace, requires dedicated classes for serialization.We decided to create a simple hybrid of the two approaches, building fromSimpleParsingwith somehydrafeatures in mind. The result,pyrallis, is a simple library that that is relatively low on features, but hopefully excels at what it does.Ifpyrallisisn't what you're looking for we strongly advise you to givehydraandsimpleParsinga try (where other interesting option includeclick,ext_argpase,jsonargparse,datargsandtap). If you do :heart:pyrallisthen welcome aboard! We're gonna have a great journey together! 🐲Tips and Design ChoicesBeware of Mutable Types (or use pyrallis.field)Dataclasses are great (really!) but using mutable fields can sometimes be confusing. For example, say we try to code the following dataclass@dataclassclassOptimConfig:worker_inds:List[int]=[]# Or the more explicit versionworker_inds:List[int]=field(default=[])As[]is mutable we would actually initialize every instance of this dataclass with the same list instance, and thus is not allowed. Insteaddataclasseswould direct you the default_factory function, which calls a factory function for generating the field in every new instance of your dataclass.worker_inds:List[int]=field(default_factory=list)Now, this works great for empty collections, but what would be the alternative forworker_inds:List[int]=field(default=[1,2,3])Well, you would have to create a dedicated factory function that regenerates the object, for exampleworker_inds:List[int]=field(default_factory=lambda:[1,2,3])Kind of annoying and could be confusing for a new guest reading your code :confused: Now, while this isn't really related to parsing/configuration we decided it could be nice to offer a sugar-syntax for such cases as part ofpyrallisfromdraccusimportfieldworker_inds:List[int]=field(default=[1,2,3],is_mutable=True)Thepyrallis.fieldbehaves like the regulardataclasses.fieldwith an additionalis_mutableflag. When toggled, thedefault_factoryis created automatically, offering the same functionally with a more reader-friendly syntax.Uniform Parsing SyntaxFor parsing files we opted foryamlas our format of choice, followinghydra, due to its concise format.
Now, let us assume we have the following.yamlfile whichyamlsuccessfully handles:compute:worker_inds:[0,2,3]Intuitively we would also want users to be able to use the same syntaxpython my_app.py --compute.worker_inds=[0,2,3]However, the more standard syntax for an argparse application would bepython my_app.py --compute.worker_inds 0 2 3We decided to use the same syntax as in theyamlfiles to avoid confusion when loading from multiple sources.Not ayamlfun?pyrallisalso supportsjsonandtomlformats usingpyrallis.set_config_type('json')orwith pyrallis.config_type('json'):TODOs:Fix error with default Dict and ListUnderlying error: No decoding function for type ~KT, consider using pyrallis.decode.registerRefine the--helpcommandFor example theoptionsargument is confusing thereAdd a test toomit_defaultsContributors ✨Thanks goes to these wonderful people (emoji key):Ido Weiss🎨🤔Yair Feldman🎨🤔This project follows theall-contributorsspecification. Contributions of any kind welcome!