Package: recap
recaprecapis a tool for providingREproducible Configurations for Any Project.Research should be reproducible.
Especially in deep learning, it is important to keep track of hyperparameters and configurations used in experiments.
This package aims at making that easier.InstallingJust install like any Python package:pipinstallrecapOverviewRecap provides two top-level concepts that would be imported as follows:fromrecapimportURI,CfgNodeasCNTheCfgNodeis a subclass ofyacs'CfgNode.
It provides some additional features for parsing configurations that are inherited between files which is not possible with yacs.Recap'sURIclass provides a mechanism for handling logical paths within your project more conveniently with an interface that is fully compatible withpathlib.Path.YAML configurationsConfigurations are definedjust like in yacs, except that you need to import theCfgNodeclass from the recap package instead of yacs.
Consider the following YAML configuration that sets default values for all configuration options we will use in our project. We shall name it_base.yamlbecause our experiments will build on these values.SYSTEM:NUM_GPUS:4NUM_WORKERS:2TRAIN:LEARNING_RATE:0.001BATCH_SIZE:32SOME_OTHER_HYPERPARAMETER:10The equivalent configuration can be obtained programatically like so:fromrecapimportCfgNodeasCNcfg=CN()cfg.SYSTEM=CN()cfg.SYSTEM.NUM_GPUS=4cfg.SYSTEM.NUM_WORKERS=2cfg.TRAIN=CN()cfg.TRAIN.LEARNING_RATE=1e-3cfg.TRAIN.BATCH_SIZE=32cfg.TRAIN.SOME_OTHER_HYPERPARAMETER=10print(cfg)Inheriting configurationsRecap provides functionality for inheriting configuration options from other configuration files by setting the top-level_BASE_key.
So, we could create a configuration fileexperiment_1.yamlfor an experiment where we try a different learning rate and batch size:_BASE_:_base.yamlTRAIN:LEARNING_RATE:1e-2BATCH_SIZE:64In our code, when we want to load the experiment configuration, we would use therecap.CfgNode.load_yaml_with_base()function:fromrecapimportCfgNodeasCNcfg=CN.load_yaml_with_base("experiment_1.yaml")print(cfg)# Will output:"""SYSTEM:NUM_GPUS: 4NUM_WORKERS: 2TRAIN:LEARNING_RATE: 0.01BATCH_SIZE: 64SOME_OTHER_HYPERPARAMETER: 10"""Note that the_BASE_keys can be arbitrarily nested; however, circular references are prohibited.Logical URIs and the path managerRecap includes a path manager for conveniently specifying paths to logical entities.
The path strings are set up like a URI where the scheme (i.e.httpin the path stringhttp://google.com) refers to a logical entity.
Each such entity needs to be set up as aPathTranslatorthat can translate the logical URI path to a physical path on the file system.For example, we could set up a path translator for thedatascheme to refer to the the path of a dataset on our file system located at/path/to/dataset. Then the recap URIdata://train/abc.txtwould be translated to/path/to/dataset/train/abc.txt.The simplest way of setting that up is using theregister_translatorfunction (although more complex setups are possible with therecap.path_manager.PathTranslatorclass, allowing you to download files from the internet, for example):fromrecap.path_managerimportregister_translatorfrompathlibimportPathregister_translator("data",Path("/path/to/dataset"))Then, we can use therecap.URIclass just like anypathlib.Pathobject:fromrecapimportURImy_uri=URI("data://train/abc.txt")# Here, str(my_uri) == "/path/to/dataset/train/abc.txt"withmy_uri.open("r")asf:print(f.read())Logical URIs in inherited configurationsTherecap.URIinterface is fully compatible with the nested configurations.
This means that you can use recapURIs within the_BASE_field for inheriting configurations.For example, you could register a path translator for theconfigscheme and then include_BASE_: config://_base.yamlin your configuration files.