Package: cfgopt
Introductioncfgoptis a solution for elegance of configuring complicated deep learning projects. Here, you configure everything only once in json, without need to write configuration and commandline parsing boilerplate code in python anymore. The solution is extremly simple, elegant, and compact. Without any 3rd-party dependency, the core implementation ofcfgoptis only around 200 LOC, andwill be(todo) fully tested.News2022/06/21cfgoptcode released.Getting StartedInstallationpip install cfgoptBasic UsageGenerally, you should have some jsonconfigfiles in a folder, and a python program that importcfgoptto parse them. The program can have command lineoptions that update specific field in the parsed results for this running. This is also why this library is called "cfgopt".To be concrete, you have two choices usingcfgopt:Use thecfgopt.parse_configs(cfg_root)API, just pass in the directory path of all the configuration files, and explore with the results! Or you should read the examples described in theFeaturessection.Usecfgoptruncommand directly or wrapcfgopt.main()function, for details please refer to the follow-upmain functionsection.Featuresbasic json bindingcfgopthas a main apiparse_configs(cfg_root)that accepts the path ofa folder of json files. All files will be (recursively) load in python and added to a rootdict, keeping the hierarchy and data types unchanged, with some exceptions described in follow-up sections.Feature first added inv0.1.block-level referencecfgoptrelease you from repeating yourself with a mountain pile of configuration files by borrowing the concept ofblock-level reference and embeddingin many modern note-taking apps such asLogseq, or more well-knownhyperlinksfor webpages.Thecfg://format URI:cfgoptfollows and expands string values matching a special syntaxcfg://<file-path>/<intra-file-uri>in the configuration files during parsing. This is one of the most repealing feature forcfgopt. You can also specifyrelativeuri which contains no substring ".json" in it.Example:file structure.
└──test_blockref_in_list├──cfg│├──data.json│└──recipes.json└──test_blockref_in_list.pydata.json:{"data1":{"meta":{"location":"/data/1/loc"}},"data2":{"meta":{"location":"/data/2/loc"}}}recipes.json:{"recipe1":{"use_data":["cfg://data.json/data1"]},"recipe2":{"use_data":["cfg://data.json/data1","cfg://data.json/data2"]}}test_blockref_in_list.pyimportcfgoptdeftest_blockref_in_list():cfg=cfgopt.parse_configs(cfg_root='test_blockref_in_list/cfg')# following lines are equivalentassertcfg["recipes.json"]["recipe2"]["use_data"][1]["meta"]["location"]=="/data/2/loc"assertcfg["recipes.json"]["recipe2/use_data/1/meta/location"]=="/data/2/loc"assertcfg["recipes.json/recipe2/use_data/1/meta/location"]=="/data/2/loc"__hostname__in URI will be translated into what you get by executinghostnamein shell. Sincev0.5.5.
Relative URI support added inv0.3.0.
Feature first added inv0.1.command-line updatecfgoptwill automatically parse command line options matching the python regex format^--(.*?)=(.*), and interpret it as an update of the parsed configuration folder. The right hand side of=should be valid json, and you might need to take care of shell escaping your special characters.For example, you can write--train.json/max_epochs=100or--train.json/max_epochs="100", since your shell would escape double-quotes, you will get an integer100in both cases.But if you write--train.json/resume=\"/path/to/ckpt\"or--train.json/resume='"/path/to/ckpt"', you should probably get a string value, which depends on your shell implementation.Unmatched uri/keys are treated as errors sincev0.7.0. Regex format updated as^--(.*?)=(.*)to match the first=symbol sincev0.7.2.Behavior changed to update existing uri before expansion and then update thereafter, this makes both features, i.e., command-line specifying expansion and assigning values after expansion, work. Sincev0.5.0.Updated regex format from^--(.*)=(.*)to^--(.*\.json.*)=(.*)inv0.2. But then changed back sincev0.4.3, be careful that unmatched uri/keys would just be ignored without warning.Feature first added inv0.1.json objects inheritanceUsers can specify a json dict, that contains a__base__field, linking to another base json dict objects with thecfg://reference format (described inblock-level reference). Then the current dict would inherit the base object, and also has its own values in normal fields. This feature is also critical to eliminate repeating, with which now you can develop multiple simillar configs from some prototypes.TODO: an example.Bugfix: changed from referencing to deepcopying the base dict inv0.2.Feature first added inv0.1.parse python objectscfgopthas a extremly flexible feature, that parse an json dict to ALMOST any python objects defined in user's code or any code python can find inPYTHON_PATH.Users can specify a json dict, that contains__module__and__class__field. The__module__field will be imported byimportlib.import_module()during parsing, and__class__field naming any pythoncallablein the imported module will be passed tofunctools.partial()along with other fields as keyword arguments. Finally, the mapped "dict" in python would be directly callable to instantiate corresponding class or get result of corresponding functions.pseudo-code of parsing:module=importlib.import_module(data["__module__"])klass=getattr(module,data["__class__"])data["__class__"]=partial(klass,**{k:vfork,vindata.items()ifnotk.startswith("__")})NOTE: The python object should not use VAR_POSITIONAL and POSITIONAL_ONLY arguments, and keywords arguments are always recommended than positional arguments.By default, when arguments has nested object that is meant to be constructed bycfgoptand you call a higher level object, the nested ones in arguments will beautomatically and recursively instantiatedas long asall its required arguments are defined. If you want certain object not to be automatically created, add a"__as_type__": truefield alongside the__class__field.Let us refer to these "dict"s with__class__and__module__fields aspython object builders. You can call into these builders with extra arguments to instantiate them. You can also pass only partial arguments for multiple times, and it won't actually instantiate until all required arguments are ready, similar to above. This is very handy to pass around the builder everywhere deep into the code to collect scattered arguments. For example define a bachnorm builder, and pass it as argument to your networks, the latter can pass the channel argument when it is figured out from the preceding layer. The builder can be used multiple times.You can also create a builder from python code, by usingcfgopt.PartialClass()function, very similar to standardfunctools.partial(), but again it allows being called multiple times until all required args are given.TODO: an example.__as_type__keyword added inv0.5.3.Important Feature:PartialClassand lazily instantiation sincev0.5.0.Supported auto instantiate of nested python objects sincev0.4.0.API enhance: user now can directly call the mapped "dict" object instead of its__class__field inv0.2.Feature first added inv0.1.main functionimportargparseimportcfgoptdefmain():parser=argparse.ArgumentParser()parser.add_argument("recipe",help="a main function uri to be execute.")parser.add_argument("-d","--cfgdir",default="cfg",help="config directory that maps to cfg:// root. (default: `cfg`)")args,unknown_args=parser.parse_known_args()cfgs=cfgopt.parse_configs(cfg_root=args.cfgdir,args=unknown_args,args_root=args.recipe,)_main=cfgs[args.recipe]return_main(recursive=False)if__name__=="__main__":main()This examplemain()function accepts the first argument as a previous describedcfg://-format uri, that parses a python callable function, and call it as the main function. This is also implemented ascfgopt.main(), and user can usecfgoptruncommand direct from shell, or just import this main function and call it in usercode with extra python arguments.TODO: an example.Behaviour changed:cfgoptruninterprets commandline arguments as relative path torecipeinstead of absolute(full) uri sincev0.6.0.Addcfgoptruncommand (entrypoint) inv0.2.Feature first added inv0.1.