Package: minydra
minydra 🦎Minimal Python command-line parser inspired by Facebook's Hydra + dot-accessible nested dictionaries.Easily parse arbitrary arguments from the command line without dependencies:pipinstallminydraminydrais tested on Python3.7,3.8and3.9.Getting Started•Forcing types•MinyDict•Save config•Prevent typos•Use default configs•ExamplesGetting Startedexamples/parser.pyfromminydra.parserimportParserif__name__=="__main__":parser=Parser(verbose=0,# print received argsallow_overwrites=False,# allow repeating args in the command-linewarn_overwrites=True,# warn repeating args if they are allowedparse_env=True,# get environment variablewarn_env=True,# warn if an environment variable is specified but not founddefaults=None,# path to a MinyDict-loadable dictionary of default values for the argsstrict=True,# if `defaults` is provided, whether to allow new keys in the command-line# or restrict to `defaults`' keyskeep_special_kwargs=True,# `defaults` and `strict` can be set from the command-line# with `@defaults=` and `@strict=`. This argument decides if# you want to keep those keys in the final arguments.)args=parser.args.pretty_print().resolve().pretty_print()# notice .resolve() transforms dotted.keys into nested dictsexamples/resolved_args.pyfromminydraimportresolved_argsif__name__=="__main__":args=resolved_args()args.pretty_print()examples/demo.pyexamples/demo.jsonfromminydraimportMinyDict,resolved_argsfrompathlibimportPathif__name__=="__main__":# parse arbitrary args in 1 lineargs=resolved_args()# override default confifargs.default:args=MinyDict.from_json(args.default).update(args)# protect args in the rest of the code executionargs.freeze()# print the args in a nice orderly fashionargs.pretty_print()# access args with dot/attribute accessprint(f'Using project "{args.log.project}" in{args.log.outdir}')# save configurationargs.to_json(Path(args.log.outdir)/f"{args.log.project}.json")examples/decorator.pyimportminydrafromminydra.dictimportMinyDict@minydra.parse_args(verbose=0,allow_overwrites=False)# Parser's init args work heredefmain(args:MinyDict)->None:args.resolve().pretty_print()if__name__=="__main__":main()ParsingSimple strings are parsed tofloatandintautomatically.A single keyword will be interpreted as a positive flag.A single keyword starting with-will be interpreted as a negative flag.Ifparse_envisTrue, environment variables are evaluated.$ python examples/decorator.py outdir=$HOME/project save -log learning_rate=1e-4 batch_size=64
╭───────────────────────────────────────────╮
│ batch_size    : 64                        │
│ learning_rate : 0.0001                    │
│ log           : False                     │
│ outdir        : /Users/victor/project     │
│ save          : True                      │
╰───────────────────────────────────────────╯dotted keys will be resolved to nested dictionary keys:$ python examples/decorator.py server.conf.port=8000
╭────────────────────╮
│ server             │
│ │conf              │
│ │ │port : 8000     │
╰────────────────────╯Usingast.literal_eval(value),minydrawill try and parse more complex values for arguments as lists or dicts. Those should be specified as strings:$ python examples/decorator.py layers="[1, 2, 3]" norms="{'conv': 'batch', 'epsilon': 1e-3}"
╭──────────────────────────────────────────────────╮
│ layers : [1, 2, 3]                               │
│ norms  : {'conv': 'batch', 'epsilon': 0.001}     │
╰──────────────────────────────────────────────────╯Forcing typesAdding___<type>to a key will force this type to the value. Notice how01is parsed to an integer1but04is parsed to a string (as specified)"04", andhellois parsed to alist, not kept as a string$ python examples/decorator.py n_jobs___str=04 job=01 chips___list=hello
╭────────────────────────────────────────╮
│ chips  : ['h', 'e', 'l', 'l', 'o']     │
│ job    : 1                             │
│ n_jobs : 04                            │
╰────────────────────────────────────────╯Known types are defined inParser.known_typesand the separator (___) inParser.type_separatorIn[1]:fromminydraimportParserIn[2]:Parser.known_typesOut[2]:{'bool','float','int','str'}In[3]:Parser.type_separatorOut[3]:'___'Command-line configurationYou can configure theParserfrom the command-line using special@arguments. In other words, all__init__(self, ...)arguments can be set from the command-line with@argname=new_value.In particular if you runpython examples/decorator.py @defaults=./examples/demo.jsonyou will see:╭──────────────────────────────────────╮
│ @defaults : ./examples/demo.json     │
│ log                                  │
│ │logger                              │
│ │ │log_level   : DEBUG               │
│ │ │logger_name : minydra             │
│ │outdir  : /some/path                │
│ │project : demo                      │
│ verbose   : False                    │
╰──────────────────────────────────────╯Butif you add@strict=false @keep_special_kwargs=falseyou will now have:$ python examples/decorator.py @defaults=./examples/demo.json @strict=false @keep_special_kwargs=false
╭──────────────────────────────╮
│ log                          │
│ │logger                      │
│ │ │log_level   : DEBUG       │
│ │ │logger_name : minydra     │
│ │outdir  : /some/path        │
│ │project : demo              │
│ verbose : False              │
╰──────────────────────────────╯(you need to have@strict=falsesince@keep_special_kwargsis unknown indemo.json. It would not be the case ifstrict=falsehad been used in the script itself (but it can be overridden from the command-line!))MinyDictMinydra's args are a custom lightweight wrapper around nativedictwhich allows for dot access (args.key), resolving dotted keys into nested dicts and pretty printing sorted keys in a box with nested dicts indented. If a key does not exist, it will not fail, rather return None (asdict.get(key, None)).aMinyDictinherits fromdictso usual methods work.keys(),.items()etc.In[1]:fromminydra.dictimportMinyDictIn[2]:args=MinyDict({"foo":"bar","yes.no.maybe":"idontknow"}).pretty_print();args╭──────────────────────────────╮│foo:bar││yes.no.maybe:idontknow│╰──────────────────────────────╯Out[2]:{'foo':'bar','yes.no.maybe':'idontknow'}In[3]:args.resolve().pretty_print();args╭──────────────────────────╮│foo:bar││yes│││no││││maybe:idontknow│╰──────────────────────────╯Out[3]:{'foo':'bar','yes':{'no':{'maybe':'idontknow'}}}In[4]:args.yes.no.maybeOut[4]:"idontknow"In[5]:"foo"inargsOut[5]:TrueIn[6]:"rick"inargsOut[6]:FalseIn[7]:args.mortyisNoneOut[7]:TrueIn[8]:args.items()Out[8]:dict_items([('foo','bar'),('yes',{'no':{'maybe':'idontknow'}})])Dumping/LoadingYou can save and readMinyDictto/from disk in 3 formats:jsonandpicklewithout dependencies,yamlwith thePyYAMLdependency (pip install minydra[yaml]).Methodsto_pickle,to_jsonandto_yamlhave 3 arguments:file_pathas astrorpathlib.Pathwhich is resolved:expand env variable ($MYDIRfor instance)expand user (~)make absolutereturn_pathwhich defaults toTrue. If those methods return the path of the created fileallow_overwriteswhich defaults toTrue. IfFalseandpathexists, aFileExistsErrorwill be raised. Otherwise creates/overwrites the file atfile_pathverbosewhich defaults to0. If>0prints the path of the created objectNote:to/from_yamlwill fail with aModuleNotFoundErrorifPyYAMLis not installed.thejsonstandard does not accept ints as keys in dictionaries so{3: 2}would be dumped -- and therefore loaded -- as{"3": 2}.In[1]:fromminydra.dictimportMinyDictIn[2]:args=MinyDict({"foo":"bar","yes.no.maybe":"idontknow"}).resolve();argsOut[2]:{'foo':'bar','yes':{'no':{'maybe':'idontknow'}}}In[3]:json_file_path=args.to_json("./args.json")In[4]:yaml_file_path=args.to_yaml("./args.yaml")In[5]:pkl_file_path=args.to_pickle("./args.pkl")In[6]:_=args.to_json("./args.json",verbose=1)# verbose argument prints the pathJsondumpedto:/Users/victor/Documents/Github/vict0rsch/minydra/args.jsonIn[7]:MinyDict.from_json("args.json")Out[7]:{'foo':'bar','yes':{'no':{'maybe':'idontknow'}}}In[8]:assert(MinyDict.from_yaml(yaml_file_path)==MinyDict.from_json(json_file_path)==MinyDict.from_pickle(pkl_file_path)==args)examples/dumps.pypython examples/dumps.py path="./myargs.pkl" format=pickle cleanup

╭────────────────────────────╮
│ cleanup : True             │
│ format  : pickle           │
│ path    : ./myargs.pkl     │
╰────────────────────────────╯
Dumped args to /Users/victor/Documents/Github/vict0rsch/minydra/myargs.pkl
Cleaning upStrict ModeTo prevent typos from the command-line, theMinyDict.updatemethod has a strict mode: updating aMinyDictwith another one usingstrict=Truewill raise aKeyErrorif the key does not already exist:fromminydraimportMinyDict,resolved_argsif__name__=="__main__":# parse arbitrary args in 1 lineargs=resolved_args()# override default confifargs.default:path=args.default# delete otherwise it will be used to update the conf which does not have# "default" as a key, therefore raising a KeyError in strict modedelargs.defaultargs=MinyDict.from_json(path).update(args,strict=True)args.pretty_print()No typo:$ python examples/strict.py default=./examples/demo.json log.logger.log_level=INFO
╭──────────────────────────────╮
│ log                          │
│ │logger                      │
│ │ │log_level   : INFO        │
│ │ │logger_name : minydra     │
│ │outdir  : /some/path        │
│ │project : demo              │
│ verbose : False              │
╰──────────────────────────────╯Typo:$ python examples/strict.py default=./examples/demo.json log.logger.log_leveel=INFO
Traceback (most recent call last):
  File "/Users/victor/Documents/Github/vict0rsch/minydra/examples/strict.py", line 13, in <module>
    args = MinyDict.from_json(path).update(args, strict=True)
  File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
    self[k].update(v, strict=strict)
  File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 111, in update
    self[k].update(v, strict=strict)
  File "/Users/victor/Documents/Github/vict0rsch/minydra/minydra/dict.py", line 100, in update
    raise KeyError(
KeyError: 'Cannot create a non-existing key in strict mode ({"log_leveel":INFO}).'Using default configurationsTheminydra.Parserclass takes adefaults=keyword argument. This can be:astror apathlib.Pathto ajsonyamlorpicklefile thatminydra.MinyDictcan load (from_X)adictor aminydra.MinyDictalistof the above types, in which case the resulting defaults will be the result of sequential updates from those defaults, enabling hierarchical defaults (first defaults are the starting point, then each subsequent defaults updates it)Whendefaultsis provided, the resultingminydra.MinyDictserves as a reference for the arguments parsed from the command-line:Ifyou setup the parser withstrict=True, arguments from the command-line will still have a higher priority but they willhave tobe present in thedefaultsto prevent typos or unknown arguments (seestrict mode)arguments not present in the command-line with fallback to values indefaultsdefaultscan actually be alistand the update order is the same as the list's. For instance:In[1]:fromminydraimportParserIn[2]:Parser(defaults=["./examples/demo.json","./examples/demo2.json"]).args.pretty_print();╭─────────────────────────────────╮│log│││logger││││log_level:INFO││││logger_name:minydra│││outdir:/some/other/path│││project:demo││new_key:3││verbose:False│╰─────────────────────────────────╯If you need to set defaults from the command-line, there's a special@defaultskeyword you can use:$ python examples/decorator.py @defaults=./examples/demo.json
╭──────────────────────────────────────╮
│ @defaults : ./examples/demo.json     │
│ log                                  │
│ │logger                              │
│ │ │log_level   : DEBUG               │
│ │ │logger_name : minydra             │
│ │outdir  : /some/path                │
│ │project : demo                      │
│ verbose   : False                    │
╰──────────────────────────────────────╯

$ python examples/decorator.py @defaults="['./examples/demo.json', './examples/demo2.json']"
╭───────────────────────────────────────────────────────────────────╮
│ @defaults : ['./examples/demo.json', './examples/demo2.json']     │
│ log                                                               │
│ │logger                                                           │
│ │ │log_level   : INFO                                             │
│ │ │logger_name : minydra                                          │
│ │outdir  : /some/other/path                                       │
│ │project : demo                                                   │
│ new_key   : 3                                                     │
│ verbose   : False                                                 │
╰───────────────────────────────────────────────────────────────────╯pretty_printPrints theMinyDictin a box, with dicts properly indented. A few arguments:indents, which defaults to2: the amount of indentation for nested dictionariessort_keys, which defaults toTrue: whether or not to alphabetically sort the keys before printingto_dictTo produce a native Pythondict, useargs.to_dict()Protected attributesMinyDict's methods (including thedictclass's) are protected, they are read-only and you cannot therefore setattributeswith there names, likeargs.get = 2. If you do need to have agetargument, you can access it throughitems:args["get"] = 2.Try withexamples/protected.py:python examples/protected.py server.conf.port=8000 get=3
╭────────────────────╮
│ get    : 3         │
│ server             │
│ │conf              │
│ │ │port : 8000     │
╰────────────────────╯
<built-in method get of MinyDict object at 0x100ccd4a0>
3
dict_items([('get', 3), ('server', {'conf': {'port': 8000}})])
{'conf': {'port': 8000}}TestsRun tests and pre-commit checks (isort,black,flake8) with$pipinstall-rrequirements-test.txt
$pre-commitrun--all-files
$pytest-vv--cov=minydratests/