Package: path-tree
pathtreeNamed path hierarchiesThe goal of this package is to make it easier to both define and understand folder structures in your scripts and pipelines. I find it difficult to visualize file structure when looking at a series ofos.path.joincommands scattered throughout a script, so instead, you define all of the paths up top, and fill in the placeholders later.Installpipinstallpath-treeTutorialQuick startNOTE: I want to update this to be more illustrative and less verbose. The problem is that this package can do a lot, but it's hard to show it concisely out of context. I'm not sure when I'll get a chance, but I'll try to get to it!Also, I apologize for not having an API Reference. I haven't quite ironed that into my deployment pipeline yet!importpathtree# instantiate the path tree - dict nesting == folder nesting !paths=pathtree.tree('./logs',{'{log_id}':{'model.h5':'model','model_spec.pkl':'model_spec','plots':{'epoch_{i_epoch:04d}':{'{plot_name}.png':'plot','':'plot_dir'}}}})...# specify variables along the way as they become availablepaths.update(log_id=12345)# usually they aren't inserted until you call format, so you can still change thempaths.update(log_id=12346)...# when you're ready to construct the path, just call .format()# with the remaining variables and it will give you the formattedplt.imwrite(paths.plot.format(i_epoch=5,plot_name='f1_score'))# writes to ./logs/12345/plots/epoch_0005/f1_score.pngThe old way - makes my brain sadUsually, you end up defining paths doing something like this (unless I'm doing something weird/dumb - lmk !!). And often,
these end up scattered throughout your project and getting a high level
picture of your path hierarchy is difficult.base_log_dir='./blah/logs'run_dir=os.path.join(base_log_dir,log_id)resources_dir=os.path.join(run_dir,'resources')...model_file=os.path.join(run_dir,'model.h5')model_spec=os.path.join(run_dir,'model_spec.pkl')...pump_file=os.path.join(resources_dir,'pump.pkl')...plot_dir=os.path.join(run_dir,'plots','epoch_{i_epoch:04d}')plot_file=os.path.join(plot_dir,'{plot_name}.png')The new way !*brain smiles*Instead, you can define your path hierarchy all in one place and give each tree node a name.importosimportpathtree# define the entire directory structure# the tree keys represent folder names.# the final non-dict key represents the name for# that directory node.## e.g.: {folder1: {folder2: name}}# paths.name => folder1/folder2## Notice the blank key under "plots". This takes advantage# of the fact that os.path.join('plots', '') == 'plots'.# So the name assigned to the blank string is naming the# directorybase_paths=pathtree.tree('./logs',{'{log_id}':{'model.h5':'model','model_spec.pkl':'model_spec','plots':{'epoch_{i_epoch:04d}':{'{plot_name}.png':'plot','':'plot_dir'# name for the directory}}}})# specify the log_id - specify returns a copy, update operates inplacepaths=base_paths.specify(log_id=12345)Basic ConceptsPaths- a collection of paths items as defined usingpathtree.tree.
Essentially, it is a wrapper around a flat dictionary of name -> path
and a data dictionary that is provided to all paths format.Path- a single path. It extendsos.PathLikeso it can be used where
path objects are expected (e.g.open(path).read()). It is a wrapper around apathlib.Pathobject and a data dictionary.
It provides basic path operations (join('subdir'),.up(2)to go up 2 parent directories,.glob()glob replacing missing fields with'*').Conversion to stringYou can access the paths using the name defined in the tree:assertstr(paths.model_spec)=='./logs/12345/model_spec.pkl'APathsobject (as defined above) is really just a dictionary of name =>pathtree.Pathobjects. This is a wrapper around string format pattern and a data dictionary.To convert aPathto a string, there are a few ways to get what you want.For fully specified strings (meaning that str.format will run without a KeyError), all three of these methods return the same thing: a fully formatted string.assertpaths.model.format()=='./logs/12345/model.h5'assertpaths.model.partial_format()=='./logs/12345/model.h5'assertpaths.model.maybe_format()=='./logs/12345/model.h5'For an underspecified string (missing data keys), the return values are different:# str.format is missing a key and will throw an exceptiontry:paths.plot.format(plot_name='f1_score')assertFalseexceptKeyError:# missing i_epochassertTrue# str.format is missing a key so the missing key will be left inassert(paths.plot.partial_format(plot_name='f1_score')=='./logs/12345/plots/epoch_{i_epoch:04d}/f1_score.png')# str.format is missing a key so it will keep it as a Path object# with the updated key `plot_name`.# this retains the ability to use data updating functionality.assertisinstance(paths.plot.maybe_format(plot_name='f1_score'),pathtree.Path)Paths are castable to string which is the same aspartial_format. You can also access the unformatted path usingPath().path.assertstr(paths.plot)==paths.plot.partial_format()assertpaths.plot.path==the_unformatted_plot_pathpathtree.Pathsubclassesos.PathLike, meaning that os.path functions know how to convert it to a path automatically.assertisinstance(paths.model,os.PathLike)assertos.path.join(paths.model)==paths.model.format()assertos.path.isfile(paths.model)Updating format dataYou can add path specificity at various points along the way. This is helpful when you need to reference subdirectories based on some loops or similar pattern. You can do that in a couple ways:# across the entire directory object# update in placepaths.update(log_id=12345)# or create a copypaths2=paths.specify(log_id=23456)assertpaths.data['log_id']==12345andpaths2.data['log_id']=23456# reverse specify - remove a data keypaths2=paths2.unspecify('log_id')assert'log_id'notinpaths2.data# or for a single path# in placeplot_file=paths.plotplot_file.update(plot_name='f1_score')# create a copyplot_file=paths.plot.specify(plot_name='f1_score')assert'plot_name'notinpaths.dataassertplot_file.data['plot_name']=='f1_score'# reverse specify - remove a data keyplot_file=plot_file.unspecify('plot_name')assert'plot_name'notinplot_file.dataAdditional FeaturesYou can automatically do glob searching. Any missing fields will be filled with a glob wildcard (asterisk). Note that this would fail using plain string format because the leading zero formatter (:04d) will throw an error if you try to insert'*'(because it's a string).plot_file=paths.plot.specify(plot_name='f1_score')assert(plot_file.partial_format()=='./logs/12345/plots/epoch_{i_epoch:04d}/f1_score.png')assert(plot_file.glob_pattern=='./logs/12345/plots/epoch_*/f1_score.png')assert(plot_file.glob()==glob.glob('./logs/12345/plots/epoch_*/f1_score.png'))You can also sometimes parse out data from a formatted string. Be warned,
it may not always work correctly because sometimes the parsing is ambiguous. Seehttps://github.com/r1chardj0n3s/parse#potential-gotchasplot_file=paths.plot.specify(root='some/logs')assert(plot_file.partial_format()=='some/logs/12345/plots/epoch_{i_epoch:04d}/{plot_name}.png')expected={'root':'some/logs''log_id':'12345','i_epoch':'0002','plot_name':'f1_score.png',}plot_data=plot_file.parse('./logs/12345/plots/0002/f1_score.png')assertset(plot_data.items())==set(expected.items())