Package: nestly
nestlyis a collection of functions designed to ease running software with combinatorial choices of parameters.
It can easily do so for “cartesian products” of parameter choices, but can do much more– arbitrary “backwards-looking” dependencies can be used.The vision here is that we take a fixed set of parameters and generate a single type of output for each defined combination, which can then be combined in some way for comparison and retrieval.
We would like to set things up tidily with nested directories for output reflecting nested parameter choices.Thefull documentationis available on ReadTheDocs.InstallingThe easiest way is withpip:$ pip install nestlyOr, for the latest commit from master:$ pip install git+git://github.com/fhcrc/nestly.git@masterPython 2.7 is required.Introductory exampleImagine you’d like to try all possible combinations of the following:OptionChoicesstrategyapproximate, exhaustiverun_count10, 100, 1000input fileany file matching inputs/file*For this we can write a littlemake_nest.py. The guts are:nest = Nest()

nest.add('strategy', ('exhaustive', 'approximate'))
nest.add('run_count', [10**i for i in xrange(3)])
nest.add('input_file', glob.glob(os.path.join(input_dir, 'file*')),
        label_func=os.path.basename)

nest.build('runs')Runningmake_nest.py, you get a directory tree like:runs
├── approximate
│   ├── 10
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   ├── 100
│   │   ├── file1
│   │   │   └── control.json
│   │   ├── file2
│   │       └── control.json
│   └── 1000
│       ├── file1
│       │   └── control.json
│       ├── file2
│           └── control.json
└── exhaustive
    ├── 10
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    ├── 100
    │   ├── file1
    │   │   └── control.json
    │   ├── file2
    │       └── control.json
    └── 1000
        ├── file1
        │   └── control.json
        ├── file2
            └── control.jsonWith the finalcontrol.jsonreading, for example:{
    "input_file": "/Users/cmccoy/Development/nestly/examples/basic_nest/inputs/file3",
    "run_count": "1000",
    "strategy": "exhaustive"
}The control files created then serve as inputs tonestrunfor template substition, for example:nestrun --save-cmd-file command.sh \
        --template='my_command -s {strategy} --count={run_count} {input_file}' \
        $(find runs -name "control.json")This command runsmy_commandin all of the tip directories with the appropriate values for the parameters.This was a “cartesian product” example.
The “meal” example in the repository exhibits a setup with more complex dependencies between the nests.Templatesnestruntakes a template and a list of control.json files with variables to
substitute. By default, substitution is performed using the Python built-instr.formatmethod. See thePython Formatter documentationfor details on syntax,
andexamples/jsonrun/do_nestrun.shfor an example.SCons integrationThere is also anestly.sconsmodule to integrate nestly with themakereplacementSCons.Running the testsRun:python setup.py testThemocklibrary is required, but will be downloaded if missing.PaperMcCoy CO, Gallagher A, Hoffman NG, Matsen FA (2013) nestly–a framework for running software with nested parameter choices and aggregating results. Bioinformatics 29: 387-388.pubmedLicensenestlysource code is freely available under theMIT License.