Package: paramsurvey
paramsurveyparamsurvey is a set of tools for creating and executing parameter
surveys, on systems ranging from laptops to supercomputing clusters.paramsurvey has a pluggable parallel backend. The supported backends at present
are python's multiprocessing module, and computing cluster softwareray.Exampleimport time
import paramsurvey


def sleep_worker(pset, system_kwargs, user_kwargs):
    time.sleep(pset['duration'])
    return {'slept': pset['duration']}


paramsurvey.init(backend='multiprocessing')  # or 'ray', if you installed it

psets = [{'duration': 0.3}] * 5

results = paramsurvey.map(sleep_worker, psets, verbose=2)

for r in results.itertuples():
    print(r.duration, r.slept)
for r in results.iterdicts():
    print(r['duration'], r['slept'])prints, in addition to some debugging output, a result from each of the 5 sleep_worker calls.Here are a few more examples:The above example, with a few notesAn example of a multi-stage computation, running severalmap()functions in a rowAn example of greedy optimization, selecting the best alternative from eachmap()resultAn example that runs a command-line program for each psetAn example of using helper functions to create the list of psets to computeAn example of using the stats to time subsections of your worker functionAn example of using results.missing to re-run missing psetsThese examples are installed with the package, so you can run them like this:$ paramsurvey-readme-example.py
$ paramsurvey-multistage-example.py
$ paramsurvey-greedy-example.pyPhilosophyA parameter survey runs begins by initializing the software,
specifying a backend ('multiprocessing' or 'ray').The user supplies a worker function, which takes a dict of parameters
(pset) and returns a dict of results.The user also supplies a list of parameter sets (psets), perhaps
constructed using the helper functionparamsurvey.params.product().Callingpararamsurvey.map()executes the worker function once for
each pset. It returns aMapResultsobject, containing the results,
performance statistics, and information about any failures.You can callparamsurvey.map()more than once.Keyword arguments to init() and map()Theparamsurveycode has a set of keyword arguments (and corresponding environment
variables) to aid debugging and testing. They are:backend="multiprocessing"-- which backend to use, currently "multiprocessing" (default) or "ray"verbose=1-- print information about the progress of the computation:0 = print nothing1 = print something every 30 seconds (default)2 = print something every second3 = print something for every actionvstats=1-- controls the verbosity of the performance statistics system, with similar values asverboselimit=0-- limits the number of psets actually computed to this number (0 meaning "all")ncores=-1-- limits the number of cores used, in this case 1 less than the number available (multiprocessing only)max_tasks_per_child=3-- the number of tasks a child will do before restarting. Useful to limit memory leaks. Default: infinitegroup_size=N-- bundle psets into groups, which is useful if a single pset's runtime is too short forrayto efficiently run them. One minute is a good runtime.Each of these has a corresponding environment variable,
e.g.PARAMSURVEY_BACKEND,PARAMSURVEY_VERBOSE.  If the environment
variable is set, it overrides the values set in the source code. If a
kwarg is set for amap()call, that value overrides any value
specified for theinit()call.For example, if you wish to debug a large computation by running a small
subset of it on a single node, the environment variables allow you to do
this without editing your source code:$ PARAMSURVEY_BACKEND=multiprocessing PARAMSURVEY_VERBOSE=3 PARAMSURVEY_LIMIT=10 ./myprogram.pyFor retrospective debugging, i.e. your run crashes and you are sad
that you specified a lower verbosity than you desire post-crash,paramsurveycreates a hidden logfile in the current directory for
every run, named.paramusurvey-DATE-TIME.log. For example, this
hidden logfile will always contain information about any exceptions
raised in your worker code.Pandas-related quirksTo preserve memory in the case of large numbers of psets, the psets and the results are stored as
a Pandas DataFrame. This creates a couple of quirks visible to user code:A key that exists in any pset or result will exist in every pset or result, with a default value ofnanIf you treatnanas a boolean in Python, it evaluates toTrueBecause of a (fixable) quirk inpandas-appender, columns are not automatically promoted from integer
to float. So if you have a large number of psets with an integer value, and then throw in a float, it will be rounded to an integer.Backend-specific argumentsBothinit()andmap()take a backend-specific keyword argument named for the backend, and
ignored by other backends. For example, to pass an argument only used by theraybackend,paramsurvey.init(..., ray={'address': 'auto'})
...
paramsurvey.map(..., ray={'num_gpus': 1})Individual psets can also have backend-specific arguments, in the case
where different units of work need different memory limits or
different thread counts. The complete list of ray arguments is:name,memory,num_cpus, andnum_gpus. There is alsonum_coresas an alias fornum_cpus.The MapResults objectThe MapResults object has several properties:resultsis a Pandas DataFrame containing the values of the pset
and the keys returned by the worker function. Iterating over these
results is documented above, as either dicts or tuples.missingis a DataFrame of psets that did not generate results,
plus extra '_exception' and '_traceback' columns if an exception was
raised in the worker function.progressis a MapProgress object with properties containing the
details of pset execution: total, active, finished, failures,
exceptions.statsis a PerfStats object containing performance statistics.Fault tolerance, and re-running failed computationsIf you have bugs in your code that raise exceptions, paramsurvey will
diligently collect the exceptions and tracebacks and will print them
into the output (verbose=2 or more) and also in the hidden logfile.paramusurvey-DATE-TIME.logmentioned above.  Any pset causing an
exception will be in theresults.missingDataFrame.In addition to exceptions thrown by user code, the paramsurvey module
and the laptop or cluster its running on can experience two kinds of
errors. The first is failures of some nodes or processes in one of the
distributed backends, likeray,raywill quietly re-run the pset
as long as the head node and the driver are still alive. So, for example,
it's safe to run most nodes in the computation as a "EC2 spot instance"
or "preemptable node" in a cluster queue system like Slurm.The other kind of error is one that can't be caught by paramsurvey.
This might include pythonmultiprocessingcompletely crashing
everything because your laptop is out of memory, orrayhaving
indigestion because some nodes are low on memory and are responding
slowly, but aren't totally dead.Debugging checklistCheck the hidden logfile for details of previous runsUse environment variables to shrink your run to a single pset, as mentioned above:$ PARAMSURVEY_BACKEND=multiprocessing PARAMSURVEY_VERBOSE=3 PARAMSURVEY_LIMIT=1 ./myprogram.pyMemory debugging:print(paramsurvey.utils.vmem())in a few places, values are in GBytesSlow memory leaks: restart children after every Nth pset by addingmax_tasks_per_child=10to theparamsurvey.init()callLook at the performance statisticsLook at the example scripts linked above, which demonstrate most features mentioned in this READMECarbon monitoringIf you'd like to monitor your paramsurvey run in real-time on one of those fancy modern dashboard thingies,
pass details toparamsurvey.init(..., carbon_server="127.0.0.1", carbon_port=2004, carbon_prefix="paramsurvey")The integrated code only knows how to sendpickleand (so far) only records progress.Worker function limitationsThe worker function runs in a different address space and possibly on a different server.
It shouldn't access any global variables.For hard-to-explain Python reasons, define the worker function before
callingparamsurvey.init(). The worker function should not be nested
inside another function. On Windows, the main program file should have
aif __name == '__main__'guard similar to the examples at the top
of the Python multprocessing
documentation.Installing$ pip install paramsurvey
$ pip install paramsurvey[ray]