Package: votakvot
votakvotA simple tool helping to track information, metrics, and files during
code development, testing, probing, experimentation and analysis.The ideaYou write Python code, annotate a function, call it.Votakvottrack what function parameters are, its result, git repo status, etc...Change your code, change parameters, try to rerun the function, experiment.Thenvotakvotmay load back all information as pandasDataFrame.Play with data and find the best combination of function parameters and version of a source code.Basic usageWrite a function and wrap it with an annotationvotakvot.track:@votakvot.track()defmy_experiment(one,two):print(one,two)returnone+twoThen callvotakvot.init()to initialize library internals:votakvot.init(path="./my-results",# path, where to store results, "." by default)Now any invocation ofmy_experiment(...)creates a new unique folder
inside./my-results. That new subfolder contains a filevotakvot.yamlwith:globally unique id (uuid4)timestamps (created, started, finished, duration)function parametersfunction resultgit info (branch, commit, work directory tree-ish)system information (machine, user, python version)traceback text on exceptionany additional ad-hoc informationAdditional information added withvotakvot.inform:@votakvot.track()defmy_experiment(one,two):...votakvot.inform(any_custom_field="any-value",other_custom_field=["structured","data"],)...Please note that any parameter, returned, or informed value is serialized
bypyyaml. It supports
all standard python types: int, float, complex, bool, str, bytes, list,
dict, tuple, set, datetime, None. Also any pickleable python class may
be serialized (including namedtuples and dataclasses), however it is not
recommended.Load reportsContent of multiplevotakvot.yamlfiles can be loaded intopandas.DataFrameby using functionvotakvot.load_report(). It gets
file path as a first argument. A path may be prefixed with a protocol
(ftp://,ssh://etc).
Some prefixes (likegs://ors3://) may require extra libraries to be installed (seefsspec protocolsfor details).Also path may contain glob patterns:*corresponds to any string
without/,**corresponds to anything.By default@votakvot.track()adds date-time of invocation into a subdirectory name:{function module} / {function name} / {yy}-{mm}-{dd} / {hour}:{minute}:{second} / {unique uuid}This allows loading results only for a particular module, function, date
or date-time only:root="/path/to/directory/with/results"# load all experiments from `root`votakvot.load_report(root)# load all experiments with additinoal fieldsvotakvot.load_report(root,full=True)# load all experiemnts for specified function onlyvotakvot.load_report(f"{root}/my_module/function_name")# load experiments for a single day 2021-05-20 (any function)votakvot.load_report(f"{root}/**/21-05-20")# load exprriments for a particular hour (any function)votakvot.load_report(f"{root}/**/21-05-20/15:*")A few dataframes may be merged with `pandas.concat`:# load results for 3 daysdf=pandas.concat(votakvot.load_report(f"{root}/**/{day}/**")fordayin["21-05-20","21-05-21","21-05-22"])ResultDataFramecan be filtered, sorted, updated, transformed,
plotted, serialized, and analyzed with all power of Pandas. Seepandasttutorial.Additionallyrawinformation may be obtained withload_trialsfunction:# load dict of {id -> votakvot.Trial}vs=load_trials(root)print("count",len(vs))print("ids:",vs.keys())# print raw content of `votakvot.yaml` filesprint("data",[v.dataforvinvs.values()])# print only git related informationprint("git commits",[v.meta.git.commitforvinvs.values()])MetricsTracked function may produce metrics:votakvot.meter(metric_name="metric value",)Metrics are stored as series of csv files and can be loaded to singlepandas.DataFrame:rep=votakvot.load_report()tid=rep.loc[0]['tid']# trial idvotakvot.metrics.load_metrics(tid)# instance of pd.DataFrameAttached filesA regular file may be created next tovotakvot.yaml. Use this to
store debug information (traceback, logs), create artifacts or even
store intermediate results of computation (see resumable tasks).@votakvot.track()defmy_experiment(one,two):...withvotakvot.attach("my-file-name.txt",mode='tw')asf:f.write("some text ...")MetadataLibrary automatically adds metadata to all generatedvotakvot.yamlfiles.
Metadata includes information about python environment, git repo
(commit, branch, and index hash), OS version. You can add extra
metadata by putting values into dictionaryvotakvot.meta.providers:# make copy of all providersmy_proivders=dict(votakvot.meta.providers)# add information about k8smy_proivders['k8s.version']=lambda:subprocess.getoutput("kubectl version")my_proivders['k8s.cluster_info']=lambda:subprocess.getoutput("kubectl cluster-info")# include list of all python librariesmy_proivders['python.pip_freeze']=lambda:subprocess.getoutput("pip freeze")# delete deafult medatata for 'git'my_proivders={k:vfork,vinvotakvot.metadata_providers.items()ifnotk.startswith("git.")}# here 'kubectl' command is invoked, but 'git' does notvotakvot.init(meta_providers=my_proivders,# use custom set of meta providers)Resumable tasksSome trials may take a lot of time to complete, it is possible to
make them resumable. A tracked function may be refactored into an iterable
pickleable object. If the program fails (or terminated manually) a pickled object
is still left on the disk andvotakvotwill automatically loaded it during the next trial run.classmy_function(votakvot.resumable_fn):snapshot_period=5# snapshot each 5sec, only in-between `self.loop()` callsdefinit(self,one,two):self.one=one...defloop(self):if...:return"result"# non-None value to finish computionelse:returnNone# repeat `self.loop()` one more time# autoresume when there is a snapshot for this id on the filesystemvotakvot.run(f"resumable_pi/n={n}/seed={s}",# id must be explicitly specified for resumable tasksmy_function,one=1,two=2,)votakvot-abVotakvot comes with basic benchmarking toolvotakvot-ab.
It behaves similar to the well knownabutility,
but instead of making HTTP calls invokes a python callback.Tool may patch sockets withgevent, allowing to
run IO-bounded code with bigger concurrency.Given filemy_module.py:importrequestsdefget_example():returnrequests.get("http://example.com/").status_codeThen call the function 1000 times in 200 "threads" (using greenlets):votakvot-ab--gevent-n1000-c200my_module.get_exampleAlso callback might be a class with custom initialization logic (usefull to create
HTTP sessions and connection pools, perform precomputations, etc):importrequestsclassHTTPGet:def__init__(self,url):self.url=urlself.session=requests.Session()# reuse connectionsdef__call__(self):returnself.session.get(self.url).status_codeInstance ofHTTPGetcreated once and then method__call__invoked 1000 times:votakvot-ab-g-n1000-c200my_module.HTTPGeturl=http://example.comSeevotakvot-ab --helpfor all parameters.LicenseVotakvotis released under the Apache 2.0 license (seeLICENSE)It is based on a project developed by Allegrohttps://github.com/allegro/votakvot