Package: benchmcmc
benchmcmc — Benchmark analysis with MCMCInstallInstalling is as alwayspip install benchmcmcIt depends onpymc3andmatplotlib, so these are also
installed.QuickstartGetting started real quick:benchmcmc --generate 69 11 10 131 10 9 --beta > bench.txt
benchmcmc bench.txtThe first command generates 200 synthetic data points and put them intobench.txt, and the second command runsbenchmcmcon the data,
displaying atrace plotof the MCMC.IntroductionThis package lets you take a series of benchmark data analyses whether
there at one point was a change in performance.Suppose that you run your benchmark tests on every commit that you have
(e.g. looping overgit-rev-list),
and you see that your performance data
is (e.g. in requests per second or in seconds, or other measures)
as follows:13.64
12.82
11.69
15.12
12.30
18.46
13.51
14.33
13.84
12.77
... (180 rows omitted)
10.93
11.02
12.45
11.78
12.12
13.51
10.66
10.18
10.81
12.19In this data, it seems to be centered around 13+ε in the beginning,
and it ends
being centered around 12+ε, or visualized:There seems to be a slight drop in values before the 100th point, but
it's not easy to determine exactly where the change occurred.Suppose that you wonder whether or not the performance at the start and
at the end are likely to be from two different distributions, and if so,
where theswitchpointwas.AnalysisRunningbenchmcmcon the data gives the above plot which shows that it
is likely that the performance went from ~13.5 to ~12.25 at or around the
69th or 75th datapoints.This helps you pin down when a performance change might have occurred.Usage$ benchmcmc bench.txt --draws 1000 --tune 1000 --cores 4 --target-accept 0.8The options available are those illustrated in the above examples and
all taken from and fed intopymc3.sampling.sample.draws— The number of samples to draw. Defaults to 1000. The number
of tuned samples are discarded by default.tune— Number of iterations to tune, defaults to 1000. Samplers
adjust the step sizes, scalings or similar during tuning. Tuning
samples will be drawn in addition to the number specified in the draws
argument, and will be discarded.cores— The number ofchainsto run in parallel.  IfNone, set
to the number of CPUs in the system, but at most 4.target_accept: float in [0, 1] — The step size is tuned such that
we approximate this acceptance rate. Higher values like 0.9 or 0.95
often work better for problematic posteriors.Generating synthetic dataYou can runbenchmcmc --generatefor generating synthetic benchmark
data.$benchmcmc--generate100153100143[--beta]>benchmarkfile.txtThis generates 200 samples, 100 fromN(mu=15, sigma=3)followed by 100
fromN(mu=14, sigma=3).If you use--beta, you get a bit more realistic performance with alower boundofmu, especially for lower values ofmu.Running a script on a historySuppose that you want to runpython script.pyon a script that is in
your Git tree.LOGFILE=/tmp/timescriptecho"">$LOGFILEforcommitin$(gitrev-listmaster)dogitcheckout$commitprintf"%s,""`(git rev-parse --short HEAD)`">>$LOGFILE/usr/bin/time-a-o$LOGFILE--format=%epythonscript.pydonetac$LOGFILEWhen run in a repository, it will output time data in the formatcommit,timeHere is an example of the output:484fde8,0.04
58a1cdb,0.04
d26b797,0.04
81f4b9a,0.04
3ae1e11,0.04
7689ca2,0.04
8c76b29,0.04
43db50c,0.04
b34b146,0.04
4c56a54,0.04
9c08050,0.07
b22278d,0.07
7a9c111,0.07
065b6a5,0.07
6cc7cdd,0.07
ec7f042,0.07
b3ba887,0.08
a32ce81,0.07
9136914,0.07
b456714,0.07
504cf73,0.07
8002774,0.07
e1f5f9f,0.09For example, you can create a scriptgitrunthat looks as follows:#!/usr/bin/bashGITRUN_EXEC=$1shiftGITRUN_ARGS=$@forcommitin$(gitrev-listmaster)dogitcheckout--quiet$commitprintf"%s,""`(git rev-parse --short HEAD)`"sh-c"$GITRUN_EXEC$GITRUN_ARGS"done