Package: laboratory
A Python library for carefully refactoring critical paths by testing in
production (inspired byGitHub’s Scientist) with support for Python 2.7, 3.3+Why?InstallationGetting startedAdding contextRamping upControlling comparisonRaise on mismatchPublishing resultsCaveatsLinksWhy?Some blocks of code are more critical than the rest. Laboratory helps us refactor important
code paths by running experiments in production and verifying the results.The value lies in its ability to give us a sense of confidence where there was none before.
Through experimentation we immediately see if candidate code is misbehaving, and at the same
time we establish a feedback loop that we can use to converge on correctness more quickly.I’ve written a blog post if you’d like to know more:Sure-footed refactoring.
The original blog post that inspired this project is worth a read too:Scientist.InstallationInstalling from pypi is recommended$ pip install laboratoryYou can also install atagged versionfrom Github$ pip install https://github.com/joealcorn/laboratory/archive/v1.0.tar.gzOr the latest development version$ pip install git+https://github.com/joealcorn/laboratory.gitGetting startedSee:Installationorpip install laboratoryWith Laboratory you conduct an experiment with your known-good code as the
control block and a new code branch as a candidate. Laboratory will:Execute the new and the old code in a randomised orderCompare the return valuesRecord timing information about old & new codeCatch (but record!) exceptions in the new codePublish all of this informationLet’s imagine you’re refactoring some authorisation code. Your existing code
is working, but it’s a fragile pile of spaghetti that is becoming hard to
maintain. You want to refactor, but this is important code and you simply can’t
afford to get this wrong or else you risk exposing user data.
Considering the state of the original code, this could be difficult to pull off,
but Laboratory is here to help.Laboratory helps us verify the correctness of our implementation even with the
cornucopia of factors that make production a unique environment (bad or legacy
data, heavy load, etc.)Let’s set up an experiment to run our old (control) and new (candidate) code:importlaboratory# set up the experiment and define control and candidate functionsexperiment=laboratory.Experiment()experiment.control(authorise_control,args=[user],kwargs={'action':action})experiment.candidate(authorise_candidate,args=[user],kwargs={'action':action})# conduct the experiment and return the control valueauthorised=experiment.conduct()Note that theExperimentclass can also be used as a decorator if the
control and candidate functions take the same arguments.defauthorise_candidate(user,action=None):returnTrue@Experiment.decorator(candidate=authorise_candidate)defauthorise_control(user,action=None):returnTrueAn experiment will always return the value of the control block.Adding contextA lot of the time there’s going to be extra context around an experiment that’s
useful to use in publishing or when verifying results. There are a couple ways
to set this.# The first is experiment-wide context, which will be set on every Observation an experiment makesexperiment=laboratory.Experiment(name='Authorisation experiment',context={'action':action})# Context can also be set on an Observation-specific basisexperiment.control(control_func,context={'strategy':1})experiment.candidate(cand_func,context={'strategy':2})Context can be retrieved using theget_contextmethod onExperimentandObservationinstances.classExperiment(laboratory.Experiment):defpublish(self,result):self.get_context()result.control.get_context()result.candidates[0].get_context()Ramping upBefore running a candidate code block Laboratory will callExperiment.enabled.
By overriding this method we can control when the candidate code will be executed.For example, if we wanted to enable the experiment for just 10% of calls, we could
do something along these lines:classMyExperiment(laboratory.Experiment):defenabled(self):returnrandom.random()<0.1This is useful for slowly ramping up the experiment, but because we have access to
the experiment context in theenabledmethod, we’re also able to do fancier
things like enabling only for specific users.classMyExperiment(laboratory.Experiment):defenabled(self):ctx=self.get_context()returnctx['user']inuser_segmentControlling comparisonNot all data is created equal. By default laboratory compares using==, but
sometimes you may need to tweak this to suit your needs.  It’s easy enough —
subclassExperimentand implement thecompare(control, observation)method.classMyExperiment(Experiment):defcompare(self,control,observation):returncontrol.value['id']==observation.value['id']Raise on mismatchTheExperimentclass accepts araise_on_mismatchargument which you can set
toTrueif you want Laboratory to raise an exception when the comparison returns
false. This may be useful in testing, for example.Publishing resultsThis data is useless unless we can do something with it. Laboratory makes no
assumptions about how to do this — it’s entirely for you to implement to suit
your needs.  For example, timing data can be sent to graphite, and mismatches
can be placed in a capped collection in redis for debugging later.The publish method is passed aResultinstance, with control and candidate
data is available inResult.controlandResult.candidatesrespectively.classMyExperiment(laboratory.Experiment):defpublish(self,result):statsd.timing('MyExperiment.control',result.control.duration)foroinresult.candidates:statsd.timing('MyExperiment.%s'%o.name,o.duration)CaveatsBecause of the way Laboratory works, there are some situations in which it should
not be used. Namely, any code with side effects, such as disk or database writes,
or other state changes, are unsuitable as they’ll lead to duplicated writes.
You could end up with buggy data or a candidate that affects the execution of
the control.You’ll also take a performance hit by running your new code in addition to the old,
so be mindful of that. You should ramp an experiment up slowly and keep an eye on
your metrics.LinksDocumentationSource codeCI serverPython Package IndexSure-footed refactoringMaintenanceLaboratory is actively maintained by Joe Alcorn (Github,Twitter)