Package: django-studies
Have you ever found yourself in a situation where you’ve made changes to
some functionality, all the tests are passing, manual tests look OK, but
you’re still not convinced that you’ve covered all of the edge-cases?You know your new implementation is faster or more stable, but you still
have the feeling you’re missing something. Wouldn’t it be great if you
could run both implementations side-by-side and compare the results?Maybe you’d want to try it out on a limited set of users for a certain
period of time just to flesh out all the cases you’ve missed. Or you
just want to run a couple of experiments and study the effects without
severely impacting the users in a negative way.Inspired byGitHub’s ScientistandRealGeeks’ lab_tech, this
project bringsJoe Alcorn’s laboratorytoDjango’s world not only to allow
you to run experiments, but to dynamically modify their impact on users.
This would give you the confirmations and the peace of mind you’re
looking forandyour users wouldn’t be inconvenienced by potential
errors.InstallationTo use this library, install it usingpippipinstalldjango-studiesregister the Django app in yoursettings.py:# project/settings.pyINSTALLED_APPS=[# ..."studies",]and run the migrations:pythonmanage.pystudiesFeaturesTo run an experiment, instantiate theExperimentclass, define thecontroland thecandidateandconductthe experiment. For example, a simple Django class-based view with an experiment would look like (the one from thedemoproject):fromstudies.experimentsimportExperimentclassViewWithMatchingResults(View):defget(self,request,*args,**kwargs):withExperiment(name="ViewWithMatchingResults",context={"context_key":"context_value"},percent_enabled=100,)asexperiment:arg="match"kwargs={"extra":"value"}experiment.control(self._get_control,context={"strategy":"control"},args=[arg],kwargs=kwargs,)experiment.candidate(self._get_candidate,context={"strategy":"candidate"},args=[arg],kwargs=kwargs,)data=experiment.conduct()returnJsonResponse(data)def_get_control(self,result,**kwargs):return{"result":result,**kwargs}def_get_candidate(self,result,**kwargs):return{"result":result,**kwargs}Adjust the percentage of users who’ll be impacted by this experiment via the admin:To add support for your own reporting system, whether it’slogging,statsdor something else, override theExperimentclass’publishmethod and make the call (another example from thedemoproject):importloggingfromstudies.experimentsimportExperimentlogger=logging.getLogger()classExperimentWithLogging(Experiment):"""
An override that provides logging support for demonstration
purposes.
"""defpublish(self,result):ifresult.match:logging.info("Experiment%(name)sis a match",{"name":result.experiment.name},)else:control_observation=result.controlcandidate_observation=result.candidates[0]logging.info(json.dumps(control_observation.__dict__,cls=ExceptionalJSONEncoder,# defined in `demo.overrides`))logging.info(json.dumps(candidate_observation.__dict__,cls=ExceptionalJSONEncoder,))logging.error("Experiment%(name)sis not a match",{"name":result.experiment.name},)Override any method fromlaboratory’sExperimentclass, includinghow you make the comparison:fromstudies.experimentsimportExperimentclassMyExperiment(Experiment):defcompare(self,control,candidate):returncontrol.value['id']==candidate.value['id']CaveatsAs always there are certain caveats that you should keep in mind. As
stated inlaboratory’sCaveats, if the
control or the candidate has a side-effect like a write operation to the
database or the cache, you could end up with erroneous data or similar
bugs.At the moment, this library doesn’t provide a safe write mechanism to
mitigate this situation, but it may in the future.ContributingTo contribute to this project, take a look atCONTRIBUTING.rst.