Package: django-proctor
django-proctordjango-proctor allows you to useProctor(an A/B testing framework) from Django by usingProctor Pipet, which exposes Proctor as a simple REST API.Proctor allows you to place your users into randomly-assigned test groups for A/B testing. It can also be used for feature toggles and gradual rollouts of new features.Using Proctor group assignments from Django templates is extremely easy by checking the bucket's name:{%ifproc.buttoncolortst.group=='blue'%}<buttonclass="blue-btn"></button>{%else%}<buttonclass="grey-btn"></button>{%endif%}And you can also use Proctor groups in Python code:ifrequest.proc.newfeaturerollout.group=='active':foo()else:bar()If you need to use Proctor groups in a cron job or some service without a request, you can do an Account test:fromproctor.identifyimportproc_by_accountidaccountid=999999ifproc_by_accountid(accountid).newfeaturerollout.group=="active":foo()else:bar()ConfigurationBefore using django-proctor, you need to set upProctor Pipet. This is the REST API that django-proctor communicates with to obtain test group assignments.You'll also need a test matrix, which defines all Proctor tests and their current allocations.Proctor Webappprovides a way to view and modify the test matrix.RequirementsTo use django-proctor, just install it with pip:$pipinstalldjango-proctorOr add it to yourrequirements.txtfile (preferably with the current version).ViewsThere are a set of private views that are available for testing and debugging. Enabling these views requires some extra configuration.Installed AppsAddproctorto your project's list of installed apps.INSTALLED_APPS+=(...proctor,...)UrlsImport the proctor urls into a private space in your project.urlpatterns=[...url(r'^private/',include('proctor.urls'))...]ShowTestMatrixThe ShowTestMatrix view allows you to see the entire test matrix for your specific PROCTOR_TESTS. This view is simply the json version of the test matrix filtered to your tests.Using the above url pattern example, the test matrix would be available athttp://<your_project_root>/private/proctor/showandhttp://<your_project_root>/private/showTestMatrix. The latter is for backwards compatibility with other projects.Force GroupsThe Force Groups view allows you to see what the current group assignments are for your session and identification and force yourself into a specific group for any test.NOTE: This template does come with a default base template, but it can be overidden. To override the default base template, you must have a base template to extend and the name of that template file should be set in PROCTOR_BASE_TEMPLATE.Using the above url pattern example, the force groups page would be available athttp://<your_project_root>/private/proctor/force.Middlewaredjango-proctor does most of its processing in a middleware. This runs before your views and addsprocto therequestobject, which lets you access test group assignments.You must subclassproctor.middleware.BaseProctorMiddlewareand override several functions to provide django-proctor with information it needs to handle group assignments.classMyProctorMiddleware(proctor.middleware.BaseProctorMiddleware):...get_identifiers()Identifiers are strings that identify users of your site uniquely. These can include tracking cookies and account ids. Proctor uses this information to keep users in the same test groups across requests.This returns a dict of identifier source keys (see Pipet configuration) to their values.If a user lacks a certain identifier, don't include it in the dict. Proctor will skip any tests using that identifier. However, make sure you always return at least one identifier like a tracking cookie.Youmustoverride this method.This method is always run after any previous middleware.defget_identifiers(self,request):ids={'USER':request.COOKIES.get('tracking')}ifrequest.user.is_authenticated:ids['acctid']=request.user.idreturnidsget_context()Context variables are properties about the user that are used in Proctor test rules to selectively enable tests or change the allocations of a test depending on whether an expression is true. This can be used to run a test only on Firefox, or you can run a test at 50% for US users and 10% for everyone else.This returns a dict of context variable source keys (see Pipet configuration) to their values, which are converted by Pipet to their final types before rule expressions are evaluated.If you don't override this method, django-proctor uses no context variables.If the Pipet configuration doesn't have a default value for a context variable, it must be included on every API request. If that is the case, make sure that context variable appears in this return value.This method is always run after any previous middleware.defget_context(self,request):return{"ua":request.META.get('HTTP_USER_AGENT',''),"loggedIn":request.user.is_authenticated,"country":geo.country(request.get_host()),}is_privileged()Returns a bool indicating whether the request is allowed to use theprforceGroupsquery parameter to force themselves into Proctor groups.If you don't override this method, it returns False, which effectively disables force groups.Use something like IP address or admin account to determine whether the user can force themselves into test groups.This methodmayrun without hitting any of the previous middleware. You must assume anything that may have been done in a middleware, like adding auserattribute torequest, may not have happened. Otherwise, you might get exceptions in strange conditions like redirects.defis_privileged(self,request):return(request.get_host().startswith('127.0.0.1')orrequest.get_host().startswith('192.168.')or(hasattr(request,'user')andrequest.user.is_staff))get_http()Returns an instance ofrequests.Session(or equivalent) that will be used when making HTTP requests to the API.If you don't override this method, it returns None, which will cause the api to use therequestsmodule.settings.pyYou must set several things in yoursettings.pyfor django-proctor to work properly:MIDDLEWAREAdd the middleware you created toMIDDLEWARE. Make sure you place it after any middleware it depends on likeAuthenticationMiddleware.TEMPLATE_CONTEXT_PROCESSORSAddproctor.context_processors.proctoTEMPLATE_CONTEXT_PROCESSORS. This makes theprocobject available in all of your Django templates.PROCTOR_API_ROOTSetPROCTOR_API_ROOTto the URL at which Proctor Pipet is running.Includehttp://orhttps://. Do not include a trailing slash.If you want production to use a different test matrix than your staging server and your developer machines, then you may want to use a different Pipet instance depending on environment.PROCTOR_API_ROOT="http://pipet.example.com"PROCTOR_TESTSPROCTOR_TESTSis a tuple or list of the Proctor tests that your Django project intends to use.Add tests to this tuple before implementing them in your templates and code, and remove tests from this tuple after removing their implementations.All tests listed here are guaranteed to exist on theprocobject.The tests listed here will also be instr(proc)(for logging test groups) if they have non-negative group values.PROCTOR_TESTS=('buttoncolortst','newfeaturerollout',)If you're only using one test, make sure you include a comma in the tuple. Otherwise, Python interprets it as just a string.PROCTOR_TESTS=('buttoncolortst',)PROCTOR_BASE_TEMPLATESetPROCTOR_BASE_TEMPLATEto the name of the base html template being used in your project.PROCTOR_BASE_TEMPLATE="base.html"PROCTOR_CACHE_METHODYou can optionally have django-proctor cache group assignments from the Proctor Pipet REST API. Ordinarily, every HTTP request that hits Django will trigger the Proctor middleware to make an HTTP request to Pipet. You can use caching to avoid this extra cost since group assignments typically stay the same.If the cache has no useful value (like when a new user visits your site), then django-proctor falls back to making an HTTP request to Proctor Pipet.IfPROCTOR_CACHE_METHODis missing or None, django-proctor will not do any caching.IfPROCTOR_CACHE_METHODis'cache', django-proctor usesDjango's cache frameworkfor caching group assignments. SeePROCTOR_CACHE_NAME.IfPROCTOR_CACHE_METHODis'session', django-proctor caches group assignments in therequest.sessiondict. This is a decent option if all of your HTTP requests get or setDjango's session objectanyway.Cache Invalidationdjango-proctor's cache invalidation is fairly smart and will not use the cache if some property of the user's request has changed, like the identifiers, context variables, or theprforceGroupsparameter. The cache will also be ignored if you change a setting likePROCTOR_API_ROOTorPROCTOR_TESTS.This means that if a user logs in, or a user changes their useragent, or you add a new test toPROCTOR_TESTS, the cached value will be skipped. You don't have to worry about outdated values.TODO: Explain matrix version detection and caching after that invalid cache issue is resolved. Current cache implementation does not work properly with multiple processes.PROCTOR_CACHE_NAMEThis setting is only meaningful ifPROCTOR_CACHE_METHODis'cache'.PROCTOR_CACHE_NAMEis the name of a cache inCACHESthat django-proctor will use.IfPROCTOR_CACHE_NAMEis missing or None, django-proctor uses thedefaultcache.PROCTOR_LAZYIfPROCTOR_LAZYisTrue, then theprocobject lazily loads its groups. Proctor group assignments are only retrieved from either the cache or the Proctor Pipet REST API on first access of theprocobject.This means that HTTP requests to your Django server that never use theprocobject don't incur the cost of getting group assignments.When measuring performance, remember that this option may move the timing of a cache access and REST API request to an unexpected place (like during template rendering).IfPROCTOR_LAZYis missing orFalse, lazy loading will not be used.UsageThe Proctor middleware adds aprocobject torequest, which allows you to easily use Proctor group assignments from any view.Group assignments can be accessed using the dot operator onproc. Every test listed inPROCTOR_TESTSis guaranteed to exist as an attribute onproc.printrequest.proc.buttoncolortst# -> GroupAssignment(group=u'blue', value=1, payload=u'#2B60DE')printrequest.proc.testnotinsettings# throws AttributeErrorEach group assignment has three attributes:group: the assigned test group name (str)value: the assigned bucket value (int)-1 typically means inactive, and 0 typically means control.payload: the assigned test group payload valueUsed to change test-specific values from Proctor instead of in code.IsNoneif the test has no payload.The payload type can be a str, long, double, or a list of one of those.If Proctor did not give an assignment for a test, then that test is unassigned. In that case: group, value, and payload are allNone.printrequest.proc.buttoncolortst# -> GroupAssignment(group=None, value=None, payload=None)This can happen if an eligibility rule was not met, if there was no matching identifier for the test type, if the test was inPROCTOR_TESTSbut not in the test matrix, or if django-proctor could not connect to Pipet (or got back an HTTP error) and set all assignments to unassigned by default.SwitchingYou can use Proctor group assignments to implement different behavior on your site based on the user's assigned test group.Suppose we have a test called "algorithmtst" in our test matrix with four test groups:'inactive','control','bogo', and'quick':ifrequest.proc.algorithmtst.group=='bogo':bogosort()elifrequest.proc.algorithmtst.group=='quick':quicksort()else:# 'control', 'inactive', and None (all default to our old sorting algorithm)# Because this covers None (unassigned), this will also be used in case of error.oldsort()Usually your'control','inactive', andNonegroups will have the same behavior, which is doing whatever your site did before you added this test or feature. It's convenient to have the else branch cover all of these groups.Ensure that your branches always cover the case that group isNone. This ensures that if your Proctor Pipet instance goes down or starts returning an HTTP error due to some misconfiguration, your site will simply fall back to default behavior.TemplatesProctor can be used from Django templates as well if you properly set upTEMPLATE_CONTEXT_PROCESSORS.Your templates have theprocobject in their context, allowing you to switch behavior based on Proctor groups:{%ifproc.buttoncolortst.group=='blue'%}<buttonclass="blue-btn"></button>{%elifproc.buttoncolortst.group=='green'%}<buttonclass="green-btn"></button>{%else%}<buttonclass="grey-btn"></button>{%endif%}{%ifproc.newfeaturerollout.group=='active'%}<button>Try our new feature!</button>{%endif%}PayloadsPayloads allow you to specify test-specific values in the test matrix instead of in your code. This allows you to try many different variations without touching Django or even redeploying your application.Here is an example for button text on a call to action:<button>{{proc.buttontexttst.payload|default_if_none:"Sign Up"}}</button>Remember that payload can beNonein many cases, including if Proctor Pipet goes down. Include adefault_if_nonefilter to ensure rational default behavior if this happens.Payloads can also be used from your views:algorithm_constants=request.proc.algoconsttst.payloadifalgorithm_constantsisNone:algorithm_constants=[2,3,42]search_ranking(algorithm_constants)Payload ArrayspayloadisNonewhen a test group is unassigned, so any attribute accesses are still an error.In Python code, make sure you check forNonebefore accessing attributes on the payload.In Django template output tags, invalid attribute accesses are interpreted as theTEMPLATE_STRING_IF_INVALIDsetting, which is the blank string by default. You can use thedefaulttemplate tag to cover these instances, but be aware that this will also match on empty arrays and other falsey values:<h1>{{proc.headertexttst.payload.0|default:"Default Title"}}</h1><p>{{proc.headertexttst.payload.1|default:"Default description text."}}</p>SeeHow invalid variables are handledin the Django template documentation for more details.JavaScriptIf you want to use Proctor test group assignments from browser-side JavaScript, you'll have to provide the values you want to use to your JavaScript through Django's templating language.A simple way to do this is to define global JavaScript variables in a script tag in your HTML template with the values your code will use:<scripttype="text/javascript">{%ifproc.buttoncolortst.groupisnotNone%}varbuttoncolortstgroup="{{proc.buttoncolortst.group|escapejs}}";{%else%}varbuttoncolortstgroup=null;{%endif%}{%ifproc.newfeaturerollout.group=='active'%}varusenewfeature=true;{%else%}varusenewfeature=false;{%endif%}varbuttontext="{{proc.buttontexttst.payload|default_if_none:"Sign Up"|escapejs}}";</script>For strings, wrap the template output tag in quotes. Use theescapejsfilter so that special characters like quotes and angle brackets are correctly placed into your JavaScript.Some people place script tags like this in a template block like "js" so that these special values appear in a consistent place alongside other script includes.You can use these global variables in your JavaScript static files to implement your tests:$(function(){if(buttoncolortstgroup==="blue"){$(".buttonone").css("background","#00f");}elseif(buttoncolortstgroup==="green"){$(".buttonone").css("background","#0f0");}else{$(".buttonone").css("background","#888");}if(usenewfeature){$(".buttontwo").show();}$(".buttonthree").text(buttontext);});This is just one way of accessing Proctor test groups from the browser. Use whatever makes the most sense for your project.Another way is templating your JavaScript directly by placing your code in HTML and mixing your Django template tags with JavaScript code. You could even template your .js files instead of serving them statically. However, these two alternatives can be messy and are not best practices.If your application is complex enough, you could even consider making a Django view that returns some test groups or payloads and have your JavaScript make an AJAX request to get them.LoggingTo compare metrics between two different test groups, you can log each request's assigned test groups in addition to any metrics you want to track.django-proctor provides a simple comma-separated representation of all the Proctor test groups that the user is in for logging purposes:printstr(request.proc)# -> "buttoncolortst1,countryalgotst0,newfeaturerollout0"This output only includes non-negative test groups, as -1 typically means inactive groups that should not be logged.Theprocobject also has a method to obtain the list of test groups before joining with a comma:printrequest.proc.get_group_string_list()# -> ['buttoncolortst1', 'countryalgotst0', 'newfeaturerollout0']prforceGroupsTo test the implementation of your test group behavior, privileged users can attach aprforceGroupsquery parameter to their site's URL to force themselves into certain test groups:http://django.example.com/?prforceGroups=buttoncolortst2,countryalgotst0The format is simply the test name followed by the bucket value (not the name), with all test groups separated by commas.The value ofprforceGroupsis set as a session cookie. Your browser will be forced into those groups until your browser is closed. You can also set an empty prforceGroups to clear the cookie:http://django.example.com/?prforceGroups=The tests and bucket values specified inprforceGroupsmust exist in the Proctor test matrix.Using Proctor from Other Python Frameworksdjango-proctor was designed primarily for Django as that is the framework that we (the Indeed Labs team) primarily use.However, these modules would be usable in other Python frameworks with some minor modifications:api,cache*,groups,identify,lazycache unfortunately has some Django mixed in for some of its subclasses. It imports django.core.cache, it uses Django in subclasses, and the abstract Cacher interface takesrequestas a parameter (becauseSessionCacherneeds it, but it can safely be None for all other subclasses).Also, identify.py imports from django settings for similar reasons when looking up account details.If this is a significant problem for you, ask us to split this into two packages: one for Python, and one for Django that has the former as a dependency. Or contribute a solution that splits the packages up.When implementing Proctor in other frameworks, usemiddleware.pyto see how we implemented this for Django. We handle providing context variables and identifiers through subclassing. Other implementations could register functions (through decorators or otherwise) to provide these details. Also, note how the prforceGroups query parameter and cookie is handled.TestingThis project usestoxfor executing tests. To run tests
locally, cd into your project and run$ toxUnderneath the hood,toxis just runningpytestfor test discovery and execution. Test arguments can
be passed through intopytestby adding--after yourtoxcommand. For example, you can
isolate a test file or test method using the following:$ tox -- proctor/tests/test_identify.py
$ tox -- proctor/tests/test_identify.py::TestIdentifyGroups::test_requested_group_resolvedwherepytestuses a double colon as a test class/method/function separatorBy default,pytestcaptures output, which prevents debugging with breakpoints. If you need to
debug the tests, you can run either of the following:$ tox -- --capture=no
$ tox -- -sYou can then add a break point to a test by adding the following to your python code:importpdb;pdb.set_trace()See AlsoProctor PipetProctor DocumentationProctor Github RepoProctor Webappfor editing the test matrix.indeedeng-proctor-usersfor questions and comments.Proctor Blog Poston the Indeed Engineering blog.Proctor Tech TalkCode of ConductThis project is governed by theContributor Covenant v 1.4.1LicenseThis project uses theApache 2.0license.