Package: side-effects
Django Side EffectsDjango app for managing external side effects.Python2/3This project is now Python3, Django 1.11+ only on master.The legacy Python2 code is now parked in the python2 branch.BackgroundThis project was created to try and bring some order to the use of external
side-effects within the YunoJuno platform. External side-effects are (as
defined by us) those actions that affect external systems, and that are not
part of the core application integrity. They fall into two main categories
within our application -notificationsandupdates, and are best
illustrated by example:NotificationsSlack messagesSMS (via Twilio)Push notificationsEmailUpdatesBase CRM (sales)Mailchimp CRM (marketing)Elasticsearch (full-text index)There are some shared aspects of all of these side-effects:They can all be processed asynchronously (queued)They can all be replayed (and are idempotent)They can be executed in any orderThey are not time criticalThey do not affect the state of the Django applicationAs we have continued to build out YunoJuno our use of these side-effects
has become ever more complex, and has in some areas left us with functions
that are 80% side-effects:deffoo():# do the thing the function is supposed to doupdate_object(obj)# spend the rest of the function working out which side-effects to fireifsettings.notify_account_handler:send_notification(obj.account_handler)ifobj.has_changed_foo():udpate_crm(obj)This results in a codebase is:Hard to readHard to testHard to document^^ Barely a week goes by without someone asking“what happens when X does Y -
I thought they got email Z?”SolutionThis project aims to address all three of the issues above by:Removing all side-effects code from core functionsSimplifying mocking / disabling of side-effects in testsSimplifying testing of side-effects onlyAutomating documentation of side-effectsIt does this with a combination of function decorators that can
be used to build up a global registry of side-effects.The first decorator,has_side_effects, is used to mark a function as one
that has side effects:# mark this function as one that has side-effects. The label# can be anything, and is used as a dict key for looking up# associated side-effects functions@side_effects.decorators.has_side_effects('update_profile')deffoo(*args,**kwargs):passDecorating view functionsBy default, thehas_side_effectsdecorator will run so long as the inner
function does not raise an exception. View functions, however, are a paticular
case where the function may run, and return a perfectly validHttpResponseobject, but you donotwant the side effects to run, as the response object
has astatus_codeof 404, 500, etc. In this case, you want to inspect the
inner function return value before deciding whether to fire the side effects
functions. In order to support this, thehas_side_effectsdecorator has
a kwargrun_on_exitwhich takes a function that takes a single parameter,
the return value from the inner function, and must returnTrueorFalsewhich determines whether to run the side effects.Thedecoratorsmodule contains the default argument for this kwarg, a
function calledhttp_response_check. This will returnFalseif the
inner function return value is anHttpResponseobject with a status
code in the 4xx-5xx range.The second decorator,is_side_effect_of, is used to bind those functions
that implement the side effects to the origin function:# bind this function to the event 'update_profile'@is_side_effect_of('update_profile')defsend_updates(*args,**kwargs):"""Update CRM system."""pass# bind this function also to 'update_profile'@is_side_effect_of('update_profile')defsend_notifications(*args,**kwargs):"""Notify account managers."""passIn the above example, the updates and notifications have been separated
out from the origin function, which is now easier to understand as it is
only responsible for its own functionality. In this example we have two
side-effects bound to the same origin, however this is an implementation
detail - you could have a single function implementing all the side-effects,
or split them out further into the individual external systems.Passing origin function return value to side-effects handlersBy default, side-effects handling functions must have the same function
signature as the origin function. (Internally the(*args, **kwargs)are
just a straight pass-through to the handler.) However, in certain cases it
is very useful to have access to the origin function return value. A common
case is where the origin function creates a new object. The framework handles
this internally by introspecting the handler function, and looking for**kwargs.This is best illustrated with an example:@has_side_effects("foo")deforigin_func(arg1:int,arg2:int)->int:returnarg1+arg2@is_side_effect_of("foo")defhandle_func1(arg1,arg2):# this func will not receive the return_value, as# no kwargs are specified@is_side_effect_of("foo")defhandle_func1(arg1,arg2,**kwargs):# this func will receive the return_value via **kwargsassert"return_value"inkwargs@is_side_effect_of("foo")defhandle_func1(arg1,arg2,return_value=None):# this func will receive the return_value@is_side_effect_of("foo")defhandle_func1(arg1,arg2,return_value):# this func will receive the return_value, as it is a named arg,# and there is no *args variable@is_side_effect_of("foo")defhandle_func1(*args,return_value):# this func will *NOT* receive the return_valueInternally, the app maintains a registry of side-effects functions bound to
origin functions using the text labels. The docstrings for all the bound functions can be grouped using these labels, and then be printed out using the
management commanddisplay_side_effects:$./manage.pydisplay_side_effectsThiscommandprintsoutthefirstlinefromthedocstringsofallfunctionsregisteredusingthe@is_side_effectdecorator,groupedbylabel.update_profile:-UpdateCRMsystem.-Notifyaccountmanagers.close_account:-Sendconfirmationemailtouser.-Notifycustomerservice.If you have a lot of side-effects wired up, you can filter the list by the label:$./manage.pydisplay_side_effects--labelupdate_profileupdate_profile:-UpdateCRMsystem.-Notifyaccountmanagers.Or by a partial match on the event label:$./manage.pydisplay_side_effects--label-containsprofileupdate_profile:-UpdateCRMsystem.-Notifyaccountmanagers.If you want to enforce docstrings on side-effect functions, then you can use the–check-docstringsoption, which will exit with a non-zero exit code if any
docstrings are missing. This can be used as part of a CI process, failing any
build that does not have all its functions documented. (The exit code is the count
of functions without docstrings).$./manage.pydisplay_side_effects--check-docstringsupdate_profile:***DOCSTRINGMISSING:update_crm***-Notifyaccountmanagers.ERROR:InvocationErrorforcommand'...'(exitedwithcode1)Why not use signals?The above solution probably looks extremely familiar - and it is very closely
related to the built-in Django signals implementation. You could easily
reproduce the output of this project using signals - this project is really
just a formalisation of the way in which a signal-like pattern could be used
to make your code clear and easy to document. The key differences are:Explicit statement that a function has side-effectsA simpler binding mechanism (using text labels)(TODO) Async processing of receiver functionsIt may well be that this project merges back in to the signals pattern in
due course - at the moment we still experimenting.InstallationThe project is available through PyPI asdjango-side-effects:$ pip install django-side-effectsAnd the main package itself is justside_effects:>>>fromside_effectsimportdecoratorsTestsThe project has pretty good test coverage (>90%) and the tests themselves run throughtox.$ pip install tox
$ toxIf you want to run the tests manually, make sure you install the requirements, and Django.$ pip install django==2.0  # your version goes here
$ toxIf you are hacking on the project, please keep coverage up.NB If you implement side-effects in your project, you will most likely want to be able to turn off the side-effects when testing your own code (so that you are not actually sending emails, updating systems), but you also probably want to know that the side-effects events that you are expecting are fired.The following code snippet shows how to use thedisable_side_effectscontext manager, which returns a list of all the side-effects events that are fired. There is a matching function decorator, which will append the events list as an arg to the decorated function, in the same manner thatunittest.mock.patchdoes.fromside_effects.decoratorsimportdisable_side_effects,has_side_effects@has_side_effects('do_foo')deffoo():passdeftest_foo():# to disable side-effects temporarily, use decoratorwithdisable_side_effects()asevents:foo()assertevents==['do_foo']foo()assertevents==['do_foo','do_foo']# events list is added to the test function as an arg@disable_side_effects()deftest_foo_without_side_effects(events):foo()assertevents==['do_foo']In addition to these testing tools there is a universal ‘kill-switch’ which can be set using the env varSIDE_EFFECTS_TEST_MODE=True. This will completely disable all side-effects events. It is a useful tool when you are migrating a project over to the side_effects pattern - as it can highlight where existing tests are relying on side-effects from firing. Use with caution.ContributingStandard GH rules apply: clone the repo to your own account, create a branch, make sure you update the tests, and submit a pull request.StatusWe are using it at YunoJuno, but ‘caveat emptor’. It does what we need it to do right now, and we will extend it as we evolve. If you need or want additional features, get involved :-).