Package: django-failover
django-failoverprovides automatic failover for services used by your
Django application. For example, if you are running a master and slave
database, you might configure the slave to failover to the master.django-failoverwill monitor the slave and, if it goes down, will modify
your settings to point the slave connection at the master. When the slave
comes back online,django-failoverwill restore the settings and re-point
the slave connection at the slave.You can configuredjango-failoverto perform monitoring, failover, and
recovery via a middleware class. You can also specify exception classes (such
as socket errors) that, when raised and logged using Python’s logging module,
will trigger monitoring.django-failoverprovides a log handler that you
can hook up to your logger(s) for this purpose.django-failoverlogs when a service goes down and when it comes back up.
It also periodically re-logs a service outage if the service remains down.Technical NoteMonitoring, failover, and recovery occur on a per-process basis. That is,django-failoverdoes not store information about which services are down,
which are up, and when they were last monitored in any global location such
as a database or cache. Rather, each process running your Django application
discovers outages for itself. Ping frequency settings (described below) only
apply to a given process; your service may be pinged more often than your
settings stipulate if multiple processes are running your Django application.
This design is intended to keepdjango-failoverlightweight and to
prevent minor network hiccups from triggering failover globally across
large-scale applications.django-failoveris best suited for environments where your webserver is
serving multiple requests per process. A service outage will trigger failover
within the process, the process will continue serving requests while in
failover, and then will resume serving requests normally once the service
comes back up. If you are working in a development environment where your
webserver is reloading your code for each request (using a setting such asMaxRequestsPerChildin Apache), you should configure your webserver not
to reload code while experimenting withdjango-failover. Otherwise,django-failoverwill start each request assuming all services are
operational.Requirementsdjango-failoverhas been tested with Django 1.3 and Python 2.6.InstallationYou can installdjango-failoverwithpip install django-failoveroreasy_install django-failoverThis will addfailoverto your Python path. Add'failover'to yourINSTALLED_APPSif you want to rundjango-failover’s test suite.Service classesTo usedjango-failover, you define and register service classes, each
class corresponding to a service used by your Django application, such as a
database, a cache, a message broker, etc. Each service class should inherit
fromfailover.services.Serviceand override it as necessary. The baseServiceclass provides the following attributes, methods, and default
behavior:Service.ping()This method pings the service to see if it is available. Must raise an
exception if the service is down, otherwise the service is assumed to be
up. Default behavior: attempts to make a socket connection usingService.SOCKET_HOST(default None) andService.SOCKET_PORT(default None). Depending on your service class, you should either define
values forSOCKET_HOSTandSOCKET_PORT, or override thepingmethod entirely to check the service in some other way.Service.failover()This method performs failover if the service is down. Default behavior:
modifiesdjango.conf.settings, setting name=value for each name/value
pair inService.FAILOVER_SETTINGS(which is an empty dictionary by
default).Service.recover()This method performs recovery when a service comes back up. Default
behavior: restores the settings that were overwritten during failover.Service.MONITORING_PING_FREQUENCYDetermines the maximum frequency (in seconds) with whichdjango-failoverwill ping the service during normal monitoring.
This means that, when monitoring is triggered,django-failoverwill not
ping the service unless at leastMONITORING_PING_FREQUENCYseconds
have passed since the last ping. Default:settings.FAILOVER_MONITORING_PING_FREQUENCY(or120if
undefined).Service.OUTAGE_PING_FREQUENCYLikeMONITORING_PING_FREQUENCY, but applies when the service is down
and is being monitored for recovery. Default:settings.FAILOVER_OUTAGE_PING_FREQUENCY(or30if undefined).Service.ERROR_PING_FREQUENCYLikeMONITORING_PING_FREQUENCY, but applies when monitoring is
triggered by an exception. Default:settings.FAILOVER_ERROR_PING_FREQUENCY(or5if undefined).Service.outage_middleware_classYou can assign a middleware class to this attribute and that middleware
will run when the service goes down (assuming you’re usingdjango-failover’s middleware). The middleware may defineprocess_requestand/orprocess_viewmethods. This could be useful
if you wanted to close down some sections of your site during the outage.
Default:None.django-failoverprovides a few ready-to-use service classes.failover.services.cache.MemcachedThis service class will ping your Memcached node(s) and let you know if
any are down. Requirespython-memcachedclient. Doesn’t perform any
failover, as the cache fails silently and thus automatically fails over
to the database. This service class is useful for notifying you of
Memcached outages.failover.services.celery.CeleryCan be used withdjango-celery. Pings your message broker (usingsettings.BROKER_HOSTandsettings.BROKER_PORT). If the message
broker is down, setsCELERY_ALWAYS_EAGERtoTrue, meaning your
celery tasks will execute locally during the outage.failover.services.db.DatabaseFails over from one of your database connections insettings.DATABASESto another. By default, fails over from a
connection with alias"slave"to a connection with alias"default". To failover using different db aliases, subclassDatabaseand defineDB_ALIASandFAILOVER_DB_ALIASon the
subclass.failover.services.db.MySQLA subclass ofDatabasethat sets sensible timeouts when pinging in
order to reduce false positives and avoid hanging. Database connection
timeout settings are backend-specific; if you use a backend other than
MySQL and want to set timeouts, subclassDatabaseand override theset_timeoutmethod.Registering service classesYou can register a service class in several ways. The easiest way is to
defineFAILOVER_SERVICESin your settings file.FAILOVER_SERVICESshould be a tuple of service classes in dot notation:FAILOVER_SERVICES = (
    "failover.services.cache.Memcached",
    "failover.services.celery.Celery",
    "myproject.myapp.MyServiceClass",
)Alternatively, importfailover.monitor.ServiceMonitorand call:ServiceMonitor.register(MyServiceClass)for each service class you want to register. You can also use this as a
decorator:@ServiceMonitor.register
class MyServiceClass(Service):Middleware configurationAddfailover.middleware.FailoverMiddlewareto yourMIDDLEWARE_CLASSES, at or near the beginning:MIDDLEWARE_CLASSES = (
    "failover.middleware.FailoverMiddleware",
    "django.middleware.common.CommonMiddleware",
    ...
)This will trigger monitoring, failover, and recovery before each request
(taking into account your ping frequency settings).Logging configurationTo trigger monitoring when certain exception classes are raised, addfailover.log.FailoverHandlerto one of your loggers. To capturedjango-failover’s own logging, add a handler to the"failover"logger. Service outages are logged asCRITICALand recoveries are logged
asINFO, so you probably want to set your logger and handler levels toINFOso you will know when services recover.LOGGING = {
    'version': 1,
    'handlers': {
        'mail_admins': {
            'level': 'INFO',
            'class': 'django.utils.log.AdminEmailHandler',
        },
        'failover': {
            'level': 'ERROR',
            'class': 'failover.log.FailoverHandler',
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins', 'failover'],
            'level': 'ERROR',
        },
        'failover': {
            'handlers': ['mail_admins'],
            'level': 'INFO',
        },
    },
}By default,FailoverHandlerwill trigger monitoring for the following
exception classes:socket.error
django.db.DatabaseError
urllib2.URLErrorTo add to this list, defineFAILOVER_OUTAGE_EXTRA_EXCEPTION_CLASSESin
settings. To override the list, defineFAILOVER_OUTAGE_EXCEPTION_CLASSES.Monitoring decoratorsYou can also trigger monitoring using several decoratorsdjango-failoverprovides. Decorating a function withfailover.decorators.monitorwill
trigger monitoring prior to the function being called. Wrapping the function
withfailover.decorators.recover_from_outageswill monitor services that
are currently down, but won’t monitor services that are operating normally.recover_from_outagesis useful in conjunction with exception-based
monitoring, as the latter provides no trigger for checking when a service
recovers. If you are only usingdjango-failover’s middleware, you
probably don’t need either of these decorators.Additional SettingsFAILOVER_OUTAGE_LOGGING_FREQUENCYControls how often, after a service outage is discovered, the outage will
be re-logged if the service remains down. Default:3600seconds (1
hour).FAILOVER_MONITORING_PING_FREQUENCYControls the ping frequency during normal monitoring on service classes
that don’t define their own value. Default:120seconds.FAILOVER_OUTAGE_PING_FREQUENCYControls the ping frequency during outages on service classes
that don’t define their own value. Default:30seconds.FAILOVER_ERROR_PING_FREQUENCYControls the ping frequency for exception-triggered monitoring on
service classes that don’t define their own value. Default:5seconds.