Package: pybreaker
PyBreaker is a Python implementation of the Circuit Breaker pattern, described
in Michael T. Nygard’s bookRelease It!.In Nygard’s words,“circuit breakers exists to allow one subsystem to fail
without destroying the entire system. This is done by wrapping dangerous
operations (typically integration points) with a component that can circumvent
calls when the system is not healthy”.FeaturesConfigurable list of excluded exceptions (e.g. business exceptions)Configurable failure threshold and reset timeoutSupport for several event listeners per circuit breakerCan guard generator functionsFunctions and properties for easy monitoring and managementThread-safeOptional redis backingOptional support for asynchronous Tornado callsRequirementsPython3.8+InstallationRun the following command line to download the latest stable version of
PyBreaker fromPyPI:$ pip install pybreakerIf you are aGituser, you might want to install the current development
version in editable mode:$ git clone git://github.com/danielfm/pybreaker.git
$ cd pybreaker
$ # run tests (on windows omit ./)
$ ./pw test
$ pip install -e .UsageThe first step is to create an instance ofCircuitBreakerfor each
integration point you want to protect against:importpybreaker# Used in database integration pointsdb_breaker=pybreaker.CircuitBreaker(fail_max=5,reset_timeout=60)CircuitBreakerinstances should live globally inside the application scope,
e.g., live across requests.NoteIntegration points to external services (i.e. databases, queues, etc) are
more likely to fail, so make sure to always use timeouts when accessing such
services if there’s support at the API level.If you’d like to use the Redis backing, initialize theCircuitBreakerwith
aCircuitRedisStorage:importpybreakerimportredisredis=redis.StrictRedis()db_breaker=pybreaker.CircuitBreaker(fail_max=5,reset_timeout=60,state_storage=pybreaker.CircuitRedisStorage(pybreaker.STATE_CLOSED,redis))Do notinitialize the Redis connection with thedecode_responsesset toTrue, this will force returning ASCII objects from redis and in Python3+ will
fail with:AttributeError: ‘str’ object has no attribute ‘decode’NoteYou may want to reuse a connection already created in your application, if you’re
usingdjango_redisfor example:importpybreakerfromdjango_redisimportget_redis_connectiondb_breaker=pybreaker.CircuitBreaker(fail_max=5,reset_timeout=60,state_storage=pybreaker.CircuitRedisStorage(pybreaker.STATE_CLOSED,get_redis_connection('default')))NoteIf you require multiple, independent CircuitBreakers and wish to store their states in Redis, it is essential to assign aunique namespacefor each
CircuitBreaker instance. This can be achieved by specifying a distinct namespace parameter in the CircuitRedisStorage constructor. for example:importpybreakerfromdjango_redisimportget_redis_connectiondb_breaker=pybreaker.CircuitBreaker(fail_max=5,reset_timeout=60,state_storage=pybreaker.CircuitRedisStorage(pybreaker.STATE_CLOSED,get_redis_connection('default'),namespace='unique_namespace'))Event ListeningThere’s no need to subclassCircuitBreakerif you just want to take action
when certain events occur. In that case, it’s better to subclassCircuitBreakerListenerinstead:classDBListener(pybreaker.CircuitBreakerListener):"Listener used by circuit breakers that execute database operations."defbefore_call(self,cb,func,*args,**kwargs):"Called before the circuit breaker `cb` calls `func`."passdefstate_change(self,cb,old_state,new_state):"Called when the circuit breaker `cb` state changes."passdeffailure(self,cb,exc):"Called when a function invocation raises a system error."passdefsuccess(self,cb):"Called when a function invocation succeeds."passclassLogListener(pybreaker.CircuitBreakerListener):"Listener used to log circuit breaker events."defstate_change(self,cb,old_state,new_state):msg="State Change: CB:{0}, New State:{1}".format(cb.name,new_state)logging.info(msg)To add listeners to a circuit breaker:# At creation time...db_breaker=pybreaker.CircuitBreaker(listeners=[DBListener(),LogListener()])# ...or laterdb_breaker.add_listeners(OneListener(),AnotherListener())What Does a Circuit Breaker Do?Let’s say you want to use a circuit breaker on a function that updates a row
in thecustomerdatabase table:@db_breakerdefupdate_customer(cust):# Do stuff here...pass# Will trigger the circuit breakerupdated_customer=update_customer(my_customer)Or if you don’t want to use the decorator syntax:defupdate_customer(cust):# Do stuff here...pass# Will trigger the circuit breakerupdated_customer=db_breaker.call(update_customer,my_customer)Or use it as a context manager and awithstatement:# Will trigger the circuit breakerwithdb_breaker.calling():# Do stuff here...passAccording to the default parameters, the circuit breakerdb_breakerwill
automatically open the circuit after 5 consecutive failures inupdate_customer.When the circuit is open, all calls toupdate_customerwill fail immediately
(raisingCircuitBreakerError) without any attempt to execute the real
operation. If you want the original error to be thrown when the circuit trips,
set thethrow_new_error_on_tripoption toFalse:pybreaker.CircuitBreaker(...,throw_new_error_on_trip=False)After 60 seconds, the circuit breaker will allow the next call toupdate_customerpass through. If that call succeeds, the circuit is closed;
if it fails, however, the circuit is opened again until another timeout elapses.Optional Tornado SupportA circuit breaker can (optionally) be used to call asynchronous Tornado functions:fromtornadoimportgen@db_breaker(__pybreaker_call_async=True)@gen.coroutinedefasync_update(cust):# Do async stuff here...passOr if you don’t want to use the decorator syntax:@gen.coroutinedefasync_update(cust):# Do async stuff here...passupdated_customer=db_breaker.call_async(async_update,my_customer)Excluding ExceptionsBy default, a failed call is any call that raises an exception. However, it’s
common to raise exceptions to also indicate business exceptions, and those
exceptions should be ignored by the circuit breaker as they don’t indicate
system errors:# At creation time...db_breaker=CircuitBreaker(exclude=[CustomerValidationError])# ...or laterdb_breaker.add_excluded_exception(CustomerValidationError)In that case, when any function guarded by that circuit breaker raisesCustomerValidationError(or any exception derived fromCustomerValidationError), that call won’t be considered a system failure.So as to cover cases where the exception class alone is not enough to determine
whether it represents a system error, you may also pass a callable rather than
a type:db_breaker=CircuitBreaker(exclude=[lambdae:type(e)==HTTPErrorande.status_code<500])You may mix types and filter callables freely.Monitoring and ManagementA circuit breaker provides properties and functions you can use to monitor and
change its current state:# Get the current number of consecutive failuresprint(db_breaker.fail_counter)# Get/set the maximum number of consecutive failuresprint(db_breaker.fail_max)db_breaker.fail_max=10# Get/set the current reset timeout period (in seconds)printdb_breaker.reset_timeoutdb_breaker.reset_timeout=60# Get the current state, i.e., 'open', 'half-open', 'closed'print(db_breaker.current_state)# Closes the circuitdb_breaker.close()# Half-opens the circuitdb_breaker.half_open()# Opens the circuitdb_breaker.open()These properties and functions might and should be exposed to the operations
staff somehow as they help them to detect problems in the system.ContributingRun tests:$ ./pw testCode formatting (black and isort) and linting (mypy)$ ./pw format
$ ./pw lintAbove commands will automatically install the necessary tools inside.pyprojectxand also install pre-commit hooks.List available commands:$ ./pw -i