Package: log-rate-limit
log-rate-limit - limit excessive log outputAlogging filterusing Python's standard logging mechanisms to rate-limit logs - i.e. suppress logs when they are being output too fast.Log commands are grouped into separatestreamsthat will each have their own rate limitation applied without affecting the logs in other streams. By default every log message is assigned a unique stream so that only repeated log messages will be suppressed.However, logs can also be assigned streams manually to achieve various outcomes:A log can be assigned to an undefined/Nonestream so that rate-limiting doesn't apply to it.Logs in different parts of the code can be grouped into the same stream so that they share a rate-limit, e.g. when they all trigger due to the same issue and only some are needed to indicate it.The default can also be changed so that rate-limiting is disabled by default and only applies to logs for which astream_idhas been manually set.Quick usageImport the filter:fromlog_rate_limitimportStreamRateLimitFilter,RateLimitUse the filter with yourlogger- with default parameters it will rate-limit all repetitive log messages:logger.addFilter(StreamRateLimitFilter(period_sec=30))All logs onloggerwill now be rate-limited, but this can be disabled per-log by setting thestream_idtoNone:logger.warning("Something went wrong!",extra=RateLimit(stream_id=None))If you have a log message that's continually changing (e.g. contains a timestamp), you can manually define how those messages are defined as similar enough to be rate-limited together by setting a customstream_id:logger.warning("Something went wrong with%sat%s!",device_id,timestamp,extra=RateLimit(stream_id=f"went_wrong_{device_id}"))Note that if the timestamps are added by alogging.Formatterthen they won't affect our rate-limiting filter as these filters run before formatting is applied (seefull workflow).Usage examplesRate-limiting by defaultExample of rate-limiting with default options where each log message is assigned to its own stream:importtimeimportloggingfromlog_rate_limitimportStreamRateLimitFilter,RateLimit# Setup logginglogging.basicConfig()logger=logging.getLogger(__name__)# Add our filterlogger.addFilter(StreamRateLimitFilter(period_sec=1))# Log many warningsfor_inrange(100):logger.warning("Wolf!")foriinrange(100):logger.warning("No really, a wolf!")ifi==98:time.sleep(1)# Prevent rate-limited by setting/overriding the stream to be undefined (None)for_inrange(3):logger.warning("Sheep!",extra=RateLimit(stream_id=None))Which only outputs the following:WARNING:__main__:Wolf!
WARNING:__main__:No really, a wolf!
WARNING:__main__:No really, a wolf!
+ skipped 98 logs due to rate-limiting
WARNING:__main__:Sheep!
WARNING:__main__:Sheep!
WARNING:__main__:Sheep!Note that (unless overridden) logs were only repeated after thesleep()call, and the repeated log also included an extra summary message added afterwards.When we override rate-limiting above, you'll see our filter reads dynamic configs from logging'sextraparameter.Be very careful not to forget theextra=name part of the argument, as then the logging framework will assume you're passing arguments meant for formatting in the logging message and your options will silently be ignored!Rate-limit only when specifiedIf you want most of your logs to be unaffected and you only have some you want to specifically rate-limit, then you can do the following:importloggingfromlog_rate_limitimportStreamRateLimitFilter,RateLimit# Setup logginglogging.basicConfig(level=logging.INFO)logger=logging.getLogger(__name__)# Add our filter, but don't assign unique streams to logs by defaultlogger.addFilter(StreamRateLimitFilter(period_sec=1,default_stream_id=None))# Normal logs are now not rate-limitedforiinrange(3):logger.info(f"Status update:{i}")# Only those we manually assign a stream will be rate-limitedfor_inrange(3):logger.warning("Issue!",extra=RateLimit(stream_id="issue"))Which only outputs the following:INFO:__main__:Status update: 0
INFO:__main__:Status update: 1
INFO:__main__:Status update: 2
WARNING:__main__:Issue!Dynamically override configuration optionsSome options set during creation of the initial filter can be overridden for individual log calls. This is done by adding theextraparameter to any specific log call, e.g.:# Override the rate-limit period for this specific log calllogger.warning("Test1",extra=RateLimit(stream_id="stream1",period_sec=30))# Override the allow_next_n value for a set of logs in the same stream so that this group of logs don't restrict one# another from occuring consecutivelylogger.warning("Test",extra=RateLimit(stream_id="stream2",allow_next_n=2))logger.info("Extra",extra=RateLimit(stream_id="stream2"))logger.debug("Info",extra=RateLimit(stream_id="stream2"))If you want to set custom options for a large group of log calls without repeatedly adding theextraparameter, it's possible to use aLoggerAdapter:importloggingfromlog_rate_limitimportStreamRateLimitFilter,RateLimitlogging.basicConfig(level=logging.INFO)logger=logging.getLogger(__name__)# Add our filterlogger.addFilter(StreamRateLimitFilter(period_sec=1))# Use LoggerAdapter to assign additional "extra" parameters to all calls using this loggerglobal_extra=RateLimit(stream_id="custom_stream",period_sec=20)logger=logging.LoggerAdapter(logger,global_extra)# Log many warningsfor_inrange(100):logger.warning("Wolf!")foriinrange(100):logger.warning("No really, a wolf!")Which merely outputs:WARNING:__main__:Wolf!Since both log calls are in the same stream.Alternatively (to a LoggerAdapter), custom options can also be added by writing your ownlogging.Filter.Dynamic stream IDDynamic stream IDs can be assigned based on any criteria you want, e.g.:logger.warning(f"Error occured on device{device_id}!",extra=RateLimit(stream_id=f"error_on_{device_id}"))InstallationInstall from PyPIWithPython 3installed on your system, you can run:pip install log-rate-limitTo test that installation worked, you can run:python -c "import log_rate_limit"and you can uninstall at any time with:pip uninstall log-rate-limitTo install with poetry:poetry add log-rate-limitInstall from GithubTo install the newest code directly from Github:pip install git+https://github.com/samuller/log-rate-limit