Package: event-count-logger
Event Count LoggerPython library (and a daemon program) allowing to count the number of various events per time interval(s) in a distributed system using shared counters in Redis.Simple example:ecl=EventCountLogger(cfg_groups)event_logger=ecl.get_group('event_group_1')...event_logger.log('event1')# increment shared counter of 'event1' by 1# Read counts in the last 5-min interval$ecl_readerecl_config.yml-gevent_group_1-i5m
event1:10
event2:2Main featuresCount any 'events' across multiple processes or machinesMultiple counters, grouped intoevent groupsPre-defined or dynamic list of event IDs in each groupMultiple time intervals (count number of events in 5 minute and 1 hour intervals at the same time, for example)Local cache for high-rate events, synchronized with the shared counters at a lower rateInstallationpipinstallevent_count_loggerPython >=3.6 is required.ArchitectureSystem
 components                      Redis                    ecl_reader
+-----------+                +------------+           +---------------+
|   log()   | -------------> |   shared   | --------> |  get_counts() |
+-----------+      +-------> |  counters  |           +---------------+
                   |         +------------+
+-----------+      |               ^               ecl_master
|   log()   | -----+               |           +----------------+
+-----------+                      +---------> |  periodically  |
                                               | reset counters |
                                               +----------------+Shared counters are stored inRedis(in-memory key-value store).
They can be safely incremented by calls toEventGroup.log(event_id)from multiple processes.A single instance ofecl_masterscript must be continuously running - it resets the counters in defined time intervals (and makes values of the last complete interval available for reading).Counters can be read byecl_readerscript, or by usingEventCountLogger.get_*()methods from any other program.How it worksMultiple types of event (identified by any string,event ID) can be logged. They are split intogroups, each group is configured and handled separately.For each group, a list of event ID is defined in a configuration file (or they can be defined dynamically, see later),
as well as the list of time intervals, in which the counters should be 'rotated'.In Redis, there are two keys with integer value for each event ID and interval length - one with the number of
logged events in the current (incomplete) interval, one with the final value of the last finished interval.
The keys are named as<group>:<interval>:<cur_or_last>:<event_id>, e.g.requests:5m:cur:req1.Logging an event means to increment thecurcounters with given<group>and<event_id>and all<intervals>.At the end of an interval, theecl_masterscript renames allcurkeys (with matching<interval>) to correspondinglastones and resets thecurones to zero.The expected usage is thatecl_readeris called periodically to fetch values of thelastcounters and
passes them to some monitoring system (e.g. Munin) or shows them in real-time in some GUI, for example.There is also a<group>:<interval>:cur:@tskey for each group and interval containing the start time of the current interval (as UNIX timestamp).Local countersNormally, when an event is logged, the counter in Redis in incremented immediately.
While Redis is quite fast, it is still a network communication which may be a problem with  too frequent events.
So, there is a possibility cache counts locally in a specific group.
In such case, only a local counter (i.e. a normal Python variable) is incremented when event is logged.
These local counters are synchronized with Redis only after some time elapses (sync_intervalconfig param)
or when one of the counters in the group counter reaches a defined value (sync_limit).Usage exampleLog event in a program:fromevent_count_loggerimportEventCountLoggercfg_groups={'requests':{'events':['req1','req2','req3','error'],'intervals':['5s','5m'],}}ecl=EventCountLogger(cfg_groups)req_logger=ecl.get_group('requests')...defhandle_request(...,req_type,...):try:ifreq_type=='req1':req_logger.log('req1')# increment counter of 'req1' event by 1process_req1()...exceptExceptionase:req_logger.log('error')# increment 'error' counter by 1handle_error(e)Read the number of events in the last finished 5-minute interval:$ecl_readerecl_config.yml-grequests-i5m
req1:54
req2:11
req3:42
error:3ConfigurationA YAML file is used to define event groups and specify Redis connection parameters.
Seeexample_config.ymlfor an example with comments.Event group specification keys:events(mandatory) - List of predefined event IDs (strings). Only those events can be logged in the group, unlessauto_declare_events=true.intervals(mandatory) - List of interval lengths in which to 'rotate' the counters. Each interval length is a string in format<number><s/m/h>(seconds, minutes, hours), e.g.5mor1h.auto_declare_events- Enable automatic declaration of events not listed inevents, i.e. a new counter is automatically created when a new event ID is logged. Such events are ephemeral, the counters are removed at the end of each interval. (true/false, default:false)sync_interval- Synchronize local counters with Redis after this number of seconds (int or float).sync_limit- Synchronize local counters with Redis when a counter reach this number (int).If none ofsync_intervalandsync_limitis specified, local counters are not used (i.e. shared counters in Redis are manipulated directly).MuninExampleMuninplugins are included for easy logging and visualization
(they must be modified for a specific case, a separate plugin should be used for each event group).Example output:NotesWritten mainly forNERD(and another internal project), but can be used in any Python system distributed to multiple processes.If you want to use it and need more info, drop me a message (bartos <at> cesnet.cz).
If there are more users, I'll write a more complete documentation.