Package: aiofluent
WARNING: This is a fork of thehttps://github.com/fluent/fluent-logger-pythonproject to work with asyncio.Many web/mobile applications generate huge amount of event logs (c,f.
login, logout, purchase, follow, etc). To analyze these event logs could
be really valuable for improving the service. However, the challenge is
collecting these logs easily and reliably.Fluentdsolves that problem by
having: easy installation, small footprint, plugins, reliable buffering,
log forwarding, etc.aiofluentis a Python library, to record the events from
Python application.RequirementsPython 3.5 or greatermsgpack-pythonInstallationThis library is distributed as ‘aiofluent’ python package. Please
execute the following command to install it.$pipinstallaiofluentConfigurationFluentd daemon must be launched with a tcp source configuration:<source>
  type forward
  port 24224
</source>To quickly test your setup, add a matcher that logs to the stdout:<match app.**>
  type stdout
</match>UsageFluentSender Interfacesender.FluentSenderis a structured event logger for Fluentd.By default, the logger assumes fluentd daemon is launched locally. You
can also specify remote logger by passing the options.fromaiofluentimportsender# for local fluentlogger=sender.FluentSender('app')# for remote fluentlogger=sender.FluentSender('app',host='host',port=24224)For sending event, callemitmethod with your event. Following example will send the event to
fluentd, with tag ‘app.follow’ and the attributes ‘from’ and ‘to’.# Use current timelogger.emit('follow',{'from':'userA','to':'userB'})# Specify optional timecur_time=int(time.time())logger.emit_with_time('follow',cur_time,{'from':'userA','to':'userB'})You can detect an error via return value ofemit. If an error happens inemit,emitreturnsFalseand get an error object usinglast_errormethod.ifnotlogger.emit('follow',{'from':'userA','to':'userB'}):print(logger.last_error)logger.clear_last_error()# clear stored error after handled errorsIf you want to shutdown the client, callclose()method.logger.close()Event-Based InterfaceThis API is a wrapper forsender.FluentSender.First, you need to callsender.setup()to create globalsender.FluentSenderlogger
instance. This call needs to be called only once, at the beginning of
the application for example.Initialization code of Event-Based API is below:fromaiofluentimportsender# for local fluentsender.setup('app')# for remote fluentsender.setup('app',host='host',port=24224)Then, please create the events like this. This will send the event to
fluentd, with tag ‘app.follow’ and the attributes ‘from’ and ‘to’.fromaiofluentimportevent# send event to fluentd, with 'app.follow' tagevent.Event('follow',{'from':'userA','to':'userB'})event.Eventhas one limitation which can’t return success/failure result.Other methods for Event-Based Interface.sender.get_global_sender# get instance of global sendersender.close# Call FluentSender#closeHandler for buffer overflowYou can inject your own custom proc to handle buffer overflow in the event of connection failure. This will mitigate the loss of data instead of simply throwing data away.importmsgpackfromioimportBytesIOdefhandler(pendings):unpacker=msgpack.Unpacker(BytesIO(pendings))forunpackedinunpacker:print(unpacked)logger=sender.FluentSender('app',host='host',port=24224,buffer_overflow_handler=handler)You should handle any exception in handler. aiofluent ignores exceptions frombuffer_overflow_handler.This handler is also called when pending events exist duringclose().Python logging.Handler interfaceThis client-library also hasFluentHandlerclass for Python logging
module.importloggingfromaiofluentimporthandlercustom_format={'host':'%(hostname)s','where':'%(module)s.%(funcName)s','type':'%(levelname)s','stack_trace':'%(exc_text)s'}logging.basicConfig(level=logging.INFO)l=logging.getLogger('fluent.test')h=handler.FluentHandler('app.follow',host='host',port=24224)formatter=handler.FluentRecordFormatter(custom_format)h.setFormatter(formatter)l.addHandler(h)l.info({'from':'userA','to':'userB'})l.info('{"from": "userC", "to": "userD"}')l.info("This log entry will be logged with the additional key: 'message'.")You can also customize formatter via logging.config.dictConfigimportlogging.configimportyamlwithopen('logging.yaml')asfd:conf=yaml.load(fd)logging.config.dictConfig(conf['logging'])A sample configurationlogging.yamlwould be:logging:version:1formatters:brief:format:'%(message)s'default:format:'%(asctime)s%(levelname)-8s%(name)-15s%(message)s'datefmt:'%Y-%m-%d%H:%M:%S'fluent_fmt:'()':fluent.handler.FluentRecordFormatterformat:level:'%(levelname)s'hostname:'%(hostname)s'where:'%(module)s.%(funcName)s'handlers:console:class:logging.StreamHandlerlevel:DEBUGformatter:defaultstream:ext://sys.stdoutfluent:class:fluent.handler.FluentHandlerhost:localhostport:24224tag:test.loggingformatter:fluent_fmtlevel:DEBUGnone:class:logging.NullHandlerloggers:amqp:handlers:[none]propagate:Falseconf:handlers:[none]propagate:False'':# root loggerhandlers:[console,fluent]level:DEBUGpropagate:FalseLicenseApache License, Version 2.01.2.9 (2020-10-22)Only log errors every 30 seconds1.2.8 (2020-05-15)Handle TypeError formatting log data1.2.7 (2020-03-09)Fix repo location1.2.6 (2020-01-06)Improve error logging
[vangheem]1.2.5 (2019-12-19)Handle event loop closed error
[vangheem]1.2.4 (2019-12-19)Increase max queue size1.2.3 (2019-04-01)Fix release1.2.2 (2019-04-01)nanosecond_precision by default
[davidonna]1.2.1 (2018-10-31)Add support for nanosecond precision timestamps
[davidonna]1.2.0 (2018-06-14)Maintain one AsyncIO queue for all logs
[vangheem]1.1.4 (2018-05-29)Handle RuntimeError on canceling tasks/cleanup
[vangheem]1.1.3 (2018-02-15)Lock calling the close method of sender
[vangheem]Increase default timeout
[vangheem]1.1.2 (2018-02-07)lock the whole method
[vangheem]1.1.1 (2018-02-07)Use lock on getting connection object
[vangheem]1.1.0 (2018-01-25)Move to using asyncio connection infrastructure instead of sockets
[vangheem]1.0.8 (2018-01-04)Always close out buffer data
[vangheem]1.0.7 (2018-01-04)Handle errors processing log queue
[vangheem]1.0.6 (2017-11-14)Prevent log queue from getting too large
[vangheem]1.0.5 (2017-10-17)Fix release to include CHANGELOG.rst file
[vangheem]1.0.4 (2017-10-10)Fix pushing initial record1.0.3 (2017-10-10)Handle Runtime error when logging done before event loop started
[vangheem]1.0.2 (2017-10-09)Fix to make normal logging call async
[vangheem]1.0.1 (2017-07-03)initial release