Package: python-logstash-async-dwn
Python logging handler for asynchronous event processing and transport to Logstash.http://logstash.net/AboutThis Python logging handler is a fork ofhttps://github.com/eht16/python-logstash-asyncIt has the following features:Asynchronous transport of log eventsTransport of events via TCP and UDPTCP transport optionally SSL-encryptedSpecial formatter ready to be used in Django projectsCompared tohttps://github.com/eht16/python-logstash-async, this fork does not try to preserve log statements if
no connection can be made to logstash.Asynchronous processingUnlike the originalpython-logstash, this handler will try to
handle log events as fast as possible so that the sending program
code can continue with its primary job.
In other words, for web applications or web services it is important
to not slow down request times due to logging delays, e.g. waiting
for network timeouts to the Logstash server or similar.So this handler will accept log events and pass them for further
processing to a separate worker thread which will try to send
the events to the configured Logstash server asynchronously.The sending intervals and timeouts can be configured in thelogstash_async.constantsmodule by the corresponding
module-level constants, see below for details.InstallationUsing pip:pip install git+https://github.com/koendeschacht/python-logstash-async.gitUsageAsynchronousLogstashHandleris a custom logging handler which
sends Logstash messages using UDP and TCP. For example:importloggingimportsysfromlogstash_async.handlerimportAsynchronousLogstashHandlerhost='localhost'port=5959test_logger=logging.getLogger('python-logstash-logger')test_logger.setLevel(logging.INFO)test_logger.addHandler(AsynchronousLogstashHandler(host,port)))# test_logger.addHandler(AsynchronousLogstashHandler(host, port))test_logger.error('python-logstash-async: test logstash error message.')test_logger.info('python-logstash-async: test logstash info message.')test_logger.warning('python-logstash-async: test logstash warning message.')# add extra field to logstash messageextra={'test_string':'python version: '+repr(sys.version_info),'test_boolean':True,'test_dict':{'a':1,'b':'c'},'test_float':1.23,'test_integer':123,'test_list':[1,2,'3'],}test_logger.info('python-logstash: test extra fields',extra=extra)When using theextrafield make sure you don’t use reserved names.
FromPython documentation:"The keys in the dictionary passed in extra should not clash
with the keys used by the logging system.
(See the `Formatter <https://docs.python.org/2/library/logging.html#logging.Formatter>`_ documentation
for more information on which keys are used by the logging system.)"You can also specify an additional extra dictionary in the logging configuration with static
values like the application name, environment, etc. These values will be merged with any
extra dictionary items passed in the logging call into the configured extra prefix.Usage with DjangoModify yoursettings.pyto integratepython-logstash-asyncwith Django’s logging:LOGGING={...'formatters':{...'logstash':{'()':'logstash_async.formatter.DjangoLogstashFormatter','message_type':'python-logstash','fqdn':False,# Fully qualified domain name. Default value: false.'extra_prefix':'dev',#'extra':{'application':PROJECT_APP,'project_path':PROJECT_APP_PATH,'environment':'production'}},},'handlers':{...'logstash':{'level':'DEBUG','class':'logstash_async.handler.AsynchronousLogstashHandler','transport':'logstash_async.transport.TcpTransport','host':'logstash.host.tld','port':5959,'ssl_enable':True,'ssl_verify':True,'ca_certs':'etc/ssl/certs/logstash_ca.crt','certfile':'/etc/ssl/certs/logstash.crt','keyfile':'/etc/ssl/private/logstash.key',},},'loggers':{'django.request':{'handlers':['logstash'],'level':'DEBUG','propagate':True,},},...}This would result in a Logstash event like the following
(note: to some extend dependent of your Logstash configuration):{"@timestamp":"2016-10-23T15:11:16.853Z","@version":"1","extra":{"application":"django_example","django_version":"1.10.2","environment":"production","func_name":"get_response","interpreter":"/home/enrico/example/venv/bin/python","interpreter_version":"2.7.12","line":152,"logger_name":"django.request","path":"/home/enrico/example/venv/lib/python2.7/site-packages/django/core/handlers/base.py","process_name":"MainProcess","project_path":"/home/enrico/example/app","req_host":"localhost","req_method":"GET","req_referer":"","req_remote_address":"127.0.0.1","req_uri":"http://localhost/hosts/nonexistent/","req_user":"enrico","req_useragent":"Mozilla/5.0 (Windows NT 6.1; WOW64; rv:40.0) Gecko/20100101 Firefox/40.1","request":"<WSGIRequest: GET '/hosts/nonexistent/'>","status_code":404,"thread_name":"Thread-1"},"host":"my.host.tld","level":"WARNING","logsource":"endor.l8failed.net","message":"Not Found: /hosts/nonexistent/","pid":23605,"port":56170,"program":"manage.py","type":"python-logstash"}ConfigurationOptions for configuring the log handlerhostThe host of the Logstash server (no default)portThe port of the Logstash server (default 5959)transportCallable or path to a compatible transport class
(default: ‘logstash_async.transport.TcpTransport’).You can specify your own transport class, e.g. to implement
a transport via Redis or the Beats protocol.
If you support pass a string, it should be a path to a
class which can be imported.
If you pass anything else, it should be an object of a class
with a similar interface aslogstash_async.transport.TcpTransport.
Especially it should provide aclose()and asend()method.ssl_enableShould SSL be enabled for the connection? (default: False)
Only used forlogstash_async.transport.TcpTransport.ssl_verifyShould the server’s SSL certificate be verified? (default: True)
Only used forlogstash_async.transport.TcpTransport.keyfileThe path to client side SSL key file (default: None)
Only used forlogstash_async.transport.TcpTransport.certfileThe path to client side SSL certificate file (default: None)
Only used forlogstash_async.transport.TcpTransport.ca_certsThe path to the file containing recognized CA certificates
(default: None)
Only used forlogstash_async.transport.TcpTransport.enableFlag to enable log processing (default is True, disabling
might be handy for local testing, etc.)Options for configuring the log formatterThe following settings are only valid for the provided formatterslogstash_async.handler.LogstashFormatterandlogstash_async.handler.DjangoLogstashFormatter.You can use any other formatter by configuring Python’s logging
system accordingly. Any other formatter’sformat()method just
should return valid JSON suitable to be sent to Logstash
(seeExample Logstash Configurationbelow).Options:message_typeThetypefield in the message sent to Logstash
(default: ‘python-logstash’)tagsAdditional tags to include in the Logstash message (default: None)fqdnUse the system’s FQDN (fully qualified domain name) in thehostfield of the message sent to Logstash.socket.getfqdn()is used to retrieve the FQDN, otherwisesocket.gethostname()is used for the default hostname.
(default: False)extra_prefixName of the field in the resulting message sent to Logstash where
all additional fields are grouped into. Consider it as some sort
of namespace for all non-standard fields in the log event.
This field will take any items passed in as extra fields via
theextraconfiguration option (see below) as well as any extra
items passed in the logging call.To disable grouping of the extra items and have them on the top
level of the log event message, simply set this option toNoneor the empty string.
(default: ‘extra’)extraDictionary with static items to be included in the message sent
to Logstash. This dictionary will be merged with any other extra
items passed in the logging call.
(default: None)Options for the asynchronous processing (in module logstash_async.constants)SOCKET_TIMEOUTTimeout in seconds for TCP connections (default: 5.0)QUEUE_CHECK_INTERVALInterval in seconds to check the internal queue for new messages (default: 0.2)TIMEOUT_SENDING_MESSAGES_AT_TERMINATIONWhen the program terminates, any remaining log messages are still send to logstash. This constant defines
for how many seconds the handler can still send messages before terminating (default: 2.0)Example Logstash ConfigurationExamplelogstash.conffor unencrypted TCP transport:input {
    tcp {
        host => "127.0.0.1"
        port => 5959
        mode => server
        codec => json
    }
}Examplelogstash.conffor SSL-encrypted TCP transport:input {
    tcp {
        host => "127.0.0.1"
        port => 5958
        mode => server
        codec => json

        ssl_enable => true
        ssl_verify => true
        ssl_extra_chain_certs => ["/etc/ssl/certs/logstash_ca.crt"]
        ssl_cert => "/etc/ssl/certs/logstash.crt"
        ssl_key => "/etc/ssl/private/logstash.key"
    }
}