Package: gocept.logging
gocept.loggingContentsgocept.loggingAdvanced usageExample configurationAdditional featuresKnown bugsDeveloping gocept.loggingChange log for gocept.logging1.0 (2023-07-18)0.8.1 (2017-01-09)0.8 (2016-03-17)0.7 (2015-09-29)0.6 (2015-09-17)0.5 (2014-02-07)0.4 (2013-09-24)0.3 (2013-09-04)0.2 (2013-08-24)0.1 (2013-08-16)This package provides infrastructure for semi-structured log messages.This means appending easily parseable information after the free-text log
message to facilitate analysis of the logs later on. Theloggingmodule of
the Python standard library already has support for this, via theextraparameter. gocept.logging provides aFormatterthat extracts theseextravalues, formats them askey=valuepairs and appends them to the
message:>>> import gocept.logging
>>> import logging
>>> import sys

>>> handler = logging.StreamHandler(sys.stdout)
>>> handler.setFormatter(gocept.logging.SyslogKeyValueFormatter())
>>> log = logging.getLogger('example')
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
Aug 24 12:10:08 localhost example: Hello, world! foo=barThis package is tested to be compatible with Python version 2.7 and 3.3.Advanced usageIf you haveextravalues that you always want to pass to your log messages
(e.g things like the current user, session id, …) you can wrap your logger
with anLoggerAdapterthat prefills these values. gocept.logging provides
one that allows both stacking adapters and overriding the prefilled values:>>> from gocept.logging.adapter import StaticDefaults
>>> import logging

>>> log = logging.getLogger('advanced')
>>> log = StaticDefaults(log, {'foo': 'bar', 'qux': 'baz'})
>>> log = StaticDefaults(log, {'blam': 'splat'})
>>> log.warning('Hello, world!', extra={'foo': 'override'})
    # yields {'foo': 'override', 'qux': 'baz', 'blam': 'splat'}Testing supportTo help inspecting theextravalues, gocept.logging comes with a
specialized handler for testing:>>> import gocept.logging
>>> import logging

>>> log = logging.getLogger('testing')
>>> handler = gocept.logging.TestingHandler()
>>> log.addHandler(handler)
>>> log.warning('Hello, world!', extra={'foo': 'bar'})
>>> handler.messages[0].extra['foo']
'bar'The TestingHandler records each log message as a namedtuple of typegocept.logging.testing.LogMessageso you an easily access all parts of the
message.Example configurationCreating semi-structured log messages is the first half of the issue, while
analysing them is the second half. We uselogstashfor that purpose.The recommended setup is:application -> syslogd on localhost -> logstash on central host (via UDP syslog input)For development you might want to leave out the middle man and configure the
application to send log messags via syslog protocol directly to logstash.Setup with ini fileIf you have a paste.ini for your application, you might use something like
this:[loggers]
keys = root

[handlers]
keys = console, syslog

[formatters]
keys = generic, keyvalue

[logger_root]
level = INFO
handlers = console, syslog

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s %(name)s: %(message)s

[handler_syslog]
class = logging.handlers.SysLogHandler
args = ()
formatter = keyvalue

[formatter_keyvalue]
class = gocept.logging.SyslogKeyValueFormatterSetup with ZConfigIf you have a Zope application, you might use something like this:<eventlog>
  <logfile>
    formatter zope.exceptions.log.Formatter
    format %(asctime)s %(levelname)-5.5s %(name)s: %(message)s
    path STDOUT
  </logfile>
  <syslog>
    formatter gocept.logging.SyslogKeyValueFormatter
  </syslog>
</eventlog>syslogd configurationrsyslog:$EscapeControlCharactersOnReceive off
$MaxMessageSize 64k
user.* @localhost:5140The first two lines are to support tracebacks, which are multiline and might
take up some space. The last line tells rsyslogd to forward all messages of theuserfacility (which is what stdliblogginguses by default) via syslog
UDP protocol to localhost port 5140 (where logstash might be listening).logstash configurationinput {
        tcp {
                host => "localhost"
                port => 5140
                type => syslog
        }
        udp {
                host => "localhost"
                port => 5140
                type => syslog
        }
}

filter {
        grok {
                type => "syslog"
                pattern => [ "(?m)<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ]
        }
        syslog_pri {
                type => "syslog"
        }
        date {
                type => "syslog"
                match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
        mutate {
                type => "syslog"
                exclude_tags => "_grokparsefailure"
                replace => [ "@source_host", "%{syslog_hostname}" ]
                replace => [ "@message", "%{syslog_program}: %{syslog_message}" ]
        }
        mutate {
                type => "syslog"
                remove => [ "syslog_hostname", "syslog_timestamp" ]
        }
        kv {
                exclude_tags => "_grokparsefailure"
                type => "syslog"
        }
}

output {
        elasticsearch { embedded => true }
}Additional featuresArgumentParserThe providedgocept.logging.ArgumentParserprovides you with the ability to
set alogginglevel in you runscripts.:from gocept.logging import ArgumentParser
parser = ArgumentParser()
# Optionally set a custom log format, defaults to ``logging.BASIC_FORMAT``
parser.LOG_FORMAT = 'LOG:%(message)s'
# add your arguments with parser.add_argument() here
options = parser.parse_args()Useyour_run_script--helpto see a help message about the arguments you
can pass to set logging level.Known bugsIf you log messages as unicode, e.g.log.info(u'foo'), the SyslogHandler
will (incorrectly) prepend a byte-order mark, which confuses the logstash
parser, resulting in “_grokparsefailure”. This is aknown bugin the Python
standard library that has been fixed in Python-2.7.4.Developing gocept.loggingAuthor:gocept<mail@gocept.com>PyPI page:http://pypi.python.org/pypi/gocept.logging/Issues:report by e-mailSource code:https://github.com/gocept/gocept.loggingCurrent change log:https://raw.githubusercontent.com/gocept/gocept.logging/master/CHANGES.rstChange log for gocept.logging1.0 (2023-07-18)Drop support for Python 2.7, 3.3, 3.4, 3.5, 3.6.Add support for Python 3.7, 3.8 3.9, 3.10, 3.11.Add an alias.warnfor.warningforStaticDefaults. (#1)0.8.1 (2017-01-09)Fixsetup.pyto use relative paths.0.8 (2016-03-17)Declare compatibility with PyPy and PyPy3.0.7 (2015-09-29)Declare Python 3.5 compatibility.0.6 (2015-09-17)Declare Python 3.4 compatibility.ArgumentParser.parse_args()now stores the computed log level on thelog_levelattribute of the return value.0.5 (2014-02-07)Allow to change log format for theArgumentParser0.4 (2013-09-24)Handle non-string log messages properly.0.3 (2013-09-04)Added sepcializedargparse.ArgumentParserwhich enables user to set thelogginglevel by default..0.2 (2013-08-24)Add timestamp and hostname to syslog messages,
this allows plugging SyslogKeyValueFormatter directly into logstash
without an intermediary syslogd.0.1 (2013-08-16)initial release