Package: gelf-formatter
GELF FormatterGraylog Extended Log Format (GELF)formatter for thePython standard libraryloggingmoduleMotivationThere are several packages available providinghandlersfor the standard library logging module that can send application logs toGraylogby TCP/UDP/HTTP (py-gelfis a good example). Although these can be useful, it's not ideal to make an application performance dependent on network requests just for the purpose of delivering logs.Alternatively, one can simply log to a file orstdoutand have a collector (likeFluentd) processing and sending those logsasynchronouslyto a remote server (and not just to Graylog, as GELF can be used as a generic log format), which is a common pattern for containerized applications. In a scenario like this all we need is a GELF loggingformatter.FeaturesSupport for arbitrary additional fields;Support for including reservedlogging.LogRecordattributes as additional fields;Exceptions detection with traceback formatting;Zero dependencies and tiny footprint.InstallationWith pip$ pip install gelf-formatterFrom source$ python setup.py installUsageSimply create agelfformatter.GelfFormatterinstance and pass it as argument tologging.Handler.setFormatter:importsysimportloggingfromgelfformatterimportGelfFormatterformatter=GelfFormatter()handler=logging.StreamHandler(sys.stdout)handler.setFormatter(formatter)Apply it globally withlogging.basicConfigto automatically format log records from third-party packages as well:logging.basicConfig(level=logging.DEBUG,handlers=[handler])Alternatively, you can configure a locallogging.Loggerinstance throughlogging.Logger.addHandler:logger=logging.getLogger('my-app')logger.addHandler(handler)That's it. You can now use the logging module as usual, all records will be formatted as GELF messages.Standard FieldsThe formatter will output all (non-deprecated) fields described in theGELF Payload Specification (version 1.1):version: String, always set to1.1;host: String, the output ofsocket.gethostnameat initialization;short_message: String, log record message;full_message(optional): String, formatted exception traceback (if any);timestamp: Number, time in seconds since the epoch as a floating point;level: Integer,syslogseverity level.None of these fields can be ignored, renamed or overridden.Examplelogging.info("Some message"){"version":"1.1","host":"my-server","short_message":"Some message","timestamp":1557342545.1067393,"level":6}ExceptionsThefull_messagefield is used to store the traceback of exceptions. You just need to log them withlogging.exception.Exampleimporturllib.requestreq=urllib.request.Request('http://www.pythonnn.org')try:urllib.request.urlopen(req)excepturllib.error.URLErrorase:logging.exception(e.reason){"version": "1.1", "short_message": "[Errno -2] Name or service not known", "timestamp": 1557342714.0695107, "level": 3, "host": "my-server", "full_message": "Traceback (most recent call last):\n  ...(truncated)... raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno -2] Name or service not known>"}Additional FieldsThe GELF specification allows arbitrary additional fields, with keys prefixed with an underscore.To include additional fields use the standard loggingextrakeyword. Keys will be automatically prefixed with an underscore (if not already).Examplelogging.info("request received",extra={"path":"/orders/1","method":"GET"}){"version": "1.1", "short_message": "request received", "timestamp": 1557343604.5892842, "level": 6, "host": "my-server", "_path": "/orders/1", "_method": "GET"}Reserved FieldsBy default the formatter ignores alllogging.LogRecordattributes. You can however opt to include them as additional fields. This can be used to display useful information like the current module, filename, line number, etc.To do so, simply pass a list ofLogRecordattribute names as value of theallowed_reserved_attrskeyword when initializing aGelfFormatter. You can also modify theallowed_reserved_attrsinstance variable of an already initialized formatter.Exampleattrs=["lineno","module","filename"]formatter=GelfFormatter(allowed_reserved_attrs=attrs)# orformatter.allowed_reserved_attrs=attrslogging.debug("starting application..."){"version": "1.1", "short_message": "starting application...", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_lineno": 175, "_module": "myapp", "_filename": "app.py"}You can optionally customize the name of these additional fields using alogging.Filter(see below).Similarily, you can choose to ignore additional attributes passed via theextrakeyword argument. This can be usefull to e.g. not log keywords namedsecretorpassword.To do so, pass a list of names to theignored_attrskeyword when initializing aGelfFormatter. You can also modify theignored_attrsinstance variable of an already initialized formatter.ExampleBut be aware: nested fields will be printed! Only the root level of keywords is filtered by theignored_attrs.attrs=["secret","password"]formatter=GelfFormatter(ignored_attrs=attrs)# orformatter.ignored_attrs=attrslogging.debug("app config",extra={"connection":"local","secret":"verySecret!","mysql":{"user":"test","password":"will_be_logged"}}){"version": "1.1", "short_message": "app config", "timestamp": 1557346554.989846, "level": 6, "host": "my-server", "_connection": "local", "_mysql": {"user": "test", "password": "will_be_logged"}}Context FieldsHaving the ability to define a set of additional fields once and have them included in all log messages can be useful to avoid repetitiveextrakey/value pairs and enable contextual logging.Python's logging module provides several options to add context to a logger, among which we highlight thelogging.LoggerAdapterandlogging.Filter.Between these we recommend alogging.Filter, which is simpler and can be attached directly to alogging.Handler. Alogging.Filtercan therefore be used locally (on alogging.Logger) or globally (throughlogging.basicConfig). If you opt for aLoggerAdapteryou'll need alogging.Loggerto wrap.You can also use alogging.Filterto reuse/rename any of the reservedlogging.LogRecordattributes.ExampleclassContextFilter(logging.Filter):deffilter(self,record):# Add any number of arbitrary additional fieldsrecord.app="my-app"record.app_version="1.2.3"record.environment=os.environ.get("APP_ENV")# Reuse any reserved `logging.LogRecord` attributesrecord.file=record.filenamerecord.line=record.linenoreturnTrueformatter=GelfFormatter()handler=logging.StreamHandler(sys.stdout)handler.setFormatter(formatter)handler.addFilter(ContextFilter())logging.basicConfig(level=logging.DEBUG,handlers=[handler])logging.info("hi",extra=dict(foo="bar")){"version": "1.1", "short_message": "hi", "timestamp": 1557431642.189755, "level": 6, "host": "my-server", "_foo": "bar", "_app": "my-app", "_app_version": "1.2.3", "_environment": "development", "_file": "app.py", "_line": 159}Pretty-PrintLooking for a GELF log pretty-printer? If so, have a look atgelf-pretty:fire:ContributionsThis project adheres to the Contributor Covenantcode of conduct. By participating, you are expected to uphold this code. Please refer to ourcontributing guidefor further information.