Package: pygelf-ex
pygelfPython logging handlers with GELF (Graylog Extended Log Format) support.Currently TCP, UDP, TLS (encrypted TCP) and HTTP logging handlers are supported.Get pygelfpipinstallpygelf-exUsagefrompygelfimportGelfTcpHandler,GelfUdpHandler,GelfTlsHandler,GelfHttpHandlerimportlogginglogging.basicConfig(level=logging.INFO)logger=logging.getLogger()logger.addHandler(GelfTcpHandler(host='127.0.0.1',port=9401))logger.addHandler(GelfUdpHandler(host='127.0.0.1',port=9402))logger.addHandler(GelfTlsHandler(host='127.0.0.1',port=9403))logger.addHandler(GelfHttpHandler(host='127.0.0.1',port=9404))logger.info('hello gelf')Message structureAccording to the GELF spec, each message has the following mandatory fields:version: ‘1.1’, can be overridden when creating a loggershort_message: the log message itselftimestamp: current timestamplevel:syslog-compliantlog level number (e.g. WARNING will be sent as 4)host: hostname of the machine that sent the messagefull_message: this field contains stack trace and is being writtenONLYwhen logging an exception, e.g.try:1/0exceptZeroDivisionErrorase:logger.exception(e)In debug mode (when handler was created with debug=True option) each message contains some extra fields (which are pretty self-explanatory):_file_line_module_func_logger_nameConfigurationEach handler has the following parameters:host: IP address of the GELF inputport: port of the GELF inputdebug(False by default): if true, each log message will include debugging info: module name, file name, line number, method nameversion(‘1.1’ by default): GELF protocol version, can be overriddeninclude_extra_fields(False by default): if true, each log message will include all the extra fields set to LogRecordjson_default(strwith exception for severaldatetimeobjects): function that is called for objects that cannot be serialized to JSON natively by python. Default implementation is custom function that returns result ofisoformat()method fordatetime.datetime,datetime.time,datetime.dateobjects and result ofstr(obj)call for other objects (which is string representation of an object with fallback torepr)Also, there are some handler-specific parameters.UDP:chunk_size(1300 by default) - maximum length of the message. If log length exceeds this value, it splits into multiple chunks (seehttps://www.graylog.org/resources/gelf/section “chunked GELF”) with the length equals to this value. This parameter must be less than theMTU. If the logs don’t seem to be delivered, try to reduce this value.compress(True by default) - if true, compress log messages before sending them to the serverTLS:validate(False by default) - if true, validate server certificate. If server provides a certificate that doesn’t exist inca_certs, you won’t be able to send logs over TLSca_certs(None by default) - path to CA bundle file. This parameter is required ifvalidateis true.certfile(None by default) - path to certificate file that will be used to identify ourselves to the remote endpoint. This is necessary when the remote server has client authentication required. Ifcertfilecontains the private key, it should be placed before the certificate.keyfile(None by default) - path to the private key. If the private key is stored incertfilethis parameter can be None.HTTP:compress(True by default) - if true, compress log messages before sending them to the serverpath(‘/gelf’ by default) - path of the HTTP input (http://docs.graylog.org/en/latest/pages/sending_data.html#gelf-via-http)timeout(5 by default) - amount of seconds that HTTP client should wait before it discards the request if the server doesn’t respondHTTPS:headers(empty by default) - default headers for the request, for example to set the content typewaitForResponse(False by default) - whether or not to wait for the response before returningStatic fieldsIf you need to include some static fields into your logs, simply pass them to the handler constructor. Each additional field should start with underscore. You can’t add field ‘_id’.Example:handler=GelfUdpHandler(host='127.0.0.1',port=9402,_app_name='pygelf',_something=11)logger.addHandler(handler)Dynamic fieldsIf you need to include some dynamic fields into your logs, add them to record by using LoggingAdapter or logging.Filter and create handler with include_extra_fields set to True.
All the non-trivial fields of the record will be sent to graylog2 with ‘_’ added before the nameExample:classContextFilter(logging.Filter):deffilter(self,record):record.job_id=threading.local().process_idreturnTruelogger.addFilter(ContextFilter())handler=GelfUdpHandler(host='127.0.0.1',port=9402,include_extra_fields=True)logger.addHandler(handler)Defining fields from environmentIf you need to include some fields from the environment into your logs, add them to record by usingadditional_env_fields.The following example will add anenvfield to the logs, taking its value from the environment variableFLASK_ENV.handler=GelfTcpHandler(host='127.0.0.1',port=9402,include_extra_fields=True,additional_env_fields={'env':'FLASK_ENV'})logger.addHandler(handler)The following can also be used in defining logging from configuration files (yaml/ini):[formatters]keys=standard[formatter_standard]class=logging.Formatterformat=%(message)s[handlers]keys=graylog[handler_graylog]class=pygelf.GelfTcpHandlerformatter=standardargs=('127.0.0.1', '12201')kwargs={'include_extra_fields': True, 'debug': True, 'additional_env_fields': {'env': 'FLASK_ENV'}}[loggers]keys=root[logger_root]level=WARNhandlers=graylogRunning testsTo run tests, you’ll needtox. After installing, simply run it:tox