Package: logging-utilities
Python logging utilitiesThis package implements some useful logging utilities. Here below are the main features of the package:JSON formatterText formatter withextraFlask request context record attributesJsonify Django request record attributeISO Time in formatYYYY-MM-DDThh:mm:ss.sss±hh:mmAdd constant record attributesLogger Level FilterAll features can be fully configured from the configuration file.NOTE:only python 3 is supported:warning:Version 3.x.x BREAKING CHANGESseeBreaking ChangesTable of contentTable of contentInstallationRelease and PublishContributionDeveloperIgnore missing log record attribute in formatterLogRecordIgnoreMissingLogging ContextLogging Context example with PyramidJSON FormatterConfigure JSON FormatJSON Formatter OptionsJSON Output - Type ConsistencyExtra FormatterExtra Formatter ConstructorExtra Formatter Config ExampleFlask Request ContextFlask Request Context Filter ConstructorFlask Request Context Config ExampleJsonify Django RequestUsageDjango Request Filter ConstructorDjango Request Config ExampleFilter out LogRecord attributes based on their typesAttribute Type Filter ConstructorAttribute Type Filter Config ExampleISO Time with TimezoneISO Time Filter ConstructorISO Time Config ExampleConstant Record AttributeConstant Record Attribute Config ExampleLogger Level FilterLogger Level Filter ConstructorLogger Level Filter Config ExampleBasic UsageCase 1. Simple JSON OutputCase 2. JSON Output Configured within Python CodeCase 3. JSON Output Configured with a YAML FileCase 4. Add Flask Request Context Attributes to JSON OutputCase 5. Add Django Request to JSON OutputCase 6. Add parts of Django Request to JSON OutputCase 7. Add all Log Extra as Dictionary to the Standard Formatter (including Django log extra)Case 8. Add Specific Log Extra to the Standard FormatterBreaking ChangesVersion 4.x.x Breaking ChangesVersion 3.x.x Breaking ChangesVersion 2.x.x Breaking ChangesCreditsInstallationlogging_utilitiesis available on PyPI.Use pip to install:pipinstalllogging-utilitiesRelease and PublishNew release and publish on PyPI is done automatically upon PR merge intomasterbranch. For bug fixes and small new features,
PR can be directly open onmaster. Then the PR title define the version bump as follow:PR title and/or commit message contains#major=> major version is bumpedPR title and/or commit message contains#patchor head branch name starts withbug-|hotfix-|bugfix-=> patch version is bumpedOtherwise by default the minor version is bumpedContributionEvery contribution to this library is welcome ! So if you find a bug or want to add a new feature everyone is welcome to open anissueor created aPull Request.Any contribution must follow thegit-flow.DeveloperYou can quickly setup your environment with the makefile:makesetupThis will create a virtual python environment with all packages required for the development.Note that for pull request, the codeMUST BEwithyapfformatted and it alsoMUST PASSthe linter. For this you can use the make targets:makeformat
makelint#ormakeformat-lintAny new feature should have its unittest class in order to be tested.Ignore missing log record attribute in formatterWhen configuring a log formatter you can provide via print style any log record attribute including extra attributes. However when using extra attribute, if this attribute is then missing (e.g. because the logger did not add that extra)
then the logging would raise aValueError: Formatting field not found in record: ....For the standard Formatter you could use theExtra Formatter, but if you have any other Formatter you
can use the globallogging_utilities.log_record.set_log_record_ignore_missing_factory()method.LogRecordIgnoreMissingTheLogRecordIgnoreMissingfactory can be used to avoidValueErrorexception when formatting a log message from
a log record that don't have the extra required by the formatter.For example:importlogginglogging.basicConfig(format="%(message)s-%(extra_param)s",level=logging.INFO,force=True)logger=logging.getLogger('my-logger')logger.info('My message',extra={'extra_param':20})Mymessage-20logger.info('My second message')---Loggingerror---Traceback(mostrecentcalllast):File"/usr/lib/python3.8/logging/__init__.py",line440,informatreturnself._format(record)File"/usr/lib/python3.8/logging/__init__.py",line436,in_formatreturnself._fmt%record.__dict__KeyError:'extra_param'Duringhandlingoftheaboveexception,anotherexceptionoccurred:Traceback(mostrecentcalllast):File"/usr/lib/python3.8/logging/__init__.py",line1085,inemitmsg=self.format(record)File"/usr/lib/python3.8/logging/__init__.py",line929,informatreturnfmt.format(record)File"/usr/lib/python3.8/logging/__init__.py",line671,informats=self.formatMessage(record)File"/usr/lib/python3.8/logging/__init__.py",line640,informatMessagereturnself._style.format(record)File"/usr/lib/python3.8/logging/__init__.py",line442,informatraiseValueError('Formatting field not found in record:%s'%e)ValueError:Formattingfieldnotfoundinrecord:'extra_param'...To avoid such crash you can useLogRecordIgnoreMissingthat will replace missing extra attributes by an empty string in the message.importloggingfromlogging_utilities.log_recordimportLogRecordIgnoreMissinglogging.setLogRecordFactory(LogRecordIgnoreMissing)logging.basicConfig(format="%(message)s-%(extra_param)s",level=logging.INFO,force=True)logger=logging.getLogger('my-logger')logger.info('My message',extra={'extra_param':20})Mymessage-20logger.info('My second message')Mysecondmessage-You can also change the default value by using the helperset_log_record_ignore_missing_factory()importloggingfromlogging_utilities.log_recordimportset_log_record_ignore_missing_factoryset_log_record_ignore_missing_factory('my-default')logging.basicConfig(format="%(message)s-%(extra_param)s",level=logging.INFO,force=True)logger=logging.getLogger('my-logger')logger.info('My message',extra={'extra_param':20})Mymessage-20logger.info('My second message')Mysecondmessage-my-default:warning:NOTE that setting the log record factory is a global action that affects every logger and formatterLogging ContextWithset_logging_context()you can add a thread based context to every log record. This can be quite usefull if
you want to globally set a context to every log record, for example a Request context in a Pyramid/Django application.Logging Context example with PyramidIn aPyramidapplication it is quite usefull to
add to every log record the Request context. This can be done as follow:# module my_app.logging_tweensfromlogging_utilities.contextimportset_logging_contextdeflogging_context_tween(handler,registry):def_logging_context_tween(request):set_logging_context({"request":{"method":request.method,"path":request.path,"headers":dict(request.headers)}})returnhandler(request)return_logging_context_tween# MAINimportloggingfromwsgiref.simple_serverimportmake_serverfrompyramid.configimportConfiguratorfrompyramid.responseimportResponselogging.basicConfig(format="%(message)s-%(context)s")logger=logging.getLogger(__name__)defhello_world(request):logger.debug('Request for hello world')returnResponse('Hello World!')if__name__=='__main__':withConfigurator()asconfig:# Register the tweenconfig.add_tween('my_app.logging_tweens.logging_context_tween')# Configure the route and viewconfig.add_route('hello','/')config.add_view(hello_world,route_name='hello')app=config.make_wsgi_app()server=make_server('0.0.0.0',6543,app)server.serve_forever()# A GET / request would produce the following log'Request for hello world - {"request": {"method": "GET", "path": "/", "headers":{}}}'For more information on Pyramid Tweens seeRegistering TweenJSON FormatterJsonFormatteris a python logging formatter that transform the log output into a json object.JSON log format is quite useful especially when the logs are sent toLogStash.This formatter supports embedded object as well as array.Configure JSON FormatThe format can be configured either using theformatconfig parameter or thefmtconstructor parameter. This parameter should be a dictionary (for Python version below 3.7, it is better to useOrderedDictto keep the attribute order). Eachkeyis taken as such askeyfor the output JSON object, while each value is transformed as follow in the output:ValueTypeTransformationExampleLogRecordattributestringThe string is aLogRecordattribute name,then the value of this attribute is used as output. See alsoType Consistency."message"LogRecordattribute dotted keystringThe string is a dotted key to access a sub key of aLogRecorddictionary attribute.For example if theLogRecordcontains a dictionary attribute added via anextra, you can use the dotted notation to access only a sub object/value of this dictionary. Note if the dotted key attribute doesn't exists it will raise aValueErrorunless you setignore_missing=Truein the Formatter config. In the latest case missing attribute will be replaced by''unless the dotted key has a trailing.then the default value will be{}instead of''.See alsoType Consistency."request.path"Named string formatstringThe string contains named string format,each named format are replaced by the correspondingLogRecordattribute value.When using the%string formatting style, you can also used dotted notation to access dictionary sub-key;%(request.headers)s. NOTE that in string format the dictionary key must be a valid python attribute name (cannot contain spaces or special characters)."%(asctime)s.%(msecs)s"ObjectdictThe object is embedded in the output with its valuefollowing the same rules as defined in this table.{"lineno": "lineno", "file": "filename", "id": "%(process)x/%(thread)x", "message": "message"}ArraylistThe list is embedded as anarrayin the output.Each value is processed using the rules from this table["created", "asctime", "message", "%(process)x/%(thread)x"]:warning:If the value doesn't match any of the table above it will raise aValueErrorunless you specifyignore_missing=Truein the configurationYou can find theLogRecordattributes list inPython DocSee below theBasic Usagefor more examples.JSON Formatter OptionsYou can change some behavior using theJsonFormatterconstructor:ParameterTypeDefaultDescriptionfmtdict{'levelname': 'levelname', 'name': 'name', 'message': 'message'}Define the output format, seeConfigure JSON FormatdatefmtstringNoneDate format forasctime, seetime.strftime()stylestring%String formatting style, seelogging.Formatteradd_always_extraboolFalseWhenTrue, logging extra (logging.log('message', extra={'my-extra': 'some value'})) are always added to the output. Otherwise they are only added if present infmt.filter_attributeslistNoneWhen the formatter is used with aLogging.Filterthat addsLogRecordattributes, they can be listed here to avoid to be treated as loggingextra.remove_emptyboolFalseWhenTrue, empty values (empty list, dict, None or empty string) are removed from output.ignore_missingboolFalseIfTrue, then all extra attributes from the log record that are missing (accessed by thefmtparameter) will be replaced by an empty string instead of raising a ValueError exception.NOTE:This has an impact on all formater not only on this one, seeLogRecordIgnoreMissing.The constructor parameters can be also be specified in the log configuration file using the()class specifier instead ofclass:formatters:json:():logging_utilities.formatters.json_formatter.JsonFormatteradd_always_extra:Truefmt:time:asctimelevel:levelnamelogger:namemodule:modulemessage:message:warning: When using the INI file format like documentedhere, you cannot use the JSON formatter options describe above and have to use the formatter using theclass,format,datefmtandstyleattributes like below[formatters]keys=my_json[formatter_my_json]class=logging_utilities.formatters.json_formatter.JsonFormatterformat:{"time":"asctime","level":"levelname","logger":"name","module":"module","function":"funcName","pid_tid":"%(process)x/%(thread)x","message":"message","exc_info":"exc_info"} # OPTIONALdatefmt=%Y-%m-%d %H:%M# OPTIONALstyle=%# OPTIONALJSON Output - Type ConsistencyWhen you useignore_missing=True, all missing attributes from the log record will be replaced by an empty string. This can be an issue if you require type consistency accross JSON logs. To avoid this, you can use the trailing dot notation.Single trailing dotattribute_name.Default to{}whenattribute_nameis missing from log recordDouble trailing dotattribute_name..Default to[]whenattribute_nameis missing from log recordThis is quite usefull if you want to add a list or an object in your JSON from a LogRecord that might be missing. For example when using theFlask Request Contextand you want to add the headers dictionary as object, you can do as follow:fmt={"message":"message","request":{"headers":"flask_request_headers."}}This way if the log record is outside a Flask request, your log output would be{"message": "this is the message", "request": {"headers": {}}}instead of{"message": "this is the message", "request": {"headers": ""}}and when the record is within a Flask context you will have{"message": "this is the message", "request": {"headers": {"Host": "www.example.com", ...}}}Extra FormatterThis formatter enhance the python standard formatter to allow working with the logextra.
When adding anextrakeyword in the format, the python standard formatter raises aValueError()when this keyword is missing from log record. This means that if you want to display a logextra, you have to make sure that every log message contains thisextra.This formatter allow you to provide anextra_fmtparameter that will add recordextrato the
log message when available. You can either add the entire extra dictionary:extra_fmt='%s'or
only some extras:extra_fmt='%(extra1)s:%(extra2)s'. In the latest case, when a key is missing
in extra, the value is replaced byextra_default.When using the wholeextradictionary, you can useextra_pretty_printto improve the
formatting, note that in this case the log might be on multiline (this usepprint.pformat).Seelogging.Logger.debugfor more infos on the loggingextraExtra Formatter ConstructorSupport the same arguments as thelogging.Formatterplus the followings:ParameterTypeDefaultDescriptionextra_fmtNone|strNoneWhen notNone, adds theextraat the end of the log message. Either uses named placeholder with the extra keywords or add the wholeextradirectory using%s.extra_defaultNone|str''Whenextra_fmtcontains named placeholders and one or more of these placeholders are not found in the log record, then the formatter uses this default value instead.extra_defaultany''When usingextra_fmtwith named placeholders and a keyword is missing in the log record, it is then replaced by this value.extra_pretty_printbooleanFalseWhenextra_fmt='%s'you can set this flag toTrueto usepprint.pformaton the dictionary.pretty_print_kwargsNone|dictNonekwargs as dictionary to pass topprint.pformatExtra Formatter Config Exampleformatters:standard:():logging_utilities.formatters.extra_formatter.ExtraFormatterformat:"%(levelname)s-%(name)s-%(message)s"extra_fmt:"-extra:\n%s"extra_pretty_print:TrueNOTE:ExtraFormatteronly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Flask Request ContextWhen using logging within aFlaskapplication, you can use thisFilterto add some context attributes to allLogRecord.AllFlask Requestattributes are supported and they are added asLogRecordwith theflask_request_prefix. SeeFlask Requestfor more details on available attributes.Flask Request Context Filter ConstructorParameterTypeDefaultDescriptionattributeslistNoneList of Flask Request attributes name to add to theLogRecordFlask Request Context Config Exampleversion:1root:handlers:-consolelevel:DEBUGpropagate:Truefilters:flask:():logging_utilities.filters.flask_attribute.FlaskRequestAttributeattributes:-url-method-headers-jsonformatters:console:format:"%(asctime)s-%(message)s-%(flask_request_url)s%(flask_request_method)s%(flask_request_headers)s:%(flask_request_json)s"handlers:console:class:logging.StreamHandlerformatter:consolestream:ext://sys.stdoutfilters:-flaskNOTE:FlaskRequestAttributeonly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Jsonify Django RequestIf you want to log theDjangoHttpRequestobject using theJSON Formatter, this filter is for made for you. It converts therecord.http_requestattribute (or the attribute specified byattr_keyin the constructor) to a valid json object if it is of typeHttpRequest.TheHttpRequestattributes that are converted can be configured using theinclude_keysand/orexclude_keysfilter parameters. This can be useful if you want to limit the log data, for example if you don't want to log Authentication headers.:warning: The django framework adds sometimes an HttpRequest or socket object underrecord.requestwhen
logging. So if you decide to use the attribute namerequestfor this filter, beware that you
will need to handle the case where the attribute is of typesocketseparately, for example by
filtering it out using the attribute type filter. (see exampleFilter out LogRecord attributes based on their types)UsageAdd the filter to the log handler and then add simply theHttpRequestto the log extra as follow:logger.info('My message',extra={'http_request':request})Django Request Filter ConstructorParameterTypeDefaultDescriptioninclude_keyslistNoneAll request attributes that match any of the dotted keys of the list will be added to the jsonifiable object. WhenNonethen all attributes are added (default behavior).exclude_keyslistNoneAll request attributes that match any of the dotted keys of the list will not be added to the jsonifiable object.NOTEthis has precedence toinclude_keyswhich means that if a key is in both lists, then it is not added.attr_keystrhttp_requestThe name of the attribute that stores the HttpRequest object. It will be replaced in place by a jsonifiable dict representing this object. (Note that django sometimes stores anHttpRequestunderattr_key: request. This is however not the default as django also stores other types of objects under this attribute name.)Django Request Config Examplefilters:django:():logging_utilities.filters.django_request.JsonDjangoRequestattr_key:'http_request'# This is the default, so it can be omittedinclude_keys:-http_request.META.REQUEST_METHOD-http_request.META.SERVER_NAME-http_request.environexclude_keys:-http_request.META.SERVER_NAME-http_request.environ.wsgiNOTE:JsonDjangoRequestonly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Filter out LogRecord attributes based on their typesIf different libraries or different parts of your code log different object types under the same
logRecord extra attribute, you can use this filter to keep only some of them (whitelist mode) or filter out
some of them (blacklist mode).Attribute Type Filter ConstructorParameterTypeDefaultDescriptiontypecheck_listdict(key, type|list of types)NoneA dictionary that maps keys to a type or a list of types. By default, it will only keep a parameter matching a key if the types match or if any of the types in the list match (white list). If in black list mode, it will only keep a parameter if the types don't match. Parameters not appearing in the dict will be ignored and passed though regardless of the mode (whitelist or blacklist).is_blacklistboolfalseWhether the list passed should be a blacklist or a whitelist. To use both, simply include this filter two times, one time with this parameter set true and one time with this parameter set false.Attribute Type Filter Config Examplefilters:type_filter:():logging_utilities.filters.attr_type_filter.AttrTypeFilteris_blacklist:False# Default value is false, so this could be left outtypecheck_list:# For each attribute listed, one type or a list of types can be specifiedrequest:# can only be a toplevel attribute (no dotted keys allowed)-django.http.request.HttpRequest# Can be a class name only or the full dotted path-builtins.dictmy_attr:myClassISO Time with TimezoneThe standard logging doesn't support the time as ISO with timezone;YYYY-MM-DDThh:mm:ss.sss±hh:mm. By defaultasctimeuses a ISO like format;YYYY-MM-DD hh:mm:ss.sss, but withoutTseparator (although this one could be configured by overriding a global variable, this can't be done by config file). You can use thedatefmtoption to specify another date format, however this one don't supports milliseconds, so you could achieve this format:YYYY-MM-DDThh:mm:ss±hh:mm.This Filter can be used to achieve the full ISO 8601 Time format including timezone and milliseconds.ISO Time Filter ConstructorParameterTypeDefaultDescriptionisotimeboolTrueAdd log local time asisotimeattribute toLogRecordwith theYYYY-MM-DDThh:mm:ss.sss±hh:mmformat.utc_isotimeboolFalseAdd log UTC time asutc_isotimeattribute toLogRecordwith theYYYY-MM-DDThh:mm:ss.sss±hh:mmformat.ISO Time Config Examplefilters:isotime:():logging_utilities.filters.TimeAttributeutc_isotime:Trueisotime:FalseNOTE:TimeAttributeonly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Constant Record AttributeSimple loggingFilterto add constant attribute to everyLogRecordConstant Record Attribute Config Examplefilters:application:():logging_utilities.filters.ConstAttributeapplication:my-applicationNOTE:ConstAttributeonly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Logger Level FilterSometimes you might want to have different log Level based on the logger and handler. The standard logging library allow to set a logger level or a handler level but not based on both. Let say you have a config with two loggers logging to two handlers, on the first handler you want all messages of both loggers and on the second handler you want all messages of the first logger but only the WARNING messages of the second logger. This is here were this filter come into play.Logger Level Filter ConstructorParameterTypeDefaultDescriptionlevelint | string'DEBUG'All messages with a lower level than this one will be filtered out.loggerstring''When non empty, only message from this logger will be filtered out based on their level.Logger Level Filter Config Exampleroot:handlers:-"console"-"file"level:"DEBUG"propagate:"True"filters:B_filter:():logging_utilities.filters.LevelFilterlevel:"WARNING"logger:'B'loggers:A:level:"DEBUG"B:level:"DEBUG"handlers:console:class:"logging.StreamHandler"file:class:"logging.handlers.RotatingFileHandler"filters:-"B_filter"NOTE:LevelFilteronly support the special key'()'factory in the configuration file (it doesn't work with the normal'class'key).Basic UsageCase 1. Simple JSON Outputimportloggingfromlogging_utilities.formatters.json_formatterimportbasic_config# default keyword parameter `format`: """{"levelname": "levelname", "name": "name", "message": "message"}"""basic_config(level=logging.INFO)logging.info('hello, json_formatter')output:{"levelname":"INFO","name":"root","message":"hello, json_formatter"}Case 2. JSON Output Configured within Python Codeimportloggingfromlogging_utilities.formatters.json_formatterimportJsonFormatter# `FORMAT` can be `json`, `OrderedDict` or `dict`.# If `FORMAT` is `dict` and python version < 3.7.0, the output order is sorted by keys, otherwise it will be the same# as the defined order.## KEY := string, can be whatever you like.# VALUE := `LogRecord` attribute name, string, formatted string (e.g. "%(asctime)s.%(msecs)s"), list or dictFORMAT={"Name":"name","Levelno":"levelno","Levelname":"levelname","Pathname":"pathname","Filename":"filename","Module":"module","Lineno":"lineno","FuncName":"funcName","Created":"created","Asctime":"asctime","Msecs":"msecs","RelativeCreated":"relativeCreated","Thread":"thread","ThreadName":"threadName","Process":"process","Message":"message"}root=logging.getLogger()root.setLevel(logging.INFO)formatter=JsonFormatter(FORMAT)sh=logging.StreamHandler()sh.setFormatter(formatter)sh.setLevel(logging.INFO)root.addHandler(sh)deftest():root.info("test%sformat",'string')test()output:{"Name":"root","Levelno":20,"Levelname":"INFO","Pathname":"test.py","Filename":"test.py","Module":"test","Lineno":75,"FuncName":"test","Created":1588185267.3198836,"Asctime":"2020-04-30 02:34:27,319","Msecs":319.8835849761963,"RelativeCreated":88.2880687713623,"Thread":16468,"ThreadName":"MainThread","Process":16828,"Message":"test string format"}Case 3. JSON Output Configured with a YAML Fileconfig.yaml:version:1root:handlers:-consolelevel:DEBUGpropagate:Trueformatters:json:class:logging_utilities.formatters.json_formatter.JsonFormatterformat:time:asctimelevel:levelnamelogger:namemodule:modulefunction:funcNameprocess:processthread:threadmessage:messagehandlers:console:class:logging.StreamHandlerformatter:jsonstream:ext://sys.stdoutThen in your python code use it as follow:importloggingimportlogging.configimportyamlconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)root=logging.getLogger()root.info('Test file config')output:{"function":"<module>","level":"INFO","logger":"root","message":"Test file config","module":"<stdin>","process":12264,"thread":139815989413696,"time":"asctime"}Case 4. Add Flask Request Context Attributes to JSON Outputconfig.yamlversion:1root:handlers:-consolelevel:DEBUGpropagate:Truefilters:isotime:():logging_utilities.filters.TimeAttributeflask:():logging_utilities.filters.flask_attribute.FlaskRequestAttributeattributes:-url-method-headers-remote_addr-jsonformatters:json:class:logging_utilities.formatters.json_formatter.JsonFormatterformat:time:isotimelevel:levelnamelogger:namemodule:modulefunction:funcNameprocess:processthread:threadrequest:# We use the "%()s" notation here to ensure a string output and also if the LogRecord has# no flask context, meaning no `flask_request_url` attribute, the "%()s" notation ensure# to have an empty string instead of treating `flask_request_url` as a string constant.url:"%(flask_request_url)s"method:"%(flask_request_method)s"# We use a trailing dot here to ensure to have a dictionary output even if the LogRecord# doesn't have a flask_request_headers attribute.headers:flask_request_headers.data:flask_request_json.remote:"%(flask_request_remote_addr)s"message:messagehandlers:console:class:logging.StreamHandlerformatter:jsonstream:ext://sys.stdoutfilters:-isotime-flaskNOTE:This require to haveflaskpackage installed otherwise it raisesImportErrorThen in your python code use it as follow:importloggingimportlogging.configimportyamlfromflaskimportFlaskconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)app=Flask('test')root=logging.getLogger()withapp.test_request_context("path/test",method='GET',headers={"Accept":"*/*"}):root.info('Test file config')output:{"time":"2022-07-20T10:09:10.765237+02:00","level":"INFO","logger":"root","module":"<stdin>","function":"<module>","process":58043,"thread":139717802334016,"request":{"url":"http://localhost/path/test","method":"GET","headers":{"Host":"localhost","Accept":"*/*"},"data":null,"remote":null},"message":"Test file config"}Case 5. Add Django Request to JSON Outputconfig.yamlversion:1root:handlers:-consolelevel:DEBUGpropagate:Truefilters:isotime:():logging_utilities.filters.TimeAttributedjango:():logging_utilities.filters.django_request.JsonDjangoRequestinclude_keys:-http_request.path-http_request.method-http_request.headersexclude_keys:-http_request.headers.Authorization-http_request.headers.Proxy-Authorizationformatters:json:class:logging_utilities.formatters.json_formatter.JsonFormatterformat:time:isotimelevel:levelnamelogger:namemodule:modulefunction:funcNameprocess:processthread:threadrequest:http_requestresponse:responsemessage:messagehandlers:console:class:logging.StreamHandlerformatter:jsonstream:ext://sys.stdoutfilters:-isotime-djangoNOTE:This require to havedjangopackage installed otherwise it raisesImportErrorThen in your python code use it as follow:importloggingimportlogging.configimportyamlfromdjango.httpimportJsonResponsefromdjango.confimportsettingsfromdjango.testimportRequestFactoryconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)logger=logging.getLogger('your_logger')defmy_page(request):answer={'success':True}logger.info('My page requested',extra={'request':request,'response':answer})returnJsonResponse(answer)settings.configure()factory=RequestFactory()my_page(factory.get('/my_page?test=true'))output:{"function":"my_page","level":"INFO","logger":"your_logger","message":"My page requested","module":"<stdin>","process":20421,"request":{"method":"GET","path":"/my_page","headers":{"Cookie":""}},"response":{"success":true},"thread":140433370822464,"time":"2020-10-12T16:44:45.374508+02:00"}Case 6. Add parts of Django Request to JSON OutputLet's say you want to log parts of the djangoHttpRequestin Json format. Django already logs it
sometimes underrecord.requestso you can use the django request filter to transform it to a jsonisable
dictionary. However django sometimes also logs an object of typesocket.socketthat you may not
want to include in the logs. In this case you could use the following configuration. This will only
keep the request attribute if it is of typeHttpRequest.config.yamlversion:1root:handlers:-consolelevel:DEBUGpropagate:Truefilters:type_filter:():logging_utilities.filters.attr_type_filter.AttrTypeFiltertypecheck_list:request:django.http.request.HttpRequestisotime:():logging_utilities.filters.TimeAttributedjango:():logging_utilities.filters.django_request.JsonDjangoRequestattr_name:requestinclude_keys:-request.path-request.method-request.headersformatters:json:class:logging_utilities.formatters.json_formatter.JsonFormatterformat:time:isotimelevel:levelnamelogger:namemodule:modulefunction:funcNameprocess:processthread:threadrequest_path:request.pathrequest_method:request.methodrequest:# NOTE: django headers name are case sensitiveheader.accept:request.headers.Acceptheader.accept-encoding:request.headers.Accept-Encodingheader.accept_language:request.headers.Accept-Languagemessage:messagehandlers:console:class:logging.StreamHandlerformatter:jsonstream:ext://sys.stdoutfilters:-isotime# Typefilter must be before django filter, as the django filter# will modify the type of the "HttpRequest" object-type_filter-djangoNOTE:This require to havedjangopackage installed otherwise it raisesImportErrorThen in your python code use it as follow:importloggingimportlogging.configimportyamlfromdjango.httpimportJsonResponsefromdjango.confimportsettingsfromdjango.testimportRequestFactoryconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)logger=logging.getLogger('your_logger')defmy_page(request):answer={'success':True}logger.info('My page requested',extra={'request':request})returnJsonResponse(answer)settings.configure()factory=RequestFactory()my_page(factory.get('/my_page?test=true',HTTP_ACCEPT='*/*',HTTP_ACCEPT_ENCODING='gzip',HTTP_ACCEPT_LANGUAGE='en'))output:{"time":"2022-07-20T12:29:19.536922+02:00","level":"INFO","logger":"your_logger","module":"<stdin>","function":"my_page","process":78479,"thread":139751209555776,"request_path":"/my_page","request_method":"GET","request":{"header.accept":"*/*","header.accept-encoding":"gzip","header.accept_language":"en"},"message":"My page requested"}Case 7. Add all Log Extra as Dictionary to the Standard Formatter (including Django log extra)config.yamlversion:1root:handlers:-consolelevel:DEBUGpropagate:Truefilters:type_filter:():logging_utilities.filters.attr_type_filter.AttrTypeFiltertypecheck_list:request:django.http.request.HttpRequestisotime:():logging_utilities.filters.TimeAttributedjango:():logging_utilities.filters.django_request.JsonDjangoRequestattr_name:requestinclude_keys:-request.path-request.method-request.headersexclude_keys:-request.headers.Authorization-request.headers.Proxy-Authorizationformatters:standard_extra:():logging_utilities.formatters.extra_formatter.ExtraFormatter# NOTE also in the constructor the parameter is `fmt` we need to use `format` hereformat:"%(isotime)s-%(levelname)s-%(name)s-%(message)s"extra_fmt:"-extra:\n%s"extra_pretty_print:Truepretty_print_kwargs:indent:2width:60handlers:console:class:logging.StreamHandlerformatter:standard_extrastream:ext://sys.stdoutfilters:-isotime# Type filter must be before django filter-type_filter-djangoNOTE:This require to havedjangopackage installed otherwise it raisesImportErrorThen in your python code use it as follow:#!.venv/bin/python3importloggingimportlogging.configimportyamlfromdjango.httpimportJsonResponsefromdjango.confimportsettingsfromdjango.testimportRequestFactoryconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)logger=logging.getLogger('your_logger')defmy_page(request):answer={'success':True}logger.info('My page requested',extra={'request':request,'response':answer})returnJsonResponse(answer)settings.configure()factory=RequestFactory()my_page(factory.get('/my_page?test=true'))output:2020-11-19T13:32:58.942568+01:00-INFO-your_logger-Mypagerequested-extra:{'request':{'headers':{'Cookie':''},'method':'GET','path':'/my_page'},'response':{'success':True}}Case 8. Add Specific Log Extra to the Standard Formatterconfig.yamlversion:1root:handlers:-consolelevel:DEBUGpropagate:Trueformatters:standard_extra:():logging_utilities.formatters.extra_formatter.ExtraFormatter# NOTE also in the constructor the parameter is `fmt` we need to use `format` hereformat:"%(asctime)s-%(levelname)s-%(name)s-%(message)s"extra_fmt:"-extra1=%(extra1)s"handlers:console:class:logging.StreamHandlerformatter:standard_extrastream:ext://sys.stdoutThen in your python code use it as follow:#!.venv/bin/python3importloggingimportlogging.configimportyamlconfig={}withopen('example-config.yaml','r')asfd:config=yaml.safe_load(fd.read())logging.config.dictConfig(config)logger=logging.getLogger('your_logger')logger.debug('My log with extras',extra={'extra1':23,'extra2':"don't add this"})output:2020-11-1913:42:29,424-DEBUG-your_logger-Mylogwithextras-extra1=23Breaking ChangesVersion 4.x.x Breaking ChangesFrom version 3.x.x to version 4.x.x there is the following breaking change:The django request filter by default now reads the attributerecord.http_requestinstead of
the attributerecord.request. There is however a new optionattr_namein the filters constructor
to manually specify the attribute name. See the exampleAdd parts of Django Request to JSON Outputfor an example on how to useattr_nameto be
backward-compatible with 3.x.xVersion 3.x.x Breaking ChangesFrom version 2.x.x to version 3.x.x there is the following breaking change:JSON Formatter doesn't support anymore string constant in thefmtparameter. Now if you want to have a string constant in all of you JSON logs output, you need to use theConstant Record Attribute Filter.Version 2.x.x Breaking ChangesFrom version 1.x.x to version 2.x.x there is the following breaking change:Flask Attribute filter do not set anymore missing Flask attribute to empty string ! So if you configure the Flask attribute you must make sure that all attribute specified in the attribute list, exists. Also if you use the filter on a logger outside of a Flask Request context, the logger will raise aValueErrorexception due to the missing Flask Request attribute. To avoid this you can use the newLogRecordIgnoreMissing.CreditsThe JSON Formatter implementation has been inspired byMyColorfulDays/jsonformatter