Package: django-structlog
django-structlogdjango-structlog is a structured logging integration forDjangoproject usingstructlogLogging will then produce additional cohesive metadata on each logs that makes it easier to track events or incidents.Additional Popular IntegrationsDjango REST frameworkDjango REST frameworkis supported by default. But when using it withrest_framework.authentication.TokenAuthentication(or other DRF authentications)user_idwill be only be inrequest_finishedandrequest_failedinstead of each logs.See#37for details.CeleryCelery’s task logging requires additional configurations, seedocumentationfor details.Logging comparisonStandard logging:>>>importlogging>>>logger=logging.get_logger(__name__)>>>logger.info("An error occurred")AnerroroccurredWell… okWith django-structlog and flat_line:>>>importstructlog>>>logger=structlog.get_logger(__name__)>>>logger.info("an_error_occurred",bar="Buz")timestamp='2019-04-13T19:39:31.089925Z'level='info'event='an_error_occurred'logger='my_awesome_project.my_awesome_module'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'bar='Buz'Then you can search with commands like:$catlogs/flat_line.log|greprequest_id='3a8f801c-072b-4805-8f38-e1337f363ed4'With django-structlog and json>>>importstructlog>>>logger=structlog.get_logger(__name__)>>>logger.info("an_error_occurred",bar="Buz"){"request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","event":"an_error_occurred","timestamp":"2019-04-13T19:39:31.089925Z","logger":"my_awesome_project.my_awesome_module","level":"info","bar":"Buz"}Then you can search with commands like:$catlogs/json.log|jq'.[] | select(.request_id="3a8f801c-072b-4805-8f38-e1337f363ed4")'-sGetting StartedThese steps will show how to integrate the middleware to your awesome application.InstallationInstall the librarypipinstalldjango-structlogAdd appINSTALLED_APP=[# ..."django_structlog",# ...]Add middlewareMIDDLEWARE=[# ..."django_structlog.middlewares.RequestMiddleware",]Add appropriate structlog configuration to yoursettings.pyimportstructlogLOGGING={"version":1,"disable_existing_loggers":False,"formatters":{"json_formatter":{"()":structlog.stdlib.ProcessorFormatter,"processor":structlog.processors.JSONRenderer(),},"plain_console":{"()":structlog.stdlib.ProcessorFormatter,"processor":structlog.dev.ConsoleRenderer(),},"key_value":{"()":structlog.stdlib.ProcessorFormatter,"processor":structlog.processors.KeyValueRenderer(key_order=['timestamp','level','event','logger']),},},"handlers":{# Important notes regarding handlers.## 1. Make sure you use handlers adapted for your project.# These handlers configurations are only examples for this library.# See python's logging.handlers: https://docs.python.org/3/library/logging.handlers.html## 2. You might also want to use different logging configurations depending of the environment.# Different files (local.py, tests.py, production.py, ci.py, etc.) or only conditions.# See https://docs.djangoproject.com/en/dev/topics/settings/#designating-the-settings"console":{"class":"logging.StreamHandler","formatter":"plain_console",},"json_file":{"class":"logging.handlers.WatchedFileHandler","filename":"logs/json.log","formatter":"json_formatter",},"flat_line_file":{"class":"logging.handlers.WatchedFileHandler","filename":"logs/flat_line.log","formatter":"key_value",},},"loggers":{"django_structlog":{"handlers":["console","flat_line_file","json_file"],"level":"INFO",},# Make sure to replace the following logger's name for yours"django_structlog_demo_project":{"handlers":["console","flat_line_file","json_file"],"level":"INFO",},}}structlog.configure(processors=[structlog.contextvars.merge_contextvars,structlog.stdlib.filter_by_level,structlog.processors.TimeStamper(fmt="iso"),structlog.stdlib.add_logger_name,structlog.stdlib.add_log_level,structlog.stdlib.PositionalArgumentsFormatter(),structlog.processors.StackInfoRenderer(),structlog.processors.format_exc_info,structlog.processors.UnicodeDecoder(),structlog.stdlib.ProcessorFormatter.wrap_for_formatter,],logger_factory=structlog.stdlib.LoggerFactory(),cache_logger_on_first_use=True,)Start logging withstructloginstead oflogging.importstructloglogger=structlog.get_logger(__name__)Extending Request Log MetadataBy default only arequest_idand theuser_idare bound from the request but pertinent log metadata may vary from a project to another.If you need to add more metadata from the request you can implement a convenient signal receiver to bind them. You can also override existing bound metadata the same way.fromdjango.contrib.sites.shortcutsimportget_current_sitefromdjango.dispatchimportreceiverfromdjango_structlogimportsignalsimportstructlog@receiver(signals.bind_extra_request_metadata)defbind_domain(request,logger,**kwargs):current_site=get_current_site(request)structlog.contextvars.bind_contextvars(domain=current_site.domain)Standard LoggersIt is also possible to log using standard python logger.In your formatters, add theforeign_pre_chainsection, and then addstructlog.contextvars.merge_contextvars:LOGGING={"version":1,"disable_existing_loggers":False,"formatters":{"json_formatter":{"()":structlog.stdlib.ProcessorFormatter,"processor":structlog.processors.JSONRenderer(),# Add this section:"foreign_pre_chain":[structlog.contextvars.merge_contextvars,# <---- add this# customize the rest as you needstructlog.processors.TimeStamper(fmt="iso"),structlog.stdlib.add_logger_name,structlog.stdlib.add_log_level,structlog.stdlib.PositionalArgumentsFormatter(),],},},...}Example outputsFlat lines file (logs/flat_lines.log)timestamp='2019-04-13T19:39:29.321453Z'level='info'event='request_started'logger='django_structlog.middlewares.request'request_id='c53dff1d-3fc5-4257-a78a-9a567c937561'user_id=1ip='0.0.0.0'request=GET/user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'timestamp='2019-04-13T19:39:29.345207Z'level='info'event='request_finished'logger='django_structlog.middlewares.request'request_id='c53dff1d-3fc5-4257-a78a-9a567c937561'user_id=1ip='0.0.0.0'code=200timestamp='2019-04-13T19:39:31.086155Z'level='info'event='request_started'logger='django_structlog.middlewares.request'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'request=POST/success_taskuser_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'timestamp='2019-04-13T19:39:31.089925Z'level='info'event='Enqueuing successful task'logger='django_structlog_demo_project.home.views'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'timestamp='2019-04-13T19:39:31.147590Z'level='info'event='task_enqueued'logger='django_structlog.middlewares.celery'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'child_task_id='6b11fd80-3cdf-4de5-acc2-3fd4633aa654'timestamp='2019-04-13T19:39:31.153081Z'level='info'event='This is a successful task'logger='django_structlog_demo_project.taskapp.celery'task_id='6b11fd80-3cdf-4de5-acc2-3fd4633aa654'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'timestamp='2019-04-13T19:39:31.160043Z'level='info'event='request_finished'logger='django_structlog.middlewares.request'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'code=201timestamp='2019-04-13T19:39:31.162372Z'level='info'event='task_succeed'logger='django_structlog.middlewares.celery'task_id='6b11fd80-3cdf-4de5-acc2-3fd4633aa654'request_id='3a8f801c-072b-4805-8f38-e1337f363ed4'user_id=1ip='0.0.0.0'result='None'Json file (logs/json.log){"request_id":"c53dff1d-3fc5-4257-a78a-9a567c937561","user_id":1,"ip":"0.0.0.0","request":"GET /","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36","event":"request_started","timestamp":"2019-04-13T19:39:29.321453Z","logger":"django_structlog.middlewares.request","level":"info"}{"request_id":"c53dff1d-3fc5-4257-a78a-9a567c937561","user_id":1,"ip":"0.0.0.0","code":200,"event":"request_finished","timestamp":"2019-04-13T19:39:29.345207Z","logger":"django_structlog.middlewares.request","level":"info"}{"request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","request":"POST /success_task","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36","event":"request_started","timestamp":"2019-04-13T19:39:31.086155Z","logger":"django_structlog.middlewares.request","level":"info"}{"request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","event":"Enqueuing successful task","timestamp":"2019-04-13T19:39:31.089925Z","logger":"django_structlog_demo_project.home.views","level":"info"}{"request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","child_task_id":"6b11fd80-3cdf-4de5-acc2-3fd4633aa654","event":"task_enqueued","timestamp":"2019-04-13T19:39:31.147590Z","logger":"django_structlog.middlewares.celery","level":"info"}{"task_id":"6b11fd80-3cdf-4de5-acc2-3fd4633aa654","request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","event":"This is a successful task","timestamp":"2019-04-13T19:39:31.153081Z","logger":"django_structlog_demo_project.taskapp.celery","level":"info"}{"request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","code":201,"event":"request_finished","timestamp":"2019-04-13T19:39:31.160043Z","logger":"django_structlog.middlewares.request","level":"info"}{"task_id":"6b11fd80-3cdf-4de5-acc2-3fd4633aa654","request_id":"3a8f801c-072b-4805-8f38-e1337f363ed4","user_id":1,"ip":"0.0.0.0","result":"None","event":"task_succeed","timestamp":"2019-04-13T19:39:31.162372Z","logger":"django_structlog.middlewares.celery","level":"info"}Upgrade GuideUpgrading to 7.0+The dependencydjango-ipwarewas upgraded to version 6. This library is used to retrieve the request’s ip address.Version 6 may have somebreaking changesif you did customizations.It should not affect most of the users but if you did some customizations, you might need to update your configurations.Upgrading to 6.0+Minimum requirementsrequires python 3.8+Changes to doAdddjango_structlogto installed appINSTALLED_APP=[# ..."django_structlog",# ...]Make sure you usedjango_structlog.middlewares.RequestMiddlewareIf you used any of the experimental async or sync middlewares, you do not need to anymore.
Make sure you usedjango_structlog.middlewares.RequestMiddlewareinstead of any of the other request middlewares commented below:MIDDLEWARE+=[# "django_structlog.middlewares.request_middleware_router", # <- remove# "django_structlog.middlewares.requests.SyncRequestMiddleware", # <- remove# "django_structlog.middlewares.requests.AsyncRequestMiddleware", # <- remove"django_structlog.middlewares.RequestMiddleware",# <- make sure you use this one](If you use celery) Make sure you useDJANGO_STRUCTLOG_CELERY_ENABLED = TrueIt is only applicable if you use celery integration.django_structlog.middlewares.CeleryMiddlewarehas been remove in favor of a django settings.MIDDLEWARE+=["django_structlog.middlewares.RequestMiddleware",# "django_structlog.middlewares.CeleryMiddleware",  # <- remove this]DJANGO_STRUCTLOG_CELERY_ENABLED=True# <-- add thisUpgrading to 5.0+Minimum requirementsrequires asgiref 3.6+Upgrading to 4.0+django-structlogdrops support of django below 3.2.Minimum requirementsrequires django 3.2+requires python 3.7+requires structlog 21.4.0+(optionally) requires celery 5.1+Changes if you useceleryYou can now installdjango-structlogexplicitly withceleryextra in order to validate the compatibility with your version ofcelery.django-structlog[celery]==4.0.0SeeInstalling “Extras”for more information about thispipfeature.Upgrading to 3.0+django-structlognow usestructlog.contextvars.bind_contextvarsinstead ofthreadlocal.Minimum requirementsrequires python 3.7+requires structlog 21.4.0+Changes you need to do1. Update structlog settingsaddstructlog.contextvars.merge_contextvarsas firstprocessorsremovecontext_class=structlog.threadlocal.wrap_dict(dict),(if you use standard loggers) addstructlog.contextvars.merge_contextvarsinforeign_pre_chain(if you use standard loggers) removedjango_structlog.processors.inject_context_dict,structlog.configure(processors=[structlog.contextvars.merge_contextvars,# <---- add thisstructlog.stdlib.filter_by_level,structlog.processors.TimeStamper(fmt="iso"),structlog.stdlib.add_logger_name,structlog.stdlib.add_log_level,structlog.stdlib.PositionalArgumentsFormatter(),structlog.processors.StackInfoRenderer(),structlog.processors.format_exc_info,structlog.processors.UnicodeDecoder(),structlog.stdlib.ProcessorFormatter.wrap_for_formatter,],# context_class=structlog.threadlocal.wrap_dict(dict), # <---- remove thislogger_factory=structlog.stdlib.LoggerFactory(),cache_logger_on_first_use=True,)# If you use standard loggingLOGGING={"version":1,"disable_existing_loggers":False,"formatters":{"json_formatter":{"()":structlog.stdlib.ProcessorFormatter,"processor":structlog.processors.JSONRenderer(),"foreign_pre_chain":[structlog.contextvars.merge_contextvars,# <---- add this# django_structlog.processors.inject_context_dict, # <---- remove thisstructlog.processors.TimeStamper(fmt="iso"),structlog.stdlib.add_logger_name,structlog.stdlib.add_log_level,structlog.stdlib.PositionalArgumentsFormatter(),],},},...}2. Replace alllogger.bindwithstructlog.contextvars.bind_contextvars@receiver(bind_extra_request_metadata)defbind_domain(request,logger,**kwargs):current_site=get_current_site(request)# logger.bind(domain=current_site.domain)structlog.contextvars.bind_contextvars(domain=current_site.domain)Upgrading to 2.0+django-structlogwas originally developed using the debug configurationExceptionPrettyPrinterwhich led to incorrect handling of exception.removestructlog.processors.ExceptionPrettyPrinter(),of your processors.make sure you havestructlog.processors.format_exc_info,in your processors if you want appropriate exception logging.Running the testsNote: For the moment redis is needed to run the tests. The easiest way is to start docker demo’s redis.dockercomposeup-dredispipinstall-rrequirements.txtenvCELERY_BROKER_URL=redis://0.0.0.0:6379DJANGO_SETTINGS_MODULE=config.settings.testpytesttest_appenvCELERY_BROKER_URL=redis://0.0.0.0:6379DJANGO_SETTINGS_MODULE=config.settings.test_demo_apppytestdjango_structlog_demo_projectdockercomposestopredisDemo appdockercomposeup--buildOpenhttp://127.0.0.1:8000/in your browser.Navigate while looking into the log files and shell’s output.AuthorsJules Robichaud-Gagnon-Initial work-jrobichaudSee also the list ofcontributorswho participated in this project.AcknowledgmentsBig thanks to@ferdfor hisbad opinionsthat inspired the author enough to spend time on this library.This issuehelped the author to figure out how to integratestructlogin Django.This stack overflow questionwas also helpful.LicenseThis project is licensed under the MIT License - see theLICENSEfile for details