Repository: jazzband/django-debug-toolbar

Issue:
Memory leak from logging panel, even when debug_toolbar middleware is not active
If `debug_toolbar` is ever imported (e.g. to add the URL) then it will set up its collection log handler (https://github.com/jazzband/django-debug-toolbar/blob/main/debug_toolbar/panels/logging.py#L69)

This happens even if the middleware is not active.

This leads to high memory usage by collecting and retaining all log entries from the root logger.

This is obviously desirable for request handling to display the logs in the toolbar, where the middleware is active, but we encountered this with a django command consuming ever more memory as it ran.

We removed it from `INSTALLED_APPS` and `MIDDLEWARE` using an ENV var to exclude it, but we had not removed it from `urlpatterns` and so `debug_toolbar.panels.logging` was still being loaded and was adding its collection logger.

Similarly, if the logger panel is in `DISABLE_PANELS` the logger is still added. It is only if the panel is not present in `DEBUG_TOOLBAR_PANELS` that it is never loaded and the collection logger is not set up.

We now exclude debug\_toolbar from our `urlpatterns` too, however it would be good if debug\_toolbar only set up this collection logger when it is actually active, rather than always adding it when the module is loaded.

---

This doesn't appear to be #906, which related explicitly to threading, but has been referenced and worked around by reef-technologies/cookiecutter-rt-django#81 in a similar way to our workaround.
Enabling debug_toolbar seems to break logging from management commands
When I add `debug_toolbar` to my `INSTALLED_APPS`, calls to `logging` `logger` seems no longer work from within a management command.



Discussion:

Can you please provide us with a test case?

https://www.dropbox.com/s/oifjoqqxsrizzew/debug_toolbar_issue.zip?dl=0
the management command is `check_log`. You can see that toggling the `debug_toolbar` app in the `settings.py` file changes the logging behavior.

This may be related to 7d29b40c70dbc1f0996b9342254f32705eba1b67.

I'm not sure if this is related to this issue (I suspect it is), but logging seems completely broken for me with DDT 1.3.0. I have numerous `logging.debug` calls scattered throughout my code, and DDT is no longer picking them up. I reverted to 1.2.2, and logging resumes working.

I'm using Python 3.4.x and Django 1.7.7.

If you want me to create a new issue, let me know.

@gfairchild Do you know what your `root.logging.level` is set to?

@tim-schilling, I'm sorry, but I googled around and can't find any references to `root.logging.level`. What is that, and how do I report my value to you?

Actually, it looks like `logging.getLogger().getEffectiveLevel()` gets that value. I missed a SO thread on it. That function returns 0, which tells me my logger's root logging level is `NOTSET` according to https://docs.python.org/3.4/library/logging.html#levels.

Sorry about that. I meant `logging.root.level`. I typed it out wrong. I'll try to recreate your scenario.

I just updated to Django 1.8, and logging still isn't working, so I'm guessing this is a DDT bug.

Yes, i catch similar bug: "No messages logged" in logging panel.
After update from Django 1.7 (logging working well) to 1.8 version.

@gfairchild @dkrnl Can you try adding the following code to your settings file?

```
import logging
logging.root.setLevel(logging.NOTSET)
```

That seems to have fixed the issue!

Glad it worked for you! That problem is actually different than the
original issue which I'm still looking into.

@cancan101 Were you using `BasicConfig` in your actual code? If so, then I don't think this is a bug with the toolbar. The reason being is that `BasicConfig` will only add a `StreamHandler` if there are no other handlers on root. The toolbar adds a handler, thus it doesn't create the `StreamHandler` and you don't see the output in the console. [Here are the docs on it](https://docs.python.org/2/library/logging.html#logging.basicConfig).

You can get the logging to work for your example if you add a django handler for that command. The code is:

```
import sys
LOGGING = {
    ...
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'stream': sys.stdout,
        }
    },
    'loggers': {
        'app.management.commands.check_log': {
            'handlers': ['console'],
            'level': 'INFO',
            'propagate': True,
        },
    }
}
```

Interesting. 

I wonder though why I don't see this same issue with other third party apps or even with Django itself. Perhaps [the call that django-debug-toolbar makes to add a Hander needs](https://github.com/django-debug-toolbar/django-debug-toolbar/blob/7d29b40c70dbc1f0996b9342254f32705eba1b67/debug_toolbar/panels/logging.py#L53) to be deferred?

It appears that the handler is added so early that I no matter where I put the call to `logging.basicConfig` in my code, it I am have included the `django-debug-toolbar`, I am always too late.

Yes, that would be the only way to solve your particular problem within the toolbar.

I know my issue above turned out not to be related to this issue, but I figured out why I needed to add this:

```
import logging
logging.root.setLevel(logging.NOTSET)
```

to my settings file. The reason is because I was logging using the logging module directly; I wasn't getting the appropriate logger. In other words, I was doing this:

```
import logging
logging.debug('message')
```

when I should have been doing this:

```
import logging
logger = logging.getLogger(__name__)
logger.debug('message')
```

DDT doesn't pick up on messages logged using the first method, but it does pick up on messages using the second method. If I use the logging module appropriately by calling `getLogger`, I don't need those 2 lines of code in the settings file. Just wanted to throw this out there in case anyone else encounters this issue.

Hi Everybody,
I'm having the same issue but can't make it that the logs appear in the logging panel.

When DjDT is not activated, my logs shows perfectly in the terminal and are written to my log file. But as soon as I activate DjDT, my app logs are not showing in the terminal anymore and nothing appears in the logging panel of DjDT... all the logs are still written to the file though

I've tried to add a handler for the command mentionned by @tim-schilling without success. Any help/hints would be greatly appreciated (if I can make logging works, DjDT would really be the ultimate debugging tool for Django)

Here are my settings for logging:

``` python
LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format' : "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt' : "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '%(levelname)s %(message)s'
        },
    },
    'handlers': {
        'file': {
            'level': 'DEBUG',
            'class': 'logging.FileHandler',
            'filename': 'mysite.log',
            'formatter': 'verbose'
        },
    },
    'loggers': {
        'django': {
            'handlers':['file'],
            'propagate': True,
            'level':'INFO',
        },
        'fullText_extraction': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    }
}
```

Can you try specifying the full path to your management command for the loggers key?

I'm not sure to understand what you mean @tim-schilling ... you want me to run manage.py with full path? or smth else? sorry about that...

I mean run it with:

```
LOGGING = {
    ...
    'loggers': {
        ...
        'python.path.to.fullText_extraction': {
            'handlers': ['file'],
            'level': 'INFO',
        },
    }
}
```

But replacing the `python.path.to.` with the path that's relevant for your project.

If I do that @tim-schilling , nothing happens anymore --> no logs in the terminal, nothing in the logging panel of DjDT and logs are not written in my log file anymore.

Hi @tim-schilling 

I am facing same issue. I tried to repro with minimal app. Issue occurs  when I add debug toolbar to installed apps.
https://github.com/tripples/logissue_with_debug_toolbar

I can still reproduce this issue with the latest version of `django-debug-toolbar` and Python 3.7.

When I include `django-debug-toolbar` in my `INSTALLED_APPS`, then the logging is broken. For example, when I run `python manage.py run_huey`, I can't see any logs. So, I did some debugging, and it turns out that it has something to do with this part of code.

---

When `django-debug-toolbar` is **not** in my `INSTALLED_APPS` and I run `python manage.py run_huey` (with logging level DEBUG), this code gets executed and everything works fine:

File `/python3.7/logging/__init__.py`
![bildschirmfoto_2018-08-19_10-38-18](https://user-images.githubusercontent.com/8258609/44307096-b4e60800-a39c-11e8-9bec-7625cbbb3360.png)

However, when `django-debug-toolbar` is in my `INSTALLED_APPS`, the try-block does not get executed and so, the loglevel "debug" is ignored and not being set. Instead, loglevel is set to "warning".

![bildschirmfoto_2018-08-19_10-26-56](https://user-images.githubusercontent.com/8258609/44307124-0d1d0a00-a39d-11e8-893c-34a4ff421959.png)



Resolving patch:
diff --git a/debug_toolbar/panels/logging.py b/debug_toolbar/panels/logging.py
deleted file mode 100644
--- a/debug_toolbar/panels/logging.py
+++ /dev/null
@@ -1,101 +0,0 @@
-import datetime
-import logging
-
-from django.utils.translation import gettext_lazy as _, ngettext
-
-from debug_toolbar.panels import Panel
-from debug_toolbar.utils import ThreadCollector
-
-try:
-    import threading
-except ImportError:
-    threading = None
-
-MESSAGE_IF_STRING_REPRESENTATION_INVALID = "[Could not get log message]"
-
-
-class LogCollector(ThreadCollector):
-    def collect(self, item, thread=None):
-        # Avoid logging SQL queries since they are already in the SQL panel
-        # TODO: Make this check whether SQL panel is enabled
-        if item.get("channel", "") == "django.db.backends":
-            return
-        super().collect(item, thread)
-
-
-class ThreadTrackingHandler(logging.Handler):
-    def __init__(self, collector):
-        logging.Handler.__init__(self)
-        self.collector = collector
-
-    def emit(self, record):
-        try:
-            message = record.getMessage()
-        except Exception:
-            message = MESSAGE_IF_STRING_REPRESENTATION_INVALID
-
-        record = {
-            "message": message,
-            "time": datetime.datetime.fromtimestamp(record.created),
-            "level": record.levelname,
-            "file": record.pathname,
-            "line": record.lineno,
-            "channel": record.name,
-        }
-        self.collector.collect(record)
-
-
-# Preserve Python's fallback log mechanism before adding ThreadTrackingHandler.
-
-# If the root logger has no handlers attached then everything that reaches it goes
-# to the "handler of last resort".
-# So a Django app that doesn't explicitly configure the root logger actually logs
-# through logging.lastResort.
-# However, logging.lastResort is not used after ThreadTrackingHandler gets added to
-# the root logger below. This means that users who have LoggingPanel enabled might
-# find their logs are gone from their app as soon as they install DDT.
-# Explicitly adding logging.lastResort to logging.root's handler sidesteps this
-# potential confusion.
-# Note that if root has already been configured, or logging.lastResort has been
-# removed, then the configuration is unchanged, so users who configured their
-# logging aren't exposed to the opposite confusion of seeing extra log lines from
-# their app.
-if not logging.root.hasHandlers() and logging.lastResort is not None:
-    logging.root.addHandler(logging.lastResort)
-
-# We don't use enable/disable_instrumentation because logging is global.
-# We can't add thread-local logging handlers. Hopefully logging is cheap.
-
-collector = LogCollector()
-logging_handler = ThreadTrackingHandler(collector)
-logging.root.addHandler(logging_handler)
-
-
-class LoggingPanel(Panel):
-    template = "debug_toolbar/panels/logging.html"
-
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        self._records = {}
-
-    nav_title = _("Logging")
-
-    @property
-    def nav_subtitle(self):
-        stats = self.get_stats()
-        record_count = len(stats["records"]) if stats else None
-        return ngettext("%(count)s message", "%(count)s messages", record_count) % {
-            "count": record_count
-        }
-
-    title = _("Log messages")
-
-    def process_request(self, request):
-        collector.clear_collection()
-        return super().process_request(request)
-
-    def generate_stats(self, request, response):
-        records = collector.get_collection()
-        self._records[threading.current_thread()] = records
-        collector.clear_collection()
-        self.record_stats({"records": records})
diff --git a/debug_toolbar/settings.py b/debug_toolbar/settings.py
--- a/debug_toolbar/settings.py
+++ b/debug_toolbar/settings.py
@@ -1,3 +1,4 @@
+import warnings
 from functools import lru_cache
 
 from django.conf import settings
@@ -64,7 +65,6 @@ def get_config():
     "debug_toolbar.panels.templates.TemplatesPanel",
     "debug_toolbar.panels.cache.CachePanel",
     "debug_toolbar.panels.signals.SignalsPanel",
-    "debug_toolbar.panels.logging.LoggingPanel",
     "debug_toolbar.panels.redirects.RedirectsPanel",
     "debug_toolbar.panels.profiling.ProfilingPanel",
 ]
@@ -76,6 +76,14 @@ def get_panels():
         PANELS = list(settings.DEBUG_TOOLBAR_PANELS)
     except AttributeError:
         PANELS = PANELS_DEFAULTS
+
+    logging_panel = "debug_toolbar.panels.logging.LoggingPanel"
+    if logging_panel in PANELS:
+        PANELS = [panel for panel in PANELS if panel != logging_panel]
+        warnings.warn(
+            f"Please remove {logging_panel} from your DEBUG_TOOLBAR_PANELS setting.",
+            DeprecationWarning,
+        )
     return PANELS
 
 


Test patch:
diff --git a/tests/panels/test_history.py b/tests/panels/test_history.py
--- a/tests/panels/test_history.py
+++ b/tests/panels/test_history.py
@@ -77,7 +77,6 @@ class HistoryViewsTestCase(IntegrationTestCase):
         "TemplatesPanel",
         "CachePanel",
         "SignalsPanel",
-        "LoggingPanel",
         "ProfilingPanel",
     }
 
diff --git a/tests/panels/test_logging.py b/tests/panels/test_logging.py
deleted file mode 100644
--- a/tests/panels/test_logging.py
+++ /dev/null
@@ -1,96 +0,0 @@
-import logging
-from unittest.mock import patch
-
-from debug_toolbar.panels.logging import (
-    MESSAGE_IF_STRING_REPRESENTATION_INVALID,
-    collector,
-)
-
-from ..base import BaseTestCase
-from ..views import regular_view
-
-
-class LoggingPanelTestCase(BaseTestCase):
-    panel_id = "LoggingPanel"
-
-    def setUp(self):
-        super().setUp()
-        self.logger = logging.getLogger(__name__)
-        collector.clear_collection()
-
-        # Assume the root logger has been configured with level=DEBUG.
-        # Previously DDT forcefully set this itself to 0 (NOTSET).
-        logging.root.setLevel(logging.DEBUG)
-
-    def test_happy_case(self):
-        def view(request):
-            self.logger.info("Nothing to see here, move along!")
-            return regular_view(request, "logging")
-
-        self._get_response = view
-        response = self.panel.process_request(self.request)
-        self.panel.generate_stats(self.request, response)
-        records = self.panel.get_stats()["records"]
-
-        self.assertEqual(1, len(records))
-        self.assertEqual("Nothing to see here, move along!", records[0]["message"])
-
-    def test_formatting(self):
-        def view(request):
-            self.logger.info("There are %d %s", 5, "apples")
-            return regular_view(request, "logging")
-
-        self._get_response = view
-        response = self.panel.process_request(self.request)
-        self.panel.generate_stats(self.request, response)
-        records = self.panel.get_stats()["records"]
-
-        self.assertEqual(1, len(records))
-        self.assertEqual("There are 5 apples", records[0]["message"])
-
-    def test_insert_content(self):
-        """
-        Test that the panel only inserts content after generate_stats and
-        not the process_request.
-        """
-
-        def view(request):
-            self.logger.info("café")
-            return regular_view(request, "logging")
-
-        self._get_response = view
-        response = self.panel.process_request(self.request)
-        # ensure the panel does not have content yet.
-        self.assertNotIn("café", self.panel.content)
-        self.panel.generate_stats(self.request, response)
-        # ensure the panel renders correctly.
-        content = self.panel.content
-        self.assertIn("café", content)
-        self.assertValidHTML(content)
-
-    def test_failing_formatting(self):
-        class BadClass:
-            def __str__(self):
-                raise Exception("Please not stringify me!")
-
-        def view(request):
-            # should not raise exception, but fail silently
-            self.logger.debug("This class is misbehaving: %s", BadClass())
-            return regular_view(request, "logging")
-
-        self._get_response = view
-        response = self.panel.process_request(self.request)
-        self.panel.generate_stats(self.request, response)
-        records = self.panel.get_stats()["records"]
-
-        self.assertEqual(1, len(records))
-        self.assertEqual(
-            MESSAGE_IF_STRING_REPRESENTATION_INVALID, records[0]["message"]
-        )
-
-    @patch("sys.stderr")
-    def test_fallback_logging(self, mock_stderr):
-        # make sure the log reaches stderr even though logging set up
-        # its own handler during its import
-        self.logger.warning("hello")
-        mock_stderr.write.assert_called_once_with("hello\n")