Repository: jazzband/django-debug-toolbar

Issue:
django-debug-toolbar is crashing postgresql by not closing connections
I am using Django 4.2.2, django-debug-toolbar 4.1.0, Python 3.11 and Postgresql 15.2
While developing with DEBUG=True and django-debug-toolbar activated in settings.py, after about 100 pages displayed, Postgresql is crashing with the messages.

```
FATAL: remaining connection slots are reserved for non-replication superuser connections
```

My max connexion is set at 100 on my Postgresql server. Nevertheless despite I am alone to request pages to Django ``runserver`` (there is no concurrency), the opened postgresql connections count is always rising up to the max, then the above message appears.

To check open connections I do : 

```
SELECT count(*) AS total_connections FROM pg_stat_activity;
```

If I use chrome to consult pages slowly or just a bench with ``ab -n 200`` : I have the problem anyway.
If I restart ``runserver``, I have again the right to read about 100 pages, but after it crashes.
The only solution I found is to remove django-debug-toolbar from settings.py : then ``total_connections`` is stable.

I guess django-debug-toolbar is not closing DB connections in some way : could you give me a workaround because your package is an absolute must.


Discussion:
Thank you for opening an issue. What version of psycopg are you using?

Are you able to investigate where in the toolbar it might be leaving a
connection open? I suspect it's mogrify, but that's only a hunch.

Another thing that would help immensely is creating a reproducible example.

On Thu, Jun 22, 2023, 8:54 AM Eric Lapouyade ***@***.***>
wrote:

> I am using Django 4.2.2, django-debug-toolbar 4.1.0, Python 3.11 and
> Postgresql 15.2
> While developing with DEBUG=True and django-debug-toolbar activated in
> settings.py, after about 100 pages displayed, Postgresql is crashing with
> the messages.
>
> FATAL: remaining connection slots are reserved for non-replication superuser connections
>
> My max connexion is set at 100 on my Postgresql server. Nevertheless
> despite I am alone to request pages to Django runserver (there is no
> concurrency), the opened postgresql connections count is always rising up
> to the max, then the above message appears.
>
> To check open connections I do :
>
> SELECT count(*) AS total_connections FROM pg_stat_activity;
>
> If I use chrome to consult pages slowly or just a bench with ab -n 200 :
> I have the problem anyway.
> If I restart runserver, I have again the right to read about 100 pages,
> but after it crashes.
> The only solution I found is to remove django-debug-toolbar from
> settings.py : then total_connections is stable.
>
> I guess django-debug-toolbar is not closing DB connections in some way :
> could you give me a workaround because your package is an absolute must.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/jazzband/django-debug-toolbar/issues/1799>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAJYZP6BYMNCH5ZBHYWO5Z3XMRFBBANCNFSM6AAAAAAZQH54FQ>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>

I checked I am using the very latest psycopg package on pypi. I will try to build a minimal project to reproduce the problem without all packages I use.
I did not have investigated yet into the django-debug-toolbar code, I will try ...
I made a minimal Django project and I reproduced the problem and I found some clues : 

It comes from the ``StaticFilesPanel.generate_stats()`` : It uses ``ThreadCollector`` to store some data in RAM that persist accross all requests. It uses ``threading.current_thread()`` as an index to a dictionary to store data.
Unfortunately, I noticed by printing ``id(thread)`` that despite ``runserver`` is mono-threaded, the thread object ID is changing almost all the time **after a while**.  I tried ``gunicorn`` : it does not have such a problem : the thread ID is always the same if I start only one worker and postgresql is not crashing.

I tried to force ``thread = 1`` like that : 

```
class ThreadCollector:
    def __init__(self):
        if threading is None:
            raise NotImplementedError(
                "threading module is not available, "
                "this panel cannot be used without it"
            )
        self.collections = {}  # a dictionary that maps threads to collections

    def get_collection(self, thread=None):
        """
        Returns a list of collected items for the provided thread, of if none
        is provided, returns a list for the current thread.
        """
        thread = 1
        if thread is None:
            thread = threading.current_thread()
        if thread not in self.collections:
            self.collections[thread] = []
        return self.collections[thread]

    def clear_collection(self, thread=None):
        thread = 1
        if thread is None:
            thread = threading.current_thread()
        if thread in self.collections:
            del self.collections[thread]

    def collect(self, item, thread=None):
        thread = 1
        self.get_collection(thread).append(item)
```

This is much better, but not perfect, I have still 30 on 200 requests that makes postgresql crash.
The relation between ``ThreadCollector`` and ``Postgresql`` is that - I guess - some data indirectly references a DB connection somewhere in staticfiles panel, as thread ID is changing, cleaning is not done in ``self.collections`` and the connection object is never processed by the garbage collector. 

I made a lot of dichotomy, this is when I by-passed ``StaticFilesPanel.generate_stats()`` that the problem totally disappeared.  
Can you share the reproducible django project?
Yes : 
```
git clone https://github.com/elapouya/ddtb-debug.git
```

Resolving patch:
diff --git a/debug_toolbar/panels/staticfiles.py b/debug_toolbar/panels/staticfiles.py
--- a/debug_toolbar/panels/staticfiles.py
+++ b/debug_toolbar/panels/staticfiles.py
@@ -1,3 +1,5 @@
+import contextlib
+from contextvars import ContextVar
 from os.path import join, normpath
 
 from django.conf import settings
@@ -7,12 +9,6 @@
 from django.utils.translation import gettext_lazy as _, ngettext
 
 from debug_toolbar import panels
-from debug_toolbar.utils import ThreadCollector
-
-try:
-    import threading
-except ImportError:
-    threading = None
 
 
 class StaticFile:
@@ -33,15 +29,8 @@ def url(self):
         return storage.staticfiles_storage.url(self.path)
 
 
-class FileCollector(ThreadCollector):
-    def collect(self, path, thread=None):
-        # handle the case of {% static "admin/" %}
-        if path.endswith("/"):
-            return
-        super().collect(StaticFile(path), thread)
-
-
-collector = FileCollector()
+# This will collect the StaticFile instances across threads.
+used_static_files = ContextVar("djdt_static_used_static_files")
 
 
 class DebugConfiguredStorage(LazyObject):
@@ -65,15 +54,16 @@ def _setup(self):
             configured_storage_cls = get_storage_class(settings.STATICFILES_STORAGE)
 
         class DebugStaticFilesStorage(configured_storage_cls):
-            def __init__(self, collector, *args, **kwargs):
-                super().__init__(*args, **kwargs)
-                self.collector = collector
-
             def url(self, path):
-                self.collector.collect(path)
+                with contextlib.suppress(LookupError):
+                    # For LookupError:
+                    # The ContextVar wasn't set yet. Since the toolbar wasn't properly
+                    # configured to handle this request, we don't need to capture
+                    # the static file.
+                    used_static_files.get().append(StaticFile(path))
                 return super().url(path)
 
-        self._wrapped = DebugStaticFilesStorage(collector)
+        self._wrapped = DebugStaticFilesStorage()
 
 
 _original_storage = storage.staticfiles_storage
@@ -97,7 +87,7 @@ def title(self):
     def __init__(self, *args, **kwargs):
         super().__init__(*args, **kwargs)
         self.num_found = 0
-        self._paths = {}
+        self.used_paths = []
 
     def enable_instrumentation(self):
         storage.staticfiles_storage = DebugConfiguredStorage()
@@ -120,18 +110,22 @@ def nav_subtitle(self):
         ) % {"num_used": num_used}
 
     def process_request(self, request):
-        collector.clear_collection()
-        return super().process_request(request)
+        reset_token = used_static_files.set([])
+        response = super().process_request(request)
+        # Make a copy of the used paths so that when the
+        # ContextVar is reset, our panel still has the data.
+        self.used_paths = used_static_files.get().copy()
+        # Reset the ContextVar to be empty again, removing the reference
+        # to the list of used files.
+        used_static_files.reset(reset_token)
+        return response
 
     def generate_stats(self, request, response):
-        used_paths = collector.get_collection()
-        self._paths[threading.current_thread()] = used_paths
-
         self.record_stats(
             {
                 "num_found": self.num_found,
-                "num_used": len(used_paths),
-                "staticfiles": used_paths,
+                "num_used": len(self.used_paths),
+                "staticfiles": self.used_paths,
                 "staticfiles_apps": self.get_staticfiles_apps(),
                 "staticfiles_dirs": self.get_staticfiles_dirs(),
                 "staticfiles_finders": self.get_staticfiles_finders(),
diff --git a/debug_toolbar/utils.py b/debug_toolbar/utils.py
--- a/debug_toolbar/utils.py
+++ b/debug_toolbar/utils.py
@@ -14,12 +14,6 @@
 
 from debug_toolbar import _stubs as stubs, settings as dt_settings
 
-try:
-    import threading
-except ImportError:
-    threading = None
-
-
 _local_data = Local()
 
 
@@ -357,33 +351,3 @@ def get_stack_trace(*, skip=0):
 def clear_stack_trace_caches():
     if hasattr(_local_data, "stack_trace_recorder"):
         del _local_data.stack_trace_recorder
-
-
-class ThreadCollector:
-    def __init__(self):
-        if threading is None:
-            raise NotImplementedError(
-                "threading module is not available, "
-                "this panel cannot be used without it"
-            )
-        self.collections = {}  # a dictionary that maps threads to collections
-
-    def get_collection(self, thread=None):
-        """
-        Returns a list of collected items for the provided thread, of if none
-        is provided, returns a list for the current thread.
-        """
-        if thread is None:
-            thread = threading.current_thread()
-        if thread not in self.collections:
-            self.collections[thread] = []
-        return self.collections[thread]
-
-    def clear_collection(self, thread=None):
-        if thread is None:
-            thread = threading.current_thread()
-        if thread in self.collections:
-            del self.collections[thread]
-
-    def collect(self, item, thread=None):
-        self.get_collection(thread).append(item)


Test patch: