Repository: jazzband/django-debug-toolbar

Issue:
Django logging not working when djdt is enabled
I recently noticed that django default logging is not working when I enable djdt.
django-debug-toolbar==4.1.0
Django==4.2.3

Is it possible to have SQL pannel and logging for sql queries work together?

Without djdt
<img width="919" alt="Screenshot 2023-07-10 at 01 00 07" src="https://github.com/jazzband/django-debug-toolbar/assets/10248817/2207582e-01ad-4f9c-bbfd-acd1b48d9839">

With djdt
<img width="914" alt="Screenshot 2023-07-10 at 01 02 08" src="https://github.com/jazzband/django-debug-toolbar/assets/10248817/d5967fc9-2db4-4efd-9b79-3cf153e1bd71">



Discussion:
Hi @forrestkouakou, can you create a reproducible environment for this?
I've just dug deep into django's guts to figure out why our console logging isn't working either. It looks like it's caused by the DjDTCursorWrapper (`debug_toolbar/panels/sql/tracking:wrap_cursor()`) overriding the default CursorDebugWrapper that django installs based on `settings.DEBUG` or an option set somehow to force it. When `wrap_cursor()` is called by djdt, it's returning an instance of `NormalCursorWrapper` which eventually extends `CursorWrapper` (not `CursorDebugWrapper` where the logging functions are).

My guess is that the resolution to this is that `DjDTCursorWrapper` needs to be mindful of what cursor django has installed, and if it's installed `CursorDebugWrapper` instead of `CursorWrapper` then it should inherit that instead. Something can probably modify `DjDTCursorWrapper.__bases__` or use a function in the class inheritance instead in order to dynamically inherit the correct cursor wrapper class.
@tim-schilling based on this would you support a revert of e7575e87dc9e2d2560b87d6fd5a123b9398cbd34?
@dannosaur thank you for that investigation! That's super helpful. Do you have a reproducible use case?

@living180 I think I'm on board for a revert, though I'd prefer to roll forward a fix.
@tim-schilling yes, I can reproduce the issue in my project.
@dannosaur Sorry, I meant can you create a way for me to reproduce the error so that I can investigate and attempt a fix?
@tim-schilling lol, sorry! I'll try and put something together quickly tomorrow that reproduces the issue. I don't think it's a complex setup though - django 4.2, any version of psycopg (2 or 3), and djdt 4.1. Leave it with me, I'll post back tomorrow with a git repo.

Edit: it's probably the case with all the database drivers, but I can only test with postgres. I don't think the engine is relevant though.
No worries. I can set up postgres on my end. Having the ability to pull a repo, set up a database and reproduce the problem makes this much, much easier to investigate. Otherwise I'm attempting random things to try to reproduce the behavior. I appreciate your work on this so far, thank you!
@tim-schilling here you go: https://github.com/dannosaur/djdttest

The django admin is installed with a user. Login `dan`/`password`. Observe the terminal running runserver - there should be some queries printed directed at `auth_user` table (there isn't though, demonstrating this issue). In settings.py I put a `DISABLE_PANELS` section with a commented out item. If you uncomment that line (158) and reload the admin user list page, you'll see the SQL query in the console.

LMK if you need anything else, happy to help!
Thank you @dannosaur, I was able to reproduce this.

Resolving patch:
diff --git a/debug_toolbar/panels/sql/tracking.py b/debug_toolbar/panels/sql/tracking.py
--- a/debug_toolbar/panels/sql/tracking.py
+++ b/debug_toolbar/panels/sql/tracking.py
@@ -5,7 +5,6 @@
 from time import perf_counter
 
 import django.test.testcases
-from django.db.backends.utils import CursorWrapper
 from django.utils.encoding import force_str
 
 from debug_toolbar.utils import get_stack_trace, get_template_info
@@ -60,34 +59,42 @@ def cursor(*args, **kwargs):
             cursor = connection._djdt_cursor(*args, **kwargs)
             if logger is None:
                 return cursor
-            wrapper = NormalCursorWrapper if allow_sql.get() else ExceptionCursorWrapper
-            return wrapper(cursor.cursor, connection, logger)
+            mixin = NormalCursorMixin if allow_sql.get() else ExceptionCursorMixin
+            return patch_cursor_wrapper_with_mixin(cursor.__class__, mixin)(
+                cursor.cursor, connection, logger
+            )
 
         def chunked_cursor(*args, **kwargs):
             # prevent double wrapping
             # solves https://github.com/jazzband/django-debug-toolbar/issues/1239
             logger = connection._djdt_logger
             cursor = connection._djdt_chunked_cursor(*args, **kwargs)
-            if logger is not None and not isinstance(cursor, DjDTCursorWrapper):
-                if allow_sql.get():
-                    wrapper = NormalCursorWrapper
-                else:
-                    wrapper = ExceptionCursorWrapper
-                return wrapper(cursor.cursor, connection, logger)
+            if logger is not None and not isinstance(cursor, DjDTCursorWrapperMixin):
+                mixin = NormalCursorMixin if allow_sql.get() else ExceptionCursorMixin
+                return patch_cursor_wrapper_with_mixin(cursor.__class__, mixin)(
+                    cursor.cursor, connection, logger
+                )
             return cursor
 
         connection.cursor = cursor
         connection.chunked_cursor = chunked_cursor
 
 
-class DjDTCursorWrapper(CursorWrapper):
+def patch_cursor_wrapper_with_mixin(base_wrapper, mixin):
+    class DjDTCursorWrapper(mixin, base_wrapper):
+        pass
+
+    return DjDTCursorWrapper
+
+
+class DjDTCursorWrapperMixin:
     def __init__(self, cursor, db, logger):
         super().__init__(cursor, db)
         # logger must implement a ``record`` method
         self.logger = logger
 
 
-class ExceptionCursorWrapper(DjDTCursorWrapper):
+class ExceptionCursorMixin(DjDTCursorWrapperMixin):
     """
     Wraps a cursor and raises an exception on any operation.
     Used in Templates panel.
@@ -97,7 +104,7 @@ def __getattr__(self, attr):
         raise SQLQueryTriggered()
 
 
-class NormalCursorWrapper(DjDTCursorWrapper):
+class NormalCursorMixin(DjDTCursorWrapperMixin):
     """
     Wraps a cursor and logs queries.
     """


Test patch:
diff --git a/tests/panels/test_sql.py b/tests/panels/test_sql.py
--- a/tests/panels/test_sql.py
+++ b/tests/panels/test_sql.py
@@ -2,12 +2,13 @@
 import datetime
 import os
 import unittest
-from unittest.mock import patch
+from unittest.mock import call, patch
 
 import django
 from asgiref.sync import sync_to_async
 from django.contrib.auth.models import User
 from django.db import connection, transaction
+from django.db.backends.utils import CursorDebugWrapper, CursorWrapper
 from django.db.models import Count
 from django.db.utils import DatabaseError
 from django.shortcuts import render
@@ -68,39 +69,59 @@ def test_recording_chunked_cursor(self):
         self.assertEqual(len(self.panel._queries), 1)
 
     @patch(
-        "debug_toolbar.panels.sql.tracking.NormalCursorWrapper",
-        wraps=sql_tracking.NormalCursorWrapper,
+        "debug_toolbar.panels.sql.tracking.patch_cursor_wrapper_with_mixin",
+        wraps=sql_tracking.patch_cursor_wrapper_with_mixin,
     )
-    def test_cursor_wrapper_singleton(self, mock_wrapper):
+    def test_cursor_wrapper_singleton(self, mock_patch_cursor_wrapper):
         sql_call()
-
         # ensure that cursor wrapping is applied only once
-        self.assertEqual(mock_wrapper.call_count, 1)
+        self.assertIn(
+            mock_patch_cursor_wrapper.mock_calls,
+            [
+                [call(CursorWrapper, sql_tracking.NormalCursorMixin)],
+                # CursorDebugWrapper is used if the test is called with `--debug-sql`
+                [call(CursorDebugWrapper, sql_tracking.NormalCursorMixin)],
+            ],
+        )
 
     @patch(
-        "debug_toolbar.panels.sql.tracking.NormalCursorWrapper",
-        wraps=sql_tracking.NormalCursorWrapper,
+        "debug_toolbar.panels.sql.tracking.patch_cursor_wrapper_with_mixin",
+        wraps=sql_tracking.patch_cursor_wrapper_with_mixin,
     )
-    def test_chunked_cursor_wrapper_singleton(self, mock_wrapper):
+    def test_chunked_cursor_wrapper_singleton(self, mock_patch_cursor_wrapper):
         sql_call(use_iterator=True)
 
         # ensure that cursor wrapping is applied only once
-        self.assertEqual(mock_wrapper.call_count, 1)
+        self.assertIn(
+            mock_patch_cursor_wrapper.mock_calls,
+            [
+                [call(CursorWrapper, sql_tracking.NormalCursorMixin)],
+                # CursorDebugWrapper is used if the test is called with `--debug-sql`
+                [call(CursorDebugWrapper, sql_tracking.NormalCursorMixin)],
+            ],
+        )
 
     @patch(
-        "debug_toolbar.panels.sql.tracking.NormalCursorWrapper",
-        wraps=sql_tracking.NormalCursorWrapper,
+        "debug_toolbar.panels.sql.tracking.patch_cursor_wrapper_with_mixin",
+        wraps=sql_tracking.patch_cursor_wrapper_with_mixin,
     )
-    async def test_cursor_wrapper_async(self, mock_wrapper):
+    async def test_cursor_wrapper_async(self, mock_patch_cursor_wrapper):
         await sync_to_async(sql_call)()
 
-        self.assertEqual(mock_wrapper.call_count, 1)
+        self.assertIn(
+            mock_patch_cursor_wrapper.mock_calls,
+            [
+                [call(CursorWrapper, sql_tracking.NormalCursorMixin)],
+                # CursorDebugWrapper is used if the test is called with `--debug-sql`
+                [call(CursorDebugWrapper, sql_tracking.NormalCursorMixin)],
+            ],
+        )
 
     @patch(
-        "debug_toolbar.panels.sql.tracking.NormalCursorWrapper",
-        wraps=sql_tracking.NormalCursorWrapper,
+        "debug_toolbar.panels.sql.tracking.patch_cursor_wrapper_with_mixin",
+        wraps=sql_tracking.patch_cursor_wrapper_with_mixin,
     )
-    async def test_cursor_wrapper_asyncio_ctx(self, mock_wrapper):
+    async def test_cursor_wrapper_asyncio_ctx(self, mock_patch_cursor_wrapper):
         self.assertTrue(sql_tracking.allow_sql.get())
         await sync_to_async(sql_call)()
 
@@ -116,7 +137,21 @@ async def task():
         await asyncio.create_task(task())
         # Because it was called in another context, it should not have affected ours
         self.assertTrue(sql_tracking.allow_sql.get())
-        self.assertEqual(mock_wrapper.call_count, 1)
+
+        self.assertIn(
+            mock_patch_cursor_wrapper.mock_calls,
+            [
+                [
+                    call(CursorWrapper, sql_tracking.NormalCursorMixin),
+                    call(CursorWrapper, sql_tracking.ExceptionCursorMixin),
+                ],
+                # CursorDebugWrapper is used if the test is called with `--debug-sql`
+                [
+                    call(CursorDebugWrapper, sql_tracking.NormalCursorMixin),
+                    call(CursorDebugWrapper, sql_tracking.ExceptionCursorMixin),
+                ],
+            ],
+        )
 
     def test_generate_server_timing(self):
         self.assertEqual(len(self.panel._queries), 0)