Repository: jazzband/django-debug-toolbar

Issue:
Duplicate queries not correctly detected
With django debug toolbar 2.2 (since 2.0) I see a lot of duplicated queries.
When turning on logging for sql queries, I see them only once, and when debugging, those queries are clearly executed only once.

I found out that this only happens when using chunked cursors, because debug toolbar wraps the cursor twice, leading to logging it twice and wrongly marking it as duplicated.

Please see this line:
https://github.com/jazzband/django-debug-toolbar/blob/2.2/debug_toolbar/panels/sql/tracking.py#L58

It wraps the cursor returned from `connection._djdt_chunked_cursor`.
For databases without a special implementation for chunked cursors (e.g. mysql), that method actually just runs `connection.cursor`:
https://github.com/django/django/blob/2.2.10/django/db/backends/base/base.py#L572

Which in turn wraps the cursor a second time:
https://github.com/jazzband/django-debug-toolbar/blob/2.2/debug_toolbar/panels/sql/tracking.py#L53






Discussion:
Do you have a test case that we can reproduce this with?
I am seeing the same, not sure how I can easily create a test case as I am seeing this in a project with a `models.py` which by itself is over 3000 lines...

It seems to only be happening for queries that involve django-filter to render dropdown/multiselect html widgets and consistenly show 2 queries, of which the first 1 is very fast:

![image](https://user-images.githubusercontent.com/4426050/83917477-404daa80-a777-11ea-8ac2-3a640fb37924.png)

Don't know if a stacktrace helps, but both queries have the exact same stacktrace:

```
/home/user/venv_dd/lib/python3.6/site-packages/django/contrib/staticfiles/handlers.py in __call__(65)
  return self.application(environ, start_response)
/home/user/venv_dd/lib/python3.6/site-packages/ddt_request_history/panels/request_history.py in patched_middleware_call(56)
  response = toolbar.process_request(request)
/home/user/venv_dd/lib/python3.6/site-packages/ddt_request_history/panels/request_history.py in process_request(195)
  return super().process_request(request)
/c/Data/django-DefectDojo/dojo/middleware.py in __call__(62)
  response = self.get_response(request)
/c/Data/django-DefectDojo/dojo/middleware.py in __call__(47)
  response = self.get_response(request)
/c/Data/django-DefectDojo/dojo/finding/views.py in open_findings(79)
  return findings(request, pid=pid, eid=eid, view=view, filter_name="Open", query_filter=OPEN_FINDINGS_QUERY)
/c/Data/django-DefectDojo/dojo/finding/views.py in findings(187)
  'jira_config': jira_config,
/home/user/venv_dd/lib/python3.6/site-packages/django/shortcuts.py in render(36)
  content = loader.render_to_string(template_name, context, request, using=using)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader.py in render_to_string(62)
  return template.render(context, request)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/backends/django.py in render(61)
  return self.template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(171)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(150)
  return compiled_parent._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(62)
  result = block.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/loader_tags.py in render(188)
  return template.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(173)
  return self._render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/test/utils.py in instrumented_test_render(96)
  return self.nodelist.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(937)
  bit = node.render_annotated(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/defaulttags.py in render(209)
  nodelist.append(node.render_annotated(context))
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_annotated(904)
  return self.render(context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render(993)
  return render_value_in_context(output, context)
/home/user/venv_dd/lib/python3.6/site-packages/django/template/base.py in render_value_in_context(972)
  value = str(value)
/home/user/venv_dd/lib/python3.6/site-packages/django/utils/html.py in <lambda>(388)
  klass.__str__ = lambda self: mark_safe(klass_str(self))
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in __str__(33)
  return self.as_widget()
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/boundfield.py in as_widget(93)
  renderer=self.form.renderer,
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in render(241)
  context = self.get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(678)
  context = super().get_context(name, value, attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in get_context(639)
  context['widget']['optgroups'] = self.optgroups(name, context['widget']['value'], attrs)
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/widgets.py in optgroups(587)
  for index, (option_value, option_label) in enumerate(self.choices):
/home/user/venv_dd/lib/python3.6/site-packages/django_filters/fields.py in __iter__(252)
  for value in iterable:
/home/user/venv_dd/lib/python3.6/site-packages/django/forms/models.py in __iter__(1137)
  for obj in queryset:
```

```
django-auditlog==0.4.7
django-filter==2.3.0
django-multiselectfield==0.1.12
django-crispy-forms==1.9.1
django_extensions==2.2.9
Django==2.2.13
uWSGI==2.0.18
vobject==0.9.6.1
whitenoise>=4.1.2
django-cachalot==2.2.0
django-debug-toolbar==2.2
```
I just hit this with SQLite. I'll try and follow-up with a sample project next week, but it's just simple model. 

Commenting out the overriding of the `chunked_cursor` (as suggested per @rubengrill's https://github.com/jazzband/django-debug-toolbar/issues/1239#issue-573597111) revolves the issue.

I'm guessing a check to see if the returned cursor is already wrapped might be sufficient. 🤔

Resolving patch:
diff --git a/debug_toolbar/panels/sql/tracking.py b/debug_toolbar/panels/sql/tracking.py
--- a/debug_toolbar/panels/sql/tracking.py
+++ b/debug_toolbar/panels/sql/tracking.py
@@ -55,9 +55,12 @@ def cursor(*args, **kwargs):
             )
 
         def chunked_cursor(*args, **kwargs):
-            return state.Wrapper(
-                connection._djdt_chunked_cursor(*args, **kwargs), connection, panel
-            )
+            # prevent double wrapping
+            # solves https://github.com/jazzband/django-debug-toolbar/issues/1239
+            cursor = connection._djdt_chunked_cursor(*args, **kwargs)
+            if not isinstance(cursor, state.Wrapper):
+                return state.Wrapper(cursor, connection, panel)
+            return cursor
 
         connection.cursor = cursor
         connection.chunked_cursor = chunked_cursor


Test patch: