Repository: jazzband/django-debug-toolbar

Issue:
SQL: Show duplicate queries as in SQL Log Middleware
SQL Log Middleware is here: http://www.djangosnippets.org/snippets/344/

Working solution proposition:

http://dumpz.org/12573/
http://dumpz.org/12575/



Discussion:
The main reason why I'm using the debug toolbar is to reduce the number of DB queries and increase the site performance.
This feature would be very handy!

I have implemented this feature in my fork (https://github.com/mikery/django-debug-toolbar/commit/27cfc7b6bdcdde4cf1e88e6502b84852915d6cf9). Once I have made it a bit more flexible I will try making a pull request to see if we can get this feature in the official master branch. If you try it out, please let me know if you have any comments or suggestions.

I've closed a stale pull request. This issue doesn't look entirely straightforward to me. For instance, how do we collapse queries that only differ by the value of some parameter? How do we handle repeated series of queries: A, B, C, A, B, C, A, etc.?

If you want to tackle this ticket, I suggest discussing a design with a maintainer before developing a solution.

The idea I had many years ago was that in the debug cursor the sql and params arguments are separate. So anywhere where the sql was the same increment a counter on that sql. The SQL parameters could be different, no problem. This might help to see that you're doing hundreds of SQL queries for the same table and that SQL is querying by ID -- which could give you a clue that you might need a `select_related` to avoid all those single pk lookups, e.g.


Resolving patch:
diff --git a/debug_toolbar/panels/sql/panel.py b/debug_toolbar/panels/sql/panel.py
--- a/debug_toolbar/panels/sql/panel.py
+++ b/debug_toolbar/panels/sql/panel.py
@@ -139,6 +139,7 @@ def disable_instrumentation(self):
     def process_response(self, request, response):
         colors = contrasting_color_generator()
         trace_colors = defaultdict(lambda: next(colors))
+        query_duplicates = defaultdict(lambda: defaultdict(int))
         if self._queries:
             width_ratio_tally = 0
             factor = int(256.0 / (len(self._databases) * 2.5))
@@ -161,6 +162,8 @@ def process_response(self, request, response):
             trans_id = None
             i = 0
             for alias, query in self._queries:
+                query_duplicates[alias][query["raw_sql"]] += 1
+
                 trans_id = query.get('trans_id')
                 last_trans_id = trans_ids.get(alias)
 
@@ -204,6 +207,30 @@ def process_response(self, request, response):
             if trans_id:
                 self._queries[(i - 1)][1]['ends_trans'] = True
 
+        # Queries are duplicates only if there's as least 2 of them.
+        # Also, to hide queries, we need to give all the duplicate groups an id
+        query_duplicates = dict(
+            (alias, dict(
+                (query, duplicate_count)
+                for query, duplicate_count in queries.items()
+                if duplicate_count >= 2
+            ))
+            for alias, queries in query_duplicates.items()
+        )
+
+        for alias, query in self._queries:
+            try:
+                duplicates_count = query_duplicates[alias][query["raw_sql"]]
+                query["duplicate_count"] = duplicates_count
+            except KeyError:
+                pass
+
+        for alias, alias_info in self._databases.items():
+            try:
+                alias_info["duplicate_count"] = sum(e for e in query_duplicates[alias].values())
+            except KeyError:
+                pass
+
         self.record_stats({
             'databases': sorted(self._databases.items(), key=lambda x: -x[1]['time_spent']),
             'queries': [q for a, q in self._queries],


Test patch: