Repository: jazzband/django-debug-toolbar

Issue:
SQL panel points a query in an included template to the same line of its parent template
Tested with:

views.py
````python
from test_debug_toolbar.models import Question

def index(request):
    return render(request, 'index.html', {'questions': Question.objects.all()})
````

index.html
````html
{% extends 'base.html' %}

{% block content %}

<h5>Hello there! This is a test for debug toolbar.</h5>

<p>Below is an included template that makes a SQL query. But in debug toolbar, it is signified in "index.html", while it should be in "inner.html".</p>

{% include 'inner.html' %}

{% endblock content %}
````

inner.html
````html
<p>Here's inner.html:</p>


<p>All questions:</p>
{{ questions }}
````

The SQL query of `Question.objects.all()` is executed at the line 5 of `inner.html`, but in the SQL panel, it is incorrectly pointed to the line 5 of its parent template `index.html`:

![image](https://cloud.githubusercontent.com/assets/8630726/21615030/c9caed1a-d1a9-11e6-8626-0b5f18f8e340.png)

Tested with current master and develop branch, and Django 1.10.4.

For the complete test Django project: https://github.com/lingxiaoyang/test_debug_toolbar




Discussion:
I investigated a little of this issue. The problem seems to exist only on Django 1.9 and later, most probably because of the following code (reference to [utils.py](https://github.com/jazzband/django-debug-toolbar/blob/master/debug_toolbar/utils.py#L117-L142)):

````python
def get_template_context(node, context, context_lines=3):
    source = getattr(node, 'source', None)
    # In Django 1.9 template Node does not have source property, Origin does
    # not reload method, so we extract contextual information from exception
    # info.
    if source:
        line, source_lines, name = get_template_source_from_source(source)
    else:
        line, source_lines, name = get_template_source_from_exception_info(
            node, context)
````

It is very possible that function `get_template_source_from_exception_info` doesn't work accurately on `include` template tag. 

Instead, on Django 1.8 the template tracking is working correctly.
Further investigation shows that Django 1.10.6 itself has the same problem when dealing with template exception, which is fixed in 1.11b1. However, Django debug toolbar gives wrong result for both versions. On an older version 1.8.11, both Django and Django debug toolbar work as expected.

Here's the comparison table:

<table>
<thead>
<th>Django version</th><th>Django</th><th>Django debug toolbar</th>
</thead>
<tbody>
<th>1.8.11</th><td><b>Correct</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060362/4fb6cfee-0b29-11e7-88c3-1bd45c68d9db.png"></td><td><b>Correct</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060396/76871a48-0b29-11e7-9545-eb3ea764e199.png"></td>
</tr>
<tr>
<th>1.10.6</th><td><b>Incorrect</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060337/35b74cf4-0b29-11e7-8e41-090d9b610920.png"></td><td><b style="color:red">Incorrect</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060419/8c21f7c4-0b29-11e7-8c19-3e943622a379.png"></td>
</tr>
<tr>
<th>1.11b1</th><td><b>Correct</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060308/1526728a-0b29-11e7-8a6a-f97770290144.png"></td><td><b>Incorrect</b><img src="https://cloud.githubusercontent.com/assets/8630726/24060444/a3626752-0b29-11e7-83d8-c738eb09ff17.png"></td>
</tr>
</table>

I'll see what Django has changed in its code and perhaps submit a PR on this later.
Questions to the package maintainers:

At the Django side, the bug concerning `{% include %}` and `{% extend %}` indeed exists in versions 1.9 and 1.10, and is being fixed in 1.11 LTS. However, this bugfix will not be backported to 1.9 and 1.10 (most probably because they will be out of support soon, according to Django roadmap). References: tickets [27584](https://code.djangoproject.com/ticket/27584) and [27956](https://code.djangoproject.com/ticket/27956).

At present, django-debug-toolbar copies the same code in Django to extract the template source. This code will need a small fix in order to work correctly in Django 1.11, but a more significant change will be required if we want it working with 1.9 and 1.10. You may compare the code of the [bugfix](https://github.com/django/django/pull/7686/files#diff-8cbe1fb6d589cb6e35b956704d7a1285R960) and [debug_toolbar/utils.py](https://github.com/jazzband/django-debug-toolbar/blob/master/debug_toolbar/utils.py).

The question is therefore: do we want this fix to only work with Django 1.11 LTS and keep how it is now with versions 1.9 and 1.10; or do we also provide another solution for these two older versions?

A 2nd question: do we need a unit test on this, or is it fine to just keep it the same with Django's code?
I think doing the same thing the Django maintainers do is fine; I'm alright with only being correct on 1.11 upwards.

That being said, if someone wants to do the work of supporting 1.9 and 1.10 as well despite Django itself being incorrect there I certainly won't stop them. 

Unit tests are always good, and if it's possible to easily construct an unit test then it should be done (tm). If it's hard or nobody wants to invest the time then following Django is fine. If we do more than Django does by fixing the reports on 1.9 and 1.10 an unit test would clearly be a requirement though, because I expect the code involved would be more complex and we would certainly want to avoid inadvertent breakage.
i'm still seeing the issue with django-debug-toolbar 1.11 and django 1.11.23 

Resolving patch:
diff --git a/debug_toolbar/panels/history/panel.py b/debug_toolbar/panels/history/panel.py
--- a/debug_toolbar/panels/history/panel.py
+++ b/debug_toolbar/panels/history/panel.py
@@ -15,7 +15,7 @@
 
 
 class HistoryPanel(Panel):
-    """ A panel to display History """
+    """A panel to display History"""
 
     title = _("History")
     nav_title = _("History")
diff --git a/debug_toolbar/utils.py b/debug_toolbar/utils.py
--- a/debug_toolbar/utils.py
+++ b/debug_toolbar/utils.py
@@ -135,7 +135,14 @@ def get_template_context(node, context, context_lines=3):
 
 
 def get_template_source_from_exception_info(node, context):
-    exception_info = context.template.get_exception_info(Exception("DDT"), node.token)
+    if context.template.origin == node.origin:
+        exception_info = context.template.get_exception_info(
+            Exception("DDT"), node.token
+        )
+    else:
+        exception_info = context.render_context.template.get_exception_info(
+            Exception("DDT"), node.token
+        )
     line = exception_info["line"]
     source_lines = exception_info["source_lines"]
     name = exception_info["name"]


Test patch:
diff --git a/tests/panels/test_sql.py b/tests/panels/test_sql.py
--- a/tests/panels/test_sql.py
+++ b/tests/panels/test_sql.py
@@ -1,4 +1,5 @@
 import datetime
+import os
 import unittest
 
 import django
@@ -396,3 +397,47 @@ def test_prettify_sql(self):
         self.panel.generate_stats(self.request, response)
         self.assertEqual(len(self.panel._queries), 1)
         self.assertEqual(pretty_sql, self.panel._queries[-1][1]["sql"])
+
+    @override_settings(
+        DEBUG=True,
+    )
+    def test_flat_template_information(self):
+        """
+        Test case for when the query is used in a flat template hierarchy
+        (without included templates).
+        """
+        self.assertEqual(len(self.panel._queries), 0)
+
+        users = User.objects.all()
+        render(self.request, "sql/flat.html", {"users": users})
+
+        self.assertEqual(len(self.panel._queries), 1)
+
+        query = self.panel._queries[0]
+        template_info = query[1]["template_info"]
+        template_name = os.path.basename(template_info["name"])
+        self.assertEqual(template_name, "flat.html")
+        self.assertEqual(template_info["context"][2]["content"].strip(), "{{ users }}")
+        self.assertEqual(template_info["context"][2]["highlight"], True)
+
+    @override_settings(
+        DEBUG=True,
+    )
+    def test_nested_template_information(self):
+        """
+        Test case for when the query is used in a nested template
+        hierarchy (with included templates).
+        """
+        self.assertEqual(len(self.panel._queries), 0)
+
+        users = User.objects.all()
+        render(self.request, "sql/nested.html", {"users": users})
+
+        self.assertEqual(len(self.panel._queries), 1)
+
+        query = self.panel._queries[0]
+        template_info = query[1]["template_info"]
+        template_name = os.path.basename(template_info["name"])
+        self.assertEqual(template_name, "included.html")
+        self.assertEqual(template_info["context"][0]["content"].strip(), "{{ users }}")
+        self.assertEqual(template_info["context"][0]["highlight"], True)
diff --git a/tests/templates/sql/flat.html b/tests/templates/sql/flat.html
new file mode 100644
--- /dev/null
+++ b/tests/templates/sql/flat.html
@@ -0,0 +1,4 @@
+{% extends "base.html" %}
+{% block content %}
+  {{ users }}
+{% endblock %}
diff --git a/tests/templates/sql/included.html b/tests/templates/sql/included.html
new file mode 100644
--- /dev/null
+++ b/tests/templates/sql/included.html
@@ -0,0 +1 @@
+{{ users }}
diff --git a/tests/templates/sql/nested.html b/tests/templates/sql/nested.html
new file mode 100644
--- /dev/null
+++ b/tests/templates/sql/nested.html
@@ -0,0 +1,4 @@
+{% extends "base.html" %}
+{% block content %}
+  {% include "sql/included.html" %}
+{% endblock %}