Repository: django-compressor/django-compressor

Issue:
Offline compression + template cached.Loader = max recursion depth error
I run "manage.py compress" with `'django.template.loaders.cached.Loader'` in TEMPLATE_LOADERS (as the first one).

I have 3 templates base.html, edit.html, choice.html containing `{% compress %}` tags, edit.html and choice.html both extend a template that extends base.html (but the intermediate one doesn't contain any compress tags. (there are more templates, but those are of interest)

compress picks up the templates in that order, and runs fine through the 1st and 2nd, but while compressing the 3rd one, it fails with the following traceback:

```
Traceback (most recent call last):
  File "../manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/.../django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File ".../django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/.../django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/.../django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/.../django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 330, in handle_noargs
    self.compress(sys.stdout, **options)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 252, in compress
    extra_context = template._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 38, in patched_render
    self._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 38, in patched_render
    self._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 101, in render
    compiled_parent = self.get_parent(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 86, in patched_get_parent
    compiled_template = self._old_get_parent(context)

(repeated hundreds of times until:)
RuntimeError: maximum recursion depth exceeded
```

If I disable cached loader, it works well.
The failing template contains a `{{block.super}}` tag, but commenting it out doesn't solve the problem, so this is probably not the issue. If I comment out all compress tags in the 3rd template, then the 4th one fails instead (it extends base.html in the same fashion)

Printing out the process, I narrowed it down to:
in `management/commands/compress.py`:

```
    def patched_render_firstnode(self, context):
        (...)
        firstnode._old_get_parent = firstnode.get_parent
        (...)
```

and the recursion starts because somehow, firstnode.get_parent is never reset to its original value.

If I patch compress.py like this:

```
             self._log.write("Caught error when rendering extend node from "
                                "template %s\n" % self)
            return None
+        firstnode.get_parent = firstnode._old_get_parent
    return extra_context

def patched_get_parent(self, context):
```

then the compression happens correctly, with or without cached loader.
So that's a solution, but I don't like the fact that I don't understand why the issue is happening.
Has anyone seen the same problem?
I'm happy testing further, if someone can point me in the right direction.

Offline compression + template cached.Loader = max recursion depth error
I run "manage.py compress" with `'django.template.loaders.cached.Loader'` in TEMPLATE_LOADERS (as the first one).

I have 3 templates base.html, edit.html, choice.html containing `{% compress %}` tags, edit.html and choice.html both extend a template that extends base.html (but the intermediate one doesn't contain any compress tags. (there are more templates, but those are of interest)

compress picks up the templates in that order, and runs fine through the 1st and 2nd, but while compressing the 3rd one, it fails with the following traceback:

```
Traceback (most recent call last):
  File "../manage.py", line 9, in <module>
    execute_from_command_line(sys.argv)
  File "/.../django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File ".../django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/.../django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/.../django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/.../django/core/management/base.py", line 371, in handle
    return self.handle_noargs(**options)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 330, in handle_noargs
    self.compress(sys.stdout, **options)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 252, in compress
    extra_context = template._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 38, in patched_render
    self._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 38, in patched_render
    self._render_firstnode(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 59, in patched_render_firstnode
    extra_context = firstnode.render(context)
  File "/.../django/template/loader_tags.py", line 101, in render
    compiled_parent = self.get_parent(context)
  File "/.../django-compressor/compressor/management/commands/compress.py", line 86, in patched_get_parent
    compiled_template = self._old_get_parent(context)

(repeated hundreds of times until:)
RuntimeError: maximum recursion depth exceeded
```

If I disable cached loader, it works well.
The failing template contains a `{{block.super}}` tag, but commenting it out doesn't solve the problem, so this is probably not the issue. If I comment out all compress tags in the 3rd template, then the 4th one fails instead (it extends base.html in the same fashion)

Printing out the process, I narrowed it down to:
in `management/commands/compress.py`:

```
    def patched_render_firstnode(self, context):
        (...)
        firstnode._old_get_parent = firstnode.get_parent
        (...)
```

and the recursion starts because somehow, firstnode.get_parent is never reset to its original value.

If I patch compress.py like this:

```
             self._log.write("Caught error when rendering extend node from "
                                "template %s\n" % self)
            return None
+        firstnode.get_parent = firstnode._old_get_parent
    return extra_context

def patched_get_parent(self, context):
```

then the compression happens correctly, with or without cached loader.
So that's a solution, but I don't like the fact that I don't understand why the issue is happening.
Has anyone seen the same problem?
I'm happy testing further, if someone can point me in the right direction.



Discussion:
Forgot to mention this is using d6d030ec3cd806bd282fd8becf65183b36918936

ah, interesting, I had the problem at some point but couldn't reproduce it any more. Can you try to make a unit test out of it ? See http://django_compressor.readthedocs.org/en/latest/contributing/#tests

I tried putting together a unit test to reproduce it, but no luck (first time doing that, I managed to get some prints in there to check what is going on, but there's a lot of unexpected output, and I don't know how to control that)

That said, here's what I've narrowed it down to, commenting out bits of templates until there's only this left:
- base.html: empty
- intermediate.html:
  `{% extends "base.html" %}`
- child1.html:

``` html
{% extends "intermediate.html" %}
{% load compress %}
{% compress js %}
<script type="text/javascript">alert("One!");</script>
{% endcompress %}
```
- child2.html:

``` html
{% extends "intermediate.html" %}
{% load compress %}
{% compress js %}
<script type="text/javascript">alert("Two!");</script>
{% endcompress %}
```

Compression happens in the order: child1.html, child2.html. I think this is important from the various attempts I made.
If one of the children extends `base.html`, no problem. If `intermediate.html` extends nothing, no problem either.
It looks like there's an issue dealing with cascaded `extends` tags.
(that's inline with my first comment: the get_parent member is never reset for the intermediate template, so second time we run by it, we hit the issue.)

That's as much as I can offer in terms of reproducing the bug. If you can help me control the order in which the unit test compresses templates, then I might get it to hit the same issue.

Forgot to mention this is using d6d030ec3cd806bd282fd8becf65183b36918936

ah, interesting, I had the problem at some point but couldn't reproduce it any more. Can you try to make a unit test out of it ? See http://django_compressor.readthedocs.org/en/latest/contributing/#tests

I tried putting together a unit test to reproduce it, but no luck (first time doing that, I managed to get some prints in there to check what is going on, but there's a lot of unexpected output, and I don't know how to control that)

That said, here's what I've narrowed it down to, commenting out bits of templates until there's only this left:
- base.html: empty
- intermediate.html:
  `{% extends "base.html" %}`
- child1.html:

``` html
{% extends "intermediate.html" %}
{% load compress %}
{% compress js %}
<script type="text/javascript">alert("One!");</script>
{% endcompress %}
```
- child2.html:

``` html
{% extends "intermediate.html" %}
{% load compress %}
{% compress js %}
<script type="text/javascript">alert("Two!");</script>
{% endcompress %}
```

Compression happens in the order: child1.html, child2.html. I think this is important from the various attempts I made.
If one of the children extends `base.html`, no problem. If `intermediate.html` extends nothing, no problem either.
It looks like there's an issue dealing with cascaded `extends` tags.
(that's inline with my first comment: the get_parent member is never reset for the intermediate template, so second time we run by it, we hit the issue.)

That's as much as I can offer in terms of reproducing the bug. If you can help me control the order in which the unit test compresses templates, then I might get it to hit the same issue.


Resolving patch:
diff --git a/compressor/management/commands/compress.py b/compressor/management/commands/compress.py
--- a/compressor/management/commands/compress.py
+++ b/compressor/management/commands/compress.py
@@ -38,6 +38,9 @@ def patched_render(self, context):
     # not, to be able to handle complex inheritance chain.
     self._render_firstnode = MethodType(patched_render_firstnode, self)
     self._render_firstnode(context)
+
+    # Cleanup, uninstall our _render monkeypatch now that it has been called
+    self._render = self._old_render
     return context
 
 
@@ -52,7 +55,10 @@ def patched_render_firstnode(self, context):
     # its get_parent method so that rendering the ExtendsNode only
     # gives us the blocks content without doing any actual rendering.
     extra_context = {}
-    firstnode = self.nodelist[0]
+    try:
+        firstnode = self.nodelist[0]
+    except IndexError:
+        firstnode = None
     if isinstance(firstnode, ExtendsNode):
         firstnode._log = self._log
         firstnode._log_verbosity = self._log_verbosity
@@ -75,8 +81,11 @@ def patched_render_firstnode(self, context):
             # as well.
             if self._log_verbosity > 0:
                 self._log.write("Caught error when rendering extend node from "
-                                "template %s\n" % self)
+                                "template %s\n" % getattr(self, 'name', self))
             return None
+        finally:
+            # Cleanup, uninstall our get_parent monkeypatch now that it has been called
+            firstnode.get_parent = firstnode._old_get_parent
     return extra_context
 
 
@@ -89,6 +98,7 @@ def patched_get_parent(self, context):
     compiled_template = self._old_get_parent(context)
     compiled_template._log = self._log
     compiled_template._log_verbosity = self._log_verbosity
+    compiled_template._old_render = compiled_template._render
     compiled_template._render = MethodType(patched_render, compiled_template)
     return compiled_template
 


Test patch:
diff --git a/compressor/tests/test_offline.py b/compressor/tests/test_offline.py
--- a/compressor/tests/test_offline.py
+++ b/compressor/tests/test_offline.py
@@ -27,14 +27,13 @@ def setUp(self):
         self._old_compress_offline = settings.COMPRESS_OFFLINE
         self._old_template_dirs = settings.TEMPLATE_DIRS
         self._old_offline_context = settings.COMPRESS_OFFLINE_CONTEXT
-        self._old_template_loaders = settings.TEMPLATE_LOADERS
         self.log = StringIO()
 
-        # Force template dirs, because it enables us to force compress to
+        # Reset template dirs, because it enables us to force compress to
         # consider only a specific directory (helps us make true,
         # independant unit tests).
         settings.TEMPLATE_DIRS = (
-            os.path.join(settings.TEMPLATE_DIRS[0], self.templates_dir),
+            os.path.join(settings.TEST_DIR, 'test_templates', self.templates_dir),
         )
         # Enable offline compress
         settings.COMPRESS_ENABLED = True
@@ -47,7 +46,6 @@ def tearDown(self):
         settings.COMPRESS_ENABLED = self._old_compress
         settings.COMPRESS_OFFLINE = self._old_compress_offline
         settings.TEMPLATE_DIRS = self._old_template_dirs
-        settings.TEMPLATE_LOADERS = self._old_template_loaders
         self.template_file.close()
         manifest_path = os.path.join('CACHE', 'manifest.json')
         if default_storage.exists(manifest_path):
@@ -73,6 +71,25 @@ class OfflineGenerationBlockSuperMultipleTestCase(OfflineTestCaseMixin, TestCase
     expected_hash = "2f6ef61c488e"
 
 
+class OfflineGenerationBlockSuperMultipleWithCachedLoaderTestCase(OfflineTestCaseMixin, TestCase):
+    templates_dir = "test_block_super_multiple_cached"
+    expected_hash = "2f6ef61c488e"
+
+    def setUp(self):
+        self._old_template_loaders = settings.TEMPLATE_LOADERS
+        settings.TEMPLATE_LOADERS = (
+            ('django.template.loaders.cached.Loader', (
+                'django.template.loaders.filesystem.Loader',
+                'django.template.loaders.app_directories.Loader',
+            )),
+        )
+        super(OfflineGenerationBlockSuperMultipleWithCachedLoaderTestCase, self).setUp()
+
+    def tearDown(self):
+        super(OfflineGenerationBlockSuperMultipleWithCachedLoaderTestCase, self).tearDown()
+        settings.TEMPLATE_LOADERS = self._old_template_loaders
+
+
 class OfflineGenerationBlockSuperTestCaseWithExtraContent(OfflineTestCaseMixin, TestCase):
     templates_dir = "test_block_super_extra"
 
diff --git a/compressor/tests/test_templates/test_block_super_multiple_cached/base.html b/compressor/tests/test_templates/test_block_super_multiple_cached/base.html
new file mode 100644
--- /dev/null
+++ b/compressor/tests/test_templates/test_block_super_multiple_cached/base.html
@@ -0,0 +1,15 @@
+{% spaceless %}
+{% block js %}
+    <script type="text/javascript">
+        alert("test using multiple inheritance and block.super");
+    </script>
+{% endblock %}
+
+{% block css %}
+    <style type="text/css">
+        body {
+            background: red;
+        }
+    </style>
+{% endblock %}
+{% endspaceless %}
diff --git a/compressor/tests/test_templates/test_block_super_multiple_cached/base2.html b/compressor/tests/test_templates/test_block_super_multiple_cached/base2.html
new file mode 100644
--- /dev/null
+++ b/compressor/tests/test_templates/test_block_super_multiple_cached/base2.html
@@ -0,0 +1,3 @@
+{% extends "base.html" %}
+
+{% block css %}{% endblock %}
diff --git a/compressor/tests/test_templates/test_block_super_multiple_cached/test_compressor_offline.html b/compressor/tests/test_templates/test_block_super_multiple_cached/test_compressor_offline.html
new file mode 100644
--- /dev/null
+++ b/compressor/tests/test_templates/test_block_super_multiple_cached/test_compressor_offline.html
@@ -0,0 +1,11 @@
+{% extends "base2.html" %}
+{% load compress %}
+
+{% block js %}{% spaceless %}
+	{% compress js %}
+	    {{ block.super }}
+	    <script type="text/javascript">
+	        alert("this alert shouldn't be alone!");
+	    </script>
+	{% endcompress %}
+{% endspaceless %}{% endblock %}