Repository: django-compressor/django-compressor

Issue:
Management command broken with 1.8
```
  File "/Users/carlton/ve/notwatch/lib/python2.7/site-packages/compressor/management/commands/compress.py", line 56, in get_loaders
    from django.template.loader import template_source_loaders
ImportError: cannot import name template_source_loaders
```

On the fly still works but offline is failing. 

Still stepping through the new 1.8 template code to be able to say more.



Discussion:
Ran into this issue too. I've dealt with changes in template loaders (so that `compress` command loop through all th templates) but stopped on `handle_extendsnode` function which I couldn't get to work.
I've changed it to `return []` and it successfully found compress tags in base templates but its still not usable coz appropriate cache keys for extended templates does not exist.

Yeah, that's not surprising given the changes around template loading in 1.8.

Sadly, there is almost no one actively working on this (or compressor itself) right now. I'll accept pull requests fixing this, but it's not trivial.

@diox — I'm happy to help maintain here. I submitted #599 to get the build passing again. 

Hmmm. I added 1.8 to tox. Executive Summary: Didn't go well. 
1. A lot of tests depend on offline compression — so need a fix here.
2. There were import errors for jingo — I wasn't able to (see how to) quickly skip this. 
3. At first glance inexplicably, a small change in one test case led to a whole load of `OfflineGenerationErrors` elsewhere (perhaps related to #579)
4. Are the `expected_hash = "a27e1d3a619a"` type values all derived by hand? 
5. All of that meant it's not easy to see where to begin even. 

I'll have a pop at the `compress` command this week. 

Regarding 4, yes - if you get different results, you should check the resulting compressed contents to see what looks suspicious.

Again, "Hmmm". :) 

I've stepped through trying to implement the [`compress` command](https://github.com/django-compressor/django-compressor/blob/develop/compressor/management/commands/compress.py#L107) with 1.8. The template loading section is simple enough (and much nicer than the existing method.) 

Like @a1tus when it gets into [compressor.offline.django.DjangoParser.walk_nodes](https://github.com/django-compressor/django-compressor/blob/develop/compressor/offline/django.py#L137) I'm currently unsure how to proceed. There's a good bit of rooting around in django.template.base to work out how to map what there to what we need. 

@aaugustin Would it be at all possible for you to look at the `walk_nodes` function linked and perhaps indicate how a 1.8 version might run? I'd be very grateful if you could! (Thank you.)

If we can get the node list then the rest of the command should run through smoothly enough (he says): the DTL `Node` and `NodeList` classes provides a `render` which will either Just Work™ or shouldn't provide too much trouble (he says)

At line 98, on Django 1.8, replace `return get_template(template_name)` by `return get_template(template_name).template`.

`get_template` returns a wrapper around the `django.template.Template`. Accessing the `template` attribute extracts the underlying template.

Brilliant. Thank you! I'll give that a run shortly. 👍


Resolving patch:
diff --git a/compressor/management/commands/compress.py b/compressor/management/commands/compress.py
--- a/compressor/management/commands/compress.py
+++ b/compressor/management/commands/compress.py
@@ -5,6 +5,7 @@
 from fnmatch import fnmatch
 from optparse import make_option
 
+import django
 from django.core.management.base import NoArgsCommand, CommandError
 import django.template
 from django.template import Context
@@ -53,24 +54,30 @@ class Command(NoArgsCommand):
     requires_model_validation = False
 
     def get_loaders(self):
-        from django.template.loader import template_source_loaders
-        if template_source_loaders is None:
-            try:
-                from django.template.loader import (
-                    find_template as finder_func)
-            except ImportError:
-                from django.template.loader import (
-                    find_template_source as finder_func)  # noqa
-            try:
-                # Force django to calculate template_source_loaders from
-                # TEMPLATE_LOADERS settings, by asking to find a dummy template
-                source, name = finder_func('test')
-            except django.template.TemplateDoesNotExist:
-                pass
-            # Reload template_source_loaders now that it has been calculated ;
-            # it should contain the list of valid, instanciated template loaders
-            # to use.
+        if django.VERSION < (1, 8):
             from django.template.loader import template_source_loaders
+            if template_source_loaders is None:
+                try:
+                    from django.template.loader import (
+                        find_template as finder_func)
+                except ImportError:
+                    from django.template.loader import (
+                        find_template_source as finder_func)  # noqa
+                try:
+                    # Force django to calculate template_source_loaders from
+                    # TEMPLATE_LOADERS settings, by asking to find a dummy template
+                    source, name = finder_func('test')
+                except django.template.TemplateDoesNotExist:
+                    pass
+                # Reload template_source_loaders now that it has been calculated ;
+                # it should contain the list of valid, instanciated template loaders
+                # to use.
+                from django.template.loader import template_source_loaders
+        else:
+            from django.template import engines
+            template_source_loaders = []
+            for e in engines.all():
+                template_source_loaders.extend(e.engine.get_template_loaders(e.engine.loaders))
         loaders = []
         # If template loader is CachedTemplateLoader, return the loaders
         # that it wraps around. So if we have
diff --git a/compressor/offline/django.py b/compressor/offline/django.py
--- a/compressor/offline/django.py
+++ b/compressor/offline/django.py
@@ -1,6 +1,7 @@
 from __future__ import absolute_import
 from copy import copy
 
+import django
 from django import template
 from django.conf import settings
 from django.template import Context
@@ -14,7 +15,7 @@
 from compressor.templatetags.compress import CompressorNode
 
 
-def handle_extendsnode(extendsnode, block_context=None):
+def handle_extendsnode(extendsnode, block_context=None, original=None):
     """Create a copy of Node tree of a derived template replacing
     all blocks tags with the nodes of appropriate blocks.
     Also handles {{ block.super }} tags.
@@ -26,6 +27,9 @@ def handle_extendsnode(extendsnode, block_context=None):
     block_context.add_blocks(blocks)
 
     context = Context(settings.COMPRESS_OFFLINE_CONTEXT)
+    if original is not None:
+        context.template = original
+
     compiled_parent = extendsnode.get_parent(context)
     parent_nodelist = compiled_parent.nodelist
     # If the parent template has an ExtendsNode it is not the root.
@@ -33,7 +37,7 @@ def handle_extendsnode(extendsnode, block_context=None):
         # The ExtendsNode has to be the first non-text node.
         if not isinstance(node, TextNode):
             if isinstance(node, ExtendsNode):
-                return handle_extendsnode(node, block_context)
+                return handle_extendsnode(node, block_context, original)
             break
     # Add blocks of the root template to block context.
     blocks = dict((n.name, n) for n in
@@ -95,7 +99,10 @@ def __init__(self, charset):
 
     def parse(self, template_name):
         try:
-            return get_template(template_name)
+            if django.VERSION < (1, 8):
+                return get_template(template_name)
+            else:
+                return get_template(template_name).template
         except template.TemplateSyntaxError as e:
             raise TemplateSyntaxError(str(e))
         except template.TemplateDoesNotExist as e:
@@ -111,15 +118,17 @@ def process_node(self, template, context, node):
         pass
 
     def render_nodelist(self, template, context, node):
+        if django.VERSION >= (1, 8):
+            context.template = template
         return node.nodelist.render(context)
 
     def render_node(self, template, context, node):
         return node.render(context, forced=True)
 
-    def get_nodelist(self, node):
+    def get_nodelist(self, node, original=None):
         if isinstance(node, ExtendsNode):
             try:
-                return handle_extendsnode(node)
+                return handle_extendsnode(node, block_context=None, original=original)
             except template.TemplateSyntaxError as e:
                 raise TemplateSyntaxError(str(e))
             except template.TemplateDoesNotExist as e:
@@ -134,10 +143,12 @@ def get_nodelist(self, node):
             nodelist = getattr(node, 'nodelist', [])
         return nodelist
 
-    def walk_nodes(self, node):
-        for node in self.get_nodelist(node):
+    def walk_nodes(self, node, original=None):
+        if django.VERSION >= (1, 8) and original is None:
+            original = node
+        for node in self.get_nodelist(node, original):
             if isinstance(node, CompressorNode) and node.is_offline_compression_enabled(forced=True):
                 yield node
             else:
-                for node in self.walk_nodes(node):
+                for node in self.walk_nodes(node, original):
                     yield node


Test patch:
diff --git a/compressor/test_settings.py b/compressor/test_settings.py
--- a/compressor/test_settings.py
+++ b/compressor/test_settings.py
@@ -21,8 +21,9 @@
 INSTALLED_APPS = [
     'compressor',
     'coffin',
-    'jingo',
 ]
+if django.VERSION < (1, 8):
+    INSTALLED_APPS.append('jingo')
 
 STATIC_URL = '/static/'
 
diff --git a/compressor/tests/test_offline.py b/compressor/tests/test_offline.py
--- a/compressor/tests/test_offline.py
+++ b/compressor/tests/test_offline.py
@@ -3,6 +3,7 @@
 import os
 import sys
 
+import django
 from django.core.management.base import CommandError
 from django.template import Template, Context
 from django.test import TestCase
@@ -44,10 +45,6 @@ class OfflineTestCaseMixin(object):
         engines = ("django",)
 
     def setUp(self):
-        self._old_compress = settings.COMPRESS_ENABLED
-        self._old_compress_offline = settings.COMPRESS_OFFLINE
-        self._old_template_dirs = settings.TEMPLATE_DIRS
-        self._old_offline_context = settings.COMPRESS_OFFLINE_CONTEXT
         self.log = StringIO()
 
         # Reset template dirs, because it enables us to force compress to
@@ -58,11 +55,18 @@ def setUp(self):
         # template to be skipped over.
         django_template_dir = os.path.join(settings.TEST_DIR, 'test_templates', self.templates_dir)
         jinja2_template_dir = os.path.join(settings.TEST_DIR, 'test_templates_jinja2', self.templates_dir)
-        settings.TEMPLATE_DIRS = (django_template_dir, jinja2_template_dir)
 
-        # Enable offline compress
-        settings.COMPRESS_ENABLED = True
-        settings.COMPRESS_OFFLINE = True
+        override_settings = {
+            'TEMPLATE_DIRS': (django_template_dir, jinja2_template_dir,),
+            'COMPRESS_ENABLED': True,
+            'COMPRESS_OFFLINE': True
+        }
+
+        if "jinja2" in self.engines:
+            override_settings["COMPRESS_JINJA2_GET_ENVIRONMENT"] = lambda: self._get_jinja2_env()
+
+        self.override_settings = self.settings(**override_settings)
+        self.override_settings.__enter__()
 
         if "django" in self.engines:
             self.template_path = os.path.join(django_template_dir, self.template_name)
@@ -70,22 +74,16 @@ def setUp(self):
             with io.open(self.template_path, encoding=settings.FILE_CHARSET) as file:
                 self.template = Template(file.read())
 
-        self._old_jinja2_get_environment = settings.COMPRESS_JINJA2_GET_ENVIRONMENT
-
         if "jinja2" in self.engines:
-            # Setup Jinja2 settings.
-            settings.COMPRESS_JINJA2_GET_ENVIRONMENT = lambda: self._get_jinja2_env()
-            jinja2_env = settings.COMPRESS_JINJA2_GET_ENVIRONMENT()
+            jinja2_env = override_settings["COMPRESS_JINJA2_GET_ENVIRONMENT"]()
             self.template_path_jinja2 = os.path.join(jinja2_template_dir, self.template_name)
 
             with io.open(self.template_path_jinja2, encoding=settings.FILE_CHARSET) as file:
                 self.template_jinja2 = jinja2_env.from_string(file.read())
 
     def tearDown(self):
-        settings.COMPRESS_JINJA2_GET_ENVIRONMENT = self._old_jinja2_get_environment
-        settings.COMPRESS_ENABLED = self._old_compress
-        settings.COMPRESS_OFFLINE = self._old_compress_offline
-        settings.TEMPLATE_DIRS = self._old_template_dirs
+        self.override_settings.__exit__(None, None, None)
+
         manifest_path = os.path.join('CACHE', 'manifest.json')
         if default_storage.exists(manifest_path):
             default_storage.delete(manifest_path)
@@ -454,6 +452,8 @@ def _test_offline(self, engine):
 # It seems there is no evidence nor indicated support for Python 3+.
 @unittest.skipIf(sys.version_info >= (3, 2),
     "Coffin does not support 3.2+")
+@unittest.skipIf(django.VERSION >= (1, 8),
+    "Import error on 1.8")
 class OfflineGenerationCoffinTestCase(OfflineTestCaseMixin, TestCase):
     templates_dir = "test_coffin"
     expected_hash = "32c8281e3346"
@@ -478,6 +478,8 @@ def _get_jinja2_env(self):
 # is also evident in its tox.ini file.
 @unittest.skipIf(sys.version_info >= (3, 2) and sys.version_info < (3, 3),
     "Jingo does not support 3.2")
+@unittest.skipIf(django.VERSION >= (1, 8),
+    "Import error on 1.8")
 class OfflineGenerationJingoTestCase(OfflineTestCaseMixin, TestCase):
     templates_dir = "test_jingo"
     expected_hash = "61ec584468eb"