Repository: django-compressor/django-compressor

Issue:
Offline compression breakage with COMPRESS_OFFLINE_CONTEXT as a generator
Hi

I have been using compressor for a while but I get this error following the update to v.3.0 (Django 3.2.10) :
```
Compressing... Traceback (most recent call last):
  File "/Users/yves/Desktop/passages/manage.py", line 23, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/compressor/management/commands/compress.py", line 294, in handle
    self.handle_inner(**options)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/compressor/management/commands/compress.py", line 317, in handle_inner
    offline_manifest, block_count, results = self.compress(engine, extensions, verbosity, follow_links, log)
  File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/compressor/management/commands/compress.py", line 234, in compress
    (len(offline_manifest), nodes_count, len(contexts)))
TypeError: object of type 'generator' has no len()
```

My settings :
```
COMPRESS_ENABLED = True
COMPRESS_OFFLINE = True
COMPRESS_OFFLINE_CONTEXT = 'config.views.offline_context'
COMPRESS_FILTERS = {
    'css': ['compressor.filters.css_default.CssAbsoluteFilter',
            'compressor.filters.cssmin.rCSSMinFilter'],
    'js': ['compressor.filters.jsmin.JSMinFilter']
}
```

Thanks !


Discussion:
Hi @yesyves — Can I ask you for more detail here please? 

I'm running `compress` with Django 3.2.10 and `django-compressor` 3.0, on Python 3.10, without issue. 

Can you dig out this line: 

```
File "/Users/yves/.virtualenvs/passages/lib/python3.10/site-packages/compressor/management/commands/compress.py", line 234, in compress
    (len(offline_manifest), nodes_count, len(contexts)))
TypeError: object of type 'generator' has no len()
```

Presumably it's `contexts` that's the generator here? 🤔

(I'm wondering if you can create a sample project demonstrating the issue?)
This is not related to your version of Django, but rather a regression with `COMPRESS_OFFLINE_CONTEXT` pointing to a generator. Likely from efb239da97f9f59b87a4a6715c477c64c781caf7
(I also suspect it only happens when `compress` is verbose)
Hi

Thanks for the follow-ups

offline_context() was missing from my report : 

```
LANGUAGES = [
    ('en', 'English'),
    ('fr-ca', 'Français (Canada)'),
    ('fr-fr', 'Français (France)'),
]

def offline_context():
    """
    Generate COMPRESS_OFFLINE_CONTEXT
    :return: context:[{}]
    """
    for code, description in settings.LANGUAGES:
        for base in ('base', 'print/print'):
            yield {'app_version': __version__,
                   'base_template': f'{base}.html',
                   'LANGUAGE_CODE': code,
                   'STATIC_URL': settings.STATIC_URL}
```

The lines in compress.py:232-235  

```
        if verbosity >= 1:
            log.write("done\nCompressed %d block(s) from %d template(s) for %d context(s).\n" %
                      (len(offline_manifest), nodes_count, len(contexts)))
        return offline_manifest, len(offline_manifest), offline_manifest.values()
```

And indeed `python manage.py compress -v 0` works OK

My question right now is mostly to know if my code which worked OK until 2.4.1 is causing the fault or if it is a v3.0 problem ?
Don't know if that can help  :
```
print(offline_manifest)
print(nodes_count)
print(contexts)
```


```
OrderedDict([('eaf286ddc8f8f627bd5aa192fafc94d5616838118690660c213959d976297376', '<link rel="stylesheet" href="/__compressor_url_placeholder__/CACHE/css/output.262821893c77.css" type="text/css">'), ('0ff473a363664d01e3ef1c7d29f6a8bd9e37ed01d51ccbbdc033b260ab322e16', '<script src="/__compressor_url_placeholder__/CACHE/js/output.714d83841691.js"></script>'), ('04c99aebb7751bcf3f932bd7960c08c1fbc70f58c8d87f6bfbcf01fd15bafada', '<script src="/__compressor_url_placeholder__/CACHE/js/output.43f1419e9133.js"></script>'), ('217dd68b1f725f4f231137d5edcb46d02e150bcac7b312d63d4825953e382397', '<script src="/__compressor_url_placeholder__/CACHE/js/output.01a356cd0074.js"></script>'), ('fb997a19b3d905c5646c7df53f4c0db2049a65f692dd68d585c796dea2802a0b', '<link rel="stylesheet" href="/__compressor_url_placeholder__/CACHE/css/output.0e30e522f63c.css" type="text/css">'), ('82ce07c5960bb242485fd1e0e780a314a30ad8a4d4cca40cfa813b4467ec99cb', '<script src="/__compressor_url_placeholder__/CACHE/js/output.b50c43bb0f37.js"></script>'), ('06487855d0ee796cb12dfcef254daeb68de8631aaa84fedbfbd479449d62eea0', '<script src="/__compressor_url_placeholder__/CACHE/js/output.b50c43bb0f37.js"></script>')])

336

<generator object offline_context at 0x109301af0>
```
output of `python mange.py compress -v 3`
[compress.log](https://github.com/django-compressor/django-compressor/files/7721762/compress.log)

This is a regression from 3.0 when `COMPRESS_OFFLINE_CONTEXT` is a generator.

Resolving patch:
diff --git a/compressor/management/commands/compress.py b/compressor/management/commands/compress.py
--- a/compressor/management/commands/compress.py
+++ b/compressor/management/commands/compress.py
@@ -187,6 +187,7 @@ def compress(self, engine, extensions, verbosity, follow_links, log):
                               "template %s\n" % template_name)
                 continue
 
+        contexts_count = 0
         nodes_count = 0
         offline_manifest = OrderedDict()
         errors = []
@@ -219,6 +220,7 @@ def compress(self, engine, extensions, verbosity, follow_links, log):
                 pool.submit(self._compress_template, offline_manifest, nodes, parser, template, errors)
 
             pool.shutdown(wait=True)
+            contexts_count += 1
 
         # If errors exist, raise the first one in the list
         if errors:
@@ -231,7 +233,7 @@ def compress(self, engine, extensions, verbosity, follow_links, log):
 
         if verbosity >= 1:
             log.write("done\nCompressed %d block(s) from %d template(s) for %d context(s).\n" %
-                      (len(offline_manifest), nodes_count, len(contexts)))
+                      (len(offline_manifest), nodes_count, contexts_count))
         return offline_manifest, len(offline_manifest), offline_manifest.values()
 
     @staticmethod


Test patch:
diff --git a/compressor/tests/test_offline.py b/compressor/tests/test_offline.py
--- a/compressor/tests/test_offline.py
+++ b/compressor/tests/test_offline.py
@@ -198,11 +198,12 @@ def _render_result(self, result, separator='\n'):
             settings.COMPRESS_URL_PLACEHOLDER, str(settings.COMPRESS_URL)
         )
 
-    def _test_offline(self, engine):
+    def _test_offline(self, engine, verbosity=0):
         hashes = self.expected_hash
         if not isinstance(hashes, (list, tuple)):
             hashes = [hashes]
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(len(hashes), count)
         self.assertEqual([self._render_script(h) for h in hashes], result)
         rendered_template = self._render_template(engine)
@@ -218,6 +219,26 @@ def test_offline_jinja2(self):
             raise SkipTest('This test class does not support jinja2 engine.')
         self._test_offline(engine='jinja2')
 
+    def test_offline_django_verbosity_1(self):
+        if 'django' not in self.engines:
+            raise SkipTest('This test class does not support django engine.')
+        self._test_offline(engine='django', verbosity=1)
+
+    def test_offline_jinja2_verbosity_1(self):
+        if 'jinja2' not in self.engines:
+            raise SkipTest('This test class does not support jinja2 engine.')
+        self._test_offline(engine='jinja2', verbosity=1)
+
+    def test_offline_django_verbosity_2(self):
+        if 'django' not in self.engines:
+            raise SkipTest('This test class does not support django engine.')
+        self._test_offline(engine='django', verbosity=2)
+
+    def test_offline_jinja2_verbosity_2(self):
+        if 'jinja2' not in self.engines:
+            raise SkipTest('This test class does not support jinja2 engine.')
+        self._test_offline(engine='jinja2', verbosity=2)
+
     def _get_jinja2_env(self):
         import jinja2.ext
         from compressor.offline.jinja2 import url_for, SpacelessExtension
@@ -328,8 +349,9 @@ def test_non_ascii_exception_messages(self, mock):
 class OfflineCompressSkipDuplicatesTestCase(OfflineTestCaseMixin, TestCase):
     templates_dir = 'test_duplicate'
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         # Only one block compressed, the second identical one was skipped.
         self.assertEqual(1, count)
         # Only 1 <script> block in returned result as well.
@@ -375,8 +397,9 @@ class OfflineCompressBlockSuperTestCaseWithExtraContent(
         SuperMixin, OfflineTestCaseMixin, TestCase):
     templates_dir = 'test_block_super_extra'
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(2, count)
         self.assertEqual([
             self._render_script('bfcec76e0f28'),
@@ -503,8 +526,9 @@ class OfflineCompressStaticUrlIndependenceTestCase(
         )
     }
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(1, count)
         self.assertEqual([self._render_script(self.expected_hash)], result)
         self.assertEqual(
@@ -548,7 +572,7 @@ class OfflineCompressTestCaseWithContextGeneratorImportError(
         OfflineTestCaseMixin, TestCase):
     templates_dir = 'test_with_context'
 
-    def _test_offline(self, engine):
+    def _test_offline(self, engine, verbosity=0):
         # Test that we are properly generating ImportError when
         # COMPRESS_OFFLINE_CONTEXT looks like a function but can't be imported
         # for whatever reason.
@@ -581,7 +605,8 @@ def _test_offline(self, engine):
             # Valid path to generator function -- no ImportError:
 
             try:
-                CompressCommand().handle_inner(engines=[engine], verbosity=0)
+                CompressCommand().handle_inner(
+                    engines=[engine], verbosity=verbosity)
             except ImportError:
                 self.fail('Valid path to offline context generator must'
                           ' not raise ImportError.')
@@ -590,8 +615,9 @@ def _test_offline(self, engine):
 class OfflineCompressTestCaseErrors(OfflineTestCaseMixin, TestCase):
     templates_dir = 'test_error_handling'
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
 
         if engine == 'django':
             self.assertEqual(2, count)
@@ -615,7 +641,7 @@ class OfflineCompressTestCaseWithError(OfflineTestCaseMixin, TestCase):
         'COMPRESS_PRECOMPILERS': (('text/coffeescript', 'nonexisting-binary'),)
     }
 
-    def _test_offline(self, engine):
+    def _test_offline(self, engine, verbosity=0):
         """
         Test that a CommandError is raised with DEBUG being False as well as
         True, as otherwise errors in configuration will never show in
@@ -623,11 +649,13 @@ def _test_offline(self, engine):
         """
         with self.settings(DEBUG=True):
             self.assertRaises(
-                CommandError, CompressCommand().handle_inner, engines=[engine], verbosity=0)
+                CommandError, CompressCommand().handle_inner, engines=[engine],
+                verbosity=verbosity)
 
         with self.settings(DEBUG=False):
             self.assertRaises(
-                CommandError, CompressCommand().handle_inner, engines=[engine], verbosity=0)
+                CommandError, CompressCommand().handle_inner, engines=[engine],
+                verbosity=verbosity)
 
 
 class OfflineCompressEmptyTag(OfflineTestCaseMixin, TestCase):
@@ -640,8 +668,8 @@ class OfflineCompressEmptyTag(OfflineTestCaseMixin, TestCase):
     templates_dir = 'basic'
     expected_hash = '822ac7501287'
 
-    def _test_offline(self, engine):
-        CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        CompressCommand().handle_inner(engines=[engine], verbosity=verbosity)
         manifest = get_offline_manifest()
         manifest[list(manifest)[0]] = ''
         self.assertEqual(self._render_template(engine), '\n')
@@ -678,8 +706,9 @@ def _render_template(self, template, engine):
         else:
             return None
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(len(self.expected_hash), count)
         for expected_hash, template in zip(self.expected_hash_offline, self.templates):
             expected = self._render_script(expected_hash)
@@ -697,8 +726,9 @@ class OfflineCompressInlineNonAsciiTestCase(OfflineTestCaseMixin, TestCase):
         }
     }
 
-    def _test_offline(self, engine):
-        _, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        _, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         rendered_template = self._render_template(engine)
         self.assertEqual(rendered_template, ''.join(result) + '\n')
 
@@ -715,8 +745,9 @@ class OfflineCompressComplexTestCase(OfflineTestCaseMixin, TestCase):
         }
     }
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(3, count)
         self.assertEqual([
             self._render_script('76a82cfab9ab'),
@@ -744,8 +775,9 @@ class OfflineCompressExtendsRecursionTestCase(OfflineTestCaseMixin, TestCase):
     ]
 
     @override_settings(INSTALLED_APPS=INSTALLED_APPS)
-    def _test_offline(self, engine):
-        count, _ = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, _ = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(count, 1)
 
 
@@ -761,7 +793,7 @@ class OfflineCompressExtendsRelativeTestCase(SuperMixin, OfflineTestCaseMixin, T
 class TestCompressCommand(OfflineTestCaseMixin, TestCase):
     templates_dir = "test_compress_command"
 
-    def _test_offline(self, engine):
+    def _test_offline(self, engine, verbosity=0):
         raise SkipTest("Not utilized for this test case")
 
     def _build_expected_manifest(self, expected):
@@ -832,8 +864,9 @@ class OfflineCompressTestCaseWithLazyStringAlikeUrls(OfflineCompressTestCaseWith
     }
     expected_hash = 'be0b1eade28b'
 
-    def _test_offline(self, engine):
-        count, result = CompressCommand().handle_inner(engines=[engine], verbosity=0)
+    def _test_offline(self, engine, verbosity=0):
+        count, result = CompressCommand().handle_inner(
+            engines=[engine], verbosity=verbosity)
         self.assertEqual(1, count)
 
         # Change ``SCRIPT_NAME`` WSGI param - it can be changed on every HTTP request,