Repository: django-compressor/django-compressor

Issue:
Make offline compression independent of STATIC_URL
I use offline compression. I do `manage.py compress` once and then deploy the project with gathered static files to multiple servers. 

On different servers I can have different `STATIC_URL`, not the one used for compression. That's problematic for current `django-compressor` implementation because:
- it calculates _offline manifest keys_ using _actual_ `STATIC_URL` when in the `manifest.json` it corresponds to `STATIC_URL` used during `manage.py compress`. And it raises an exception - "the key is not found in the offline manifest"
- another problem is that in offline manifest you store pieces of HTML that also contain _obsolete_ `STATIC_URL` which doesn't work on deployed site instance because `STATIC_URL` is different now

You could change 3 things to make it work:
- when calculating manifest keys: substitute all `STATIC_URL` entries with some constant, e.g.: `/static/` → `$STATIC_URL`
- when generating offline manifest via `manage.py compress` - don't put actual `STATIC_URL` (e.g. `/static/`) into `manifest.json` but put some constant like `$STATIC_URL` instead
- when reading the manifest - replace all `$STATIC_URL` with actual `STATIC_URL`, e.g.: `$STATIC_URL` → `/new/static/`

UnicodeDecodeError when creating error message string
Bug happens in [compressor/management/commands/compress.py](https://github.com/django-compressor/django-compressor/blob/master/compressor/management/commands/compress.py#L230-L231)

I ran in to this when the output of uglify-js (the compressor we're using) is outputting something that can't be read as a regular Python string.

When I was debugging:

```
(Pdb) "An error occured during rendering %s: %s" % (template.template_name, e.message)
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 124: ordinal not in range(128)
(Pdb) "An error occured during rendering {template}: {error}".format(template=template.template_name, error=e)
'An error occured during rendering /Users/mrpi/Workspace/.../assets.html: util.error: Use console.error instead\nParse error at -:29,7\nutil.error: Use console.error instead\nUnexpected token operator \xc2\xab!\xc2\xbb, expected punc \xc2\xab(\xc2\xbb\nutil.error: Use console.error instead\nError\n    at new JS_Parse_Error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:196:18)\n    at js_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:204:11)\n    at croak (/usr/local/lib/node_modules/uglify-js/lib/parse.js:679:41)\n    at token_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:683:9)\n    at expect_token (/usr/local/lib/node_modules/uglify-js/lib/parse.js:696:9)\n    at expect (/usr/local/lib/node_modules/uglify-js/lib/parse.js:699:36)\n    at parenthesised (/usr/local/lib/node_modules/uglify-js/lib/parse.js:713:9)\n    at if_ (/usr/local/lib/node_modules/uglify-js/lib/parse.js:985:20)\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:813:24\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:722:24\n'
```

I've created a PR for this which can be found here: https://github.com/django-compressor/django-compressor/pull/610



Discussion:
I've created [a gist](https://gist.github.com/kottenator/fe4b0a05d1645af6986fbfdd407c57a7) that _patches_ `django-compressor` - it's a quick-and-dirty solution for my project. Maybe you'll find it useful.

may i ask, why do you have different `STATIC_URL`s?

Sure, for example, I host my Django app on multiple sub-paths of the same domain, like:
- http://example.com/dev/my-app/
- http://example.com/staging/my-app/

Unfortunately, we can't use sub-domains for that. So the static files are hosted at:
- http://example.com/dev/my-app/static/
- http://example.com/staging/my-app/static/

I configure `STATIC_URL` via environment variable, so it's `/dev/my-app/static/` and `/staging/my-app/static/` correspondingly.

But `django-compressor` has generated static files with `STATIC_URL = '/static/'`.

Can't you use a different offline manifest for both environments ?

I build Django project distribution once and deploy it on multiple hosts. The dist doesn't know about where it'll be deployed - it's a matter of configuration.

We have similar use-case - app is built completely once and then we need to deploy it at arbitrary locations (sub-paths) with different configuration (generally following 12factor approach). Hence we set `STATIC_URL` for each deployment via configuration and it doesn't work with `django-compressor` as it is now. Also we cannot 'pre-build' manifests for 'different environments' since (1) they're arbitrary and (2) we follow the policy of immutable artifacts (exactly the same code and files deployed each time).

Are there any issues with omitting `STATIC_URL` from values in manifests?..

Having same issue in similar use-case.

+1 for my friend :)

Please don't comment if you have nothing meaningful to add. If you care about this functionality, implement it, with good tests and care for backwards-compatibility, and submit a pull request. Thanks.

I'm going to prepare a PR on Thursday or Friday. It may look _hacky_ but it works :)

Pull-request created.
@karyon - I saw you've added `needinfo` label. Do you need any information from me? Can I help?
yeah i just wanted to follow up regarding the pull request. now you've created one :)


Resolving patch:
diff --git a/compressor/base.py b/compressor/base.py
--- a/compressor/base.py
+++ b/compressor/base.py
@@ -14,7 +14,7 @@
 from compressor.exceptions import (CompressorError, UncompressableFileError,
         FilterDoesNotExist)
 from compressor.filters import CachedCompilerFilter
-from compressor.filters.css_default import CssAbsoluteFilter
+from compressor.filters.css_default import CssAbsoluteFilter, CssRelativeFilter
 from compressor.storage import compressor_file_storage
 from compressor.signals import post_compress
 from compressor.utils import get_class, get_mod_func, staticfiles
@@ -211,6 +211,8 @@ def hunks(self, forced=False):
                 # on precompiled css files even if compression is disabled.
                 if CssAbsoluteFilter in self.cached_filters:
                     value = self.filter(value, [CssAbsoluteFilter], **options)
+                elif CssRelativeFilter in self.cached_filters:
+                    value = self.filter(value, [CssRelativeFilter], **options)
                 yield self.handle_output(kind, value, forced=True,
                                          basename=basename)
             else:
diff --git a/compressor/filters/css_default.py b/compressor/filters/css_default.py
--- a/compressor/filters/css_default.py
+++ b/compressor/filters/css_default.py
@@ -77,8 +77,8 @@ def add_suffix(self, url):
             suffix = get_hashed_content(filename)
         else:
             raise FilterError('COMPRESS_CSS_HASHING_METHOD is configured '
-                                  'with an unknown method (%s).' %
-                                  settings.COMPRESS_CSS_HASHING_METHOD)
+                              'with an unknown method (%s).' %
+                              settings.COMPRESS_CSS_HASHING_METHOD)
         fragment = None
         if "#" in url:
             url, fragment = url.rsplit("#", 1)
@@ -99,7 +99,14 @@ def _converter(self, url):
                                                 url]))
         if self.has_scheme:
             full_url = "%s%s" % (self.protocol, full_url)
-        return self.add_suffix(full_url)
+        full_url = self.add_suffix(full_url)
+        return self.post_process_url(full_url)
+
+    def post_process_url(self, url):
+        """
+        Extra URL processing, to be overridden in subclasses.
+        """
+        return url
 
     def url_converter(self, matchobj):
         quote = matchobj.group(1)
@@ -110,3 +117,38 @@ def src_converter(self, matchobj):
         quote = matchobj.group(1)
         converted_url = self._converter(matchobj.group(2))
         return "src=%s%s%s" % (quote, converted_url, quote)
+
+
+class CssRelativeFilter(CssAbsoluteFilter):
+    """
+    Do similar to ``CssAbsoluteFilter`` URL processing
+    but add a *relative URL prefix* instead of ``settings.COMPRESS_URL``.
+    """
+    def post_process_url(self, url):
+        """
+        Replace ``settings.COMPRESS_URL`` URL prefix with  '../' * (N + 1)
+        where N is the *depth* of ``settings.COMPRESS_OUTPUT_DIR`` folder.
+
+        E.g. by default ``settings.COMPRESS_OUTPUT_DIR == 'CACHE'``,
+        the depth is 1, and the prefix will be '../../'.
+
+        If ``settings.COMPRESS_OUTPUT_DIR == 'my/compiled/data'``,
+        the depth is 3, and the prefix will be '../../../../'.
+
+        Example:
+
+        - original file URL: '/static/my-app/style.css'
+        - it has an image link: ``url(images/logo.svg)``
+        - compiled file URL: '/static/CACHE/css/abcdef123456.css'
+        - replaced image link URL: ``url(../../my-app/images/logo.svg)``
+        """
+        old_prefix = self.url
+        if self.has_scheme:
+            old_prefix = '{}{}'.format(self.protocol, old_prefix)
+        # One level up from 'css' / 'js' folder
+        new_prefix = '..'
+        # N levels up from ``settings.COMPRESS_OUTPUT_DIR``
+        new_prefix += '/..' * len(list(filter(
+            None, os.path.normpath(settings.COMPRESS_OUTPUT_DIR).split(os.sep)
+        )))
+        return re.sub('^{}'.format(old_prefix), new_prefix, url)


Test patch:
diff --git a/compressor/tests/test_filters.py b/compressor/tests/test_filters.py
--- a/compressor/tests/test_filters.py
+++ b/compressor/tests/test_filters.py
@@ -15,7 +15,7 @@
 from compressor.css import CssCompressor
 from compressor.filters.base import CompilerFilter, CachedCompilerFilter
 from compressor.filters.cssmin import CSSCompressorFilter, rCSSMinFilter
-from compressor.filters.css_default import CssAbsoluteFilter
+from compressor.filters.css_default import CssAbsoluteFilter, CssRelativeFilter
 from compressor.filters.jsmin import JSMinFilter
 from compressor.filters.template import TemplateFilter
 from compressor.filters.closure import ClosureCompilerFilter
@@ -209,6 +209,11 @@ class CssAbsolutizingTestCase(TestCase):
     hashing_func = staticmethod(get_hashed_mtime)
     template = ("p { background: url('%(url)simg/python.png%(query)s%(hash)s%(frag)s') }"
                 "p { filter: Alpha(src='%(url)simg/python.png%(query)s%(hash)s%(frag)s') }")
+    filter_class = CssAbsoluteFilter
+
+    @property
+    def expected_url_prefix(self):
+        return settings.COMPRESS_URL
 
     def setUp(self):
         self.override_settings = self.settings(COMPRESS_CSS_HASHING_METHOD=self.hashing_method)
@@ -222,10 +227,10 @@ def test_css_no_hash(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         content = self.template % blankdict(url='../../')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter(self):
@@ -233,11 +238,11 @@ def test_css_absolute_filter(self):
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
         content = self.template % blankdict(url='../../')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
             'hash': '?' + self.hashing_func(imagefilename),
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_url_fragment(self):
@@ -245,24 +250,24 @@ def test_css_absolute_filter_url_fragment(self):
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
         content = self.template % blankdict(url='../../', frag='#foo')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
             'hash': '?' + self.hashing_func(imagefilename),
             'frag': '#foo',
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_only_url_fragment(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         content = "p { background: url('#foo') }"
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(content, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_only_url_fragment_wrap_double_quotes(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         content = 'p { background: url("#foo") }'
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(content, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_querystring(self):
@@ -270,12 +275,12 @@ def test_css_absolute_filter_querystring(self):
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
         content = self.template % blankdict(url='../../', query='?foo')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
             'query': '?foo',
             'hash': '&' + self.hashing_func(imagefilename),
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_https(self):
@@ -287,21 +292,21 @@ def test_css_absolute_filter_relative_path(self):
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
         content = self.template % blankdict(url='../../')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
             'hash': '?' + self.hashing_func(imagefilename),
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_filename_outside_compress_root(self):
         filename = '/foo/bar/baz/test.css'
         content = self.template % blankdict(url='../qux/')
         params = blankdict({
-            'url': settings.COMPRESS_URL + 'bar/qux/',
+            'url': self.expected_url_prefix + 'bar/qux/',
         })
         output = self.template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='bar/baz/test.css'))
 
     def test_css_hunks(self):
@@ -314,7 +319,7 @@ def test_css_hunks(self):
 p { background: url(%(compress_url)simg/python.png?%(hash)s); }
 p { background: url('%(compress_url)simg/python.png?%(hash)s'); }
 p { filter: progid:DXImageTransform.Microsoft.AlphaImageLoader(src='%(compress_url)simg/python.png?%(hash)s'); }
-""" % dict(compress_url=settings.COMPRESS_URL, hash=hash_python_png)
+""" % dict(compress_url=self.expected_url_prefix, hash=hash_python_png)
 
         css2 = """\
 p { background: url('%(compress_url)simg/add.png?%(hash)s'); }
@@ -322,7 +327,7 @@ def test_css_hunks(self):
 p { background: url(%(compress_url)simg/add.png?%(hash)s); }
 p { background: url('%(compress_url)simg/add.png?%(hash)s'); }
 p { filter: progid:DXImageTransform.Microsoft.AlphaImageLoader(src='%(compress_url)simg/add.png?%(hash)s'); }
-""" % dict(compress_url=settings.COMPRESS_URL, hash=hash_add_png)
+""" % dict(compress_url=self.expected_url_prefix, hash=hash_add_png)
 
         css = """
         <link rel="stylesheet" href="/static/css/url/url1.css" type="text/css">
@@ -336,7 +341,7 @@ def test_guess_filename(self):
         url = '%s/img/python.png' % settings.COMPRESS_URL.rstrip('/')
         path = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
         content = "p { background: url('%s') }" % url
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(path, filter.guess_filename(url))
 
     def test_filenames_with_space(self):
@@ -347,23 +352,23 @@ def test_filenames_with_space(self):
 
         content = template % blankdict(url='../../')
         params = blankdict({
-            'url': settings.COMPRESS_URL,
+            'url': self.expected_url_prefix,
             'hash': '?' + self.hashing_func(imagefilename),
         })
         output = template % params
-        filter = CssAbsoluteFilter(content)
+        filter = self.filter_class(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_does_not_change_nested_urls(self):
         css = """body { background-image: url("data:image/svg+xml;utf8,<svg><rect fill='url(%23gradient)'/></svg>");}"""
-        filter = CssAbsoluteFilter(css, filename="doesntmatter")
+        filter = self.filter_class(css, filename="doesntmatter")
         self.assertEqual(css, filter.input(filename="doesntmatter", basename="doesntmatter"))
 
     def test_does_not_change_quotes_in_src(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         hash_add_png = self.hashing_func(os.path.join(settings.COMPRESS_ROOT, 'img/add.png'))
         css = """p { filter: Alpha(src="/img/add.png%(hash)s") }"""
-        filter = CssAbsoluteFilter(css % dict(hash=""))
+        filter = self.filter_class(css % dict(hash=""))
         expected = css % dict(hash='?' + hash_add_png)
         self.assertEqual(expected, filter.input(filename=filename, basename='css/url/test.css'))
 
@@ -378,6 +383,31 @@ class CssAbsolutizingTestCaseWithHash(CssAbsolutizingTestCase):
     hashing_func = staticmethod(get_hashed_content)
 
 
+@override_settings(
+    COMPRESS_ENABLED=True,
+    COMPRESS_URL='/static/',
+    COMPRESS_CSS_FILTERS=['compressor.filters.css_default.CssRelativeFilter']
+)
+class CssRelativizingTestCase(CssAbsolutizingTestCase):
+    filter_class = CssRelativeFilter
+    expected_url_prefix = '../../'
+
+    @override_settings(
+        COMPRESS_CSS_HASHING_METHOD=None,
+        COMPRESS_OUTPUT_DIR='CACHE/in/depth'
+    )
+    def test_nested_cache_dir(self):
+        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
+        content = self.template % blankdict(url='../../')
+        params = blankdict({
+            'url': '../../../../',
+        })
+        output = self.template % params
+        filter = self.filter_class(content)
+        self.assertEqual(output, filter.input(filename=filename,
+                                              basename='css/url/test.css'))
+
+
 @override_settings(
     COMPRESS_ENABLED=True,
     COMPRESS_CSS_FILTERS=[
diff --git a/compressor/tests/test_mtime_cache.py b/compressor/tests/test_mtime_cache.py
--- a/compressor/tests/test_mtime_cache.py
+++ b/compressor/tests/test_mtime_cache.py
@@ -8,7 +8,7 @@ class TestMtimeCacheCommand(TestCase):
     # FIXME: add actual tests, improve the existing ones.
 
     exclusion_patterns = [
-        '*CACHE*', '*custom*', '*d728fc7f9301.js', 'test.txt*'
+        '*CACHE*', '*custom*', '*066cd253eada.js', '*d728fc7f9301.js', 'test.txt*'
     ]
 
     def default_ignore(self):