Repository: django-compressor/django-compressor

Issue:
CssAbsoluteFilter only seems to ever work with settings.DEBUG enabled
We have a virtualenv that holds a project and its dependencies. All dependencies use `django.contrib.staticfiles` and the files in question are deployed directly to S3.

The checks at the beginning of the `input()` method of `CssAbsoluteFilter` to always fail.

Relevant code:

``` python
class CssAbsoluteFilter(FilterBase):

    # ...

    def input(self, filename=None, basename=None, **kwargs):
        if filename is not None:
            filename = os.path.normcase(os.path.abspath(filename))
        if (not (filename and filename.startswith(self.root)) and
                not self.find(basename)):
            # this will always fail:
            # filename = '/opt/envs/foo/src/some-dependency/app/static/css/example.css'
            # self.root = '/opt/envs/foo/src/project/static'
            return self.content
        # ...

    def find(self, basename):
        # settings.DEBUG is False so this method is irrelevant
        if settings.DEBUG and basename and staticfiles.finders:
            return staticfiles.finders.find(basename)
```

Why does it need to check whether the file lives inside of `COMPRESS_ROOT` if it seems to be mainly an output directory?



Discussion:
The purpose of `CssAbsoluteFilter` is to replace relative paths in CSS with absolute paths, considering that the compressed CSS will live in a location that's different from the original CSS path.  It depends on `settings.COMPRESS_ROOT` because that's assumed to be a path that's being served by--or at least reflects the content of--the static file server.  The `staticfiles.finders` bit is reserved to `settings.DEBUG` because production servers will have run `manage.py collectstatic` rather than using Django staticfiles to serve from app directories.

Just laying the groundwork there; hope I'm not just reiterating things you already know.  Have you seen http://django-compressor.readthedocs.org/en/latest/remote-storages/ ?

I know how compressor works, I've pasted parts of its source code. The problem is `COMPRESS_ROOT` is a destination path. There is no reason for source CSS files to live in it, especially given how `staticfiles` works (source files scattered across applications).

Yes, I have seen the remote storage documentation. In fact that's how I got S3 to work in the first place.

I don't think you understand the problem. I know the `find()` function is irrelevant. The comments in the pasted code are mine. The problem is none of my source files are ever going to have `COMPRESS_ROOT` as their prefix because that would require moving all of the static files into a single directory which is the opposite of why we use `django.contrib.staticfiles`.

Actually it's not clear that you understand how `staticfiles` is intended to work in production with `collectstatic`.  The first line of https://docs.djangoproject.com/en/dev/ref/contrib/staticfiles/ says "django.contrib.staticfiles collects static files from each of your applications (and any other places you specify) into a single location that can easily be served in production."  Note the "single location" part. That should be the same as `COMPRESS_ROOT` which is why it defaults to `STATIC_ROOT`

If I'm misunderstanding the problem, perhaps you could propose how you think it should work differently?

In case of S3 that "single location" is actually Amazon. That's why I'm asking about the need of having the prefix check. If you change that entire condition to `if not filename:` then it works properly even if no files are ever copied on the local disk (and why should they?).


Resolving patch:
diff --git a/compressor/filters/css_default.py b/compressor/filters/css_default.py
--- a/compressor/filters/css_default.py
+++ b/compressor/filters/css_default.py
@@ -5,7 +5,6 @@
 from compressor.cache import get_hashed_mtime, get_hashed_content
 from compressor.conf import settings
 from compressor.filters import FilterBase, FilterError
-from compressor.utils import staticfiles
 
 URL_PATTERN = re.compile(r'url\(([^\)]+)\)')
 SRC_PATTERN = re.compile(r'src=([\'"])(.+?)\1')
@@ -22,10 +21,7 @@ def __init__(self, *args, **kwargs):
         self.has_scheme = False
 
     def input(self, filename=None, basename=None, **kwargs):
-        if filename is not None:
-            filename = os.path.normcase(os.path.abspath(filename))
-        if (not (filename and filename.startswith(self.root)) and
-                not self.find(basename)):
+        if not filename:
             return self.content
         self.path = basename.replace(os.sep, '/')
         self.path = self.path.lstrip('/')
@@ -40,10 +36,6 @@ def input(self, filename=None, basename=None, **kwargs):
         return SRC_PATTERN.sub(self.src_converter,
             URL_PATTERN.sub(self.url_converter, self.content))
 
-    def find(self, basename):
-        if settings.DEBUG and basename and staticfiles.finders:
-            return staticfiles.finders.find(basename)
-
     def guess_filename(self, url):
         local_path = url
         if self.has_scheme:


Test patch:
diff --git a/compressor/tests/test_filters.py b/compressor/tests/test_filters.py
--- a/compressor/tests/test_filters.py
+++ b/compressor/tests/test_filters.py
@@ -1,4 +1,5 @@
 from __future__ import with_statement, unicode_literals
+from collections import defaultdict
 import io
 import os
 import sys
@@ -20,6 +21,10 @@
 from compressor.tests.test_base import test_dir
 
 
+def blankdict(*args, **kwargs):
+    return defaultdict(lambda: '', *args, **kwargs)
+
+
 @unittest.skipIf(find_command(settings.COMPRESS_CSSTIDY_BINARY) is None,
                  'CSStidy binary %r not found' % settings.COMPRESS_CSSTIDY_BINARY)
 class CssTidyTestCase(TestCase):
@@ -99,8 +104,8 @@ def test_cssmin_filter(self):
 class CssAbsolutizingTestCase(TestCase):
     hashing_method = 'mtime'
     hashing_func = staticmethod(get_hashed_mtime)
-    content = ("p { background: url('../../img/python.png') }"
-               "p { filter: Alpha(src='../../img/python.png') }")
+    template = ("p { background: url('%(url)simg/python.png%(query)s%(hash)s%(frag)s') }"
+                "p { filter: Alpha(src='%(url)simg/python.png%(query)s%(hash)s%(frag)s') }")
 
     def setUp(self):
         self.old_enabled = settings.COMPRESS_ENABLED
@@ -123,37 +128,34 @@ def tearDown(self):
     def test_css_absolute_filter(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
-        params = {
+        content = self.template % blankdict(url='../../')
+        params = blankdict({
             'url': settings.COMPRESS_URL,
-            'hash': self.hashing_func(imagefilename),
-        }
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
-        filter = CssAbsoluteFilter(self.content)
+            'hash': '?' + self.hashing_func(imagefilename),
+        })
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = params['url'] = 'http://static.example.com/'
-        filter = CssAbsoluteFilter(self.content)
-        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_url_fragment(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
-        params = {
+        content = self.template % blankdict(url='../../', frag='#foo')
+        params = blankdict({
             'url': settings.COMPRESS_URL,
-            'hash': self.hashing_func(imagefilename),
-        }
-        content = "p { background: url('../../img/python.png#foo') }"
-
-        output = "p { background: url('%(url)simg/python.png?%(hash)s#foo') }" % params
+            'hash': '?' + self.hashing_func(imagefilename),
+            'frag': '#foo',
+        })
+        output = self.template % params
         filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = params['url'] = 'http://media.example.com/'
+        output = self.template % params
         filter = CssAbsoluteFilter(content)
-        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
-        output = "p { background: url('%(url)simg/python.png?%(hash)s#foo') }" % params
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_only_url_fragment(self):
@@ -163,62 +165,72 @@ def test_css_absolute_filter_only_url_fragment(self):
         self.assertEqual(content, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = 'http://media.example.com/'
         filter = CssAbsoluteFilter(content)
-        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         self.assertEqual(content, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_querystring(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
-        params = {
+        content = self.template % blankdict(url='../../', query='?foo')
+        params = blankdict({
             'url': settings.COMPRESS_URL,
-            'hash': self.hashing_func(imagefilename),
-        }
-        content = "p { background: url('../../img/python.png?foo') }"
-
-        output = "p { background: url('%(url)simg/python.png?foo&%(hash)s') }" % params
+            'query': '?foo',
+            'hash': '&' + self.hashing_func(imagefilename),
+        })
+        output = self.template % params
         filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = params['url'] = 'http://media.example.com/'
+        output = self.template % params
         filter = CssAbsoluteFilter(content)
-        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
-        output = "p { background: url('%(url)simg/python.png?foo&%(hash)s') }" % params
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_https(self):
         filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
-        params = {
+        content = self.template % blankdict(url='../../')
+        params = blankdict({
             'url': settings.COMPRESS_URL,
-            'hash': self.hashing_func(imagefilename),
-        }
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
-        filter = CssAbsoluteFilter(self.content)
+            'hash': '?' + self.hashing_func(imagefilename),
+        })
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = params['url'] = 'https://static.example.com/'
-        filter = CssAbsoluteFilter(self.content)
-        filename = os.path.join(settings.COMPRESS_ROOT, 'css/url/test.css')
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
     def test_css_absolute_filter_relative_path(self):
         filename = os.path.join(settings.TEST_DIR, 'whatever', '..', 'static', 'whatever/../css/url/test.css')
         imagefilename = os.path.join(settings.COMPRESS_ROOT, 'img/python.png')
-        params = {
+        content = self.template % blankdict(url='../../')
+        params = blankdict({
             'url': settings.COMPRESS_URL,
-            'hash': self.hashing_func(imagefilename),
-        }
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
-        filter = CssAbsoluteFilter(self.content)
+            'hash': '?' + self.hashing_func(imagefilename),
+        })
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
         settings.COMPRESS_URL = params['url'] = 'https://static.example.com/'
-        filter = CssAbsoluteFilter(self.content)
-        output = ("p { background: url('%(url)simg/python.png?%(hash)s') }"
-                  "p { filter: Alpha(src='%(url)simg/python.png?%(hash)s') }") % params
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
         self.assertEqual(output, filter.input(filename=filename, basename='css/url/test.css'))
 
+    def test_css_absolute_filter_filename_outside_compress_root(self):
+        filename = '/foo/bar/baz/test.css'
+        content = self.template % blankdict(url='../qux/')
+        params = blankdict({
+            'url': settings.COMPRESS_URL + 'bar/qux/',
+        })
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
+        self.assertEqual(output, filter.input(filename=filename, basename='bar/baz/test.css'))
+        settings.COMPRESS_URL = 'https://static.example.com/'
+        params['url'] = settings.COMPRESS_URL + 'bar/qux/'
+        output = self.template % params
+        filter = CssAbsoluteFilter(content)
+        self.assertEqual(output, filter.input(filename=filename, basename='bar/baz/test.css'))
+
     def test_css_hunks(self):
         hash_dict = {
             'hash1': self.hashing_func(os.path.join(settings.COMPRESS_ROOT, 'img/python.png')),