Repository: django-compressor/django-compressor

Issue:
UnicodeDecodeError when creating error message string
Bug happens in [compressor/management/commands/compress.py](https://github.com/django-compressor/django-compressor/blob/master/compressor/management/commands/compress.py#L230-L231)

I ran in to this when the output of uglify-js (the compressor we're using) is outputting something that can't be read as a regular Python string.

When I was debugging:

```
(Pdb) "An error occured during rendering %s: %s" % (template.template_name, e.message)
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 124: ordinal not in range(128)
(Pdb) "An error occured during rendering {template}: {error}".format(template=template.template_name, error=e)
'An error occured during rendering /Users/mrpi/Workspace/.../assets.html: util.error: Use console.error instead\nParse error at -:29,7\nutil.error: Use console.error instead\nUnexpected token operator \xc2\xab!\xc2\xbb, expected punc \xc2\xab(\xc2\xbb\nutil.error: Use console.error instead\nError\n    at new JS_Parse_Error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:196:18)\n    at js_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:204:11)\n    at croak (/usr/local/lib/node_modules/uglify-js/lib/parse.js:679:41)\n    at token_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:683:9)\n    at expect_token (/usr/local/lib/node_modules/uglify-js/lib/parse.js:696:9)\n    at expect (/usr/local/lib/node_modules/uglify-js/lib/parse.js:699:36)\n    at parenthesised (/usr/local/lib/node_modules/uglify-js/lib/parse.js:713:9)\n    at if_ (/usr/local/lib/node_modules/uglify-js/lib/parse.js:985:20)\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:813:24\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:722:24\n'
```

I've created a PR for this which can be found here: https://github.com/django-compressor/django-compressor/pull/610

UnicodeDecodeError when creating error message string
Bug happens in [compressor/management/commands/compress.py](https://github.com/django-compressor/django-compressor/blob/master/compressor/management/commands/compress.py#L230-L231)

I ran in to this when the output of uglify-js (the compressor we're using) is outputting something that can't be read as a regular Python string.

When I was debugging:

```
(Pdb) "An error occured during rendering %s: %s" % (template.template_name, e.message)
*** UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 124: ordinal not in range(128)
(Pdb) "An error occured during rendering {template}: {error}".format(template=template.template_name, error=e)
'An error occured during rendering /Users/mrpi/Workspace/.../assets.html: util.error: Use console.error instead\nParse error at -:29,7\nutil.error: Use console.error instead\nUnexpected token operator \xc2\xab!\xc2\xbb, expected punc \xc2\xab(\xc2\xbb\nutil.error: Use console.error instead\nError\n    at new JS_Parse_Error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:196:18)\n    at js_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:204:11)\n    at croak (/usr/local/lib/node_modules/uglify-js/lib/parse.js:679:41)\n    at token_error (/usr/local/lib/node_modules/uglify-js/lib/parse.js:683:9)\n    at expect_token (/usr/local/lib/node_modules/uglify-js/lib/parse.js:696:9)\n    at expect (/usr/local/lib/node_modules/uglify-js/lib/parse.js:699:36)\n    at parenthesised (/usr/local/lib/node_modules/uglify-js/lib/parse.js:713:9)\n    at if_ (/usr/local/lib/node_modules/uglify-js/lib/parse.js:985:20)\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:813:24\n    at /usr/local/lib/node_modules/uglify-js/lib/parse.js:722:24\n'
```

I've created a PR for this which can be found here: https://github.com/django-compressor/django-compressor/pull/610



Discussion:

Resolving patch:
diff --git a/compressor/management/commands/compress.py b/compressor/management/commands/compress.py
--- a/compressor/management/commands/compress.py
+++ b/compressor/management/commands/compress.py
@@ -1,3 +1,4 @@
+from __future__ import unicode_literals
 # flake8: noqa
 import os
 import sys
@@ -11,6 +12,7 @@
 import django.template
 from django.template import Context
 from django.utils import six
+from django.utils.encoding import smart_text
 from django.template.loader import get_template  # noqa Leave this in to preload template locations
 from django.template import engines
 
@@ -119,7 +121,7 @@ def compress(self, log=None, **options):
                         'get_template_sources', None)
                     if get_template_sources is None:
                         get_template_sources = loader.get_template_sources
-                    paths.update(str(origin) for origin in get_template_sources(''))
+                    paths.update(smart_text(origin) for origin in get_template_sources(''))
                 except (ImportError, AttributeError, TypeError):
                     # Yeah, this didn't work out so well, let's move on
                     pass
@@ -177,7 +179,7 @@ def compress(self, log=None, **options):
                 continue
             except TemplateSyntaxError as e:  # broken template -> ignore
                 if verbosity > 0:
-                    log.write("Invalid template %s: %s\n" % (template_name, e))
+                    log.write("Invalid template %s: %s\n" % (template_name, smart_text(e)))
                 continue
             except TemplateDoesNotExist:  # non existent template -> ignore
                 if verbosity > 0:
@@ -197,7 +199,8 @@ def compress(self, log=None, **options):
                 except (TemplateDoesNotExist, TemplateSyntaxError) as e:
                     # Could be an error in some base template
                     if verbosity > 0:
-                        log.write("Error parsing template %s: %s\n" % (template_name, e))
+                        log.write("Error parsing template %s: %s\n" %
+                                  (template_name, smart_text(e)))
                     continue
                 if nodes:
                     template.template_name = template_name
@@ -244,7 +247,7 @@ def compress(self, log=None, **options):
                         result = parser.render_node(template, context, node)
                     except Exception as e:
                         raise CommandError("An error occurred during rendering %s: "
-                                           "%s" % (template.template_name, e))
+                                           "%s" % (template.template_name, smart_text(e)))
                     offline_manifest[key] = result
                     context.pop()
                     results.append(result)


Test patch:
diff --git a/compressor/tests/test_offline.py b/compressor/tests/test_offline.py
--- a/compressor/tests/test_offline.py
+++ b/compressor/tests/test_offline.py
@@ -265,6 +265,12 @@ def test_get_loaders(self):
             self.assertTrue(isinstance(loaders[0], FileSystemLoader))
             self.assertTrue(isinstance(loaders[1], AppDirectoriesLoader))
 
+    @patch("compressor.offline.django.DjangoParser.render_node",
+           side_effect=Exception(b"non-ascii character here:\xc3\xa4"))
+    def test_non_ascii_exception_messages(self, mock):
+        with self.assertRaises(CommandError):
+            CompressCommand().handle()
+
 
 class OfflineCompressSkipDuplicatesTestCase(OfflineTestCaseMixin, TestCase):
     templates_dir = 'test_duplicate'