Repository: python-babel/babel
Pull request #399 (2016-04-27 19:29:30)

Default to UTF-8 source encoding (PEP8: python3)
The following python3 source file:

```python3
print(_('☃'))
```

has UTF-8 encoding (by default without a pragma).  The code I touched (before my change) caused this to be scanned as latin-1 (mojibake)

The section of PEP8 which covers this: http://legacy.python.org/dev/peps/pep-0008/#source-file-encoding

jtwang: Changes look good to me. I have no idea why codecov croaked. @akx ?
asottile: forgot to import pytest, maybe that's why it croaked /me rebases
akx: This is technically fine as a patch -- thanks @asottile -- but I'm wondering... Should we maybe default to UTF-8 on _both_ Py2 and Py3?

I kind of dislike the idea that Babel acts differently depending on the platform it's being run on, especially considering all of the other compat stuff we have that makes it work equivalently on both!

I kinda do think it'd be better to assume UTF-8 on _every_ .py file regardless of interpreter version, unless told otherwise. Is there something I'm missing with that approach?
asottile: I thought about that as well too, the only issue I have with that is it would cause babel to potentially scan files which are syntactically incorrect -- not sure how much of an issue that is though.  In most tools I've written I've defaulted to UTF-8 in both to simplify code so maybe it's OK here too?
akx: I think it's okay. Really, as an user of Babel, if you have a source file that

* has internationalizable strings
* is using an esoteric encoding
* doesn't declare the encoding in the standard-format header

then you have bigger problems already. :grin: 
asottile: @akx now defaulting to UTF-8 everywhere :)

Patch:
diff --git a/babel/messages/extract.py b/babel/messages/extract.py
index 7162627..4dc56a4 100644
--- a/babel/messages/extract.py
+++ b/babel/messages/extract.py
@@ -398,7 +398,7 @@ def extract_python(fileobj, keywords, comment_tags, options):
     in_def = in_translator_comments = False
     comment_tag = None
 
-    encoding = parse_encoding(fileobj) or options.get('encoding', 'iso-8859-1')
+    encoding = parse_encoding(fileobj) or options.get('encoding', 'UTF-8')
 
     if PY2:
         next_line = fileobj.readline


Test patch:
diff --git a/tests/messages/test_extract.py b/tests/messages/test_extract.py
index db03af2..9d78d92 100644
--- a/tests/messages/test_extract.py
+++ b/tests/messages/test_extract.py
@@ -50,6 +50,21 @@ msg10 = dngettext(getDomain(), 'Page', 'Pages', 3)
             (10, 'dngettext', (None, u'Page', u'Pages', None), [])],
             messages)
 
+    def test_extract_default_encoding_ascii(self):
+        buf = BytesIO(b'_("a")')
+        messages = list(extract.extract_python(
+            buf, list(extract.DEFAULT_KEYWORDS), [], {},
+        ))
+        # Should work great in both py2 and py3
+        self.assertEqual([(1, '_', 'a', [])], messages)
+
+    def test_extract_default_encoding_utf8(self):
+        buf = BytesIO(u'_("☃")'.encode('UTF-8'))
+        messages = list(extract.extract_python(
+            buf, list(extract.DEFAULT_KEYWORDS), [], {},
+        ))
+        self.assertEqual([(1, '_', u'☃', [])], messages)
+
     def test_nested_comments(self):
         buf = BytesIO(b"""\
 msg = ngettext('pylon',  # TRANSLATORS: shouldn't be