Repository: gratipay/gratipay.com

Issue:
we choke on country codes in i18n
I tried deploying the `pt_BR` translation and our Heroku app crashed with this error ([IRC](https://botbot.me/freenode/gratipay/2014-11-03/?msg=24767397&page=3)):

```
2014-11-03T21:46:51.529221+00:00 app[web.1]: 2014-11-03 21:46:51 [2] [INFO] Reason: Worker failed to boot.
2014-11-03T21:46:51.390578+00:00 app[web.1]: 2014-11-03 21:46:51 [7] [ERROR] Exception in worker process:
2014-11-03T21:46:51.390584+00:00 app[web.1]: Traceback (most recent call last):
2014-11-03T21:46:51.390586+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2014-11-03T21:46:51.390587+00:00 app[web.1]:     worker.init_process()
2014-11-03T21:46:51.390589+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2014-11-03T21:46:51.390590+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2014-11-03T21:46:51.390592+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2014-11-03T21:46:51.390593+00:00 app[web.1]:     self.callable = self.load()
2014-11-03T21:46:51.390595+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2014-11-03T21:46:51.390596+00:00 app[web.1]:     return self.load_wsgiapp()
2014-11-03T21:46:51.390597+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2014-11-03T21:46:51.390599+00:00 app[web.1]:     return util.import_app(self.app_uri)
2014-11-03T21:46:51.390600+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2014-11-03T21:46:51.390602+00:00 app[web.1]:     __import__(module)
2014-11-03T21:46:51.390603+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/wsgi.py", line 20, in <module>
2014-11-03T21:46:51.427791+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/honestbleeps/public.json?callback=_req8" host=gratipay.com request_id=dc67af37-017c-43c7-a0eb-64e44dbde4d2 fwd="67.185.195.170" dyno=web.1 connect=1ms service=528ms status=503 bytes=0
2014-11-03T21:46:51.390605+00:00 app[web.1]:     website = Website([])
2014-11-03T21:46:51.390606+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/website.py", line 35, in __init__
2014-11-03T21:46:51.390607+00:00 app[web.1]:     self.configure(argv)
2014-11-03T21:46:51.390609+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/configuration/__init__.py", line 295, in configure
2014-11-03T21:46:51.390610+00:00 app[web.1]:     self.run_config_scripts()
2014-11-03T21:46:51.390612+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/configuration/__init__.py", line 338, in run_config_scripts
2014-11-03T21:46:51.390613+00:00 app[web.1]:     execfile(filepath, {'website': self})
2014-11-03T21:46:51.390615+00:00 app[web.1]:   File "/app/configure-aspen.py", line 12, in <module>
2014-11-03T21:46:51.390616+00:00 app[web.1]:     from gratipay.utils import cache_static, i18n, set_cookie, timer
2014-11-03T21:46:51.390618+00:00 app[web.1]:   File "/app/gratipay/utils/i18n.py", line 102, in <module>
2014-11-03T21:46:51.390619+00:00 app[web.1]:     LOCALES = load_langs("i18n/core")
2014-11-03T21:46:51.390620+00:00 app[web.1]:   File "/app/gratipay/utils/i18n.py", line 89, in load_langs
2014-11-03T21:46:51.390622+00:00 app[web.1]:     l = langs[lang] = Locale(lang)
2014-11-03T21:46:51.390623+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/babel/core.py", line 150, in __init__
2014-11-03T21:46:51.390624+00:00 app[web.1]:     raise UnknownLocaleError(identifier)
2014-11-03T21:46:51.390626+00:00 app[web.1]: UnknownLocaleError: unknown locale 'pt_br'
```

We're currently rolled back on Heroku.

we choke on country codes in i18n
I tried deploying the `pt_BR` translation and our Heroku app crashed with this error ([IRC](https://botbot.me/freenode/gratipay/2014-11-03/?msg=24767397&page=3)):

```
2014-11-03T21:46:51.529221+00:00 app[web.1]: 2014-11-03 21:46:51 [2] [INFO] Reason: Worker failed to boot.
2014-11-03T21:46:51.390578+00:00 app[web.1]: 2014-11-03 21:46:51 [7] [ERROR] Exception in worker process:
2014-11-03T21:46:51.390584+00:00 app[web.1]: Traceback (most recent call last):
2014-11-03T21:46:51.390586+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker
2014-11-03T21:46:51.390587+00:00 app[web.1]:     worker.init_process()
2014-11-03T21:46:51.390589+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process
2014-11-03T21:46:51.390590+00:00 app[web.1]:     self.wsgi = self.app.wsgi()
2014-11-03T21:46:51.390592+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi
2014-11-03T21:46:51.390593+00:00 app[web.1]:     self.callable = self.load()
2014-11-03T21:46:51.390595+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load
2014-11-03T21:46:51.390596+00:00 app[web.1]:     return self.load_wsgiapp()
2014-11-03T21:46:51.390597+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp
2014-11-03T21:46:51.390599+00:00 app[web.1]:     return util.import_app(self.app_uri)
2014-11-03T21:46:51.390600+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app
2014-11-03T21:46:51.390602+00:00 app[web.1]:     __import__(module)
2014-11-03T21:46:51.390603+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/wsgi.py", line 20, in <module>
2014-11-03T21:46:51.427791+00:00 heroku[router]: at=error code=H13 desc="Connection closed without response" method=GET path="/honestbleeps/public.json?callback=_req8" host=gratipay.com request_id=dc67af37-017c-43c7-a0eb-64e44dbde4d2 fwd="67.185.195.170" dyno=web.1 connect=1ms service=528ms status=503 bytes=0
2014-11-03T21:46:51.390605+00:00 app[web.1]:     website = Website([])
2014-11-03T21:46:51.390606+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/website.py", line 35, in __init__
2014-11-03T21:46:51.390607+00:00 app[web.1]:     self.configure(argv)
2014-11-03T21:46:51.390609+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/configuration/__init__.py", line 295, in configure
2014-11-03T21:46:51.390610+00:00 app[web.1]:     self.run_config_scripts()
2014-11-03T21:46:51.390612+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/aspen/configuration/__init__.py", line 338, in run_config_scripts
2014-11-03T21:46:51.390613+00:00 app[web.1]:     execfile(filepath, {'website': self})
2014-11-03T21:46:51.390615+00:00 app[web.1]:   File "/app/configure-aspen.py", line 12, in <module>
2014-11-03T21:46:51.390616+00:00 app[web.1]:     from gratipay.utils import cache_static, i18n, set_cookie, timer
2014-11-03T21:46:51.390618+00:00 app[web.1]:   File "/app/gratipay/utils/i18n.py", line 102, in <module>
2014-11-03T21:46:51.390619+00:00 app[web.1]:     LOCALES = load_langs("i18n/core")
2014-11-03T21:46:51.390620+00:00 app[web.1]:   File "/app/gratipay/utils/i18n.py", line 89, in load_langs
2014-11-03T21:46:51.390622+00:00 app[web.1]:     l = langs[lang] = Locale(lang)
2014-11-03T21:46:51.390623+00:00 app[web.1]:   File "/app/.heroku/python/lib/python2.7/site-packages/babel/core.py", line 150, in __init__
2014-11-03T21:46:51.390624+00:00 app[web.1]:     raise UnknownLocaleError(identifier)
2014-11-03T21:46:51.390626+00:00 app[web.1]: UnknownLocaleError: unknown locale 'pt_br'
```

We're currently rolled back on Heroku.



Discussion:

Resolving patch:
diff --git a/configure-aspen.py b/configure-aspen.py
--- a/configure-aspen.py
+++ b/configure-aspen.py
@@ -67,6 +67,7 @@ def _set_cookie(response, *args, **kw):
 gratipay.wireup.billing(env)
 gratipay.wireup.username_restrictions(website)
 gratipay.wireup.nanswers(env)
+gratipay.wireup.load_i18n(website)
 gratipay.wireup.other_stuff(website, env)
 gratipay.wireup.accounts_elsewhere(website, env)
 
diff --git a/gratipay/utils/i18n.py b/gratipay/utils/i18n.py
--- a/gratipay/utils/i18n.py
+++ b/gratipay/utils/i18n.py
@@ -1,23 +1,24 @@
+# encoding: utf8
 from __future__ import print_function, unicode_literals
 
 from io import BytesIO
-import os
 import re
 from unicodedata import combining, normalize
 
 from aspen.resources.pagination import parse_specline, split_and_escape
 from aspen.utils import utcnow
-from babel.core import Locale
+from babel.core import LOCALE_ALIASES
 from babel.dates import format_timedelta
-from babel.messages.pofile import Catalog, read_po
 from babel.messages.extract import extract_python
 from babel.numbers import (
     format_currency, format_decimal, format_number, format_percent,
-    get_decimal_symbol, parse_decimal, parse_pattern
+    get_decimal_symbol, parse_decimal
 )
 import jinja2.ext
 
-from gratipay.utils import COUNTRIES, COUNTRIES_MAP
+
+ALIASES = {k: v.lower() for k, v in LOCALE_ALIASES.items()}
+ALIASES_R = {v: k for k, v in ALIASES.items()}
 
 
 ternary_re = re.compile(r'^\(? *(.+?) *\? *(.+?) *: *(.+?) *\)?$')
@@ -71,60 +72,50 @@ def to_age(dt, loc):
 
 
 def regularize_locale(loc):
-    return loc.split('-', 1)[0].lower()
+    if loc == 'no':
+
+        # There are two forms of written Norwegian, Bokmål and Nynorsk, and
+        # while ISO 639 includes `no` as a "macrolanguage", the CLDR (upon
+        # which Babel, our i18n/l10n library, depends), does not include it at
+        # all. Therefore, if a client sends `no` we interpret it as `nb_NO`.
+
+        loc = 'nb_NO'
+    return loc.replace('-', '_').lower()
+
+
+def regularize_locales(locales):
+    """Yield locale strings in the same format as they are in website.locales.
+    """
+    locales = [regularize_locale(loc) for loc in locales]
+    locales_set = set(locales)
+    for loc in locales:
+        yield loc
+        parts = loc.split('_')
+        if len(parts) > 1 and parts[0] not in locales_set:
+            # Insert "fr" after "fr_fr" if it's not somewhere in the list
+            yield parts[0]
+        alias = ALIASES.get(loc)
+        if alias and alias not in locales_set:
+            # Insert "fr_fr" after "fr" if it's not somewhere in the list
+            yield alias
 
 
 def strip_accents(s):
     return ''.join(c for c in normalize('NFKD', s) if not combining(c))
 
 
-def load_langs(localeDir):
-    key = lambda t: strip_accents(t[1])
-    langs = {}
-    for file in os.listdir(localeDir):
-        parts = file.split(".")
-        if len(parts) == 2 and parts[1] == "po":
-            lang = regularize_locale(parts[0])
-            with open(os.path.join(localeDir, file)) as f:
-                l = langs[lang] = Locale(lang)
-                c = l.catalog = read_po(f)
-                c.plural_func = get_function_from_rule(c.plural_expr)
-                try:
-                    l.countries_map = {k: l.territories[k] for k in COUNTRIES_MAP}
-                    l.countries = sorted(l.countries_map.items(), key=key)
-                except KeyError:
-                    l.countries_map = COUNTRIES_MAP
-                    l.countries = COUNTRIES
-    return langs
-
-
-# Load the locales
-LOCALES = load_langs("i18n/core")
-
-# Add the default English locale
-LOCALE_EN = LOCALES['en'] = Locale('en')
-LOCALE_EN.catalog = Catalog('en')
-LOCALE_EN.catalog.plural_func = lambda n: n != 1
-LOCALE_EN.countries = COUNTRIES
-LOCALE_EN.countries_map = COUNTRIES_MAP
-
-# Patch the locales to look less formal
-LOCALES['fr'].currency_formats[None] = parse_pattern('#,##0.00\u202f\xa4')
-LOCALES['fr'].currency_symbols['USD'] = '$'
-
-
-def get_locale_for_request(request):
+def get_locale_for_request(request, website):
     accept_lang = request.headers.get("Accept-Language", "")
     languages = (lang.split(";", 1)[0] for lang in accept_lang.split(","))
+    languages = request.accept_langs = regularize_locales(languages)
     for lang in languages:
-        lang = regularize_locale(lang)
-        loc = LOCALES.get(lang)
+        loc = website.locales.get(lang)
         if loc:
             return loc
-    return LOCALE_EN
+    return website.locale_en
 
 
-def format_currency_with_options(number, currency, locale=LOCALE_EN, trailing_zeroes=True):
+def format_currency_with_options(number, currency, locale='en', trailing_zeroes=True):
     s = format_currency(number, currency, locale=locale)
     if not trailing_zeroes:
         s = s.replace(get_decimal_symbol(locale)+'00', '')
@@ -133,7 +124,7 @@ def format_currency_with_options(number, currency, locale=LOCALE_EN, trailing_ze
 
 def add_helpers_to_context(website, request):
     context = request.context
-    loc = context['locale'] = get_locale_for_request(request)
+    loc = context['locale'] = get_locale_for_request(request, website)
     context['decimal_symbol'] = get_decimal_symbol(locale=loc)
     context['_'] = lambda s, *a, **kw: get_text(request, loc, s, *a, **kw)
     context['ngettext'] = lambda *a, **kw: n_get_text(website, request, loc, *a, **kw)
diff --git a/gratipay/wireup.py b/gratipay/wireup.py
--- a/gratipay/wireup.py
+++ b/gratipay/wireup.py
@@ -8,6 +8,9 @@
 
 import aspen
 from aspen.testing.client import Client
+from babel.core import Locale
+from babel.messages.pofile import Catalog, read_po
+from babel.numbers import parse_pattern
 import balanced
 import gratipay
 import gratipay.billing.payday
@@ -28,7 +31,9 @@
 from gratipay.models.participant import Participant
 from gratipay.models.email_address_with_confirmation import EmailAddressWithConfirmation
 from gratipay.models import GratipayDB
+from gratipay.utils import COUNTRIES, COUNTRIES_MAP
 from gratipay.utils.cache_static import asset_etag
+from gratipay.utils.i18n import ALIASES, ALIASES_R, get_function_from_rule, strip_accents
 
 
 def canonical(env):
@@ -246,6 +251,49 @@ def compile_assets(website):
         os.rename(tmpfpath, filepath)
 
 
+def load_i18n(website):
+    # Load the locales
+    key = lambda t: strip_accents(t[1])
+    localeDir = os.path.join(website.project_root, 'i18n', 'core')
+    locales = website.locales = {}
+    for file in os.listdir(localeDir):
+        try:
+            parts = file.split(".")
+            if not (len(parts) == 2 and parts[1] == "po"):
+                continue
+            lang = parts[0]
+            with open(os.path.join(localeDir, file)) as f:
+                l = locales[lang.lower()] = Locale(lang)
+                c = l.catalog = read_po(f)
+                c.plural_func = get_function_from_rule(c.plural_expr)
+                try:
+                    l.countries_map = {k: l.territories[k] for k in COUNTRIES_MAP}
+                    l.countries = sorted(l.countries_map.items(), key=key)
+                except KeyError:
+                    l.countries_map = COUNTRIES_MAP
+                    l.countries = COUNTRIES
+        except Exception as e:
+            website.tell_sentry(e)
+
+    # Add the default English locale
+    locale_en = website.locale_en = locales['en'] = Locale('en')
+    locale_en.catalog = Catalog('en')
+    locale_en.catalog.plural_func = lambda n: n != 1
+    locale_en.countries = COUNTRIES
+    locale_en.countries_map = COUNTRIES_MAP
+
+    # Add aliases
+    for k, v in list(locales.items()):
+        locales.setdefault(ALIASES.get(k, k), v)
+        locales.setdefault(ALIASES_R.get(k, k), v)
+    for k, v in list(locales.items()):
+        locales.setdefault(k.split('_', 1)[0], v)
+
+    # Patch the locales to look less formal
+    locales['fr'].currency_formats[None] = parse_pattern('#,##0.00\u202f\xa4')
+    locales['fr'].currency_symbols['USD'] = '$'
+
+
 def other_stuff(website, env):
     website.cache_static = env.gratipay_cache_static
     website.compress_assets = env.gratipay_compress_assets


Test patch:
diff --git a/tests/py/test_utils.py b/tests/py/test_utils.py
--- a/tests/py/test_utils.py
+++ b/tests/py/test_utils.py
@@ -79,17 +79,17 @@ def test_long_difference_not_expiring(self):
 
     def test_format_currency_without_trailing_zeroes(self):
         expected = '$16'
-        actual = i18n.format_currency_with_options(16, 'USD', locale=i18n.Locale('en'), trailing_zeroes=False)
+        actual = i18n.format_currency_with_options(16, 'USD', locale='en', trailing_zeroes=False)
         assert actual == expected
 
     def test_format_currency_with_trailing_zeroes(self):
         expected = '$16.00'
-        actual = i18n.format_currency_with_options(16, 'USD', locale=i18n.Locale('en'), trailing_zeroes=True)
+        actual = i18n.format_currency_with_options(16, 'USD', locale='en', trailing_zeroes=True)
         assert actual == expected
 
     def test_format_currency_defaults_to_trailing_zeroes(self):
         expected = '$16.00'
-        actual = i18n.format_currency_with_options(16, 'USD', locale=i18n.Locale('en'))
+        actual = i18n.format_currency_with_options(16, 'USD', locale='en')
         assert actual == expected