Repository: getnikola/nikola

Issue:
Problems with multiple languages in theme hierarchy with partial message files
I have a chain of three themes:
- `mybase` (parent of `concrete-site`)
- `concrete-site` (parent of `concrete-site-test`)
- `concrete-site-test`

The base theme has full `messages_xx.py` files; `concrete-site` only has some strings defined in `messages_xx.py` which are needed resp. differ for that theme, and `concrete-site-test` has few more (or even none).

The test blog which uses `concrete-site-test` has more than one language. I now experienced the problem that sometimes when building, strings of some languages are replaced by the english strings, and sometimes not. Which languages are affected changes from build to build. (I'm using Python 3, so the reason is probably some `dict`/`set` ordering which is randomized in Python 3.)

I've tracked this down to `load_messages` in `utils.py`. It goes through the hierarchy of themes (in order `mybase`, `concrete-site` and `concrete-site-test`), loads the messages for each language as well as the english strings, updates the current messages dictionary for the language first with the english strings and then with the language-dependent strings.

The code looks like this:

``` .py
    for theme_name in themes[::-1]:
        msg_folder = os.path.join(get_theme_path(theme_name), 'messages')
        default_folder = os.path.join(get_theme_path('base'), 'messages')
        sys.path.insert(0, default_folder)
        sys.path.insert(0, msg_folder)
        english = __import__('messages_en')
        for lang in list(translations.keys()):
            try:
                translation = __import__('messages_' + lang)
                # If we don't do the reload, the module is cached
                _reload(translation)
                if sorted(translation.MESSAGES.keys()) !=\
                        sorted(english.MESSAGES.keys()) and \
                        lang not in language_incomplete_warned:
                    language_incomplete_warned.append(lang)
                    LOGGER.warn("Incomplete translation for language "
                                "'{0}'.".format(lang))
                messages[lang].update(english.MESSAGES)
                for k, v in translation.MESSAGES.items():
                    if v:
                        messages[lang][k] = v
                del(translation)
            except ImportError as orig:
                raise LanguageNotFoundError(lang, orig)
```

As you can see, the code takes special care to ensure that `translation` is not cached: if `messages_de` (or any other language) is imported three times (while modifying the import path every time), you will get the same version (the one imported first) if you don't do this.

Unfortunately, for the english strings, this is not done. So the english strings always end up being the ones for the base theme. Hence, all strings for the derived themes coming from parent themes are overwritten by the english strings of the parent themes.

This only affects the situation when you don't define all strings in every theme, but only the ones changing with respect to the parent. But that is what for example the default themes (`base-jinja`, `bootstrap3` and `bootstrap3-jinja`) do. For this reason, I think `load_messages` should be changed as follows:

``` .py
    for theme_name in themes[::-1]:
        msg_folder = os.path.join(get_theme_path(theme_name), 'messages')
        default_folder = os.path.join(get_theme_path('base'), 'messages')
        sys.path.insert(0, default_folder)
        sys.path.insert(0, msg_folder)
        english = __import__('messages_en')
        # If we don't do the reload, the module is cached
        _reload(english)
        for lang in list(translations.keys()):
            try:
                translation = __import__('messages_' + lang)
                # If we don't do the reload, the module is cached
                _reload(translation)
                if sorted(translation.MESSAGES.keys()) !=\
                        sorted(english.MESSAGES.keys()) and \
                        lang not in language_incomplete_warned:
                    language_incomplete_warned.append(lang)
                    LOGGER.warn("Incomplete translation for language "
                                "'{0}'.".format(lang))
                messages[lang].update(english.MESSAGES)
                for k, v in translation.MESSAGES.items():
                    if v:
                        messages[lang][k] = v
                del(translation)
            except ImportError as orig:
                raise LanguageNotFoundError(lang, orig)
        del(english)
```

The difference is

``` .py
        [...]
        # If we don't do the reload, the module is cached
        _reload(english)
        [...]
        del(english)
```

@ralsina, @Kwpolska: did I overlook something, and was the caching of the english translations intended? Or is this indeed a bug?

(I won't really be online the next days, so if this is a bug and the fix above is valid, I'd be happy if someone else creates a PR for this.)



Discussion:
Actually, for the default themes this won't be a problem since they don't define `messages_xx` at all for inherited themes, so in every stage it will get the `messages_xx` module from the base theme. This is only a problem if you specify only parts of the strings in inherited themes.

I think this is a bug, but I’m not sure if we should support inheritance like this (even if it’s 2 extra lines of code). It just feels weird that somehow the dicts compliment each other.

It's nice to allow inheritance, so in case you want to replace a specific term by another one for a subtheme (for whatever reason), you just have to include the changed strings and not all strings. This avoids a lot of duplication.

Also, if you just want to add a few theme-specific strings, you don't have to copy everything again. That would be unnecessary duplication to me.

There's another reason to fix this bug anyway: if an inherited theme has more strings, and one language has not the right amount, the incomplete warning will trigger or not trigger depending on the order of the languages, and the missing strings for the incomplete language might be replaced by the english version or not depending on the order of the languages. (Order of the languages: if the language comes before `en` in the dict order, it might not emit a warning and will miss the new strings; if it comes after `en`, everything's fine and as expected.)


Resolving patch:
diff --git a/nikola/utils.py b/nikola/utils.py
--- a/nikola/utils.py
+++ b/nikola/utils.py
@@ -649,6 +649,8 @@ def load_messages(themes, translations, default_lang):
         sys.path.insert(0, default_folder)
         sys.path.insert(0, msg_folder)
         english = __import__('messages_en')
+        # If we don't do the reload, the module is cached
+        _reload(english)
         for lang in list(translations.keys()):
             try:
                 translation = __import__('messages_' + lang)
@@ -667,6 +669,7 @@ def load_messages(themes, translations, default_lang):
                 del(translation)
             except ImportError as orig:
                 raise LanguageNotFoundError(lang, orig)
+        del(english)
     sys.path = oldpath
     return messages
 


Test patch: