Repository: ckan/ckan

Issue:
Flask-Babel does not translate zh_TW and zh_CN
### CKAN Version if known (or site URL)

≧ 2.8.0

### Please describe the expected behaviour

When switching to ``zh_TW`` or ``zh_CN`` languages on the pages written in Flask (ex. ``/`` and ``/user``), the pages should be shown in corresponding languages.

### Please describe the actual behaviour

It shows English instead of ``zh_TW`` or ``zh_CN``.

### What steps can be taken to reproduce the issue? 

Switch language to Chinese on the above-mentioned pages.


Discussion:
The problem is caused by the ``Flask-Babel`` module which is used to translate pages written in Flask.

Pylons uses Python's ``gettext`` module which is [directory-based](https://github.com/python/cpython/blob/2.7/Lib/gettext.py#L524). However, ``Flask-Babel`` tries to find the directory containing translated strings through Babel's ``Locale.parse()`` function ([ref1](https://github.com/python-babel/flask-babel/blob/v0.11.2/flask_babel/__init__.py#L227), [ref2](https://github.com/python-babel/flask-babel/blob/v0.11.2/flask_babel/__init__.py#L243)), which "normalizes" the ``zh_TW`` and ``zh_CN`` to ``zh_Hant_TW`` and ``zh_Hans_CN`` respectively. Since there is no dirs named ``zh_Hant_TW`` and ``zh_Hans_CN``, those pages will not shown in desired languages.

A [similar situation](https://github.com/flask-admin/flask-admin/issues/403) happened on ``flask-admin`` and they decided to rename the dirs. But I think renaming dirs will break backwards compatibility in CKAN if people bookmark urls with old ``/zh_TW`` or ``/zh_CN`` path. Any ideas or suggestions are welcomed.
@u10313335 Please check #4402. I think its fixing this issue
@tino097 thanks. But #4402 is about the "label" shown on the language switcher. The problem I have described in this issue is the language of a CKAN site.
Sorry, i didnt read your comment, just went through the issue description  :/ 
flask-babel needs to allow passing through args to babel.parse, specifically `resolve_likely_subtags=False` would fix this problem.
@TkTech thanks.

But I got the ``babel.core.UnknownLocaleError: unknown locale 'zh_TW'`` exception after setting ``resolve_likely_subtags=False``.
@TkTech any pointers on how to progress with this?
I didn't change resolve_likely_subtags=False
I just cp zh_CN(or zh_TW) zh_Hans_CN(zh_Hant_TW)
And it works!
@ltf9651 You should set the ``resolve_likely_subtags`` argument to babel.parse in flask-babel just like @TkTech said. However, I have tried that but no luck.

And, renaming dirs resolve this issue but it will break backwards compatibility in CKAN if people bookmark urls with old /zh_TW or /zh_CN path.

Please refer to my previous comment here: https://github.com/ckan/ckan/issues/4413#issuecomment-414532805
@u10313335 
cd /ckan/i18n
copy zh_CN zh_Hans_CN
My problem fixed, you should have a try
@ltf9651 I know that renaming dirs will resolve this issue. But it will break the links with /zh_TW or /zh_CN in the older versions of CKAN.
> 
> 
> @u10313335
> cd /ckan/i18n
> copy zh_CN zh_Hans_CN
> My problem fixed, you should have a try

It does work properly, thanks
> 
> 
> @u10313335
> cd /ckan/i18n
> copy zh_CN zh_Hans_CN
> My problem fixed, you should have a try

Thanks. It is working.
> 
> 
> @ltf9651 I know that renaming dirs will resolve this issue. But it will break the links with /zh_TW or /zh_CN in the older versions of CKAN.

The links remain zh_CN or zh_TW when using the workaround above provided by @u10313335.
For example, the URL is http://127.0.0.1:5000/zh_CN/dataset/example_xxx_dataset
rather than http://127.0.0.1:5000/zh_Hans_CN/dataset/example_xxx_dataset

When switch the language to zh_CN, an error occurs with the details page of an organization:
...
File "/usr/local/lib64/python3.6/site-packages/markupsafe/init.py", line 249, in getitem
return self._kwargs[key]
KeyError: 'organization'

[**Note**: 

_This is the only problem After renaming the the locale folder 'zh_CN' to 'zh_Hans_CN'  and Update the locale name from 'zh_CN' to 'zh_Hans_CN' in the config file ckan.ini but not changing any other things.  That is to say,  everything is okay but displaying the details page of an organization in this locale and the page can be  normally displayed in the other locales._

The page shows "Error [500] - CKAN" on the page tile bar and "Internal server error " in the section of the oganization details.
]

The details page of an organization is okay when switch the language to other language such as ja or zh_TW.
And ckan/i18n uses the folder named 'zh_Hans_CN' and 'zh_Hans_TW'.
But the po and mo files remain intact as the original files released by the CKAN GitHub project.

The error is still there even after the database has been cleared and initialized.
It's okay with other languages such as en, ja or zh_TW.
> I didn't change resolve_likely_subtags=False
> I just cp zh_CN(or zh_TW) zh_Hans_CN(zh_Hant_TW)
> And it works!

Did you encounter the internal server error about showing the details page of an organization?
> The problem is caused by the `Flask-Babel` module which is used to translate pages written in Flask.
>  ... Any ideas or suggestions are welcomed.

Is there any solution or workaround for this issue?  Thanks.
> 
> 
> flask-babel needs to allow passing through args to babel.parse, specifically `resolve_likely_subtags=False` would fix this problem.
@u10313335
How to set resolve_likely_subtags=False?  The python file 'core.py' in the folder '/usr/local/lib/python3.6/site-packages/Babel'?  Thanks

Workaround to be fully evaluated:
Temporarily COMMENT the line for calling the feeds snippet in the read-base of "organization".  That's all.  Any idea or suggestion?  THX.
The formal solution seems  within the snippet feeds.html

Resolving patch:
diff --git a/ckan/views/home.py b/ckan/views/home.py
--- a/ckan/views/home.py
+++ b/ckan/views/home.py
@@ -1,6 +1,6 @@
 # encoding: utf-8
 
-from flask import Blueprint, abort
+from flask import Blueprint, abort, redirect
 
 import ckan.model as model
 import ckan.logic as logic
@@ -82,9 +82,36 @@ def about():
     return base.render(u'home/about.html', extra_vars={})
 
 
+def redirect_locale(target_locale, path=None):
+    target = f'/{target_locale}/{path}' if path else f'/{target_locale}'
+    return redirect(target, code=308)
+
+
 util_rules = [
     (u'/', index),
     (u'/about', about)
 ]
 for rule, view_func in util_rules:
     home.add_url_rule(rule, view_func=view_func)
+
+locales_mapping = [
+    ('zh_TW', 'zh_Hant_TW'),
+    ('zh_CN', 'zh_Hans_CN'),
+]
+
+for locale in locales_mapping:
+
+    legacy_locale = locale[0]
+    new_locale = locale[1]
+
+    home.add_url_rule(
+        f'/{legacy_locale}/',
+        view_func=redirect_locale,
+        defaults={'target_locale': new_locale}
+    )
+
+    home.add_url_rule(
+        f'/{legacy_locale}/<path:path>',
+        view_func=redirect_locale,
+        defaults={'target_locale': new_locale}
+    )


Test patch:
diff --git a/ckan/tests/controllers/test_home.py b/ckan/tests/controllers/test_home.py
--- a/ckan/tests/controllers/test_home.py
+++ b/ckan/tests/controllers/test_home.py
@@ -117,3 +117,30 @@ def test_right_option_is_selected_on_language_selector(self, app):
                 assert option["selected"] == "selected"
             else:
                 assert not option.has_attr("selected")
+
+    def test_redirects_legacy_locales(self, app):
+        locales_mapping = [
+            ('zh_TW', 'zh_Hant_TW'),
+            ('zh_CN', 'zh_Hans_CN'),
+        ]
+
+        for locale in locales_mapping:
+
+            legacy_locale = locale[0]
+            new_locale = locale[1]
+
+            response = app.get(f'/{legacy_locale}/', follow_redirects=False)
+
+            assert response.status_code == 308
+            assert (
+                response.headers['Location'] ==
+                f'http://test.ckan.net/{new_locale}'
+            )
+
+            response = app.get(f'/{legacy_locale}/dataset', follow_redirects=False)
+
+            assert response.status_code == 308
+            assert (
+                response.headers['Location'] ==
+                f'http://test.ckan.net/{new_locale}/dataset'
+            )