Repository: ckan/ckan

Issue:
i18n js files are empty and datasets are not shown
**CKAN 2.9.2**

Ubuntu 18.04

**Describe the bug**
The error occurs after upgrading from Python 2.7 to 3.6.
When navigating to a dataset the data isn't shown, but a loading spinner animation.
The devtools reveal there was an XHR call to `/api/i18n/de` which responded with a 500 error status code.
The web server logs contain an error message:
`File "/usr/lib/ckan/default/lib/python3.6/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)`

I think the cause is that the js files in /usr/lib/ckan/default/src/ckan/ckan/public/base/i18n are empty. First there was a file permission problem I had to solve. Then you could watch them being created one by one on the first run. The files in the old Python 2 environment are not empty. I tried to copy `de.js` from Python 2 to Python 3 environment, but this doesn't work and results in a different error:
`UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 156: ordinal not in range(128)`

_But how do I know if the XHR call is related to the dataset problem?_

I guess there is some sort of unicode incompatibility between Python2 and Python3. To avoid unicode problems I replaced the german i18n file (`de.js`) in the new Python 3 env by the english file (`en.js`) from Python 2. After that the dataset is shown and the error 500 is gone.

**Steps to reproduce**
- Upgrade Python 2 to Python 3 (https://docs.ckan.org/en/2.9/maintaining/upgrading/upgrade-to-python3.html)
- Run application (`sudo systemctl start apache2`)
- Open page with dataset

**Expected behavior**
- dataset is being shown
- Status 200 for XHR request `/api/i18n/de`
- i18n js files not to be empty 

**Additional details**
Stack trace:
```
2021-04-28 10:05:39,522 ERROR [ckan.config.middleware.flask_app] Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/flask/app.py", line 1949, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/flask/app.py", line 1935, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/ckan/default/src/ckan/ckan/config/middleware/../../views/api.py", line 482, in i18n_js_translations
    translations = json.load(open(source, u'r'))
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/simplejson/__init__.py", line 459, in load
    use_decimal=use_decimal, **kw)
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/ckan/default/lib/python3.6/site-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
```
Screenshot:

![grafik](https://user-images.githubusercontent.com/13943237/116380884-ad957e80-a814-11eb-8d56-869114b30f93.png)



Discussion:

Resolving patch:
diff --git a/ckan/lib/i18n.py b/ckan/lib/i18n.py
--- a/ckan/lib/i18n.py
+++ b/ckan/lib/i18n.py
@@ -295,7 +295,7 @@ def _build_js_translation(lang, source_filenames, entries, dest_filename):
                 ordered_plural = sorted(entry.msgstr_plural.items())
                 for order, msgstr in ordered_plural:
                     plural.append(msgstr)
-    with open(dest_filename, u'w') as f:
+    with open(dest_filename, u'w', encoding='utf-8') as f:
         s = json.dumps(result, sort_keys=True, indent=2, ensure_ascii=False)
         f.write(six.ensure_str(s))
 
diff --git a/ckan/views/api.py b/ckan/views/api.py
--- a/ckan/views/api.py
+++ b/ckan/views/api.py
@@ -479,7 +479,7 @@ def i18n_js_translations(lang, ver=API_REST_DEFAULT_VERSION):
                              u'base', u'i18n', u'{0}.js'.format(lang)))
     if not os.path.exists(source):
         return u'{}'
-    translations = json.load(open(source, u'r'))
+    translations = json.load(open(source, u'r', encoding='utf-8'))
     return _finish_ok(translations)
 
 


Test patch: