Repository: searx/searx

Issue:
Display wrong language in the search interface.
<!-- PLEASE FILL THESE FIELDS, IT REALLY HELPS THE MAINTAINERS OF SEARX -->

**Version of Searx, commit number if you are using on master branch and stipulate if you forked Searx**
<!-- If you are running on master branch using git execute this command
in order to fetch the latest commit ID:
```
git log -1
``` 
If you are using searx-docker then look at the bottom of the Searx page
and check for the version after "Powered by searx"

Please also stipulate if you are using a forked version of Searx and
include a link to the fork source code.
-->
**How did you install Searx?**
<!-- Did you install Searx using the official wiki or using searx-docker
or manually by executing the searx/webapp.py file? -->
Just use it in the web browser.
**What happened?**
<!-- A clear and concise description of what the bug is. -->
The UI language is wrong.
**How To Reproduce**
<!-- How can we reproduce this issue? (as minimally and as precisely as possible) -->
Open searx search page.
**Expected behavior**
<!-- A clear and concise description of what you expected to happen. -->
See the right language.
**Screenshots & Logs**
<!-- If applicable, add screenshots, logs to help explain your problem. -->
![圖片](https://user-images.githubusercontent.com/9532105/106374524-08a70900-63bf-11eb-8112-3fba816bd635.png)

**Additional context**
<!-- Add any other context about the problem here. -->
I'm using Traditional Chinese, but it displays the Simplified Chinese.
https://www.transifex.com/asciimoo/searx/dashboard/
It is fully translated to my language, but it still displays Simplified Chinese.
It also searches lots of websites from China, although I have selected Taiwan.


Discussion:
I can change the UI language in settings, searx doesn't recognize my language automatically.
It is painful when I am browsing the web in incognito mode
![圖片](https://user-images.githubusercontent.com/9532105/106376781-4e6dcc80-63d3-11eb-9344-339f63d00f38.png)

What's the `Accept-Language` header in your browser?
@MarcAbonce Where to see it? I'm using Firefox.

```
❯ pacman -Qs firefox
local/firefox 85.0-1
    Standalone web browser from mozilla.org
local/firefox-i18n-zh-tw 85.0-1
    Chinese (Traditional) language pack for Firefox
```
In `about:config` it appears as `intl.accept_languages`.
@MarcAbonce 

`intl.accept_languages | zh-tw`

Resolving patch:
diff --git a/searx/webapp.py b/searx/webapp.py
--- a/searx/webapp.py
+++ b/searx/webapp.py
@@ -142,6 +142,7 @@
 
 rtl_locales = ['ar', 'arc', 'bcc', 'bqi', 'ckb', 'dv', 'fa', 'fa_IR', 'glk', 'he',
                'ku', 'mzn', 'pnb', 'ps', 'sd', 'ug', 'ur', 'yi']
+ui_locale_codes = [l.replace('_', '-') for l in settings['locales'].keys()]
 
 # used when translating category names
 _category_names = (gettext('files'),
@@ -175,6 +176,9 @@ def _get_browser_or_settings_language(request, lang_list):
     for lang in request.headers.get("Accept-Language", "en").split(","):
         if ';' in lang:
             lang = lang.split(';')[0]
+        if '-' in lang:
+            lang_parts = lang.split('-')
+            lang = "{}-{}".format(lang_parts[0], lang_parts[-1].upper())
         locale = match_language(lang, lang_list, fallback=None)
         if locale is not None:
             return locale
@@ -194,13 +198,10 @@ def get_locale():
         locale_source = 'preferences'
     else:
         # use local from the browser
-        locale = _get_browser_or_settings_language(request, settings['locales'].keys())
+        locale = _get_browser_or_settings_language(request, ui_locale_codes)
+        locale = locale.replace('-', '_')
         locale_source = 'browser'
 
-    #
-    if locale == 'zh_TW':
-        locale = 'zh_Hant_TW'
-
     # see _get_translations function
     # and https://github.com/searx/searx/pull/1863
     if locale == 'oc':


Test patch:
diff --git a/tests/unit/test_webapp.py b/tests/unit/test_webapp.py
--- a/tests/unit/test_webapp.py
+++ b/tests/unit/test_webapp.py
@@ -198,6 +198,20 @@ def test_preferences(self):
             result.data
         )
 
+    def test_browser_locale(self):
+        result = self.app.get('/preferences', headers={'Accept-Language': 'zh-tw;q=0.8'})
+        self.assertEqual(result.status_code, 200)
+        self.assertIn(
+            b'<option value="zh_TW" selected="selected">',
+            result.data,
+            'Interface locale ignored browser preference.'
+        )
+        self.assertIn(
+            b'<option value="zh-TW" selected="selected">',
+            result.data,
+            'Search language ignored browser preference.'
+        )
+
     def test_stats(self):
         result = self.app.get('/stats')
         self.assertEqual(result.status_code, 200)