Repository: pallets/werkzeug

Issue:
LanguageAccept.best_match() - better fallback
```
>>> parse_accept_header('en-us', LanguageAccept).best_match(['en', 'de'])
>>> parse_accept_header('en', LanguageAccept).best_match(['en-us', 'de'])
```

Both those calls return `None`. I'm not sure if there's a RFC specifying the behaviour for this but shouldn't a territory-less language _always_ work if there's no exact match available? I.e. if the client specified `en-us`, `en` would be fine and more important, if the client specified `en` and the server just knows `en-us` that one should be used.

This is important in cases like the following one: There's an application using Flask-Babel which has english as its native language (i.e. hardcoded) because it's nice to keep code in english and a translation, e.g. german. That language is the default language though (because the main audience of the site speaks german). However, if a client sends an Accept-Languages for `en` (or any  `en-*`) the english "translation" should be used. But without specifying `en` and all `en-*` language codes when calling `best_match()` it will return `None` (or `de` depending on the passed arguments) instead - even though the user's header clearly says he prefers english.



Discussion:
Babel has a function `negotiate_locale` which does exactly what I'm suggesting here.

```
@babel.localeselector
def get_current_locale():
    preferred = [x.replace('-', '_') for x in request.accept_languages.values()]
    return negotiate_locale(preferred, AVAILABLE_LOCALES)
```

Since the function is rather big it probably wouldn't be a good idea to include it in werkzeug. Might be worth mentioning it in the docs of [Flask-Babel](https://github.com/mitsuhiko/flask-babel) though since it's more useful than the current example using `best_match()`.

This leaves out matching based on quality though.  So this solution isn't technically correct.  I'm not really sure whether it is preferred for the fix to be in werkzeug or in babel.  best_match does do quality matching, but babel doesn't. 

I had problems matching various german accept_languages, foreinstance de-de, de-at. This seems to work for me (all de-de or de-at will be 'de'):

`

```
@babel.localeselector
def get_locale():
    locale = 'en'

    for lang in request.accept_languages.values():
        if lang[:2] in ['de', 'en']:
            locale = lang[:2]
            break

    g.locale = locale
    return locale
```

`

> Babel has a function negotiate_locale which does exactly what I'm suggesting here.
> 
> ```
> @babel.localeselector
> def get_current_locale():
>     preferred = [x.replace('-', '_') for x in request.accept_languages.values()]
>     return negotiate_locale(preferred, AVAILABLE_LOCALES)
> ```

I'm not sure this covers every case.

If my application defines locales as lang_territory and the browser sends a request as lang_territory where only lang matches:

``` py
from babel import negotiate_locale
AVAILABLE_LOCALES = ['en_US', 'de_DE']
negotiate_locale(['en_EN'], AVAILABLE_LOCALES)  # returns None
```

Isn't this the typical use case?

Or am I missing something?

> This leaves out matching based on quality though. So this solution isn't technically correct. I'm not really sure whether it is preferred for the fix to be in werkzeug or in babel. best_match does do quality matching, but babel doesn't. — @budlight

No, the @ThiefMaster snippet is correct as an [`Accept`](https://github.com/pallets/werkzeug/blob/b7c88324054e99c1a8bf124173c3ab3c14cebc28/werkzeug/datastructures.py#L1601) object is automatically sorted by quality (from 1 to 0) so [`negotiate_locale`](https://github.com/python-babel/babel/blob/ead6ed984fee9019b94700068f553223ad30b012/babel/core.py#L979) tries to match the most preferred first.

## 

The @lafrech case also should work.
But that's a Babel issue.

Just stumbled upon this behaviour and thought/think it is a bug, as it does not fall back to e.g. 'de' if only 'de-AT' is in request.accept_languages.
I tracked it down to the 
`class LanguageAccept(Accept):` in datastructures.py (see [here](https://github.com/arthurwhite/werkzeug/blob/2f610f2d5f510a9ace14cdc26dbc6cecc9ca64a6/werkzeug/datastructures.py#L1806))
where I think it should be `return _locale_delim_re.split(language.lower())[0]` with the **`[0]`** added inside
`
def _value_matches(self, value, item):
  def _normalize(language):
`
Will add more info later, see [this gist](https://gist.github.com/kramred/b25a0eda206f83a8f07f0a289cdfacd1) for testing
Edit1: adjusted the [gist](https://gist.github.com/kramred/b25a0eda206f83a8f07f0a289cdfacd1) to include the examples from first post and I think it does what I want it to do – should I maybe create a pull req. or could you add the fix after further testing?
I've already submitted a PR: https://github.com/pallets/werkzeug/pull/1020. ⌛️

Any updates on this?
I thought https://github.com/pallets/werkzeug/pull/1065 has done the work. cc @ThiefMaster 
Though #1065 merged, this still persist in 0.14.1. I am now using `babel. negotiate_locale` as a workaround.

Resolving patch:
diff --git a/src/werkzeug/datastructures.py b/src/werkzeug/datastructures.py
--- a/src/werkzeug/datastructures.py
+++ b/src/werkzeug/datastructures.py
@@ -1840,14 +1840,63 @@ def accept_json(self):
         return "application/json" in self
 
 
+def _normalize_lang(value):
+    """Process a language tag for matching."""
+    return _locale_delim_re.split(value.lower())
+
+
 class LanguageAccept(Accept):
-    """Like :class:`Accept` but with normalization for languages."""
+    """Like :class:`Accept` but with normalization for language tags."""
 
     def _value_matches(self, value, item):
-        def _normalize(language):
-            return _locale_delim_re.split(language.lower())
+        return item == "*" or _normalize_lang(value) == _normalize_lang(item)
 
-        return item == "*" or _normalize(value) == _normalize(item)
+    def best_match(self, matches, default=None):
+        """Given a list of supported values, finds the best match from
+        the list of accepted values.
+
+        Language tags are normalized for the purpose of matching, but
+        are returned unchanged.
+
+        If no exact match is found, this will fall back to matching
+        the first subtag (primary language only), first with the
+        accepted values then with the match values. This partial is not
+        applied to any other language subtags.
+
+        The default is returned if no exact or fallback match is found.
+
+        :param matches: A list of supported languages to find a match.
+        :param default: The value that is returned if none match.
+        """
+        # Look for an exact match first. If a client accepts "en-US",
+        # "en-US" is a valid match at this point.
+        result = super(LanguageAccept, self).best_match(matches)
+
+        if result is not None:
+            return result
+
+        # Fall back to accepting primary tags. If a client accepts
+        # "en-US", "en" is a valid match at this point. Need to use
+        # re.split to account for 2 or 3 letter codes.
+        fallback = Accept(
+            [(_locale_delim_re.split(item[0], 1)[0], item[1]) for item in self]
+        )
+        result = fallback.best_match(matches)
+
+        if result is not None:
+            return result
+
+        # Fall back to matching primary tags. If the client accepts
+        # "en", "en-US" is a valid match at this point.
+        fallback_matches = [_locale_delim_re.split(item, 1)[0] for item in matches]
+        result = super(LanguageAccept, self).best_match(fallback_matches)
+
+        # Return a value from the original match list. Find the first
+        # original value that starts with the matched primary tag.
+        if result is not None:
+            return next(item for item in matches if item.startswith(result))
+
+        return default
 
 
 class CharsetAccept(Accept):


Test patch:
diff --git a/tests/test_datastructures.py b/tests/test_datastructures.py
--- a/tests/test_datastructures.py
+++ b/tests/test_datastructures.py
@@ -37,6 +37,7 @@
 from werkzeug._compat import itervalues
 from werkzeug._compat import PY2
 from werkzeug._compat import text_type
+from werkzeug.datastructures import LanguageAccept
 from werkzeug.datastructures import Range
 from werkzeug.exceptions import BadRequestKeyError
 
@@ -911,12 +912,12 @@ def make_call_asserter(func=None):
 
     :param func: Additional callback for each function call.
 
-    >>> assert_calls, func = make_call_asserter()
-    >>> with assert_calls(2):
-    ...    func()
-    ...    func()
+    .. code-block:: python
+        assert_calls, func = make_call_asserter()
+        with assert_calls(2):
+            func()
+            func()
     """
-
     calls = [0]
 
     @contextmanager
@@ -1100,6 +1101,30 @@ def test_accept_wildcard_specificity(self):
         assert accept.best_match(["text/plain", "image/png"]) == "image/png"
 
 
+class TestLanguageAccept(object):
+    @pytest.mark.parametrize(
+        ("values", "matches", "default", "expect"),
+        (
+            ([("en-us", 1)], ["en"], None, "en"),
+            ([("en", 1)], ["en_US"], None, "en_US"),
+            ([("en-GB", 1)], ["en-US"], None, None),
+            ([("de_AT", 1), ("de", 0.9)], ["en"], None, None),
+            ([("de_AT", 1), ("de", 0.9), ("en-US", 0.8)], ["de", "en"], None, "de"),
+            ([("de_AT", 0.9), ("en-US", 1)], ["en"], None, "en"),
+            ([("en-us", 1)], ["en-us"], None, "en-us"),
+            ([("en-us", 1)], ["en-us", "en"], None, "en-us"),
+            ([("en-GB", 1)], ["en-US", "en"], "en-US", "en"),
+            ([("de_AT", 1)], ["en-US", "en"], "en-US", "en-US"),
+            ([("aus-EN", 1)], ["aus"], None, "aus"),
+            ([("aus", 1)], ["aus-EN"], None, "aus-EN"),
+        ),
+    )
+    def test_best_match_fallback(self, values, matches, default, expect):
+        accept = LanguageAccept(values)
+        best = accept.best_match(matches, default=default)
+        assert best == expect
+
+
 class TestFileStorage(object):
     storage_class = datastructures.FileStorage