Repository: pallets/werkzeug

Issue:
MIMEAccept doesn't respect specificity
Hello!

I've been using the helpful MIMEAccept class in `werkzeug.datastructures` and I noticed that while it does respect quality, it doesn't respect specificity as outlined in RFC 2616.

> Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence. For example,
> 
> Accept: text/_, text/html, text/html;level=1, */_
>     have the following precedence:
> 
> 1) text/html;level=1
> 2) text/html
> 3) text/*
> 4) */*

Is this something that people are interested in seeing fixed? I'd be happy to open a pull.



Discussion:
:+1: I just noticed this as well.  The `_accept_re` in _http.py_ does not take in to account media type parameters

https://github.com/mitsuhiko/werkzeug/pull/205 is kind-of related.

What is prioritized -- quality or specificity?

Can we agree that even though `_accept_re` now notices media type parameters, they should still be treated as opaque, so that one set of parameters isn't considered more specific than another (non-empty) set? If application code wants to examine these it's welcome to do so.

The regex can't do much about this. This is supposed to be fixed in `MIMEAccept.__init__`, where the sorting happens. I'm currently working on it, but i have two questions: The one written above, and how specificity is defined.

As I understand it, specificity only governs how a representation with a particular _media type_ is associated with a particular _media range_. So the server application should assign each representation of the current resource it can possibly send to one particular range that was sent in the `Accept` header, if it can. (If none of the possible representations fits into any of the specified media ranges, the server may **406** or arbitrarily pick a representation.) Specificity in this case means that any particular representation should go in the most specific range that includes that representation.

Quality/weight is only used **after** that assignment is done. It could have the result of a representation that had a _less specific_ match having a _higher quality_ and being chosen. If so, that's just what the client requested, e.g. `Accept: text/*, text/html;q=0.5` would result in a `text/plain` representation being sent instead of a `text/html` representation, if the server application supports both plain text and html.

I just realized that if more than one representation is assigned to the "winning" range, the server application still has to arbitrarily pick one. Oh, well.

As for a previous comment of mine:

> This is supposed to be fixed in MIMEAccept.**init**, where the sorting happens

After trying to implement specificity, i don't think anymore this could be done there.


Resolving patch:
diff --git a/src/werkzeug/datastructures.py b/src/werkzeug/datastructures.py
--- a/src/werkzeug/datastructures.py
+++ b/src/werkzeug/datastructures.py
@@ -29,8 +29,6 @@
 from ._internal import _missing
 from .filesystem import get_filesystem_encoding
 
-_locale_delim_re = re.compile(r"[_-]")
-
 
 def is_immutable(self):
     raise TypeError("%r objects are immutable" % self.__class__.__name__)
@@ -1785,40 +1783,58 @@ def best(self):
             return self[0][0]
 
 
+_mime_split_re = re.compile(r"/|(?:\s*;\s*)")
+
+
+def _normalize_mime(value):
+    return _mime_split_re.split(value.lower())
+
+
 class MIMEAccept(Accept):
     """Like :class:`Accept` but with special methods and behavior for
     mimetypes.
     """
 
     def _specificity(self, value):
-        return tuple(x != "*" for x in value.split("/", 1))
+        return tuple(x != "*" for x in _mime_split_re.split(value))
 
     def _value_matches(self, value, item):
-        def _normalize(x):
-            x = x.lower()
-            return ("*", "*") if x == "*" else x.split("/", 1)
+        # item comes from the client, can't match if it's invalid.
+        if "/" not in item:
+            return False
 
-        # this is from the application which is trusted.  to avoid developer
-        # frustration we actually check these for valid values
+        # value comes from the application, tell the developer when it
+        # doesn't look valid.
         if "/" not in value:
             raise ValueError("invalid mimetype %r" % value)
-        value_type, value_subtype = _normalize(value)
+
+        # Split the match value into type, subtype, and a sorted list of parameters.
+        normalized_value = _normalize_mime(value)
+        value_type, value_subtype = normalized_value[:2]
+        value_params = sorted(normalized_value[2:])
+
+        # "*/*" is the only valid value that can start with "*".
         if value_type == "*" and value_subtype != "*":
             raise ValueError("invalid mimetype %r" % value)
 
-        if "/" not in item:
-            return False
-        item_type, item_subtype = _normalize(item)
+        # Split the accept item into type, subtype, and parameters.
+        normalized_item = _normalize_mime(item)
+        item_type, item_subtype = normalized_item[:2]
+        item_params = sorted(normalized_item[2:])
+
+        # "*/not-*" from the client is invalid, can't match.
         if item_type == "*" and item_subtype != "*":
             return False
+
         return (
-            item_type == item_subtype == "*" or value_type == value_subtype == "*"
+            (item_type == "*" and item_subtype == "*")
+            or (value_type == "*" and value_subtype == "*")
         ) or (
             item_type == value_type
             and (
                 item_subtype == "*"
                 or value_subtype == "*"
-                or item_subtype == value_subtype
+                or (item_subtype == value_subtype and item_params == value_params)
             )
         )
 
@@ -1840,6 +1856,9 @@ def accept_json(self):
         return "application/json" in self
 
 
+_locale_delim_re = re.compile(r"[_-]")
+
+
 def _normalize_lang(value):
     """Process a language tag for matching."""
     return _locale_delim_re.split(value.lower())


Test patch:
diff --git a/tests/test_datastructures.py b/tests/test_datastructures.py
--- a/tests/test_datastructures.py
+++ b/tests/test_datastructures.py
@@ -38,6 +38,7 @@
 from werkzeug._compat import PY2
 from werkzeug._compat import text_type
 from werkzeug.datastructures import LanguageAccept
+from werkzeug.datastructures import MIMEAccept
 from werkzeug.datastructures import Range
 from werkzeug.exceptions import BadRequestKeyError
 
@@ -1084,21 +1085,48 @@ def test_accept_wildcard_specificity(self):
 
 
 class TestMIMEAccept(object):
-    storage_class = datastructures.MIMEAccept
-
-    def test_accept_wildcard_subtype(self):
-        accept = self.storage_class([("text/*", 1)])
-        assert accept.best_match(["text/html"], default=None) == "text/html"
-        assert accept.best_match(["image/png", "text/plain"]) == "text/plain"
-        assert accept.best_match(["image/png"], default=None) is None
-
-    def test_accept_wildcard_specificity(self):
-        accept = self.storage_class([("*/*", 1), ("text/html", 1)])
-        assert accept.best_match(["image/png", "text/html"]) == "text/html"
-        assert accept.best_match(["image/png", "text/plain"]) == "image/png"
-        accept = self.storage_class([("*/*", 1), ("text/html", 1), ("image/*", 1)])
-        assert accept.best_match(["image/png", "text/html"]) == "text/html"
-        assert accept.best_match(["text/plain", "image/png"]) == "image/png"
+    @pytest.mark.parametrize(
+        ("values", "matches", "default", "expect"),
+        [
+            ([("text/*", 1)], ["text/html"], None, "text/html"),
+            ([("text/*", 1)], ["image/png"], "text/plain", "text/plain"),
+            ([("text/*", 1)], ["image/png"], None, None),
+            (
+                [("*/*", 1), ("text/html", 1)],
+                ["image/png", "text/html"],
+                None,
+                "text/html",
+            ),
+            (
+                [("*/*", 1), ("text/html", 1)],
+                ["image/png", "text/plain"],
+                None,
+                "image/png",
+            ),
+            (
+                [("*/*", 1), ("text/html", 1), ("image/*", 1)],
+                ["image/png", "text/html"],
+                None,
+                "text/html",
+            ),
+            (
+                [("*/*", 1), ("text/html", 1), ("image/*", 1)],
+                ["text/plain", "image/png"],
+                None,
+                "image/png",
+            ),
+            (
+                [("text/html", 1), ("text/html; level=1", 1)],
+                ["text/html;level=1"],
+                None,
+                "text/html;level=1",
+            ),
+        ],
+    )
+    def test_mime_accept(self, values, matches, default, expect):
+        accept = MIMEAccept(values)
+        match = accept.best_match(matches, default=default)
+        assert match == expect
 
 
 class TestLanguageAccept(object):
diff --git a/tests/test_http.py b/tests/test_http.py
--- a/tests/test_http.py
+++ b/tests/test_http.py
@@ -61,6 +61,13 @@ def test_accept_matches(self):
         assert a.best_match(["foo/bar", "bar/foo"], default="foo/bar") == "foo/bar"
         assert a.best_match(["application/xml", "text/xml"]) == "application/xml"
 
+    def test_accept_mime_specificity(self):
+        a = http.parse_accept_header(
+            "text/*, text/html, text/html;level=1, */*", datastructures.MIMEAccept
+        )
+        assert a.best_match(["text/html; version=1", "text/html"]) == "text/html"
+        assert a.best_match(["text/html", "text/html; level=1"]) == "text/html; level=1"
+
     def test_charset_accept(self):
         a = http.parse_accept_header(
             "ISO-8859-1,utf-8;q=0.7,*;q=0.7", datastructures.CharsetAccept