Repository: marshmallow-code/webargs

Issue:
Errors while validating arguments in headers result in a flask crash
If you make a view with header arguments `@bp.arguments(someschema, location='headers')`
Then feed it headers that are not defined in the schema, it will (rightfully) cause a schema validation error, however the error created includes the entire header tuple as a dictionary key, instead of just the 'key' (tuple position 0). This causes flask to error out while trying to convert the response to a valid JSON response.

This is the response returned (grabbed this with a pydb)
```python
{
    "code": 422,
    "status": "Unprocessable Entity",
    "errors": {"headers": {("Someheader", "someval"): ["Unknown field."]}},
}
```
This is the stack trace flask produces, which I have included so people searching it will hopefully find their way here.

```python
../../../../miniconda3/lib/python3.7/site-packages/werkzeug/test.py:1006: in get
    return self.open(*args, **kw)
nomitall/api/tests/_client.py:37: in open
    return super().open(*args, **kwargs)
../../../../miniconda3/lib/python3.7/site-packages/flask/testing.py:227: in open
    follow_redirects=follow_redirects,
../../../../miniconda3/lib/python3.7/site-packages/werkzeug/test.py:970: in open
    response = self.run_wsgi_app(environ.copy(), buffered=buffered)
../../../../miniconda3/lib/python3.7/site-packages/werkzeug/test.py:861: in run_wsgi_app
    rv = run_wsgi_app(self.application, environ, buffered=buffered)
../../../../miniconda3/lib/python3.7/site-packages/werkzeug/test.py:1096: in run_wsgi_app
    app_rv = app(environ, start_response)
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:2463: in __call__
    return self.wsgi_app(environ, start_response)
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:2449: in wsgi_app
    response = self.handle_exception(e)
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:1866: in handle_exception
    reraise(exc_type, exc_value, tb)
../../../../miniconda3/lib/python3.7/site-packages/flask/_compat.py:39: in reraise
    raise value
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:2446: in wsgi_app
    response = self.full_dispatch_request()
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:1952: in full_dispatch_request
    return self.finalize_request(rv)
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:1967: in finalize_request
    response = self.make_response(rv)
../../../../miniconda3/lib/python3.7/site-packages/flask/app.py:2111: in make_response
    rv = jsonify(rv)
../../../../miniconda3/lib/python3.7/site-packages/flask/json/__init__.py:370: in jsonify
    dumps(data, indent=indent, separators=separators) + "\n",
../../../../miniconda3/lib/python3.7/site-packages/flask/json/__init__.py:211: in dumps
    rv = _json.dumps(obj, **kwargs)
../../../../miniconda3/lib/python3.7/site-packages/simplejson/__init__.py:412: in dumps
    **kw).encode(obj)
../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:298: in encode
    chunks = list(chunks)
../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:696: in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:652: in _iterencode_dict
    for chunk in chunks:
../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:652: in _iterencode_dict
    for chunk in chunks:
../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:598: in _iterencode_dict
    k = _stringify_key(k)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

key = ("Someheader", "someval")

    def _stringify_key(key):
        if isinstance(key, string_types): # pragma: no cover
            pass
        elif _PY3 and isinstance(key, bytes) and _encoding is not None:
            key = str(key, _encoding)
        elif isinstance(key, float):
            key = _floatstr(key)
        elif key is True:
            key = 'true'
        elif key is False:
            key = 'false'
        elif key is None:
            key = 'null'
        elif isinstance(key, integer_types):
            if type(key) not in integer_types:
                # See marshmallow-code/flask-smorest#118, do not trust custom str/repr
                key = int(key)
            key = str(key)
        elif _use_decimal and isinstance(key, Decimal):
            key = str(key)
        elif _skipkeys:
            key = None
        else:
            raise TypeError('keys must be str, int, float, bool or None, '
>                           'not %s' % key.__class__.__name__)
E           TypeError: keys must be str, int, float, bool or None, not tuple

../../../../miniconda3/lib/python3.7/site-packages/simplejson/encoder.py:568: TypeError
```

I'm not sure there is a good workaround apart from disabling schema validation.

If this issue is unclear please ask for further explanation and i'll sink some time into creating some self contained reproduction code.


Discussion:
I think this is a webargs issue, so I'll transfer to webargs.

When using a marshmallow 3 schema to deserialize headers, one should always specify `unknown=EXCLUDE` because they won't be defining all possible headers as fields in the schema. Doing so would avoid erroring on unknown keys.

The issue might still occur for any shema-validation error, I guess, so it should be investigated.
Might be worth a small line in the documentation about that, I got caught by it and this error made tracing the source much more difficult.
Absolutely. I meant to mention this in my comment then forgot.
Back to the Flask crash issue.

Quick and dirty repro case:

```py
def test_flask_headers():
    import marshmallow as ma
    from flask import request
    from webargs.multidictproxy import MultiDictProxy

    app = Flask("testapp")
    app.config['DEBUG'] = True
    app.config['TESTING'] = True

    class UserSchema(ma.Schema):
        username = fields.Str()

    @app.route("/test")
    def head():
        mdp = MultiDictProxy(request.headers, UserSchema())
        UserSchema().load(mdp)
        return "Hello"

    resp = app.test_client().get("/test", headers={"test": "12"})
```

    marshmallow.exceptions.ValidationError: {('Test', '12'): ['Unknown field.'], ('Host', 'localhost'): ['Unknown field.'], ('User-Agent', 'werkzeug/0.16.0'): ['Unknown field.']}

Looks like this is related to how marshmallow deserializes `MultiDictProxy`.

I didn't get the time to dig any further yet.
I just got some time to work on this.
I'm pretty sure the issue is that marshmallow is calling `set(MultiDictProxy)`, expecting that to produce the set of dict keys which it will treat as fieldnames.

MultiDictProxy calls `iter(request.headers)`, and this is the result.

So I think the solution might be for MultiDictProxy to get more intelligent about how it exposes `iter`. I'm fiddling with this now. Hope to have something reviewable soon.

Resolving patch:
diff --git a/src/webargs/multidictproxy.py b/src/webargs/multidictproxy.py
--- a/src/webargs/multidictproxy.py
+++ b/src/webargs/multidictproxy.py
@@ -62,7 +62,13 @@ def __getattr__(self, name):
         return getattr(self.data, name)
 
     def __iter__(self):
-        return iter(self.data)
+        for x in iter(self.data):
+            # special case for header dicts which produce an iterator of tuples
+            # instead of an iterator of strings
+            if isinstance(x, tuple):
+                yield x[0]
+            else:
+                yield x
 
     def __contains__(self, x):
         return x in self.data


Test patch:
diff --git a/tests/apps/flask_app.py b/tests/apps/flask_app.py
--- a/tests/apps/flask_app.py
+++ b/tests/apps/flask_app.py
@@ -126,6 +126,14 @@ def echo_headers():
     return J(parser.parse(hello_exclude_schema, location="headers"))
 
 
+@app.route("/echo_headers_raising")
+@use_args(HelloSchema(**strict_kwargs), location="headers")
+def echo_headers_raising(args):
+    # as above, but in this case, don't use the exclude schema (so unexpected
+    # headers will raise errors)
+    return J(args)
+
+
 @app.route("/echo_cookie")
 def echo_cookie():
     return J(parser.parse(hello_args, request, location="cookies"))
diff --git a/tests/test_flaskparser.py b/tests/test_flaskparser.py
--- a/tests/test_flaskparser.py
+++ b/tests/test_flaskparser.py
@@ -67,6 +67,21 @@ def test_nested_many_with_data_key(self, testapp):
         res = testapp.post_json("/echo_nested_many_data_key", {})
         assert res.json == {}
 
+    # regression test for
+    # https://github.com/marshmallow-code/webargs/issues/500
+    def test_parsing_unexpected_headers_when_raising(self, testapp):
+        res = testapp.get(
+            "/echo_headers_raising", expect_errors=True, headers={"X-Unexpected": "foo"}
+        )
+        # under marshmallow 2 this is allowed and works
+        if MARSHMALLOW_VERSION_INFO[0] < 3:
+            assert res.json == {"name": "World"}
+        # but on ma3 it's supposed to be a validation error
+        else:
+            assert res.status_code == 422
+            assert "headers" in res.json
+            assert "X-Unexpected" in set(res.json["headers"].keys())
+
 
 @mock.patch("webargs.flaskparser.abort")
 def test_abort_called_on_validation_error(mock_abort):