Repository: falconry/falcon

Issue:
Custom media handlers: Unexpected issue when providing custom json handler
This is in falcon-2.0

Look at the documentation [here][1] for using rapidjson for encoding/decoding json. By providing:

`extra_handlers={'application/json': json_handler}` we are still left with the default handler for content-type `application-json; charset=UTF-8`. This results in an unexpected behaviour when some client library (e.g. Retrofit for Android) includes the charset in the header. 

While the documentation should be updated, the expected behaviour is that if the handler for `application/json` is updated - it should also update the handler for variant with charset (or at least throw a warning) otherwise there is a possibility of hidden bugs. 

[1]: https://falcon.readthedocs.io/en/stable/api/media.html


Discussion:
Hi :wave:,

Thanks for using Falcon. The large amount of time and effort needed to
maintain the project and develop new features is not sustainable without
the generous financial support of community members like you.

Please consider helping us secure the future of the Falcon framework with a
one-time or recurring donation.

Thank you for your support!

I agree we should make this better since it violates the principle of least-surprise. Scheduled for 3.0 in case someone wants to tackle it right away, otherwise it will need to be pushed to 3.1.
FWIW, our media type negotiation machinery seems to automatically choose the `application/json` handler for requests sent as `application-json; charset=UTF-8`, even if an explicit handler for the latter is missing. In that light, could we just remove the explicit `application-json; charset=UTF-8` handler from our codebase and the docs, now that we are defaulting to just `application/json` ourselves.

As a bonus, that would eliminate confusion in situations like @adsahay is describing.

I was testing the app below, and it seems to just work:
```python
import falcon


class Resource:

    def on_post(self, req, resp):
        resp.media = req.get_media()


app = falcon.App()
app.req_options.media_handlers = falcon.media.Handlers({
    'application/json': falcon.media.JSONHandler(),
})
app.add_route('/mirror', Resource())
```

Thoughts?
Yes this sounds great. 

Also, it might be a stylistic preference, but in such situations if we can have a constant for all handlers (falcon.MIME_JSON) or something it will have the additional benefit of discoverability of available handlers in documentation / code completion, and avoid typos. 
If you want I can fix and send PR for this (my first falcon contribution!) 
Hi again @adsahay ,
We do have such constants (the stable version already has), maybe we just haven't updated all the docs with them yet. Or, we may have left explicit strings in selected places for clarity.

```python
>>> import falcon
>>> falcon.MEDIA_JSON
'application/json'
>>> falcon.MEDIA_PNG
'image/png'
>>> falcon.MEDIA_TEXT
'text/plain; charset=utf-8'
```

@adsahay PRs are always welcome :slightly_smiling_face: :+1:
To be honest, I haven't fully explored the ramifications of removing the explicit `UTF-8` JSON handler, it's just a quick idea that crossed my head. But maybe we could start a draft PR anyway?

@kgriffs thoughts on my suggestion to simplify the default JSON handlers by only keeping the `application/json` handler?
It could be that the key including the charset was left in to speed up the selection algorithm... regardless, I agree that it is not worth maintaining it since in most cases the charset will not be present in the string to begin with. 
@adsahay just checking, would you like to work on this by removing `application-json; charset=UTF-8` from the default handlers?

As we proceed to remove it though, I would like to see a note added to the docs regarding speeding up the selection algorithm (needs to be verified with a benchmark!), as hinted by @kgriffs above. So that people could duplicate their handlers for a specific parameter set, if they expect that to be their common use case that needs to be optimized.
Ok sure. 
@adsahay just checking again :slightly_smiling_face: Have you had a chance to do any work on this one?
Not yet, slipped through the cracks. Will wrap it up tomorrow.

On Tue, 17 Nov 2020 at 12:36, Vytautas Liuolia <notifications@github.com>
wrote:

> @adsahay <https://github.com/adsahay> just checking again 🙂 Have you had
> a chance to do any work on this one?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/falconry/falcon/issues/1717#issuecomment-728733045>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAC3X52J53AIVPNXEU26FPLSQIOIXANCNFSM4MN7TATQ>
> .
>


-- 
Aditya Sahay
@adsahay <http://twitter.com/adsahay> | sahay.co

Awesome, looking forward to!

Resolving patch:
diff --git a/falcon/media/handlers.py b/falcon/media/handlers.py
--- a/falcon/media/handlers.py
+++ b/falcon/media/handlers.py
@@ -1,7 +1,7 @@
 from collections import UserDict
 
 from falcon import errors
-from falcon.constants import MEDIA_MULTIPART, MEDIA_URLENCODED
+from falcon.constants import MEDIA_JSON, MEDIA_MULTIPART, MEDIA_URLENCODED
 from falcon.media.json import JSONHandler
 from falcon.media.multipart import MultipartFormHandler, MultipartParseOptions
 from falcon.media.urlencoded import URLEncodedFormHandler
@@ -12,8 +12,7 @@ class Handlers(UserDict):
     """A :class:`dict`-like object that manages Internet media type handlers."""
     def __init__(self, initial=None):
         handlers = initial or {
-            'application/json': JSONHandler(),
-            'application/json; charset=UTF-8': JSONHandler(),
+            MEDIA_JSON: JSONHandler(),
             MEDIA_MULTIPART: MultipartFormHandler(),
             MEDIA_URLENCODED: URLEncodedFormHandler(),
         }
@@ -60,7 +59,6 @@ def find_by_media_type(self, media_type, default):
 
 # NOTE(vytas): An ugly way to work around circular imports.
 MultipartParseOptions._DEFAULT_HANDLERS = Handlers({
-    'application/json': JSONHandler(),
-    'application/json; charset=UTF-8': JSONHandler(),
+    MEDIA_JSON: JSONHandler(),
     MEDIA_URLENCODED: URLEncodedFormHandler(),
 })  # type: ignore


Test patch:
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -10,8 +10,10 @@
 import pytest
 
 import falcon
+from falcon import media
 from falcon import testing
 from falcon import util
+from falcon.constants import MEDIA_JSON
 from falcon.util import deprecation, json, misc, structures, uri
 
 from _util import create_app, to_coroutine  # NOQA
@@ -874,6 +876,81 @@ def test_override_method_with_extras(self, asgi):
         assert result.status_code == 200
         assert result.text == 'test'
 
+    @pytest.mark.parametrize('content_type', [
+        'application/json',
+        'application/json; charset=UTF-8',
+        'application/yaml',
+    ])
+    def test_simulate_content_type(self, content_type):
+        class MediaMirror():
+            def on_post(self, req, resp):
+                resp.media = req.media
+
+        app = create_app(asgi=False)
+        app.add_route('/', MediaMirror())
+
+        client = testing.TestClient(app)
+        headers = {'Content-Type': content_type}
+        payload = b'{"hello": "world"}'
+
+        resp = client.simulate_post('/', headers=headers, body=payload)
+
+        if MEDIA_JSON in content_type:
+            assert resp.status_code == 200
+            assert resp.json == {'hello': 'world'}
+        else:
+            # JSON handler should not have been called for YAML
+            assert resp.status_code == 415
+
+    @pytest.mark.parametrize('content_type', [
+        'application/json',
+        'application/json; charset=UTF-8',
+        'application/yaml'
+    ])
+    def test_simulate_content_type_extra_handler(self, asgi, content_type):
+        class TrackingJSONHandler(media.JSONHandler):
+            def __init__(self):
+                super().__init__()
+                self.deserialize_count = 0
+
+            def deserialize(self, *args, **kwargs):
+                result = super().deserialize(*args, **kwargs)
+                self.deserialize_count += 1
+                return result
+
+            async def deserialize_async(self, *args, **kwargs):
+                result = await super().deserialize_async(*args, **kwargs)
+                self.deserialize_count += 1
+                return result
+
+        resource = testing.SimpleTestResourceAsync() if asgi else testing.SimpleTestResource()
+        app = create_app(asgi)
+        app.add_route('/', resource)
+
+        handler = TrackingJSONHandler()
+        extra_handlers = {'application/json': handler}
+        app.req_options.media_handlers.update(extra_handlers)
+        app.resp_options.media_handlers.update(extra_handlers)
+
+        client = testing.TestClient(app)
+        headers = {
+            'Content-Type': content_type,
+            'capture-req-media': 'y',
+        }
+        payload = b'{"hello": "world"}'
+        resp = client.simulate_post('/', headers=headers, body=payload)
+
+        if MEDIA_JSON in content_type:
+            # Test that our custom deserializer was called
+            assert handler.deserialize_count == 1
+            assert resource.captured_req_media == {'hello': 'world'}
+            assert resp.status_code == 200
+        else:
+            # YAML should not get handled
+            assert handler.deserialize_count == 0
+            assert resource.captured_req_media is None
+            assert resp.status_code == 415
+
 
 class TestNoApiClass(testing.TestCase):
     def test_something(self):