Repository: aio-libs/aiohttp

Issue:
json request for certain media types fail
## Long story short

The `resp.json()` call reports a bug if the return type is something like `application/XXX+json` (as opposed to `application/json`). However, `application/XXX+json` is a perfectly fine media type, "extending" the _semantics_ (not the syntax) of the core json syntax, ie, should be parsed as bona fide json...



## Your environment

<!-- Describe the environment you have that lead to your issue.
     This includes project version, OS, proxy server and other bits that
     are related to your case. -->

OS X 10.11.6, Python 3.6.2



Discussion:
You can pass a custom content_type parameter to the ClientResponse.json method, although I agree that this isn't perhaps the most sensible API.  I just switched some code from using Requests to using this library and while Requests could decode the JSON without a problem, aiohttp reported an error which will trip some people up.
Related to #2174
Extending the list of accepted json content types is fair easy change.
Please propose a Pull Request. `application/XXX+json` is not acceptable but I pretty sure you could grab a list of wide spread json types from `requests` library.
use `content_type=None` for disabling the check
@asvetlov, unfortunately I missed the [reply on Sep 11](https://github.com/aio-libs/aiohttp/issues/2206#issuecomment-328630051); probably my fault.

However, I am not sure I agree with your statement that "`application/XXX+json` is not acceptable". In my view, it is perfectly acceptable (and future proof) if the code simply ignores the `XXX+` part, whatever `XXX` is. Looking at [RFC 6839](https://tools.ietf.org/html/rfc6839#section-2):

> At the same time, using the suffix allows receivers of the media types to do generic processing of the underlying representation in cases where
>
>* they do not need to perform special handling of the particular semantics of the exact media type, and
>
>* there is no special knowledge needed by such a generic processor in order to parse that underlying representation other than what would be needed to parse any example of that underlying representation.
>

I.e., there is no reason for aiohttp to maintain whatever specific list. This is valid for any conversion step, not only for JSON.

---

Yes, I agree that [using `content_type=None` can be done](https://github.com/aio-libs/aiohttp/issues/2206#issuecomment-366452898), but that is clearly just a hack...
What to do?
On one hand we want to prevent processing non-JSON data in `.json()` method.
On other we don't want to be too strict.

Maybe you can pass a custom content type when fetching a data from your server?
Or create a custom json reading function on top of `resp.read()`?
@asvetlov,

The fundamental fact is that data coming with `XXX+json` **is** JSON. From the point of view of aiohttp that all that matters. A properly registered `XXX+json` media type means that the content _is_ JSON but it may have some additional syntactic or structural constraint _within the framework of JSON_. This may mean imposing a particular JSON Schema, or some more complicated specification on top of what JSON means, or may be no extra syntactic restriction whatsoever. 

To take a specific example. `application/ld+json` is a bona fide media type defined through the structural syntax; the content is defined via the [JSON-LD Recommendation](https://www.w3.org/TR/json-ld/). That specification adds some additional semantics to individual terms, requires some specific structure to the JSON data, but the essential point is that a JSON-LD content is a perfectly fine JSON content. (This is probably one of the most complex examples. Others may be a few paragraphs only.)

In my view, this means that the handling of the `XXX+json` (or `XXX+xml`, etc.) is simple, as far as aiohttp goes: strip the `XXX+` string altogether (the `+` character is not allowed elsewhere in a media type name), and handle the content as `json` (or `xml`). You are done. All other processing is in the application domain.

(You will find an abundance of such media types on the [media type registry](https://www.iana.org/assignments/media-types/media-types.xhtml).)
You would say that any `application/xxx+json` is a valid JSON type.
Or parsing all JSON registered types from the media registry is better?
I'd say both statements are true.
By sending `application/xxx+json` client explicitly tells server to expect valid json payload and vice versa
@asvetlov : parsing the json types from the media registry is of course nicer and better but... it is not that obvious. Would aiohttp check that registry any time the "read" function is called? Would set up some sort of a local cache for the registry that has a finite lifetime? The registry is a changing target after all...

I believe just saying that "any `application/xxx+json` is a valid JSON type" is good enough. 
Reopen the issue.
@iherman would you provide a Pull Request with the first option (accepting `application/xxx+json`)?
@asvetlov : I have no idea when I could do that; I have never looked into the internal of the aiohttp code, ie, it would take me quite a while to familiarize myself with it to produce a proper PR. I can try at some point unless somebody beats me into it, but it may take a while:-(
@asvetlov, I have looked at the code and I _think_ I have found the few changes that are required. However, I simply will not have the time to make a proper pull request, ie, running all the necessary tests on the code overall. It would require a major setup on my machine, understanding how the test harness works, etc. Sorry, but that will not work.

I will look at the code again tomorrow just to be sure, and I could then forward you (tomorrow or on Wednesday) the file where I have made some changes: `client_reqrep.py`. Would that work?

Resolving patch:
diff --git a/aiohttp/client_reqrep.py b/aiohttp/client_reqrep.py
--- a/aiohttp/client_reqrep.py
+++ b/aiohttp/client_reqrep.py
@@ -2,6 +2,7 @@
 import codecs
 import io
 import json
+import re
 import sys
 import traceback
 import warnings
@@ -39,6 +40,9 @@
 __all__ = ('ClientRequest', 'ClientResponse', 'RequestInfo', 'Fingerprint')
 
 
+json_re = re.compile('^application/(?:[\w.+-]+?\+)?json')
+
+
 @attr.s(frozen=True, slots=True)
 class ContentDisposition:
     type = attr.ib(type=str)
@@ -131,6 +135,12 @@ def _merge_ssl_params(ssl, verify_ssl, ssl_context, fingerprint):
 ConnectionKey = namedtuple('ConnectionKey', ['host', 'port', 'ssl'])
 
 
+def _is_expected_content_type(response_content_type, expected_content_type):
+    if expected_content_type == 'application/json':
+        return json_re.match(response_content_type)
+    return expected_content_type in response_content_type
+
+
 class ClientRequest:
     GET_METHODS = {
         hdrs.METH_GET,
@@ -859,7 +869,7 @@ async def json(self, *, encoding=None, loads=json.loads,
 
         if content_type:
             ctype = self.headers.get(hdrs.CONTENT_TYPE, '').lower()
-            if content_type not in ctype:
+            if not _is_expected_content_type(ctype, content_type):
                 raise ContentTypeError(
                     self.request_info,
                     self.history,


Test patch:
diff --git a/tests/test_client_response.py b/tests/test_client_response.py
--- a/tests/test_client_response.py
+++ b/tests/test_client_response.py
@@ -345,6 +345,45 @@ def side_effect(*args, **kwargs):
     assert response._connection is None
 
 
+async def test_json_extended_content_type(loop, session):
+    response = ClientResponse('get', URL('http://def-cl-resp.org'))
+    response._post_init(loop, session)
+
+    def side_effect(*args, **kwargs):
+        fut = loop.create_future()
+        fut.set_result('{"тест": "пройден"}'.encode('cp1251'))
+        return fut
+
+    response.headers = {
+        'Content-Type':
+            'application/this.is-1_content+subtype+json;charset=cp1251'}
+    content = response.content = mock.Mock()
+    content.read.side_effect = side_effect
+
+    res = await response.json()
+    assert res == {'тест': 'пройден'}
+    assert response._connection is None
+
+
+async def test_json_custom_content_type(loop, session):
+    response = ClientResponse('get', URL('http://def-cl-resp.org'))
+    response._post_init(loop, session)
+
+    def side_effect(*args, **kwargs):
+        fut = loop.create_future()
+        fut.set_result('{"тест": "пройден"}'.encode('cp1251'))
+        return fut
+
+    response.headers = {
+        'Content-Type': 'custom/type;charset=cp1251'}
+    content = response.content = mock.Mock()
+    content.read.side_effect = side_effect
+
+    res = await response.json(content_type='custom/type')
+    assert res == {'тест': 'пройден'}
+    assert response._connection is None
+
+
 async def test_json_custom_loader(loop, session):
     response = ClientResponse('get', URL('http://def-cl-resp.org'))
     response._post_init(loop, session)