Repository: scrapy/scrapy

Issue:
response.json()?
python-requests have response.json() that decodes json body and returns appropriate Python objects. Does it make sense to have something like this in Scrapy? 
Add json response
My first PR for Scrapy that is supposed to fix GH https://github.com/scrapy/scrapy/issues/2444

Adds a `JsonResponse` class and a `json()` function similar to the one for the "requests" library [here](https://github.com/psf/requests/blob/master/requests/models.py#L874-L898)

Test manually by 
```
# from the Scrapy package itself install Scrapy first  
python setup.py install

# then run the Scrapy shell
scrapy shell https://api.github.com/events
>> response.json
```

Fixes #2444



Discussion:
@eliasdorneles proposed it here: https://github.com/scrapy/scrapy/issues/1729#issuecomment-175195461. I think that's fine to add it. Not sure if we need JsonResponse - maybe it'd be nice to have, to be able to check if json is available. But maybe it doesn't worths it, I'm not sure.
The question is whether should response.json raise AttributeError if response is not JSON; if it should (like .text, .css and .xpath) then we need JsonResponse. If it shouldn't then we don't need JsonResponse.
python-requests provide it by default and just fail with ValueError if body is not json. Of course their implementation is not necessarily ideal, maybe we can do better. We could check for response headers and check if they have `application/json` content type, but there's a risk that some responses won't have json content type and they will be json (it's probably not standard compliant but I imagine it might be possible). It would also make code somewhat more complicated, e.g.:

```python
# some spider callback
if isinstance(response, scrapy.response.JSONResponse):
    data = response.json()
else:
   raise ValueError('expected JSON')
```

instead of:

```python
data = response.json()
```
@kmike oops, only `.text` is raising AttributeError: https://github.com/scrapy/scrapy/blob/master/scrapy/http/response/__init__.py#L86-L103 -- should I change that for the case people do `hasattr(response, 'css')` as well?
About `response.json`, I'd avoid creating a new subclass (JsonResponse) if we don't have a clear use for it.

I've been thinking that the `.json()` method could be on the selectors as well, as a shortcut to extract() -> json.loads, but that idea has to mature a little bit. :)
If we do include the `response.json` then could we also include a method for JmesQuery. Something like this `response.jmes('[::].urls').extract()`. Although, I don't think we will need `extract_first()` and just a simple `extract()` will do.
Regarding jmespath - see also https://github.com/scrapy/parsel/pull/27.

As for .json(), is it just a shortcut for `json.loads(response.text)`? 

Supporting .json() it in selectors sounds good, but I'd wait for real use cases / examples for that.
mark
I was thinking about `.json` and `.jmespath` this week.
I think that something like `response.as_json()` or `response.css('script::text').as_json()` would be nice, since sometimes we're handling JSON data and apis.

However, I don't think it should be only a wrapper for `json.loads(response.text)`. Otherwise we would end up writing some code like
```
data = response.json()
item['field'] = data['property']['property']
```
But, it would be better to load the JSON as a python object and then write
```
data = response.json()
item['field'] = data.property.property
```
Would love to have this feature.
I have added a [PR](https://github.com/scrapy/scrapy/pull/4460) which is being reviewed by @elacuesta at the moment, please let me know if it goes in the right direction.
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/4460?src=pr&el=h1) Report
> Merging [#4460](https://codecov.io/gh/scrapy/scrapy/pull/4460?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/scrapy/commit/8845773d44329194ee73fa4985a94e768bf664e7&el=desc) will **increase** coverage by `0.04%`.
> The diff coverage is `90.90%`.


```diff
@@            Coverage Diff             @@
##           master    #4460      +/-   ##
==========================================
+ Coverage   84.75%   84.79%   +0.04%     
==========================================
  Files         164      165       +1     
  Lines        9889     9900      +11     
  Branches     1469     1470       +1     
==========================================
+ Hits         8381     8395      +14     
+ Misses       1250     1248       -2     
+ Partials      258      257       -1     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/4460?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scrapy/responsetypes.py](https://codecov.io/gh/scrapy/scrapy/pull/4460/diff?src=pr&el=tree#diff-c2NyYXB5L3Jlc3BvbnNldHlwZXMucHk=) | `94.11% <ø> (ø)` | |
| [scrapy/http/response/json.py](https://codecov.io/gh/scrapy/scrapy/pull/4460/diff?src=pr&el=tree#diff-c2NyYXB5L2h0dHAvcmVzcG9uc2UvanNvbi5weQ==) | `90.00% <90.00%> (ø)` | |
| [scrapy/http/\_\_init\_\_.py](https://codecov.io/gh/scrapy/scrapy/pull/4460/diff?src=pr&el=tree#diff-c2NyYXB5L2h0dHAvX19pbml0X18ucHk=) | `100.00% <100.00%> (ø)` | |
| [scrapy/utils/misc.py](https://codecov.io/gh/scrapy/scrapy/pull/4460/diff?src=pr&el=tree#diff-c2NyYXB5L3V0aWxzL21pc2MucHk=) | `97.11% <0.00%> (+1.92%)` | :arrow_up: |
| [scrapy/utils/defer.py](https://codecov.io/gh/scrapy/scrapy/pull/4460/diff?src=pr&el=tree#diff-c2NyYXB5L3V0aWxzL2RlZmVyLnB5) | `95.65% <0.00%> (+2.17%)` | :arrow_up: |

thanks for the good review @elacuesta , feedback has been applied 

Resolving patch:
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -5,6 +5,7 @@
 See documentation in docs/topics/request-response.rst
 """
 
+import json
 import warnings
 from contextlib import suppress
 from typing import Generator
@@ -21,10 +22,13 @@
 from scrapy.utils.python import memoizemethod_noargs, to_unicode
 from scrapy.utils.response import get_base_url
 
+_NONE = object()
+
 
 class TextResponse(Response):
 
     _DEFAULT_ENCODING = 'ascii'
+    _cached_decoded_json = _NONE
 
     def __init__(self, *args, **kwargs):
         self._encoding = kwargs.pop('encoding', None)
@@ -68,6 +72,14 @@ def body_as_unicode(self):
                       ScrapyDeprecationWarning)
         return self.text
 
+    def json(self):
+        """
+        Deserialize a JSON document to a Python object.
+        """
+        if self._cached_decoded_json is _NONE:
+            self._cached_decoded_json = json.loads(self.text)
+        return self._cached_decoded_json
+
     @property
     def text(self):
         """ Body as unicode """


Test patch:
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -1,4 +1,5 @@
 import unittest
+from unittest import mock
 from warnings import catch_warnings
 
 from w3lib.encoding import resolve_encoding
@@ -669,6 +670,26 @@ def test_body_as_unicode_deprecation_warning(self):
             self.assertEqual(len(warnings), 1)
             self.assertEqual(warnings[0].category, ScrapyDeprecationWarning)
 
+    def test_json_response(self):
+        json_body = b"""{"ip": "109.187.217.200"}"""
+        json_response = self.response_class("http://www.example.com", body=json_body)
+        self.assertEqual(json_response.json(), {'ip': '109.187.217.200'})
+
+        text_body = b"""<html><body>text</body></html>"""
+        text_response = self.response_class("http://www.example.com", body=text_body)
+        with self.assertRaises(ValueError):
+            text_response.json()
+
+    def test_cache_json_response(self):
+        json_valid_bodies = [b"""{"ip": "109.187.217.200"}""", b"""null"""]
+        for json_body in json_valid_bodies:
+            json_response = self.response_class("http://www.example.com", body=json_body)
+
+            with mock.patch('json.loads') as mock_json:
+                for _ in range(2):
+                    json_response.json()
+                mock_json.assert_called_once_with(json_body.decode())
+
 
 class HtmlResponseTest(TextResponseTest):