Repository: scrapy/scrapy

Issue:
response.follow_all or SelectorList.follow_all shortcut
What do you think about adding response.follow_all shortcut, which returns a list of requests? This is inspired by this note in docs:

> `response.follow(response.css('li.next a'))` is not valid because response.css returns a list-like object with selectors for all results, not a single selector. A for loop like in the example above, or response.follow(response.css('li.next a')[0]) is fine.

So instead of 

```py
for href in response.css('li.next a::attr(href)'):
    yield response.follow(href, callback=self.parse)
```

users would be able to write (in Python 3)

```py
yield from response.follow_all(response.css('li.next a::attr(href)'), self.parse)
```

We can also add 'css' and 'xpath' support to it, as keyword arguments; it would shorten the code to this:

```py
yield from response.follow_all(css='li.next a::attr(href)', callback=self.parse)
```

(this is a follow-up to https://github.com/scrapy/scrapy/issues/1940 and https://github.com/scrapy/scrapy/issues/2540)


Discussion:
An alternative is to implement it on SelectorList:

```py
yield from response.css('li.next a').follow_all(self.parse, meta={...})
```
For Python 2 users we may add RequestSet container class, so that `yield` works. It could be helpful for await API as well - to allow waiting for several responses at the same time, while executing them in parallel.

```py
page_response = await response.css('.pages a::attr(href)').follow_all()
# page_responses is a list of scrapy.Response objects for all pages
```

or 

```py
responses = await RequestSet([Request(url1), Request(url2)])
```

I'm not sure we can guarantee the order in the ``responses`` array.
But this async API is likely a separate question, for a far future :)
A RequestSet class is useful regardless of whether or not you are using Python 3. 

It's useful to track state that's shared among a subset of requests, but not all of them.  E.g. you are parsing a subsection of a site and you want to track duplicates or some statistics for that subsection only.  You need to store some info which becomes irrelevant as soon as the last request from that subset is dealt with. If you store that info on the spider, you need to carefully track the set of active requests using callbacks & errbacks. The implementation is error-prone and hard to do right on the first try, so it would be nice to have it shipped in well-tested state with the package.
@immerrr do you have an idea on how this RequestSet API could look like? 
It is also somewhat related to https://github.com/scrapy/scrapy/issues/1226, though in an indirect way.
@kmike, only a rough one.

I think of RequestSet as something that:
- has a `DeferredList`-like API (`asyncio.gather` has the drawback of being a function rather than an object)
- knows that it contains Requests as deferreds
- ~~has its own `callback` to be run when the request set is dealt with, probably an `errback` too, for symmetry~~ has a Deferred that would fire when the requestset is being cleaned up
- has its own `'meta'` dictionary, much like the one shared between Request & Response objects
- since it knows that it contains Requests, it can piggyback on the first received value and do `response.meta['request_set'] = self` so that the callbacks can access the shared data
- (maybe) it should silently copy fields from `request_set.meta` to `response.meta` if they are unset in `request.meta`, or maybe even make `request.meta` a ChainDict with fallback to `request_set.meta`
- it should wrap requests coming from its respective response callbacks unless specifically asked not to do that, e.g. with `request.meta['request_set'] = None`
- (maybe) it should be possible to return other RequestSet from response callbacks 
- (maybe) returned RequestSets should be made nestable, i.e. to keep the parent RequestSet alive during their lifetime if not explicitly asked not to with `request_set.meta['request_set'] = None` (if nesting is considered, the `request_set` metadata key seems redundant and we might consider `parent_set` instead.
- not sure if it's worth it to make them nestable, i.e. if a certain response callback produces a different RequestSet, should it be owned by the parent request set?

One more thing to consider is cross-referencing RequestSets, i.e. when two requests that should belong to one RequestSet are produced by different callbacks and thus have different scopes. Maybe a simple `WeakValueDictionary` would suffice to lookup the sets and ensure the references are cleaned up as necessary. But then you'd have the usual get-or-create operation, that might be worth creating an etalon implementation for.
@immerrr could you please coopy-paste this to a new issue?
@kmike done: #2600

Resolving patch:
diff --git a/scrapy/http/response/__init__.py b/scrapy/http/response/__init__.py
--- a/scrapy/http/response/__init__.py
+++ b/scrapy/http/response/__init__.py
@@ -4,14 +4,15 @@
 
 See documentation in docs/topics/request-response.rst
 """
+from typing import Generator
 from urllib.parse import urljoin
 
-from scrapy.http.request import Request
+from scrapy.exceptions import NotSupported
+from scrapy.http.common import obsolete_setter
 from scrapy.http.headers import Headers
+from scrapy.http.request import Request
 from scrapy.link import Link
 from scrapy.utils.trackref import object_ref
-from scrapy.http.common import obsolete_setter
-from scrapy.exceptions import NotSupported
 
 
 class Response(object_ref):
@@ -41,8 +42,8 @@ def _set_url(self, url):
         if isinstance(url, str):
             self._url = url
         else:
-            raise TypeError('%s url must be str, got %s:' % (type(self).__name__,
-                type(url).__name__))
+            raise TypeError('%s url must be str, got %s:' %
+                            (type(self).__name__, type(url).__name__))
 
     url = property(_get_url, obsolete_setter(_set_url, 'url'))
 
@@ -123,14 +124,51 @@ def follow(self, url, callback=None, method='GET', headers=None, body=None,
         elif url is None:
             raise ValueError("url can't be None")
         url = self.urljoin(url)
-        return Request(url, callback,
-                       method=method,
-                       headers=headers,
-                       body=body,
-                       cookies=cookies,
-                       meta=meta,
-                       encoding=encoding,
-                       priority=priority,
-                       dont_filter=dont_filter,
-                       errback=errback,
-                       cb_kwargs=cb_kwargs)
+        return Request(
+            url=url,
+            callback=callback,
+            method=method,
+            headers=headers,
+            body=body,
+            cookies=cookies,
+            meta=meta,
+            encoding=encoding,
+            priority=priority,
+            dont_filter=dont_filter,
+            errback=errback,
+            cb_kwargs=cb_kwargs,
+        )
+
+    def follow_all(self, urls, callback=None, method='GET', headers=None, body=None,
+                   cookies=None, meta=None, encoding='utf-8', priority=0,
+                   dont_filter=False, errback=None, cb_kwargs=None):
+        # type: (...) -> Generator[Request, None, None]
+        """
+        Return an iterable of :class:`~.Request` instances to follow all links
+        in ``urls``. It accepts the same arguments as ``Request.__init__`` method,
+        but elements of ``urls`` can be relative URLs or :class:`~scrapy.link.Link` objects,
+        not only absolute URLs.
+
+        :class:`~.TextResponse` provides a :meth:`~.TextResponse.follow_all`
+        method which supports selectors in addition to absolute/relative URLs
+        and Link objects.
+        """
+        if not hasattr(urls, '__iter__'):
+            raise TypeError("'urls' argument must be an iterable")
+        return (
+            self.follow(
+                url=url,
+                callback=callback,
+                method=method,
+                headers=headers,
+                body=body,
+                cookies=cookies,
+                meta=meta,
+                encoding=encoding,
+                priority=priority,
+                dont_filter=dont_filter,
+                errback=errback,
+                cb_kwargs=cb_kwargs,
+            )
+            for url in urls
+        )
diff --git a/scrapy/http/response/text.py b/scrapy/http/response/text.py
--- a/scrapy/http/response/text.py
+++ b/scrapy/http/response/text.py
@@ -5,17 +5,19 @@
 See documentation in docs/topics/request-response.rst
 """
 
+from contextlib import suppress
+from typing import Generator
 from urllib.parse import urljoin
 
 import parsel
-from w3lib.encoding import html_to_unicode, resolve_encoding, \
-    html_body_declared_encoding, http_content_type_encoding
+from w3lib.encoding import (html_body_declared_encoding, html_to_unicode,
+                            http_content_type_encoding, resolve_encoding)
 from w3lib.html import strip_html5_whitespace
 
-from scrapy.http.request import Request
+from scrapy.http import Request
 from scrapy.http.response import Response
-from scrapy.utils.response import get_base_url
 from scrapy.utils.python import memoizemethod_noargs, to_unicode
+from scrapy.utils.response import get_base_url
 
 
 class TextResponse(Response):
@@ -40,7 +42,7 @@ def _set_body(self, body):
         if isinstance(body, str):
             if self._encoding is None:
                 raise TypeError('Cannot convert unicode body - %s has no encoding' %
-                    type(self).__name__)
+                                type(self).__name__)
             self._body = body.encode(self._encoding)
         else:
             super(TextResponse, self)._set_body(body)
@@ -86,8 +88,8 @@ def _body_inferred_encoding(self):
         if self._cached_benc is None:
             content_type = to_unicode(self.headers.get(b'Content-Type', b''))
             benc, ubody = html_to_unicode(content_type, self.body,
-                    auto_detect_fun=self._auto_detect_fun,
-                    default_encoding=self._DEFAULT_ENCODING)
+                                          auto_detect_fun=self._auto_detect_fun,
+                                          default_encoding=self._DEFAULT_ENCODING)
             self._cached_benc = benc
             self._cached_ubody = ubody
         return self._cached_benc
@@ -126,13 +128,14 @@ def follow(self, url, callback=None, method='GET', headers=None, body=None,
         It accepts the same arguments as ``Request.__init__`` method,
         but ``url`` can be not only an absolute URL, but also
 
-        * a relative URL;
-        * a scrapy.link.Link object (e.g. a link extractor result);
-        * an attribute Selector (not SelectorList) - e.g.
+        * a relative URL
+        * a :class:`~scrapy.link.Link` object, e.g. the result of
+          :ref:`topics-link-extractors`
+        * a :class:`~scrapy.selector.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
+          ``response.css('a.my_link')[0]``
+        * an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
           ``response.css('a::attr(href)')[0]`` or
-          ``response.xpath('//img/@src')[0]``.
-        * a Selector for ``<a>`` or ``<link>`` element, e.g.
-          ``response.css('a.my_link')[0]``.
+          ``response.xpath('//img/@src')[0]``
 
         See :ref:`response-follow-example` for usage examples.
         """
@@ -141,7 +144,9 @@ def follow(self, url, callback=None, method='GET', headers=None, body=None,
         elif isinstance(url, parsel.SelectorList):
             raise ValueError("SelectorList is not supported")
         encoding = self.encoding if encoding is None else encoding
-        return super(TextResponse, self).follow(url, callback,
+        return super(TextResponse, self).follow(
+            url=url,
+            callback=callback,
             method=method,
             headers=headers,
             body=body,
@@ -154,6 +159,69 @@ def follow(self, url, callback=None, method='GET', headers=None, body=None,
             cb_kwargs=cb_kwargs,
         )
 
+    def follow_all(self, urls=None, callback=None, method='GET', headers=None, body=None,
+                   cookies=None, meta=None, encoding=None, priority=0,
+                   dont_filter=False, errback=None, cb_kwargs=None,
+                   css=None, xpath=None):
+        # type: (...) -> Generator[Request, None, None]
+        """
+        A generator that produces :class:`~.Request` instances to follow all
+        links in ``urls``. It accepts the same arguments as the :class:`~.Request`'s
+        ``__init__`` method, except that each ``urls`` element does not need to be
+        an absolute URL, it can be any of the following:
+
+        * a relative URL
+        * a :class:`~scrapy.link.Link` object, e.g. the result of
+          :ref:`topics-link-extractors`
+        * a :class:`~scrapy.selector.Selector` object for a ``<link>`` or ``<a>`` element, e.g.
+          ``response.css('a.my_link')[0]``
+        * an attribute :class:`~scrapy.selector.Selector` (not SelectorList), e.g.
+          ``response.css('a::attr(href)')[0]`` or
+          ``response.xpath('//img/@src')[0]``
+
+        In addition, ``css`` and ``xpath`` arguments are accepted to perform the link extraction
+        within the ``follow_all`` method (only one of ``urls``, ``css`` and ``xpath`` is accepted).
+
+        Note that when passing a ``SelectorList`` as argument for the ``urls`` parameter or
+        using the ``css`` or ``xpath`` parameters, this method will not produce requests for
+        selectors from which links cannot be obtained (for instance, anchor tags without an
+        ``href`` attribute)
+        """
+        arg_count = len(list(filter(None, (urls, css, xpath))))
+        if arg_count != 1:
+            raise ValueError('Please supply exactly one of the following arguments: urls, css, xpath')
+        if not urls:
+            if css:
+                urls = self.css(css)
+            if xpath:
+                urls = self.xpath(xpath)
+        if isinstance(urls, parsel.SelectorList):
+            selectors = urls
+            urls = []
+            for sel in selectors:
+                with suppress(_InvalidSelector):
+                    urls.append(_url_from_selector(sel))
+        return super(TextResponse, self).follow_all(
+            urls=urls,
+            callback=callback,
+            method=method,
+            headers=headers,
+            body=body,
+            cookies=cookies,
+            meta=meta,
+            encoding=encoding,
+            priority=priority,
+            dont_filter=dont_filter,
+            errback=errback,
+            cb_kwargs=cb_kwargs,
+        )
+
+
+class _InvalidSelector(ValueError):
+    """
+    Raised when a URL cannot be obtained from a Selector
+    """
+
 
 def _url_from_selector(sel):
     # type: (parsel.Selector) -> str
@@ -161,12 +229,12 @@ def _url_from_selector(sel):
         # e.g. ::attr(href) result
         return strip_html5_whitespace(sel.root)
     if not hasattr(sel.root, 'tag'):
-        raise ValueError("Unsupported selector: %s" % sel)
+        raise _InvalidSelector("Unsupported selector: %s" % sel)
     if sel.root.tag not in ('a', 'link'):
-        raise ValueError("Only <a> and <link> elements are supported; got <%s>" %
-                         sel.root.tag)
+        raise _InvalidSelector("Only <a> and <link> elements are supported; got <%s>" %
+                               sel.root.tag)
     href = sel.root.get('href')
     if href is None:
-        raise ValueError("<%s> element has no href attribute: %s" %
-                         (sel.root.tag, sel))
+        raise _InvalidSelector("<%s> element has no href attribute: %s" %
+                               (sel.root.tag, sel))
     return strip_html5_whitespace(href)


Test patch:
diff --git a/tests/sample_data/link_extractor/sgml_linkextractor.html b/tests/sample_data/link_extractor/linkextractor.html
similarity index 100%
rename from tests/sample_data/link_extractor/sgml_linkextractor.html
rename to tests/sample_data/link_extractor/linkextractor.html
diff --git a/tests/sample_data/link_extractor/linkextractor_no_href.html b/tests/sample_data/link_extractor/linkextractor_no_href.html
new file mode 100644
--- /dev/null
+++ b/tests/sample_data/link_extractor/linkextractor_no_href.html
@@ -0,0 +1,25 @@
+<html>
+  <head>
+    <base href='http://example.com' />
+    <title>Sample page with anchor tags containing no href attribute, to test the TextResponse.follow_all method</title>
+  </head>
+
+<body>
+  <div class="quote">
+    <span class="text">“The world as we have created it is a process of our
+      thinking. It cannot be changed without changing our thinking.”</span>
+    <span>
+      by <small class="author">Albert Einstein</small>
+      <a href="/author/Albert-Einstein">(about)</a>
+    </span>
+    <div id="pagination" class="pagination">
+      Tags:
+      <a href="/page/1/">Page 1</a>
+      <a>Current</a>
+      <a href="/page/3/">Page 3</a>
+      <a href="/page/4/">Page 4</a>
+    </div>
+  </div>
+</body>
+
+</html>
\ No newline at end of file
diff --git a/tests/test_http_response.py b/tests/test_http_response.py
--- a/tests/test_http_response.py
+++ b/tests/test_http_response.py
@@ -141,6 +141,8 @@ def test_shortcut_attributes(self):
             r.css('body')
             r.xpath('//body')
 
+    # Response.follow
+
     def test_follow_url_absolute(self):
         self._assert_followed_url('http://foo.example.com',
                                   'http://foo.example.com')
@@ -164,6 +166,72 @@ def test_follow_whitespace_url(self):
     def test_follow_whitespace_link(self):
         self._assert_followed_url(Link('http://example.com/foo '),
                                   'http://example.com/foo%20')
+
+    # Response.follow_all
+
+    def test_follow_all_absolute(self):
+        url_list = ['http://example.org', 'http://www.example.org',
+                    'http://example.com', 'http://www.example.com']
+        self._assert_followed_all_urls(url_list, url_list)
+
+    def test_follow_all_relative(self):
+        relative = ['foo', 'bar', 'foo/bar', 'bar/foo']
+        absolute = [
+            'http://example.com/foo',
+            'http://example.com/bar',
+            'http://example.com/foo/bar',
+            'http://example.com/bar/foo',
+        ]
+        self._assert_followed_all_urls(relative, absolute)
+
+    def test_follow_all_links(self):
+        absolute = [
+            'http://example.com/foo',
+            'http://example.com/bar',
+            'http://example.com/foo/bar',
+            'http://example.com/bar/foo',
+        ]
+        links = map(Link, absolute)
+        self._assert_followed_all_urls(links, absolute)
+
+    def test_follow_all_invalid(self):
+        r = self.response_class("http://example.com")
+        if self.response_class == Response:
+            with self.assertRaises(TypeError):
+                list(r.follow_all(urls=None))
+            with self.assertRaises(TypeError):
+                list(r.follow_all(urls=12345))
+            with self.assertRaises(ValueError):
+                list(r.follow_all(urls=[None]))
+        else:
+            with self.assertRaises(ValueError):
+                list(r.follow_all(urls=None))
+            with self.assertRaises(TypeError):
+                list(r.follow_all(urls=12345))
+            with self.assertRaises(ValueError):
+                list(r.follow_all(urls=[None]))
+
+    def test_follow_all_whitespace(self):
+        relative = ['foo ', 'bar ', 'foo/bar ', 'bar/foo ']
+        absolute = [
+            'http://example.com/foo%20',
+            'http://example.com/bar%20',
+            'http://example.com/foo/bar%20',
+            'http://example.com/bar/foo%20',
+        ]
+        self._assert_followed_all_urls(relative, absolute)
+
+    def test_follow_all_whitespace_links(self):
+        absolute = [
+            'http://example.com/foo ',
+            'http://example.com/bar ',
+            'http://example.com/foo/bar ',
+            'http://example.com/bar/foo ',
+        ]
+        links = map(Link, absolute)
+        expected = [u.replace(' ', '%20') for u in absolute]
+        self._assert_followed_all_urls(links, expected)
+
     def _assert_followed_url(self, follow_obj, target_url, response=None):
         if response is None:
             response = self._links_response()
@@ -171,8 +239,21 @@ def _assert_followed_url(self, follow_obj, target_url, response=None):
         self.assertEqual(req.url, target_url)
         return req
 
+    def _assert_followed_all_urls(self, follow_obj, target_urls, response=None):
+        if response is None:
+            response = self._links_response()
+        followed = response.follow_all(follow_obj)
+        for req, target in zip(followed, target_urls):
+            self.assertEqual(req.url, target)
+            yield req
+
     def _links_response(self):
-        body = get_testdata('link_extractor', 'sgml_linkextractor.html')
+        body = get_testdata('link_extractor', 'linkextractor.html')
+        resp = self.response_class('http://example.com/index', body=body)
+        return resp
+
+    def _links_response_no_href(self):
+        body = get_testdata('link_extractor', 'linkextractor_no_href.html')
         resp = self.response_class('http://example.com/index', body=body)
         return resp
 
@@ -481,6 +562,53 @@ def test_follow_encoding(self):
         )
         self.assertEqual(req.encoding, 'cp1251')
 
+    def test_follow_all_css(self):
+        expected = [
+            'http://example.com/sample3.html',
+            'http://example.com/innertag.html',
+        ]
+        response = self._links_response()
+        extracted = [r.url for r in response.follow_all(css='a[href*="example.com"]')]
+        self.assertEqual(expected, extracted)
+
+    def test_follow_all_css_skip_invalid(self):
+        expected = [
+            'http://example.com/page/1/',
+            'http://example.com/page/3/',
+            'http://example.com/page/4/',
+        ]
+        response = self._links_response_no_href()
+        extracted1 = [r.url for r in response.follow_all(css='.pagination a')]
+        self.assertEqual(expected, extracted1)
+        extracted2 = [r.url for r in response.follow_all(response.css('.pagination a'))]
+        self.assertEqual(expected, extracted2)
+
+    def test_follow_all_xpath(self):
+        expected = [
+            'http://example.com/sample3.html',
+            'http://example.com/innertag.html',
+        ]
+        response = self._links_response()
+        extracted = response.follow_all(xpath='//a[contains(@href, "example.com")]')
+        self.assertEqual(expected, [r.url for r in extracted])
+
+    def test_follow_all_xpath_skip_invalid(self):
+        expected = [
+            'http://example.com/page/1/',
+            'http://example.com/page/3/',
+            'http://example.com/page/4/',
+        ]
+        response = self._links_response_no_href()
+        extracted1 = [r.url for r in response.follow_all(xpath='//div[@id="pagination"]/a')]
+        self.assertEqual(expected, extracted1)
+        extracted2 = [r.url for r in response.follow_all(response.xpath('//div[@id="pagination"]/a'))]
+        self.assertEqual(expected, extracted2)
+
+    def test_follow_all_too_many_arguments(self):
+        response = self._links_response()
+        with self.assertRaises(ValueError):
+            response.follow_all(css='a[href*="example.com"]', xpath='//a[contains(@href, "example.com")]')
+
 
 class HtmlResponseTest(TextResponseTest):
 
diff --git a/tests/test_linkextractors.py b/tests/test_linkextractors.py
--- a/tests/test_linkextractors.py
+++ b/tests/test_linkextractors.py
@@ -19,7 +19,7 @@ class LinkExtractorTestCase(unittest.TestCase):
         escapes_whitespace = False
 
         def setUp(self):
-            body = get_testdata('link_extractor', 'sgml_linkextractor.html')
+            body = get_testdata('link_extractor', 'linkextractor.html')
             self.response = HtmlResponse(url='http://example.com/index', body=body)
 
         def test_urls_type(self):