Repository: scrapy/scrapy

Issue:
Should we delete the deprecated code in Scrapy 2.0?
I recopilated some of the old (before 2018) deprecation warnings to discuss if we can delete them in the upcoming **Scrapy 2.0** release (chronological order):

**`BaseSgmlLinkExtractor` and `SgmlLinkExtractor` (classes)**
* ~~Deprecation warnings: 07/2014~~
* ~~Links:~~
    * ~~https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/sgml.py#L23~~
    * ~~https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/sgml.py#L116~~
* PR: #4679

**`Crawler.spiders` (property)**
*  ~~Deprecation warning: 08/2014~~
*  ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/crawler.py#L74~~
* PR: https://github.com/scrapy/scrapy/pull/4398

**`HtmlParserLinkExtractor` (class)**
* ~~Deprecation warning: 05/2015~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/linkextractors/htmlparser.py#L22~~
* PR: #4674

**`stringify_dict` (function)**
* ~~Deprecation warning: 07/2015~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L296~~
* PR: https://github.com/scrapy/scrapy/pull/4362

**`is_writable` (function)**
* ~~Deprecation warning: 07/2015~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L311~~
* PR: https://github.com/scrapy/scrapy/pull/4362

**`setattr_default` (function)**
* ~~Deprecation warning: 07/2015~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L322~~
* PR: https://github.com/scrapy/scrapy/pull/4362

**Initialising `scrapy.extensions.feedexport.S3FeedStorage` without AWS keys is deprecated. (procedure)**
* ~~Deprecation warning: 11/2015~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/extensions/feedexport.py#L103~~
* PR: https://github.com/scrapy/scrapy/pull/4411

**`scrapy.utils.datatypes.MultiValueDict`, `MultiValueDictKeyError` and  `SiteNode` (classes)**
* ~~Deprecation warnings: 01/2016~~
* ~~Links~~:
    * ~~https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L46~~
    * ~~https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L19~~
    * ~~https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/datatypes.py#L161~~
* PR: https://github.com/scrapy/scrapy/pull/4400/

**`isbinarytext` (function)**
* ~~Deprecation warning: 03/2016~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/utils/python.py#L168~~
* PR: https://github.com/scrapy/scrapy/pull/4362 

**`scrapy.downloadermiddlewares.chunked` (module)**
* ~~Deprecation warning: 11/2016~~
* ~~Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/downloadermiddlewares/chunked.py#L7~~
* PR: https://github.com/scrapy/scrapy/pull/4431/

**`Spider.make_requests_from_url` (method)**
* Deprecation warning: 02/2017
* Link: https\://github.com/scrapy/scrapy/blob/c4ee4b6075fbb616272788b14b70f0af63dc2877/scrapy/spiders/__init__.py#L66 
* PR: https://github.com/scrapy/scrapy/pull/4178

There are other deprecation warnings but as they are from 2018 to the present I think we can leave it there for a while. However, as we are going to release a major version maybe is a good moment to do it.

**Do you see any inconvenience?**

**Any thoughts?** Let me know what you think.


Discussion:

Resolving patch:
diff --git a/scrapy/spiders/__init__.py b/scrapy/spiders/__init__.py
--- a/scrapy/spiders/__init__.py
+++ b/scrapy/spiders/__init__.py
@@ -4,14 +4,12 @@
 See documentation in docs/topics/spiders.rst
 """
 import logging
-import warnings
 from typing import Optional
 
 from scrapy import signals
 from scrapy.http import Request
 from scrapy.utils.trackref import object_ref
 from scrapy.utils.url import url_is_from_spider
-from scrapy.utils.deprecate import method_is_overridden
 
 
 class Spider(object_ref):
@@ -57,34 +55,13 @@ def _set_crawler(self, crawler):
         crawler.signals.connect(self.close, signals.spider_closed)
 
     def start_requests(self):
-        cls = self.__class__
         if not self.start_urls and hasattr(self, 'start_url'):
             raise AttributeError(
                 "Crawling could not start: 'start_urls' not found "
                 "or empty (but found 'start_url' attribute instead, "
                 "did you miss an 's'?)")
-        if method_is_overridden(cls, Spider, 'make_requests_from_url'):
-            warnings.warn(
-                "Spider.make_requests_from_url method is deprecated; it "
-                "won't be called in future Scrapy releases. Please "
-                "override Spider.start_requests method instead "
-                f"(see {cls.__module__}.{cls.__name__}).",
-            )
-            for url in self.start_urls:
-                yield self.make_requests_from_url(url)
-        else:
-            for url in self.start_urls:
-                yield Request(url, dont_filter=True)
-
-    def make_requests_from_url(self, url):
-        """ This method is deprecated. """
-        warnings.warn(
-            "Spider.make_requests_from_url method is deprecated: "
-            "it will be removed and not be called by the default "
-            "Spider.start_requests method in future Scrapy releases. "
-            "Please override Spider.start_requests method instead."
-        )
-        return Request(url, dont_filter=True)
+        for url in self.start_urls:
+            yield Request(url, dont_filter=True)
 
     def _parse(self, response, **kwargs):
         return self.parse(response, **kwargs)


Test patch:
diff --git a/tests/test_spider.py b/tests/test_spider.py
--- a/tests/test_spider.py
+++ b/tests/test_spider.py
@@ -584,39 +584,6 @@ def test_crawl_spider(self):
         assert issubclass(CrawlSpider, Spider)
         assert isinstance(CrawlSpider(name='foo'), Spider)
 
-    def test_make_requests_from_url_deprecated(self):
-        class MySpider4(Spider):
-            name = 'spider1'
-            start_urls = ['http://example.com']
-
-        class MySpider5(Spider):
-            name = 'spider2'
-            start_urls = ['http://example.com']
-
-            def make_requests_from_url(self, url):
-                return Request(url + "/foo", dont_filter=True)
-
-        with warnings.catch_warnings(record=True) as w:
-            # spider without overridden make_requests_from_url method
-            # doesn't issue a warning
-            spider1 = MySpider4()
-            self.assertEqual(len(list(spider1.start_requests())), 1)
-            self.assertEqual(len(w), 0)
-
-            # spider without overridden make_requests_from_url method
-            # should issue a warning when called directly
-            request = spider1.make_requests_from_url("http://www.example.com")
-            self.assertTrue(isinstance(request, Request))
-            self.assertEqual(len(w), 1)
-
-            # spider with overridden make_requests_from_url issues a warning,
-            # but the method still works
-            spider2 = MySpider5()
-            requests = list(spider2.start_requests())
-            self.assertEqual(len(requests), 1)
-            self.assertEqual(requests[0].url, 'http://example.com/foo')
-            self.assertEqual(len(w), 2)
-
 
 class NoParseMethodSpiderTest(unittest.TestCase):