Repository: scrapy/scrapy
Pull request: Update Python version references after dropping support for 3.5
Covers everything but:

- CI (#4732)
- An `if` statement that needs to check the PyPy version instead of the Python version (#4732)
- `py36.py` (#4467)

Resolves #4733

Problem statement:
Review references to Python 3.5 and 3.6
Now that we are dropping Python 3.5 support, such references in the code and documentation are likely to need an update.

Make sure to also look for the `version_info` equivalents (i.e. `(3, 5)`, `(3, 6)`).

Hints from issue discussion:

Commit message: Remove unused import

Patch to review:
diff --git a/README.rst b/README.rst
index 0e3939e9b4d..a8f2ba52b64 100644
--- a/README.rst
+++ b/README.rst
@@ -40,7 +40,7 @@ including a list of features.
 Requirements
 ============
 
-* Python 3.5.2+
+* Python 3.6+
 * Works on Linux, Windows, macOS, BSD
 
 Install
diff --git a/docs/intro/install.rst b/docs/intro/install.rst
index 6d65ae2ee8c..8b4240bf647 100644
--- a/docs/intro/install.rst
+++ b/docs/intro/install.rst
@@ -9,8 +9,8 @@ Installation guide
 Supported Python versions
 =========================
 
-Scrapy requires Python 3.5.2+, either the CPython implementation (default) or
-the PyPy 5.9+ implementation (see :ref:`python:implementations`).
+Scrapy requires Python 3.6+, either the CPython implementation (default) or
+the PyPy 7.2.0+ implementation (see :ref:`python:implementations`).
 
 
 Installing Scrapy
diff --git a/docs/intro/tutorial.rst b/docs/intro/tutorial.rst
index f96c788873f..b3b8b47069c 100644
--- a/docs/intro/tutorial.rst
+++ b/docs/intro/tutorial.rst
@@ -405,8 +405,6 @@ to get all of them:
 
   from sys import version_info
 
-.. skip: next if(version_info < (3, 6), reason="Only Python 3.6+ dictionaries match the output")
-
 Having figured out how to extract each bit, we can now iterate over all the
 quotes elements and put them together into a Python dictionary:
 
diff --git a/docs/topics/coroutines.rst b/docs/topics/coroutines.rst
index a0952d323f9..3b1549bd3e0 100644
--- a/docs/topics/coroutines.rst
+++ b/docs/topics/coroutines.rst
@@ -17,19 +17,14 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   :class:`~scrapy.http.Request` callbacks.
 
-    The following are known caveats of the current implementation that we aim
-    to address in future versions of Scrapy:
-
-    -   The callback output is not processed until the whole callback finishes.
+    .. note:: The callback output is not processed until the whole callback
+        finishes.
 
         As a side effect, if the callback raises an exception, none of its
         output is processed.
 
-    -   Because `asynchronous generators were introduced in Python 3.6`_, you
-        can only use ``yield`` if you are using Python 3.6 or later.
-
-        If you need to output multiple items or requests and you are using
-        Python 3.5, return an iterable (e.g. a list) instead.
+        This is a known caveat of the current implementation that we aim to
+        address in a future version of Scrapy.
 
 -   The :meth:`process_item` method of
     :ref:`item pipelines <topics-item-pipeline>`.
@@ -44,8 +39,6 @@ hence use coroutine syntax (e.g. ``await``, ``async for``, ``async with``):
 
 -   :ref:`Signal handlers that support deferreds <signal-deferred>`.
 
-.. _asynchronous generators were introduced in Python 3.6: https://www.python.org/dev/peps/pep-0525/
-
 Usage
 =====
 
diff --git a/scrapy/__init__.py b/scrapy/__init__.py
index f0259a9b79a..4326ca4aa32 100644
--- a/scrapy/__init__.py
+++ b/scrapy/__init__.py
@@ -28,8 +28,8 @@
 
 
 # Check minimum required Python version
-if sys.version_info < (3, 5, 2):
-    print("Scrapy %s requires Python 3.5.2" % __version__)
+if sys.version_info < (3, 6):
+    print("Scrapy %s requires Python 3.6+" % __version__)
     sys.exit(1)
 
 
diff --git a/scrapy/utils/defer.py b/scrapy/utils/defer.py
index a3950db75b3..21ba02a0b5d 100644
--- a/scrapy/utils/defer.py
+++ b/scrapy/utils/defer.py
@@ -124,18 +124,11 @@ def iter_errback(iterable, errback, *a, **kw):
             errback(failure.Failure(), *a, **kw)
 
 
-def _isfuture(o):
-    # workaround for Python before 3.5.3 not having asyncio.isfuture
-    if hasattr(asyncio, 'isfuture'):
-        return asyncio.isfuture(o)
-    return isinstance(o, asyncio.Future)
-
-
 def deferred_from_coro(o):
     """Converts a coroutine into a Deferred, or returns the object as is if it isn't a coroutine"""
     if isinstance(o, defer.Deferred):
         return o
-    if _isfuture(o) or inspect.isawaitable(o):
+    if asyncio.isfuture(o) or inspect.isawaitable(o):
         if not is_asyncio_reactor_installed():
             # wrapping the coroutine directly into a Deferred, this doesn't work correctly with coroutines
             # that use asyncio, e.g. "await asyncio.sleep(1)"
@@ -167,7 +160,7 @@ def maybeDeferred_coro(f, *args, **kw):
 
     if isinstance(result, defer.Deferred):
         return result
-    elif _isfuture(result) or inspect.isawaitable(result):
+    elif asyncio.isfuture(result) or inspect.isawaitable(result):
         return deferred_from_coro(result)
     elif isinstance(result, failure.Failure):
         return defer.fail(result)
diff --git a/scrapy/utils/gz.py b/scrapy/utils/gz.py
index fbd7bd18fc2..11d433cf59b 100644
--- a/scrapy/utils/gz.py
+++ b/scrapy/utils/gz.py
@@ -6,11 +6,10 @@
 from scrapy.utils.decorators import deprecated
 
 
-# - Python>=3.5 GzipFile's read() has issues returning leftover
-#   uncompressed data when input is corrupted
-#   (regression or bug-fix compared to Python 3.4)
+# - GzipFile's read() has issues returning leftover uncompressed data when
+#   input is corrupted
 # - read1(), which fetches data before raising EOFError on next call
-#   works here but is only available from Python>=3.3
+#   works here
 @deprecated('GzipFile.read1')
 def read1(gzf, size=-1):
     return gzf.read1(size)
diff --git a/setup.py b/setup.py
index 52a27c3684a..0c22814001b 100644
--- a/setup.py
+++ b/setup.py
@@ -82,7 +82,6 @@ def has_environment_marker_platform_impl_support():
         'Operating System :: OS Independent',
         'Programming Language :: Python',
         'Programming Language :: Python :: 3',
-        'Programming Language :: Python :: 3.5',
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
@@ -92,7 +91,7 @@ def has_environment_marker_platform_impl_support():
         'Topic :: Software Development :: Libraries :: Application Frameworks',
         'Topic :: Software Development :: Libraries :: Python Modules',
     ],
-    python_requires='>=3.5.2',
+    python_requires='>=3.6',
     install_requires=install_requires,
     extras_require=extras_require,
 )
diff --git a/tests/requirements-py3.txt b/tests/requirements-py3.txt
index 44ddcded8b2..002e46fcc05 100644
--- a/tests/requirements-py3.txt
+++ b/tests/requirements-py3.txt
@@ -3,7 +3,6 @@ attrs
 dataclasses; python_version == '3.6'
 mitmproxy; python_version >= '3.7'
 mitmproxy >= 4, < 5; python_version >= '3.6' and python_version < '3.7'
-mitmproxy < 4; python_version < '3.6'
 pyftpdlib
 # https://github.com/pytest-dev/pytest-twisted/issues/93
 pytest != 5.4, != 5.4.1
diff --git a/tests/test_crawl.py b/tests/test_crawl.py
index 642c2465156..c1b918baf7f 100644
--- a/tests/test_crawl.py
+++ b/tests/test_crawl.py
@@ -1,6 +1,5 @@
 import json
 import logging
-import sys
 from ipaddress import IPv4Address
 from socket import gethostbyname
 from urllib.parse import urlparse
@@ -405,7 +404,6 @@ def _on_item_scraped(item):
         self.assertIn("Got response 200", str(log))
         self.assertIn({"foo": 42}, items)
 
-    @mark.skipif(sys.version_info < (3, 6), reason="Async generators require Python 3.6 or higher")
     @mark.only_asyncio()
     @defer.inlineCallbacks
     def test_async_def_asyncgen_parse(self):
@@ -417,7 +415,6 @@ def test_async_def_asyncgen_parse(self):
         itemcount = crawler.stats.get_value('item_scraped_count')
         self.assertEqual(itemcount, 1)
 
-    @mark.skipif(sys.version_info < (3, 6), reason="Async generators require Python 3.6 or higher")
     @mark.only_asyncio()
     @defer.inlineCallbacks
     def test_async_def_asyncgen_parse_loop(self):
@@ -437,7 +434,6 @@ def _on_item_scraped(item):
         for i in range(10):
             self.assertIn({'foo': i}, items)
 
-    @mark.skipif(sys.version_info < (3, 6), reason="Async generators require Python 3.6 or higher")
     @mark.only_asyncio()
     @defer.inlineCallbacks
     def test_async_def_asyncgen_parse_complex(self):
diff --git a/tests/test_item.py b/tests/test_item.py
index 66fa761f0d6..78d204e3441 100644
--- a/tests/test_item.py
+++ b/tests/test_item.py
@@ -1,4 +1,3 @@
-import sys
 import unittest
 from unittest import mock
 from warnings import catch_warnings
@@ -7,9 +6,6 @@
 from scrapy.item import ABCMeta, _BaseItem, BaseItem, DictItem, Field, Item, ItemMeta
 
 
-PY36_PLUS = (sys.version_info.major >= 3) and (sys.version_info.minor >= 6)
-
-
 class ItemTest(unittest.TestCase):
 
     def assertSortedEqual(self, first, second, msg=None):
@@ -280,14 +276,6 @@ def test_new_method_propagates_classcell(self):
         with mock.patch.object(base, '__new__', new_mock):
 
             class MyItem(Item):
-                if not PY36_PLUS:
-                    # This attribute is an internal attribute in Python 3.6+
-                    # and must be propagated properly. See
-                    # https://docs.python.org/3.6/reference/datamodel.html#creating-the-class-object
-                    # In <3.6, we add a dummy attribute just to ensure the
-                    # __new__ method propagates it correctly.
-                    __classcell__ = object()
-
                 def f(self):
                     # For rationale of this see:
                     # https://github.com/python/cpython/blob/ee1a81b77444c6715cbe610e951c655b6adab88b/Lib/test/test_super.py#L222
diff --git a/tests/test_proxy_connect.py b/tests/test_proxy_connect.py
index a56e3c39a1f..6f70c4267a4 100644
--- a/tests/test_proxy_connect.py
+++ b/tests/test_proxy_connect.py
@@ -7,7 +7,6 @@
 from urllib.parse import urlsplit, urlunsplit
 from unittest import skipIf
 
-import pytest
 from testfixtures import LogCapture
 from twisted.internet import defer
 from twisted.trial.unittest import TestCase
@@ -58,8 +57,6 @@ def _wrong_credentials(proxy_url):
     return urlunsplit(bad_auth_proxy)
 
 
-@skipIf(sys.version_info < (3, 5, 4),
-        "requires mitmproxy < 3.0.0, which these tests do not support")
 @skipIf("pypy" in sys.executable,
         "mitmproxy does not support PyPy")
 @skipIf(platform.system() == 'Windows' and sys.version_info < (3, 7),
@@ -88,14 +85,6 @@ def test_https_connect_tunnel(self):
             yield crawler.crawl(self.mockserver.url("/status?n=200", is_secure=True))
         self._assert_got_response_code(200, log)
 
-    @pytest.mark.xfail(reason='Python 3.6+ fails this earlier', condition=sys.version_info >= (3, 6))
-    @defer.inlineCallbacks
-    def test_https_connect_tunnel_error(self):
-        crawler = get_crawler(SimpleSpider)
-        with LogCapture() as log:
-            yield crawler.crawl("https://localhost:99999/status?n=200")
-        self._assert_got_tunnel_error(log)
-
     @defer.inlineCallbacks
     def test_https_tunnel_auth_error(self):
         os.environ['https_proxy'] = _wrong_credentials(os.environ['https_proxy'])
diff --git a/tox.ini b/tox.ini
index dec0d75e8e0..12e40295c38 100644
--- a/tox.ini
+++ b/tox.ini
@@ -89,7 +89,7 @@ deps =
 basepython = python3
 deps =
     {[pinned]deps}
-    # First lxml version that includes a Windows wheel for Python 3.5, so we do
+    # First lxml version that includes a Windows wheel for Python 3.6, so we do
     # not need to build lxml from sources in a CI Windows job:
     lxml==3.8.0
 


Code review comments:
diff_hunk: @@ -3,7 +3,6 @@ attrs
 dataclasses; python_version == '3.6'
 mitmproxy; python_version >= '3.7'
 mitmproxy >= 4, < 5; python_version >= '3.6' and python_version < '3.7'
path: tests/requirements-py3.txt
text: @user1:
Just in case, a suggestion:

```suggestion
mitmproxy >= 4.0.4, < 5; python_version >= '3.6' and python_version < '3.7'
```

There is a CVE-2018-14505 on earlier mitmproxy versions.

@user1:
It got reverted back after the force-push :)

@author:
:facepalm: