Repository: scrapy/scrapy

Issue:
is_generator_with_return_value raises IndentationError with a flush left doc string
### Description

Code that is accepted by the python interpreter raises when fed through `textwrap.dedent`

### Steps to Reproduce

1. Create `is_generator_bug.py` with the content below (which I simplified from [the `is_generator_with_return_value` method body](https://github.com/scrapy/scrapy/blob/2.0.1/scrapy/utils/misc.py#L186-L187)
2. Run `python is_generator_bug.py`
3. Observe the kaboom

```python
import ast
import inspect
from textwrap import dedent
class Bob:
    def doit(self):
        """
this line is flush left
        """
        if True:
            yield 1234

if __name__ == '__main__':
    b = Bob()
    c = b.doit
    if inspect.isgeneratorfunction(c):
        tree = ast.parse(dedent(inspect.getsource(c)))
```

**Expected behavior:** [What you expect to happen]

No Error

**Actual behavior:** [What actually happens]

```console
$ python3.7 is_generator_bug.py
Traceback (most recent call last):
  File "is_generator_bug.py", line 16, in <module>
    tree = ast.parse(dedent(inspect.getsource(c)))
  File "/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    def doit(self):
    ^
IndentationError: unexpected indent
```

**Reproduces how often:** [What percentage of the time does it reproduce?]

100%

### Versions

```
Scrapy       : 2.0.1
lxml         : 4.5.0.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 20.3.0
Python       : 3.7.7 (default, Mar 11 2020, 23:30:22) - [Clang 10.0.0 (clang-1000.11.45.5)]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019)
cryptography : 2.8
Platform     : Darwin-17.7.0-x86_64-i386-64bit
```

### Additional context



Discussion:
I can reproduce:

```python
import scrapy


class GeneratorSpider(scrapy.Spider):
    name = "generator"
    start_urls = ["https://example.org"]

    def parse(self, response):
        """
docstring
        """
        yield {"url": response.url}
```

```
2020-04-08 10:28:07 [scrapy.core.scraper] ERROR: Spider error processing <GET https://example.org> (referer: None)
Traceback (most recent call last):
  File "/.../scrapy/venv-scrapy/lib/python3.6/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/.../scrapy/scrapy/core/downloader/middleware.py", line 42, in process_request
    defer.returnValue((yield download_func(request=request, spider=spider)))
  File "/.../scrapy/venv-scrapy/lib/python3.6/site-packages/twisted/internet/defer.py", line 1362, in returnValue
    raise _DefGen_Return(val)
twisted.internet.defer._DefGen_Return: <200 https://example.org>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/.../scrapy/scrapy/utils/defer.py", line 55, in mustbe_deferred
    result = f(*args, **kw)
  File "/.../scrapy/scrapy/core/spidermw.py", line 60, in process_spider_input
    return scrape_func(response, request, spider)
  File "/.../scrapy/scrapy/core/scraper.py", line 148, in call_spider
    warn_on_generator_with_return_value(spider, callback)
  File "/.../scrapy/scrapy/utils/misc.py", line 202, in warn_on_generator_with_return_value
    if is_generator_with_return_value(callable):
  File "/.../scrapy/scrapy/utils/misc.py", line 187, in is_generator_with_return_value
    tree = ast.parse(dedent(inspect.getsource(callable)))
  File "/usr/lib/python3.6/ast.py", line 35, in parse
    return compile(source, filename, mode, PyCF_ONLY_AST)
  File "<unknown>", line 1
    def parse(self, response):
    ^
IndentationError: unexpected indent
```

```
Scrapy       : 2.0.1
lxml         : 4.5.0.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.5.2
w3lib        : 1.21.0
Twisted      : 19.10.0
Python       : 3.6.9 (default, Nov  7 2019, 10:44:02) - [GCC 8.3.0]
pyOpenSSL    : 19.1.0 (OpenSSL 1.1.1d  10 Sep 2019)
cryptography : 2.8
Platform     : Linux-4.15.0-91-generic-x86_64-with-LinuxMint-19.1-tessa
```

Seems to me like the following should work fine for docstrings:
```diff
diff --git scrapy/utils/misc.py scrapy/utils/misc.py
index 52cfba20..fa3d31cc 100644
--- scrapy/utils/misc.py
+++ scrapy/utils/misc.py
@@ -184,7 +184,10 @@ def is_generator_with_return_value(callable):
         return value is None or isinstance(value, ast.NameConstant) and value.value is None
 
     if inspect.isgeneratorfunction(callable):
-        tree = ast.parse(dedent(inspect.getsource(callable)))
+        code = inspect.getsource(callable)
+        if callable.__doc__:
+            code = code.replace(callable.__doc__, "")
+        tree = ast.parse(dedent(code))
         for node in ast.walk(tree):
             if isinstance(node, ast.Return) and not returns_none(node):
                 _generator_callbacks_cache[callable] = True
```

Unfortunately, I found that multiline strings are also a problem. This snippet also raises the exception:
```python
import scrapy


class GeneratorSpider(scrapy.Spider):
    name = "generator"
    start_urls = ["https://example.org"]

    def parse(self, response):
        url = """
docstring
        """
        yield {"url": url}
```

The following works, but I don't really like it:
```diff
diff --git scrapy/utils/misc.py scrapy/utils/misc.py
index 52cfba20..881e6c07 100644
--- scrapy/utils/misc.py
+++ scrapy/utils/misc.py
@@ -184,7 +184,8 @@ def is_generator_with_return_value(callable):
         return value is None or isinstance(value, ast.NameConstant) and value.value is None
 
     if inspect.isgeneratorfunction(callable):
-        tree = ast.parse(dedent(inspect.getsource(callable)))
+        code = "class _:\n" + dedent(inspect.getsource(callable))
+        tree = ast.parse(code)
         for node in ast.walk(tree):
             if isinstance(node, ast.Return) and not returns_none(node):
                 _generator_callbacks_cache[callable] = True
```
Thoughts?

It appears based on some lightweight testing that just removing all the leading indentation from the `def` cures it:

```python
    if inspect.isgeneratorfunction(callable):
        code = inspect.getsource(callable)
        code = re.sub(r"^\s+", "", code)
        tree = ast.parse(code)
```
since it turns this:
```python
    def parse(self, response):
        url = """
docstring
        """
        yield {"url": url}
```
into
```python
def parse(self, response):
        url = """
docstring
        """
        yield {"url": url}
```

@elacuesta I think your workaround is quite smart. Although I wonder if maybe it could fail in some corner cases, such as callbacks that are not actually spider methods but external functions. Still, if it solves more problems than it causes, it may be worth it.

Resolving patch:
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -9,7 +9,6 @@
 from contextlib import contextmanager
 from importlib import import_module
 from pkgutil import iter_modules
-from textwrap import dedent
 
 from w3lib.html import replace_entities
 
@@ -227,7 +226,8 @@ def returns_none(return_node):
         return value is None or isinstance(value, ast.NameConstant) and value.value is None
 
     if inspect.isgeneratorfunction(callable):
-        tree = ast.parse(dedent(inspect.getsource(callable)))
+        code = re.sub(r"^[\t ]+", "", inspect.getsource(callable))
+        tree = ast.parse(code)
         for node in walk_callable(tree):
             if isinstance(node, ast.Return) and not returns_none(node):
                 _generator_callbacks_cache[callable] = True
@@ -242,12 +242,23 @@ def warn_on_generator_with_return_value(spider, callable):
     Logs a warning if a callable is a generator function and includes
     a 'return' statement with a value different than None
     """
-    if is_generator_with_return_value(callable):
+    try:
+        if is_generator_with_return_value(callable):
+            warnings.warn(
+                f'The "{spider.__class__.__name__}.{callable.__name__}" method is '
+                'a generator and includes a "return" statement with a value '
+                'different than None. This could lead to unexpected behaviour. Please see '
+                'https://docs.python.org/3/reference/simple_stmts.html#the-return-statement '
+                'for details about the semantics of the "return" statement within generators',
+                stacklevel=2,
+            )
+    except IndentationError:
+        callable_name = spider.__class__.__name__ + "." + callable.__name__
         warnings.warn(
-            f'The "{spider.__class__.__name__}.{callable.__name__}" method is '
-            'a generator and includes a "return" statement with a value '
-            'different than None. This could lead to unexpected behaviour. Please see '
-            'https://docs.python.org/3/reference/simple_stmts.html#the-return-statement '
-            'for details about the semantics of the "return" statement within generators',
+            f'Unable to determine whether or not "{callable_name}" is a generator with a return value. '
+            'This will not prevent your code from working, but it prevents Scrapy from detecting '
+            f'potential issues in your implementation of "{callable_name}". Please, report this in the '
+            'Scrapy issue tracker (https://github.com/scrapy/scrapy/issues), '
+            f'including the code of "{callable_name}"',
             stacklevel=2,
         )


Test patch:
diff --git a/tests/test_utils_misc/test_return_with_argument_inside_generator.py b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
--- a/tests/test_utils_misc/test_return_with_argument_inside_generator.py
+++ b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
@@ -1,35 +1,116 @@
 import unittest
+import warnings
+from unittest import mock
 
-from scrapy.utils.misc import is_generator_with_return_value
+from scrapy.utils.misc import is_generator_with_return_value, warn_on_generator_with_return_value
+
+
+def _indentation_error(*args, **kwargs):
+    raise IndentationError()
+
+
+def top_level_return_something():
+    """
+docstring
+    """
+    url = """
+https://example.org
+"""
+    yield url
+    return 1
+
+
+def top_level_return_none():
+    """
+docstring
+    """
+    url = """
+https://example.org
+"""
+    yield url
+    return
+
+
+def generator_that_returns_stuff():
+    yield 1
+    yield 2
+    return 3
 
 
 class UtilsMiscPy3TestCase(unittest.TestCase):
 
-    def test_generators_with_return_statements(self):
-        def f():
+    def test_generators_return_something(self):
+        def f1():
             yield 1
             return 2
 
-        def g():
+        def g1():
+            yield 1
+            return "asdf"
+
+        def h1():
             yield 1
-            return 'asdf'
 
-        def h():
+            def helper():
+                return 0
+
+            yield helper()
+            return 2
+
+        def i1():
+            """
+docstring
+            """
+            url = """
+https://example.org
+        """
+            yield url
+            return 1
+
+        assert is_generator_with_return_value(top_level_return_something)
+        assert is_generator_with_return_value(f1)
+        assert is_generator_with_return_value(g1)
+        assert is_generator_with_return_value(h1)
+        assert is_generator_with_return_value(i1)
+
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, top_level_return_something)
+            self.assertEqual(len(w), 1)
+            self.assertIn('The "NoneType.top_level_return_something" method is a generator', str(w[0].message))
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, f1)
+            self.assertEqual(len(w), 1)
+            self.assertIn('The "NoneType.f1" method is a generator', str(w[0].message))
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, g1)
+            self.assertEqual(len(w), 1)
+            self.assertIn('The "NoneType.g1" method is a generator', str(w[0].message))
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, h1)
+            self.assertEqual(len(w), 1)
+            self.assertIn('The "NoneType.h1" method is a generator', str(w[0].message))
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, i1)
+            self.assertEqual(len(w), 1)
+            self.assertIn('The "NoneType.i1" method is a generator', str(w[0].message))
+
+    def test_generators_return_none(self):
+        def f2():
             yield 1
             return None
 
-        def i():
+        def g2():
             yield 1
             return
 
-        def j():
+        def h2():
             yield 1
 
-        def k():
+        def i2():
             yield 1
-            yield from g()
+            yield from generator_that_returns_stuff()
 
-        def m():
+        def j2():
             yield 1
 
             def helper():
@@ -37,20 +118,56 @@ def helper():
 
             yield helper()
 
-        def n():
-            yield 1
-
-            def helper():
-                return 0
+        def k2():
+            """
+docstring
+            """
+            url = """
+https://example.org
+        """
+            yield url
+            return
 
-            yield helper()
-            return 2
+        def l2():
+            return
 
-        assert is_generator_with_return_value(f)
-        assert is_generator_with_return_value(g)
-        assert not is_generator_with_return_value(h)
-        assert not is_generator_with_return_value(i)
-        assert not is_generator_with_return_value(j)
-        assert not is_generator_with_return_value(k)  # not recursive
-        assert not is_generator_with_return_value(m)
-        assert is_generator_with_return_value(n)
+        assert not is_generator_with_return_value(top_level_return_none)
+        assert not is_generator_with_return_value(f2)
+        assert not is_generator_with_return_value(g2)
+        assert not is_generator_with_return_value(h2)
+        assert not is_generator_with_return_value(i2)
+        assert not is_generator_with_return_value(j2)  # not recursive
+        assert not is_generator_with_return_value(k2)  # not recursive
+        assert not is_generator_with_return_value(l2)
+
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, top_level_return_none)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, f2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, g2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, h2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, i2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, j2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, k2)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, l2)
+            self.assertEqual(len(w), 0)
+
+    @mock.patch("scrapy.utils.misc.is_generator_with_return_value", new=_indentation_error)
+    def test_indentation_error(self):
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, top_level_return_none)
+            self.assertEqual(len(w), 1)
+            self.assertIn('Unable to determine', str(w[0].message))