Repository: scrapy/scrapy

Issue:
Warning: Unable to determine whether or not callable is a generator with a return value [unable to parse decorated class methods]
### Description

If you have a decorated spider method, `scrapy.utils.misc` [throws a warning](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L240-L264) saying that it cannot determine if the callable is a generator with a return value.

`ast.parse()` fails [here](https://github.com/scrapy/scrapy/blob/master/scrapy/utils/misc.py#L228-L230) when called with a decorated method.

### Steps to Reproduce

I just copied the logic from `misc.py` and used it to analyze a class with the same overall code structure:

```python
import re
import ast
import inspect

class Foo:
    @classmethod
    def func(self):
        """
        Description of func
        """  
        return

code = re.sub(r"^[\t ]+", "", inspect.getsource(Foo.func))
tree = ast.parse(code)
```

```
python3 test.py
> IndentationError: unexpected indent
```

The regex replacement isn't accounting for a possible decorator, so the code ends up looking like:

```
@classmethod
    def func(self):
        """
        Description of func
        """  
        return
```

**Expected behavior:** I'd like to be able to use decorated methods without dealing with noisy logs.

**Actual behavior:** My container logs are filled with tons of warning messages. The only workaround is to avoid the usage of decorators.

**Reproduces how often:** 100% of the time to my knowledge

### Versions

```
$ scrapy version --verbose

Scrapy       : 2.5.1
lxml         : 4.6.4.0
libxml2      : 2.9.10
cssselect    : 1.1.0
parsel       : 1.6.0
w3lib        : 1.22.0
Twisted      : 21.7.0
Python       : 3.8.2 (default, Dec 21 2020, 15:06:04) - [Clang 12.0.0 (clang-1200.0.32.29)]
pyOpenSSL    : 21.0.0 (OpenSSL 1.1.1l  24 Aug 2021)
cryptography : 35.0.0
Platform     : macOS-10.15.7-x86_64-i386-64bit
```

### Additional context

This is my first time filing a Scrapy issue. I'm happy to add more context if necessary, and apologies in advance if this has already been discussed elsewhere (fwiw I couldn't find anything).


Discussion:
When this features was introduced in #3869, it used `dedent`, which would have avoided this issue. In #4935, this was changed to a simple regex, because apparently some people write code like:

```python
class Bob:
    def doit(self):
        """
this line is flush left
        """
        if True:
            yield 1234
```

Anyway, to support both decorators and non-indented heredocs, then the regex needs to be made smarter.
```python test.py
import re
import ast
import inspect

class Foo:
    @classmethod
    def func(self):
        """
        Description of func
        """  
        return

    @classmethod
    def test(self):
        """
    Description of test
        """
        return


def parse(func):
    src = inspect.getsource(func)
    pattern = r"^[\t ]+"
    matches = re.findall(pattern, src)  # finds indentation
    code = re.sub(pattern, "", src)
    if matches:
        code = re.sub(f"\n{matches[0]}", "\n", code)  # remove indentation

    tree = ast.parse(code)

parse(Foo.func)
parse(Foo.test)
parse(parse)
```

Finding indentation of callable and removing it works.

Resolving patch:
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -226,7 +226,14 @@ def returns_none(return_node):
         return value is None or isinstance(value, ast.NameConstant) and value.value is None
 
     if inspect.isgeneratorfunction(callable):
-        code = re.sub(r"^[\t ]+", "", inspect.getsource(callable))
+        src = inspect.getsource(callable)
+        pattern = re.compile(r"(^[\t ]+)")
+        code = pattern.sub("", src)
+
+        match = pattern.match(src)  # finds indentation
+        if match:
+            code = re.sub(f"\n{match.group(0)}", "\n", code)  # remove indentation
+
         tree = ast.parse(code)
         for node in walk_callable(tree):
             if isinstance(node, ast.Return) and not returns_none(node):


Test patch:
diff --git a/tests/test_utils_misc/test_return_with_argument_inside_generator.py b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
--- a/tests/test_utils_misc/test_return_with_argument_inside_generator.py
+++ b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
@@ -165,6 +165,89 @@ def l2():
             warn_on_generator_with_return_value(None, l2)
             self.assertEqual(len(w), 0)
 
+    def test_generators_return_none_with_decorator(self):
+        def decorator(func):
+            def inner_func():
+                func()
+            return inner_func
+
+        @decorator
+        def f3():
+            yield 1
+            return None
+
+        @decorator
+        def g3():
+            yield 1
+            return
+
+        @decorator
+        def h3():
+            yield 1
+
+        @decorator
+        def i3():
+            yield 1
+            yield from generator_that_returns_stuff()
+
+        @decorator
+        def j3():
+            yield 1
+
+            def helper():
+                return 0
+
+            yield helper()
+
+        @decorator
+        def k3():
+            """
+docstring
+            """
+            url = """
+https://example.org
+        """
+            yield url
+            return
+
+        @decorator
+        def l3():
+            return
+
+        assert not is_generator_with_return_value(top_level_return_none)
+        assert not is_generator_with_return_value(f3)
+        assert not is_generator_with_return_value(g3)
+        assert not is_generator_with_return_value(h3)
+        assert not is_generator_with_return_value(i3)
+        assert not is_generator_with_return_value(j3)  # not recursive
+        assert not is_generator_with_return_value(k3)  # not recursive
+        assert not is_generator_with_return_value(l3)
+
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, top_level_return_none)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, f3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, g3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, h3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, i3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, j3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, k3)
+            self.assertEqual(len(w), 0)
+        with warnings.catch_warnings(record=True) as w:
+            warn_on_generator_with_return_value(None, l3)
+            self.assertEqual(len(w), 0)
+
     @mock.patch("scrapy.utils.misc.is_generator_with_return_value", new=_indentation_error)
     def test_indentation_error(self):
         with warnings.catch_warnings(record=True) as w: