Repository: scrapy/scrapy

Issue:
Smarter generator check for combined yield/return statements: ignore nested functions
## Summary
Currently, if spider methods are generators that yield results and contain nested function, then the following warning is issued:

```
[py.warnings] WARNING: /Library/Python/3.7/site-packages/scrapy/core/scraper.py:148: UserWarning: The "MySpider.parse" method is a generator and includes a "return" statement with a value different than None. This could lead to unexpected behaviour. Please see https://docs.python.org/3/reference/simple_stmts.html#the-return-statement for details about the semantics of the "return" statement within generators
  warn_on_generator_with_return_value(spider, callback)
```

The example of a simple spider that results in the warning:
```
import scrapy

class MySpider(scrapy.Spider):
    name = "MySpider"
    start_urls = ["https://scrapy.org"]
    
    def parse(self, response):
        
        def is_external(url):
            href = url.css('::attr(href)').get()
            return href.startswith('http') and 'scrapy.org' not in href
        
        links = [link for link in response.css('a') if is_external(link)]
        for link in links:
            yield {'link': link.css('::attr(href)').get(), 'text': link.css('::text').get()}
```

I know it's a bit artificial example as the nested function can be moved, but there is nothing wrong with nested function conceptually.

## Motivation

I have a midsize spider function that includes some nested helper functions that I'd like to keep close to where they are called.

## Describe alternatives you've considered

Moving nested function out of the generator is an easy fix, but it constrains expressivity of the code.

## Additional context

Related function: is_generator_with_return_value



Discussion:

Resolving patch:
diff --git a/scrapy/utils/misc.py b/scrapy/utils/misc.py
--- a/scrapy/utils/misc.py
+++ b/scrapy/utils/misc.py
@@ -5,6 +5,7 @@
 import re
 import hashlib
 import warnings
+from collections import deque
 from contextlib import contextmanager
 from importlib import import_module
 from pkgutil import iter_modules
@@ -184,6 +185,22 @@ def set_environ(**kwargs):
                 os.environ[k] = v
 
 
+def walk_callable(node):
+    """Similar to ``ast.walk``, but walks only function body and skips nested
+    functions defined within the node.
+    """
+    todo = deque([node])
+    walked_func_def = False
+    while todo:
+        node = todo.popleft()
+        if isinstance(node, ast.FunctionDef):
+            if walked_func_def:
+                continue
+            walked_func_def = True
+        todo.extend(ast.iter_child_nodes(node))
+        yield node
+
+
 _generator_callbacks_cache = LocalWeakReferencedCache(limit=128)
 
 
@@ -201,7 +218,7 @@ def returns_none(return_node):
 
     if inspect.isgeneratorfunction(callable):
         tree = ast.parse(dedent(inspect.getsource(callable)))
-        for node in ast.walk(tree):
+        for node in walk_callable(tree):
             if isinstance(node, ast.Return) and not returns_none(node):
                 _generator_callbacks_cache[callable] = True
                 return _generator_callbacks_cache[callable]


Test patch:
diff --git a/tests/test_utils_misc/test_return_with_argument_inside_generator.py b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
--- a/tests/test_utils_misc/test_return_with_argument_inside_generator.py
+++ b/tests/test_utils_misc/test_return_with_argument_inside_generator.py
@@ -29,9 +29,28 @@ def k():
             yield 1
             yield from g()
 
+        def m():
+            yield 1
+
+            def helper():
+                return 0
+
+            yield helper()
+
+        def n():
+            yield 1
+
+            def helper():
+                return 0
+
+            yield helper()
+            return 2
+
         assert is_generator_with_return_value(f)
         assert is_generator_with_return_value(g)
         assert not is_generator_with_return_value(h)
         assert not is_generator_with_return_value(i)
         assert not is_generator_with_return_value(j)
         assert not is_generator_with_return_value(k)  # not recursive
+        assert not is_generator_with_return_value(m)
+        assert is_generator_with_return_value(n)