Repository: scrapy/scrapy

Issue:
Make sure all type annotations relevant to web-poet are resolvable
As noticed recently (https://github.com/scrapinghub/scrapy-poet/issues/221) in Scrapy 2.12.0 `scrapy.spiders.sitemap.SitemapSpider._parse_sitemap()` has a type annotation that cannot be resolved at the run time because `from collections.abc import Iterable` is under `if TYPE_CHECKING:`. It doesn't seem possible to handle this at the web-poet side so we should make sure no Scrapy code causes this problem. For this we should fix all callbacks that are defined in the Scrapy code and can be used directly (so e.g. `scrapy.spiders.Spider.parse()` doesn't matter). I wonder if there are also any classes whose constructors can be used in a plan.

Discussion:
codecov[bot]: ## [Codecov](https://app.codecov.io/gh/scrapy/scrapy/pull/6671?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 33.34%. Comparing base [(`00167ed`)](https://app.codecov.io/gh/scrapy/scrapy/commit/00167edca0677fd9657a8c5995d30600dd37fe37?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) to head [(`54faceb`)](https://app.codecov.io/gh/scrapy/scrapy/commit/54facebb3eaa0763c09ff77e70befbceac20bb92?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy).

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff             @@
##           master    #6671       +/-   ##
===========================================
- Coverage   91.36%   33.34%   -58.02%     
===========================================
  Files         162      162               
  Lines       12047    12048        +1     
  Branches     1551     1551               
===========================================
- Hits        11007     4018     -6989     
- Misses        735     7902     +7167     
+ Partials      305      128      -177     
```

| [Files with missing lines](https://app.codecov.io/gh/scrapy/scrapy/pull/6671?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/spiders/sitemap.py](https://app.codecov.io/gh/scrapy/scrapy/pull/6671?src=pr&el=tree&filepath=scrapy%2Fspiders%2Fsitemap.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3NwaWRlcnMvc2l0ZW1hcC5weQ==) | `32.53% <100.00%> (-52.84%)` | :arrow_down: |

... and [147 files with indirect coverage changes](https://app.codecov.io/gh/scrapy/scrapy/pull/6671/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy)

</details>

Fix (patch):
diff --git a/scrapy/spiders/sitemap.py b/scrapy/spiders/sitemap.py
index 91c7e3be9..39033ac3c 100644
--- a/scrapy/spiders/sitemap.py
+++ b/scrapy/spiders/sitemap.py
@@ -2,6 +2,9 @@ from __future__ import annotations
 
 import logging
 import re
+
+# Iterable is needed at the run time for the SitemapSpider._parse_sitemap() annotation
+from collections.abc import Iterable, Sequence  # noqa: TC003
 from typing import TYPE_CHECKING, Any, cast
 
 from scrapy.http import Request, Response, XmlResponse
@@ -11,8 +14,6 @@ from scrapy.utils.gz import gunzip, gzip_magic_number
 from scrapy.utils.sitemap import Sitemap, sitemap_urls_from_robots
 
 if TYPE_CHECKING:
-    from collections.abc import Iterable, Sequence
-
     # typing.Self requires Python 3.11
     from typing_extensions import Self
 


Test changes:
diff --git a/tests/test_poet.py b/tests/test_poet.py
new file mode 100644
index 000000000..9601c75a1
--- /dev/null
+++ b/tests/test_poet.py
@@ -0,0 +1,20 @@
+"""Tests that make sure parts needed for the scrapy-poet stack work."""
+
+from typing import get_type_hints
+
+from scrapy import Spider
+from scrapy.spiders import CrawlSpider, CSVFeedSpider, SitemapSpider, XMLFeedSpider
+
+
+def test_callbacks():
+    """Making sure annotations on all non-abstract callbacks can be resolved."""
+
+    for cb in [
+        Spider._parse,
+        CrawlSpider._parse,
+        CrawlSpider._callback,
+        XMLFeedSpider._parse,
+        CSVFeedSpider._parse,
+        SitemapSpider._parse_sitemap,
+    ]:
+        get_type_hints(cb)