Repository: scrapy/scrapy

Issue:
Add count to spider_exceptions stats
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your pull request, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#writing-patches and https://doc.scrapy.org/en/latest/contributing.html#submitting-patches

-->

## Summary

Today, we only increase the specific exception under `spider_exceptions`, e.g., [`spider_exceptions/AttributeError`](https://github.com/scrapy/scrapy/blob/master/scrapy/core/scraper.py#L250). We should also have a total count like we have for downloader, e.g., [`downloader/exception_count`](https://github.com/scrapy/scrapy/blob/master/scrapy/downloadermiddlewares/stats.py#L80).

## Motivation

This information is useful when creating visualizations of stats.

Discussion:

Solution patch:
diff --git a/scrapy/core/scraper.py b/scrapy/core/scraper.py
index b664b61f6..496adb500 100644
--- a/scrapy/core/scraper.py
+++ b/scrapy/core/scraper.py
@@ -246,6 +246,7 @@ class Scraper:
             spider=spider,
         )
         assert self.crawler.stats
+        self.crawler.stats.inc_value("spider_exceptions/count", spider=spider)
         self.crawler.stats.inc_value(
             f"spider_exceptions/{_failure.value.__class__.__name__}", spider=spider
         )


Test patch:
diff --git a/tests/test_closespider.py b/tests/test_closespider.py
index 476662789..4a17b254b 100644
--- a/tests/test_closespider.py
+++ b/tests/test_closespider.py
@@ -88,6 +88,7 @@ class TestCloseSpider(TestCase):
         assert reason == "closespider_errorcount"
         key = f"spider_exceptions/{crawler.spider.exception_cls.__name__}"
         errorcount = crawler.stats.get_value(key)
+        assert crawler.stats.get_value("spider_exceptions/count") >= close_on
         assert errorcount >= close_on
 
     @defer.inlineCallbacks