Repository: scrapy/scrapy

Issue:
Exception when using DummyStatsCollector
### Description

Using the DummyStatsCollector results in an exception:

```
2019-09-09 13:51:23 [scrapy.utils.signal] ERROR: Error caught on signal handler: <bound method CoreStats.spider_closed of <scrapy.extensions.corestats.CoreStats object at 0x7f86269cac18>>
Traceback (most recent call last):
  File ".../lib/python3.6/site-packages/twisted/internet/defer.py", line 150, in maybeDeferred
    result = f(*args, **kw)
  File ".../lib/python3.6/site-packages/pydispatch/robustapply.py", line 55, in robustApply
    return receiver(*arguments, **named)
  File ".../lib/python3.6/site-packages/scrapy/extensions/corestats.py", line 28, in spider_closed
    elapsed_time = finish_time - self.stats.get_value('start_time')
TypeError: unsupported operand type(s) for -: 'datetime.datetime' and 'NoneType'
```

This problem has been introduced in aa46e1995cd5cb1099aba17535372b538bd656b3.

### Steps to Reproduce

Set `STATS_CLASS = "scrapy.statscollectors.DummyStatsCollector"` in the settings module as described in the documentation (https://docs.scrapy.org/en/latest/topics/stats.html#dummystatscollector).

**Expected behavior:** no exception
**Actual behavior:** exception thrown
**Reproduces how often:** always

### Versions

At least master as of 534de7395da3a53b5a2c89960db9ec5d8fdab60c

### Fix

A possible fix is to use the elapsed time as a default argument so that `get_value()` does not return None. I can prepare a PR if needed.

```diff
--- a/scrapy/extensions/corestats.py
+++ b/scrapy/extensions/corestats.py
@@ -25,7 +25,7 @@ class CoreStats(object):
 
     def spider_closed(self, spider, reason):
         finish_time = datetime.datetime.utcnow()
-        elapsed_time = finish_time - self.stats.get_value('start_time')
+        elapsed_time = finish_time - self.stats.get_value('start_time', finish_time)
         elapsed_time_seconds = elapsed_time.total_seconds()
         self.stats.set_value('elapsed_time_seconds', elapsed_time_seconds, spider=spider)
         self.stats.set_value('finish_time', finish_time, spider=spider)
```

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/4052?src=pr&el=h1) Report
> Merging [#4052](https://codecov.io/gh/scrapy/scrapy/pull/4052?src=pr&el=desc) into [master](https://codecov.io/gh/scrapy/scrapy/commit/7632e375c268d31c570904452902799db180c415?src=pr&el=desc) will **decrease** coverage by `2.74%`.
> The diff coverage is `100%`.


```diff
@@            Coverage Diff             @@
##           master    #4052      +/-   ##
==========================================
- Coverage   85.68%   82.94%   -2.75%     
==========================================
  Files         165      165              
  Lines        9732     9734       +2     
  Branches     1463     1463              
==========================================
- Hits         8339     8074     -265     
- Misses       1136     1394     +258     
- Partials      257      266       +9
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/4052?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [scrapy/extensions/corestats.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2V4dGVuc2lvbnMvY29yZXN0YXRzLnB5) | `100% <100%> (ø)` | :arrow_up: |
| [scrapy/linkextractors/sgml.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmtleHRyYWN0b3JzL3NnbWwucHk=) | `0% <0%> (-96.81%)` | :arrow_down: |
| [scrapy/linkextractors/regex.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmtleHRyYWN0b3JzL3JlZ2V4LnB5) | `0% <0%> (-95.66%)` | :arrow_down: |
| [scrapy/linkextractors/htmlparser.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmtleHRyYWN0b3JzL2h0bWxwYXJzZXIucHk=) | `0% <0%> (-92.07%)` | :arrow_down: |
| [scrapy/robotstxt.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L3JvYm90c3R4dC5weQ==) | `57.89% <0%> (-39.48%)` | :arrow_down: |
| [scrapy/extensions/statsmailer.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2V4dGVuc2lvbnMvc3RhdHNtYWlsZXIucHk=) | `0% <0%> (-30.44%)` | :arrow_down: |
| [scrapy/\_monkeypatches.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L19tb25rZXlwYXRjaGVzLnB5) | `54.54% <0%> (-18.19%)` | :arrow_down: |
| [scrapy/link.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L2xpbmsucHk=) | `86.36% <0%> (-13.64%)` | :arrow_down: |
| [scrapy/utils/gz.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L3V0aWxzL2d6LnB5) | `91.89% <0%> (-8.11%)` | :arrow_down: |
| [scrapy/utils/python.py](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree#diff-c2NyYXB5L3V0aWxzL3B5dGhvbi5weQ==) | `78.26% <0%> (-5.98%)` | :arrow_down: |
| ... and [14 more](https://codecov.io/gh/scrapy/scrapy/pull/4052/diff?src=pr&el=tree-more) | |


Patch:
diff --git a/scrapy/extensions/corestats.py b/scrapy/extensions/corestats.py
index 8cc5e18ac..20adfbe4b 100644
--- a/scrapy/extensions/corestats.py
+++ b/scrapy/extensions/corestats.py
@@ -1,14 +1,16 @@
 """
 Extension for collecting core stats like items scraped and start/finish times
 """
-import datetime
+from datetime import datetime
 
 from scrapy import signals
 
+
 class CoreStats(object):
 
     def __init__(self, stats):
         self.stats = stats
+        self.start_time = None
 
     @classmethod
     def from_crawler(cls, crawler):
@@ -21,11 +23,12 @@ class CoreStats(object):
         return o
 
     def spider_opened(self, spider):
-        self.stats.set_value('start_time', datetime.datetime.utcnow(), spider=spider)
+        self.start_time = datetime.utcnow()
+        self.stats.set_value('start_time', self.start_time, spider=spider)
 
     def spider_closed(self, spider, reason):
-        finish_time = datetime.datetime.utcnow()
-        elapsed_time = finish_time - self.stats.get_value('start_time')
+        finish_time = datetime.utcnow()
+        elapsed_time = finish_time - self.start_time
         elapsed_time_seconds = elapsed_time.total_seconds()
         self.stats.set_value('elapsed_time_seconds', elapsed_time_seconds, spider=spider)
         self.stats.set_value('finish_time', finish_time, spider=spider)


Test patch:
diff --git a/tests/test_stats.py b/tests/test_stats.py
index 9f950ebc9..2033dbe07 100644
--- a/tests/test_stats.py
+++ b/tests/test_stats.py
@@ -1,10 +1,59 @@
+from datetime import datetime
 import unittest
 
+try:
+    from unittest import mock
+except ImportError:
+    import mock
+
+from scrapy.extensions.corestats import CoreStats
 from scrapy.spiders import Spider
 from scrapy.statscollectors import StatsCollector, DummyStatsCollector
 from scrapy.utils.test import get_crawler
 
 
+class CoreStatsExtensionTest(unittest.TestCase):
+
+    def setUp(self):
+        self.crawler = get_crawler(Spider)
+        self.spider = self.crawler._create_spider('foo')
+
+    @mock.patch('scrapy.extensions.corestats.datetime')
+    def test_core_stats_default_stats_collector(self, mock_datetime):
+        fixed_datetime = datetime(2019, 12, 1, 11, 38)
+        mock_datetime.utcnow = mock.Mock(return_value=fixed_datetime)
+        self.crawler.stats = StatsCollector(self.crawler)
+        ext = CoreStats.from_crawler(self.crawler)
+        ext.spider_opened(self.spider)
+        ext.item_scraped({}, self.spider)
+        ext.response_received(self.spider)
+        ext.item_dropped({}, self.spider, ZeroDivisionError())
+        ext.spider_closed(self.spider, 'finished')
+        self.assertEqual(
+            ext.stats._stats,
+            {
+                'start_time': fixed_datetime,
+                'finish_time': fixed_datetime,
+                'item_scraped_count': 1,
+                'response_received_count': 1,
+                'item_dropped_count': 1,
+                'item_dropped_reasons_count/ZeroDivisionError': 1,
+                'finish_reason': 'finished',
+                'elapsed_time_seconds': 0.0,
+            }
+        )
+
+    def test_core_stats_dummy_stats_collector(self):
+        self.crawler.stats = DummyStatsCollector(self.crawler)
+        ext = CoreStats.from_crawler(self.crawler)
+        ext.spider_opened(self.spider)
+        ext.item_scraped({}, self.spider)
+        ext.response_received(self.spider)
+        ext.item_dropped({}, self.spider, ZeroDivisionError())
+        ext.spider_closed(self.spider, 'finished')
+        self.assertEqual(ext.stats._stats, {})
+
+
 class StatsCollectorTest(unittest.TestCase):
 
     def setUp(self):