Repository: scrapy/scrapy

Issue:
Scrapy does not use a non-zero exit code when pipeline's open_spider throws the exception
<!--

Thanks for taking an interest in Scrapy!

If you have a question that starts with "How to...", please see the Scrapy Community page: https://scrapy.org/community/.
The GitHub issue tracker's purpose is to deal with bug reports and feature requests for the project itself.

Keep in mind that by filing an issue, you are expected to comply with Scrapy's Code of Conduct, including treating everyone with respect: https://github.com/scrapy/scrapy/blob/master/CODE_OF_CONDUCT.md

The following is a suggested template to structure your issue, you can find more guidelines at https://doc.scrapy.org/en/latest/contributing.html#reporting-bugs

-->

### Description
In our case, we execute command `scrapy crawl` in airflow task and the exit code would be used to judge this task success or failure. I agree that `scrapy crawl` ignores spider exceptions because it's unpredictable in the crawling process. 

Back to our case, we export data to file or database in the pipeline and we create the directory or database connection in `open_spider(self, spider)`. I think if there is an exception happens during this function, it's reasonable to propagate a non-zero exit code. it because we normally do some initialization in this function.

### Steps to Reproduce

- scrapy startproject test_spider
- cd test_spider
- scrapy genspider example example.com
- modify spiders/example.py to  
```
# -*- coding: utf-8 -*-
import scrapy


class ExampleSpider(scrapy.Spider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com/']

    custom_settings = {
        'ITEM_PIPELINES': {
            'test_spider.pipelines.TestSpiderPipeline': 300
        }
    }

    def parse(self, response):
        pass
```
- modify pipelines.py to 
```
# -*- coding: utf-8 -*-

# Define your item pipelines here
#
# Don't forget to add your pipeline to the ITEM_PIPELINES setting
# See: https://docs.scrapy.org/en/latest/topics/item-pipeline.html


class TestSpiderPipeline(object):

    def open_spider(self, spider):
        raise Exception('error')

    def process_item(self, item, spider):
        return item
```
- scrapy crawl example
- echo $?  

**Expected behavior:** [What you expect to happen]
return non-zero exit code

**Actual behavior:** [What actually happens]
return zero exit code

**Reproduces how often:** [What percentage of the time does it reproduce?]
100%

### Versions
Scrapy       : 1.8.0
lxml         : 4.3.3.0
libxml2      : 2.9.9
cssselect    : 1.0.3
parsel       : 1.5.1
w3lib        : 1.20.0
Twisted      : 19.2.0
Python       : 3.7.3 (default, Mar 27 2019, 09:23:39) - [Clang 10.0.0 (clang-1000.11.45.5)]
pyOpenSSL    : 19.0.0 (OpenSSL 1.1.1b  26 Feb 2019)
cryptography : 2.6.1
Platform     : Darwin-18.5.0-x86_64-i386-64bit

### Additional context

I could get the expected behavior if I change `def run(self, args, opts)` in scrapy/commands/crawl.py  to 
```
    def run(self, args, opts):
        if len(args) < 1:
            raise UsageError()
        elif len(args) > 1:
            raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
        spname = args[0]

        res = self.crawler_process.crawl(spname, **opts.spargs)

        if hasattr(res, 'result') and res.result is not None and issubclass(res.result.type, Exception):
            self.exitcode = 1
        else:
            self.crawler_process.start()

            if self.crawler_process.bootstrap_failed:
                self.exitcode = 1
```
original `def run(self, args, opts)`
```
    def run(self, args, opts):
        if len(args) < 1:
            raise UsageError()
        elif len(args) > 1:
            raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
        spname = args[0]

        self.crawler_process.crawl(spname, **opts.spargs)
        self.crawler_process.start()

        if self.crawler_process.bootstrap_failed:
            self.exitcode = 1
```

Is it the proper way to modify the code for achieving this purpose? if it is, could I create a PR request for this issue?



Discussion:
Would a **try..except** in your `open_spider` method that uses `sys.exit` in case of an exception achieve this purpose?
@Gallaecio `sys.exit` doesn't work. I've tried before. thanks for your reply
Oh, right, this is Twisted. See if [this](https://stackoverflow.com/q/5344123/939364) helps.
> Oh, right, this is Twisted. See if [this](https://stackoverflow.com/q/5344123/939364) helps.

this doesn't work either. thanks for your suggestion

Resolving patch:
diff --git a/scrapy/commands/crawl.py b/scrapy/commands/crawl.py
--- a/scrapy/commands/crawl.py
+++ b/scrapy/commands/crawl.py
@@ -54,8 +54,13 @@ def run(self, args, opts):
             raise UsageError("running 'scrapy crawl' with more than one spider is no longer supported")
         spname = args[0]
 
-        self.crawler_process.crawl(spname, **opts.spargs)
-        self.crawler_process.start()
+        crawl_defer = self.crawler_process.crawl(spname, **opts.spargs)
 
-        if self.crawler_process.bootstrap_failed:
+        if getattr(crawl_defer, 'result', None) is not None and issubclass(crawl_defer.result.type, Exception):
             self.exitcode = 1
+        else:
+            self.crawler_process.start()
+
+            if self.crawler_process.bootstrap_failed or \
+                    (hasattr(self.crawler_process, 'has_exception') and self.crawler_process.has_exception):
+                self.exitcode = 1


Test patch:
diff --git a/tests/test_cmdline_crawl_with_pipeline/__init__.py b/tests/test_cmdline_crawl_with_pipeline/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/__init__.py
@@ -0,0 +1,20 @@
+import os
+import sys
+import unittest
+from subprocess import Popen, PIPE
+
+
+class CmdlineCrawlPipelineTest(unittest.TestCase):
+
+    def _execute(self, spname):
+        args = (sys.executable, '-m', 'scrapy.cmdline', 'crawl', spname)
+        cwd = os.path.dirname(os.path.abspath(__file__))
+        proc = Popen(args, stdout=PIPE, stderr=PIPE, cwd=cwd)
+        proc.communicate()
+        return proc.returncode
+
+    def test_open_spider_normally_in_pipeline(self):
+        self.assertEqual(self._execute('normal'), 0)
+
+    def test_exception_at_open_spider_in_pipeline(self):
+        self.assertEqual(self._execute('exception'), 1)
diff --git a/tests/test_cmdline_crawl_with_pipeline/scrapy.cfg b/tests/test_cmdline_crawl_with_pipeline/scrapy.cfg
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/scrapy.cfg
@@ -0,0 +1,2 @@
+[settings]
+default = test_spider.settings
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/__init__.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/__init__.py
new file mode 100644
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/pipelines.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/pipelines.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/test_spider/pipelines.py
@@ -0,0 +1,16 @@
+class TestSpiderPipeline(object):
+
+    def open_spider(self, spider):
+        pass
+
+    def process_item(self, item, spider):
+        return item
+
+
+class TestSpiderExceptionPipeline(object):
+
+    def open_spider(self, spider):
+        raise Exception('exception')
+
+    def process_item(self, item, spider):
+        return item
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/settings.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/settings.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/test_spider/settings.py
@@ -0,0 +1,2 @@
+BOT_NAME = 'test_spider'
+SPIDER_MODULES = ['test_spider.spiders']
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/__init__.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/__init__.py
new file mode 100644
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/exception.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/exception.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/exception.py
@@ -0,0 +1,14 @@
+import scrapy
+
+
+class ExceptionSpider(scrapy.Spider):
+    name = 'exception'
+
+    custom_settings = {
+        'ITEM_PIPELINES': {
+            'test_spider.pipelines.TestSpiderExceptionPipeline': 300
+        }
+    }
+
+    def parse(self, response):
+        pass
diff --git a/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/normal.py b/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/normal.py
new file mode 100644
--- /dev/null
+++ b/tests/test_cmdline_crawl_with_pipeline/test_spider/spiders/normal.py
@@ -0,0 +1,14 @@
+import scrapy
+
+
+class NormalSpider(scrapy.Spider):
+    name = 'normal'
+
+    custom_settings = {
+        'ITEM_PIPELINES': {
+            'test_spider.pipelines.TestSpiderPipeline': 300
+        }
+    }
+
+    def parse(self, response):
+        pass