Repository: scrapy/scrapy

Issue:
Command parse unhandled error :AttributeError: 'NoneType' object has no attribute 'start_requests'
Scrapy version :1.5.0
When i run the command **scrapy parse http://www.baidu.com**, and the url www.baidu.com dosn't  have spider matched , then i got the error:

> 2018-03-11 16:23:35 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: DouTu)
> 2018-03-11 16:23:35 [scrapy.utils.log] INFO: Versions: lxml 4.2.1.0, libxml2 2.9.8, cssselect 1.0.3, parsel 1.4.0, w3lib 1.19.0, Twisted 17.9.0, Python 2.7.12 (default, Dec  4 2017, 14:50:18) - [GCC 5.4.0 20160609], pyOpenSSL 17.5.0 (OpenSSL 1.1.0h  27 Mar 2018), cryptography 2.2.2, Platform Linux-4.13.0-38-generic-x86_64-with-Ubuntu-16.04-xenial
> 2018-05-18 16:23:35 [scrapy.commands.parse] ERROR: Unable to find spider for: http://www.baidu.com
> Traceback (most recent call last):
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/cmdline.py", line 239, in <module>
>     execute(['scrapy','parse','http://www.baidu.com'])
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/cmdline.py", line 168, in execute
>     _run_print_help(parser, _run_command, cmd, args, opts)
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/cmdline.py", line 98, in _run_print_help
>     func(*a, **kw)
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/cmdline.py", line 176, in _run_command
>     cmd.run(args, opts)
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/commands/parse.py", line 250, in run
>     self.set_spidercls(url, opts)
>   File "/home/wangsir/code/sourceWorkSpace/scrapy/commands/parse.py", line 151, in set_spidercls
>     self.spidercls.start_requests = _start_requests
> AttributeError: 'NoneType' object has no attribute 'start_requests'.

The failed reason should be follwing code(scrapy/commands/parse.py line 151):
         **`self.spidercls.start_requests = _start_requests`**
because the url www.baidu.com dosn't  have spider matched,so self.spidercls is none,so self.spidercls.start_requests throw the error.
Fix command parse unhandled error :AttributeError: 'NoneType' object has no attribute 'start_requests'(#3264)
Reopening @wangrenlei's PR 
Fixes #3264


Discussion:
For those who need a quick-and-dirty workaround, specify the spider `--spider=NAME_OF_MY_SPIDER`
The original PR, #3265, looks correct to me, someone just needs to create a new one, import the commit from that one and publish it.
Hello , I think i can help with this issue if someone could give me a little insight .
hi @PushanAgrawal , as you can see above your comment there is already a PR open for it.
# [Codecov](https://codecov.io/gh/scrapy/scrapy/pull/5376?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) Report
> Merging [#5376](https://codecov.io/gh/scrapy/scrapy/pull/5376?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (837038d) into [master](https://codecov.io/gh/scrapy/scrapy/commit/92764d68e272079b6004c194bf237f0e1c8ee95d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) (92764d6) will **decrease** coverage by `0.46%`.
> The diff coverage is `91.89%`.

> :exclamation: Current head 837038d differs from pull request most recent head e7ccec1. Consider uploading reports for the commit e7ccec1 to get more accurate results


```diff
@@            Coverage Diff             @@
##           master    #5376      +/-   ##
==========================================
- Coverage   88.58%   88.11%   -0.47%     
==========================================
  Files         163      163              
  Lines       10635    10628       -7     
  Branches     1812     1810       -2     
==========================================
- Hits         9421     9365      -56     
- Misses        938      985      +47     
- Partials      276      278       +2     
```

| [Impacted Files](https://codecov.io/gh/scrapy/scrapy/pull/5376?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | Coverage Δ | |
|---|---|---|
| [scrapy/commands/parse.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2NvbW1hbmRzL3BhcnNlLnB5) | `20.11% <0.00%> (-0.12%)` | :arrow_down: |
| [scrapy/http/response/text.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2h0dHAvcmVzcG9uc2UvdGV4dC5weQ==) | `100.00% <ø> (ø)` | |
| [scrapy/downloadermiddlewares/stats.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2Rvd25sb2FkZXJtaWRkbGV3YXJlcy9zdGF0cy5weQ==) | `91.66% <91.66%> (-0.93%)` | :arrow_down: |
| [scrapy/exporters.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2V4cG9ydGVycy5weQ==) | `100.00% <100.00%> (ø)` | |
| [scrapy/extensions/feedexport.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2V4dGVuc2lvbnMvZmVlZGV4cG9ydC5weQ==) | `91.23% <100.00%> (-3.85%)` | :arrow_down: |
| [scrapy/item.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L2l0ZW0ucHk=) | `98.36% <100.00%> (-0.39%)` | :arrow_down: |
| [scrapy/settings/\_\_init\_\_.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3NldHRpbmdzL19faW5pdF9fLnB5) | `93.06% <100.00%> (+0.08%)` | :arrow_up: |
| [scrapy/spiders/feed.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3NwaWRlcnMvZmVlZC5weQ==) | `73.84% <100.00%> (+7.69%)` | :arrow_up: |
| [scrapy/utils/misc.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL21pc2MucHk=) | `97.72% <100.00%> (ø)` | |
| [scrapy/utils/response.py](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy#diff-c2NyYXB5L3V0aWxzL3Jlc3BvbnNlLnB5) | `90.19% <100.00%> (+0.40%)` | :arrow_up: |
| ... and [13 more](https://codecov.io/gh/scrapy/scrapy/pull/5376/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=scrapy) | |

Hey @wRAR, I am new to this, in fact this is my first PR. Is there something else I need to do here? Thanks.
Ideally, the pull requests should pass all tests, and if not you should find out why and address any issue.

It is possible that some of the broken tests are unrelated to your changes though. Let me close and reopen the pull request to re-trigger the tests, and see if they pass.
The tests indeed fail for different reasons but this code is not covered by tests. I think it shouldn't be hard to make a test that runs the failing command (even though all existing tests in test_command_parse.py pass --spider explicitly).

Resolving patch:
diff --git a/scrapy/commands/parse.py b/scrapy/commands/parse.py
--- a/scrapy/commands/parse.py
+++ b/scrapy/commands/parse.py
@@ -146,7 +146,8 @@ def set_spidercls(self, url, opts):
 
         def _start_requests(spider):
             yield self.prepare_request(spider, Request(url), opts)
-        self.spidercls.start_requests = _start_requests
+        if self.spidercls:
+            self.spidercls.start_requests = _start_requests
 
     def start_parsing(self, url, opts):
         self.crawler_process.crawl(self.spidercls, **opts.spargs)


Test patch:
diff --git a/tests/test_command_parse.py b/tests/test_command_parse.py
--- a/tests/test_command_parse.py
+++ b/tests/test_command_parse.py
@@ -1,6 +1,7 @@
 import os
 import argparse
 from os.path import join, abspath, isfile, exists
+
 from twisted.internet import defer
 from scrapy.commands import parse
 from scrapy.settings import Settings
@@ -222,6 +223,11 @@ def test_crawlspider_no_matching_rule(self):
         self.assertRegex(_textmode(out), r"""# Scraped Items  -+\n\[\]""")
         self.assertIn("""Cannot find a rule that matches""", _textmode(stderr))
 
+    @defer.inlineCallbacks
+    def test_crawlspider_not_exists_with_not_matched_url(self):
+        status, out, stderr = yield self.execute([self.url('/invalid_url')])
+        self.assertEqual(status, 0)
+
     @defer.inlineCallbacks
     def test_output_flag(self):
         """Checks if a file was created successfully having