Issue 1:
Description

Using spider.crawler.engine.close_spide() in pipeline will throw an exception

Steps to Reproduce

class IPVerification:
@classmethod
def from_settings(cls, settings):
return cls(settings.get('REDIS_URL'))

def __init__(self, redis_url):
connect_pool = ConnectionPool.from_url(redis_url)
self.redis = Redis(connection_pool=connect_pool, max_connections=128)
self.count = 0

def process_item(self, item, spider: Spider):
ipv4 = item.get('ipv4')
port = item.get('port')
if ipv4 is None or port is None:
return
fingerprint = md5(ipv4.encode() + port.encode()).hexdigest()
if self.redis.sadd('Item', fingerprint):
return item
if self.count > 10:
spider.crawler.engine.close_spider(spider, 'Multiple data repetitions')
else:
self.count += 1
raise DropItem('Data duplication')

Reproduces how often: 100%

Versions

Scrapy : 2.6.1
lxml : 4.6.4.0
libxml2 : 2.9.5
cssselect : 1.1.0
parsel : 1.6.0
w3lib : 1.22.0
Twisted : 22.2.0
Python : 3.10.0 (tags/v3.10.0:b494f59, Oct 4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)]
pyOpenSSL : 22.0.0 (OpenSSL 1.1.1m 14 Dec 2021)
cryptography : 36.0.1
Platform : Windows-10-10.0.22000-SP0

Error

'start_time': datetime.datetime(2022, 4, 6, 1, 52, 16, 182409)}
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Spider closed (Multiple data repetitions)
2022-04-06 09:52:17 [scrapy.core.engine] INFO: Error while scheduling new request
Traceback (most recent call last):
File "D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\task.py", line 526, in _oneWorkUnit
result = next(self._iterator)
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "D:\Program Files\Python\Python310\lib\site-packages\twisted\internet\defer.py", line 857, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "D:\Program Files\Python\Python310\lib\site-packages\scrapy\core\engine.py", line 187, in <lambda>
d.addBoth(lambda _: self.slot.nextcall.schedule())
AttributeError: 'NoneType' object has no attribute 'nextcall'

Process finished with exit code 0

Issue 2:
Sometimes pages depend on certain HTTP request headers sent, for rendering the expected result, and it's a manual and tedious job to find out which headers those are.

So, here's an idea for automating this probing mechanism: create a new scrapy command probe which takes a url as argument and a text to look for.

Scrapy then tries several combinations of HTTP headers (user-agent, accept, etc) and return a set that works (where works mean that the text passed is found).

Here's a real world example to illustrate:

http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm

The page should contain a string 'var sFeatures', but that string is not returned with the default Scrapy HTTP request headers. So we run scrapy probe on it:

$ scrapy probe http://www.storage-cabinets-online.com/IVG2/N/ProductID-118021.htm 'var sFeatures'
Found set of working headers:
{'Host': 'www.storage-cabinets-online.com', 'User-Agent': 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.10) Gecko/20100915 Ubuntu/10.04 (lucid)
Firefox/3.6.10', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Accept-Encoding': 'gzip,deflate',
'Accept-Charset': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'Keep-Alive': '115'}

The Scrapy probe command would try a different list of well known user-agents, along with Accept headers.