Question: I have created a new scrapy projects with a spider (multiple to be added). The spider works without any issues if started with scrapy crawl myspider However, when I try to run the scraper from a custom script, it does not start. I have broken down the script to a bare minimum that does not work: from scrapy.spiderloader import SpiderLoader from scrapy.crawler import CrawlerRunner from scrapy.utils.project import get_project_settings from scrapy.utils.log import configure_logging from twisted.internet import reactor settings = get_project_settings() configure_logging(settings) runner = CrawlerRunner(settings) spider_loader = SpiderLoader.from_settings(settings) for spider in spider_loader.list(): print(f"Adding Spider: {spider}") runner.crawl(spider_loader.load(spider)) d = runner.join() d.addBoth(lambda _: reactor.stop()) reactor.run()  The output of the script is: $ python3 minimal.py Adding Spider: myspider 2024-12-06 14:52:02 [scrapy.addons] INFO: Enabled addons: []  The script than hangs, and no additional messages from the spider are printed. I confirmed that no network traffic related to crawling is observed. The code is very close to the documentation, so I am little bit clueless on what the problem could be and where to look. Edit: This is a minimal crawler that does not run: import scrapy class GoogleSpider(scrapy.Spider): name = "google" allowed_domains = ["google.com"] start_urls = ["https://www.google.com"] def parse(self, response): pass  # Scrapy settings for fwscraper project # # For simplicity, this file contains only settings considered important or # commonly used. You can find more settings consulting the documentation: # # https://docs.scrapy.org/en/latest/topics/settings.html # https://docs.scrapy.org/en/latest/topics/downloader-middleware.html # https://docs.scrapy.org/en/latest/topics/spider-middleware.html BOT_NAME = "fwscraper" SPIDER_MODULES = ["fwscraper.spiders"] NEWSPIDER_MODULE = "fwscraper.spiders" # Crawl responsibly by identifying yourself (and your website) on the user-agent #USER_AGENT = "fwscraper (+http://www.yourdomain.com)" # Obey robots.txt rules ROBOTSTXT_OBEY = True # Configure maximum concurrent requests performed by Scrapy (default: 16) #CONCURRENT_REQUESTS = 32 # Configure a delay for requests for the same website (default: 0) # See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay # See also autothrottle settings and docs #DOWNLOAD_DELAY = 3 # The download delay setting will honor only one of: #CONCURRENT_REQUESTS_PER_DOMAIN = 16 #CONCURRENT_REQUESTS_PER_IP = 16 # Disable cookies (enabled by default) #COOKIES_ENABLED = False # Disable Telnet Console (enabled by default) #TELNETCONSOLE_ENABLED = False # Override the default request headers: #DEFAULT_REQUEST_HEADERS = { # "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", # "Accept-Language": "en", #} # Enable or disable spider middlewares # See https://docs.scrapy.org/en/latest/topics/spider-middleware.html #SPIDER_MIDDLEWARES = { # "fwscraper.middlewares.FwscraperSpiderMiddleware": 543, #} # Enable or disable downloader middlewares # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html #DOWNLOADER_MIDDLEWARES = { # "fwscraper.middlewares.FwscraperDownloaderMiddleware": 543, #} # Enable or disable extensions # See https://docs.scrapy.org/en/latest/topics/extensions.html #EXTENSIONS = { # "scrapy.extensions.telnet.TelnetConsole": None, #} # Configure item pipelines # See https://docs.scrapy.org/en/latest/topics/item-pipeline.html #ITEM_PIPELINES = { # "fwscraper.pipelines.FwscraperPipeline": 300, #} # Enable and configure the AutoThrottle extension (disabled by default) # See https://docs.scrapy.org/en/latest/topics/autothrottle.html #AUTOTHROTTLE_ENABLED = True # The initial download delay #AUTOTHROTTLE_START_DELAY = 5 # The maximum download delay to be set in case of high latencies #AUTOTHROTTLE_MAX_DELAY = 60 # The average number of requests Scrapy should be sending in parallel to # each remote server #AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0 # Enable showing throttling stats for every response received: #AUTOTHROTTLE_DEBUG = False # Enable and configure HTTP caching (disabled by default) # See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings #HTTPCACHE_ENABLED = True #HTTPCACHE_EXPIRATION_SECS = 0 #HTTPCACHE_DIR = "httpcache" #HTTPCACHE_IGNORE_HTTP_CODES = [] #HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage" # Set settings whose default value is deprecated to a future-proof value TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor" FEED_EXPORT_ENCODING = "utf-8" 

Answer: You requested a non-default reactor in your settings so you need to install it explicitly when using CrawlerRunner (this is mentioned in https://docs.scrapy.org/en/latest/topics/asyncio.html#installing-the-asyncio-reactor). You also need to do it before importing twisted.internet.reactor as that installs the default reactor (this is mentioned in https://docs.scrapy.org/en/latest/topics/asyncio.html#handling-a-pre-installed-reactor). A correct code example is shown in https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script: import scrapy from scrapy.crawler import CrawlerRunner from scrapy.utils.reactor import install_reactor install_reactor("twisted.internet.asyncioreactor.AsyncioSelectorReactor") runner = CrawlerRunner() d = runner.crawl(MySpider) from twisted.internet import reactor d.addBoth(lambda _: reactor.stop()) reactor.run()