Package: os-scrapy-rq-crawler
os-scrapy-rq-crawlerThis project provide Crawler for RQ mode. Based on Scrapy 2.0+, require Python 3.6+TheScrapyframework is used for crawling specific sites. It is not good for"Broad Crawls". The Scrapy built-in schedule mechanism is not for many domains, it use one channel queue for requests of all different domains. The scheduler can not decide to crawl request of the specified domain.The RQ mode is the key mechanism/concept for broad crawls. The key point is RQ(request queue), actually it is a banch of queues, requests of different domains in the different sub-queues.Deploy with theos-rq-podandos-rq-hubyou can build a large scalable distributed "Broad Crawls" system.Quick StartWe offer Crawler for RQ mode. Because the Scrapy framework can not custom Crawler class, so you can useos-scrapy(installed with this project) to start crawling.installpip install os-scrapy-rq-crawlerstart your projectos-scrapy startproject <your_project>set Crawler class andenable asyncio reactorin project setting.pyCRAWLER_CLASS = "os_scrapy_rq_crawler.asyncio.Crawler"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"start crawling example spideros-scrapy crawl exampleInstallpip install os-scrapy-rq-crawlerWhen you already installed os-scrapy, you can run example spider directly in this project root pathos-scrapy crawl exampleUsageThis project offer Crawler class can be used in your project by config in settings.py file. And it can also be used as a scrapy project directly.Crawler ClassThis project provide Crawler to enable RQ mode. The Scrapy framework can not config Crawler class, so you can useos-scrapyto specifyin the settings.pyCRAWLER_CLASS = "os_scrapy_rq_crawler.asyncio.Crawler"or with-ccommand line optionos-scrapy crawl -c os_scrapy_rq_crawler.asyncio.Crawler exampleBecause the Crawler deponds on asyncio reactor, so you need to enable itin the settings.pyTWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"or with-rcommand line optionos-scrapy crawl -r asyncio exampleRequest QueueRQ is just a conception can be implements in different ways.config in the settings.py fileSCHEDULER_REQUEST_QUEUE = "os_scrapy_rq_crawler.AsyncRequestQueue"os_scrapy_rq_crawler.MemoryRequestQueuethe default requet queueall requests in memorythe spider closed when no requests in the queue andspider is idleos_scrapy_rq_crawler.AsyncRequestQueuecan be used withos-rq-podoros-rq-hubas external request queuethe spider will not closed even if no requests to crawlconfig the pod/hub http api in the settings.pyRQ_API = "http://localhost:6789/api/"config the api timeoutRQ_API_TIMEOUT = 3.0os_scrapy_rq_crawler.MultiUpstreamRequestQueuesame asos_scrapy_rq_crawler.AsyncRequestQueue, can configure multi upstreams request queuesconfig the pod/hub http apis in the settings.pyRQ_API = ["http://server01:6789/api/", "http://server02:6789/api/"]FYIOur RQ mode Crawler is a substitute of the Scrapy built-in Crawler. Most of the Scrapy functionalities(middleware/extension) can also be used as normal.There are some tips:the request queue is fifo, priority is not supported yetthe requests come from downloader middlewares will push to the head of the request queuethe request of the same domain will not crawl concurrently,CONCURRENT_REQUESTScontrol the max concurrency andDOWNLOAD_DELAYcontrol the download delyyou can set request delay: request.meta["download_delay"] = 3.0Unit Testssh scripts/test.shLicenseMIT licensed.