Package: scrapy-redis
Scrapy-RedisRedis-based components for Scrapy.Usage:https://github.com/rmax/scrapy-redis/wiki/UsageDocumentation:https://github.com/rmax/scrapy-redis/wiki.Release:https://github.com/rmax/scrapy-redis/wiki/HistoryContribution:https://github.com/rmax/scrapy-redis/wiki/Getting-StartedLICENSE: MIT licenseFeaturesDistributed crawling/scrapingYou can start multiple spider instances that share a single redis queue.
Best suitable for broad multi-domain crawls.Distributed post-processingScraped items gets pushed into a redis queued meaning that you can start as
many as needed post-processing processes sharing the items queue.Scrapy plug-and-play componentsScheduler + Duplication Filter, Item Pipeline, Base Spiders.In this forked version: addedjsonsupported data in Redisdata containsurl,`meta`and other optional parameters.metais a nested json which contains sub-data.
this function extract this data and send another FormRequest withurl,metaand additionformdata.For example:{"url":"https://exaple.com","meta":{"job-id":"123xsd","start-date":"dd/mm/yy"},"url_cookie_key":"fertxsas"}this data can be accessed inscrapy spiderthrough response.
like:request.url,request.meta,request.cookiesNoteThis features cover the basic case of distributing the workload across multiple workers. If you need more features like URL expiration, advanced URL prioritization, etc., we suggest you to take a look at theFronteraproject.RequirementsPython 3.7+Redis >= 5.0Scrapy>=  2.0redis-py>= 4.0InstallationFrom pippipinstallscrapy-redisFrom GitHubgitclonehttps://github.com/darkrho/scrapy-redis.gitcdscrapy-redispythonsetup.pyinstallNoteFor using this json supported data feature, please make sure you have not installed the scrapy-redis through pip. If you already did it, you first uninstall that one.pipuninstallscrapy-redisAlternative ChoiceFronterais a web crawling framework consisting ofcrawl frontier, and distribution/scaling primitives, allowing to build a large scale online web crawler.History0.7.3 (2022-07-21)Move docs to GitHub WikiUpdate tox and support dynamic testsUpdate support for json dataRefactor max idle timeAdd support for python3.7~python3.10Deprecate python2.x support0.7.2 (2021-12-27)Fix RedisStatsCollector._get_key()Fix redis-py dependency versionAdded maximum idle waiting time MAX_IDLE_TIME_BEFORE_CLOSE0.7.1 (2021-03-27)Fixes datetime parse error for redis-py 3.x.Add support for stats extensions.0.7.1-rc1 (2021-03-27)Fixes datetime parse error for redis-py 3.x.0.7.1-b1 (2021-03-22)Add support for stats extensions.0.7.0-dev (unreleased)Unreleased.0.6.8 (2017-02-14)Fixed automated release due to not matching registered email.0.6.7 (2016-12-27)Fixes bad formatting in logging message.0.6.6 (2016-12-20)Fixes wrong message on dupefilter duplicates.0.6.5 (2016-12-19)Fixed typo in default settings.0.6.4 (2016-12-18)Fixed data decoding in Python 3.x.AddedREDIS_ENCODINGsetting (defaultutf-8).Default toCONCURRENT_REQUESTSvalue forREDIS_START_URLS_BATCH_SIZE.Renamed queue classes to a proper naming conventiong (backwards compatible).0.6.3 (2016-07-03)AddedREDIS_START_URLS_KEYsetting.Fixed spider methodfrom_crawlersignature.0.6.2 (2016-06-26)Supportredis_clsparameter inREDIS_PARAMSsetting.Python 3.x compatibility fixed.AddedSCHEDULER_SERIALIZERsetting.0.6.1 (2016-06-25)Backwards incompatible change:Require explicitDUPEFILTER_CLASSsetting.AddedSCHEDULER_FLUSH_ON_STARTsetting.AddedREDIS_START_URLS_AS_SETsetting.AddedREDIS_ITEMS_KEYsetting.AddedREDIS_ITEMS_SERIALIZERsetting.AddedREDIS_PARAMSsetting.AddedREDIS_START_URLS_BATCH_SIZEspider attribute to read start urls
in batches.AddedRedisCrawlSpider.0.6.0 (2015-07-05)Updated code to be compatible with Scrapy 1.0.Added-a domain=…option for example spiders.0.5.0 (2013-09-02)AddedREDIS_URLsetting to support Redis connection string.AddedSCHEDULER_IDLE_BEFORE_CLOSEsetting to prevent the spider closing too
quickly when the queue is empty. Default value is zero keeping the previous
behavior.Schedule preemptively requests on item scraped.This version is the latest release compatible with Scrapy 0.24.x.0.4.0 (2013-04-19)AddedRedisSpiderandRedisMixinclasses as building blocks for spiders
to be fed through a redis queue.Added redis queue stats.Let the encoder handle the item as it comes instead converting it to a dict.0.3.0 (2013-02-18)Added support for different queue classes.Changed requests serialization frommarshaltocPickle.0.2.0 (2013-02-17)Improved backward compatibility.Added example project.0.1.0 (2011-09-01)First release on PyPI.