Saravanan666/pypi-org huggingface dataset:
Package: scrapy-proxy-rotator
This is aScrapydownloader middleware that sets proxy
server for requests.
Currently proxy servers are rotated randomly.
In the future more rotation strategies will be supported.ConfigurationScrapy settings file:DOWNLOADER_MIDDLEWARES = {
    'scrapy_proxy_rotator.ProxyMiddleware': 1,
}

PROXY_ROTATOR = {
    'username': 'user1',
    'password': 'pass1',
    'proxies_file': 'proxies.txt',
}In proxies file proxy server must be formatted like thishttp://host:port.
Package: scrapy-proxy-validation
OverviewScrapy is a great framework for web crawling. This package provides a highly
customized way to deal with the exceptions happening in the downloader
middleware because of the proxy, and uses a signal to note relatives to treat
the invalidated proxies (e.g. moving to blacklist, renew the proxy pool).There are two types of signals this package support:traditional signal, syncdeferred signal, asyncPlease refer to the scrapy and twisted documents:Core API — Scrapy 1.4.0 documentationSignals — Scrapy 1.4.0 documentationDeferred Reference — Twisted 17.9.0 documentationRequirementsScrapyTests on Python 3.5Tests on Linux, but it is a pure python module, should work on any other
platforms with official python and twisted supportInstallationThe quick way:pip install -U scrapy-proxy-validationOr put this middleware just beside the scrapy project.DocumentationSet this middleware inITEMPIPELINESinsettings.py, for example:from scrapy_proxy_validation.downloadermiddlewares.proxy_validation import Validation

DOWNLOADER_MIDDLEWARES.update({
    'scrapy_proxy_validation.downloadmiddlewares.proxy_validation.ProxyValidation': 751
})

SIGNALS = [Validation(exception='twisted.internet.error.ConnectionRefusedError',
                      signal='scrapy.signals.spider_closed'),
           Validation(exception='twisted.internet.error.ConnectionLost',
                      signal='scrapy.signals.spider_closed',
                      signal_deferred='scrapy.signals.spider_closed',
                      limit=5)]

RECYCLE_REQUEST = 'scrapy_proxy_validation.utils.recycle_request.recycle_request'Settings ReferenceSIGNALSA list of the class Validation with the exception it wants to deal with, the
sync signal it sends, the async signal it sends and the limit it touches.RECYCLE_REQUESTA function to recycle the request which have trouble with the proxy, the input
argument isrequest, and the output isrequesttoo.Note: remember to set ``dont_filter`` to be ``True``, or the middleware
``duplicate_fitler`` will remove this request.Built-in Functionsscrapy_proxy_validation.utils.recycle_request.recycle_requestThis is a built-in function to recycle the request which has a problem with
the proxy.This function will remove the proxy keyword in meta and setdont_filterto
beTrue.To use this function, insettings.py:RECYCLE_REQUEST = 'scrapy_proxy_validation.utils.recycle_request.recycle_request'NoteThere could be many different problems about the proxy, thus it will take some
to collect them all and add toSIGNALS. Please be patient, this is not a
once-time solution middleware for this case.TODONo idea, please let me know if you have!
Package: scrapyq
Scrapyq is designed to replace the SQLite backend by a Redis backend.
In other words, all the queue management will be done using Redis.Scrapyq is a fork of the originalhttps://github.com/speakol-ads/scrapyd-redisimplementation.Installpip install scrapyqConfigTo start using this library you just need to override
thespiderqueueoption in yourscrapyd.conffile:[scrapyd]
spiderqueue = scrapyq.spiderqueue.RedisSpiderQueue
...If you want to customize the access to the database,
you can add into yourscrapyd.conffile:[scrapyq]
queue_prefix = scrapyq.queue.
redis_db = 0
redis_host = localhost
redis_port = 6379
redis_username = 'admin'  # (Optional)
redis_password = 'password'  # (Optional)
...
Package: scrapy-qiniu
scrapy_qiniu=====Scrapy中的media pipeline机制，可以方便地将静态资源资源（文件、图像等）下载到本地，然后进行处理。scrapy-qiniu扩展了这一机制，可以将资源存储到七牛云存储上面。并且，实现了以下几个特性：* 支持缓存，可以避免静态资源的重复下载* 采用fetch模式，让七牛服务器代为下载，而不用像默认的FilesPipeline那样，先下载到爬虫所在主机，然后再上传到七牛服务器
Package: scrapy-qos
Scrapy-QOSQOS components for ScrapyUsageActive theQosDownloaderMiddlewarein settings.pyDOWNLOADER_MIDDLEWARES={"scrapy_qos.QosDownloaderMiddleware":543}Config following option in settings.pyQOS_IOPS_ENABLEDdefaultFalsesetTrueto enable IOPS limiterQOS_IOPS_CAPACITYdefault1burst IO count per secondsQOS_IOPS_LIMITdefault1/ show many requests sent per secondsQOS_BPS_ENABLEDdefaultFalsesetTrueto enable BPS limiterQOS_BPS_CAPACITYdefault1048576Bytesburst IO Bytes per secondsQOS_BPS_LIMITdefault1048576Bytes / show many response Bytes receive per secondsQOS_SMALL_RESPONSE_SIZEdefault1048576Bytesguess next response size filter response less than this valueRequirementsPython 3.7+Scrapy >= 2.0asyncioInstallationFrom pippipinstallscrapy-qosFrom Giteegit clone https://gitee.com/hgdsdq/scrapy_qos.git
cd scrapy_qos
python setup.py installImplementationBasic implement QOS with Token Bucket AlgorithmFor scrapy, QosDownloaderMiddleware will guess next response body size that used for BPS limiterα=0.8guess_response_size=(1-α)*guess_response_size+α*guess_response_size
Package: scrapy-querycleaner
This is a Scrapy spider middleware to clean up the request URL GET query parameters
at the output of the spider in accordance with the patterns provided by the user.InstallationInstall scrapy-querycleaner usingpip:$ pip install scrapy-querycleanerConfigurationAddQueryCleanerMiddlewareby including it inSPIDER_MIDDLEWARESin yoursettings.pyfile:SPIDER_MIDDLEWARES = {
    'scrapy_querycleaner.QueryCleanerMiddleware': 100,
}Here, priority100is just an example.
Set its value depending on other middlewares you may have enabled already.Enable the middleware using eitherQUERYCLEANER_REMOVEorQUERYCLEANER_KEEP(or both) in yoursetting.py.UsageAt least one of the following settings needs to be present for the
middleware to be enabled.NoteYou can specify a list of parameter names by using the|(OR) regex operator.For example, the patternsearch|login|postidwill match query parameterssearch,loginandpostid.
This is by far the most common usage case.And by settingQUERYCLEANER_REMOVEvalue to.*you can completely remove all URL query parameters.Supported settingsQUERYCLEANER_REMOVEa pattern (regular expression) that a query parameter name must match
in order to be removed from the URL. (All the others will be accepted.)QUERYCLEANER_KEEPa pattern that a query parameter name must match in order to be kept in the URL.
(All the others will be removed.)You can combine both if some query parameters patterns should be kept and some should not.Theremovepattern has precedence over thekeepone.ExampleLet’s suppose that the spider extracts URLs like:http://www.example.com/product.php?pid=135&cid=12&ttda=12and we want to leave only the parameterpid.To achieve this objective we can use eitherQUERYCLEANER_REMOVEorQUERYCLEANER_KEEP:In the first case, the pattern would becid|ttda:QUERYCLEANER_REMOVE = 'cid|ttda'In the second case,pid:QUERYCLEANER_KEEP = 'pid'The best solution depends on a particular case, that is,
how the query filters will affect any other URL that the spider is expected to extract.
Package: scrapy-rabbit
Scrapy-Rabbit1.简介基于Scrapy爬虫框架与RabbitMQ消息队列结合，一款新的分布式采集框架2.快速入门安装pip install scrapy_rabbit创建项目scrapy_rabbit startproject ProjectNameProjectName：要创建的项目名称创建爬虫scrapy_rabbit genspider SpidersClass SpiderName UrlSpidersClass：爬虫分类SpiderName：爬虫名字Url：初始连接是否使用代理以及是否清空队列都可以在Spider中单独配置修改配置在conf下config.ini中设置rabbit的链接方式即可。指定运行方式run方法需要传一个列表，列表里第一个元素为当前爬虫分类名字，第二元素为爬虫名字，第三元素为运行模式（auto(自动)：所有请求生产完成后直接开始消费。m(生产模式)：只生产不消费。w(消费模式)：只消费不生产），第四元素为并发量。运行
接下来就可以直接右键运行，或者使用命令行python spider.py运行，spider.py为当前爬虫文件。整体目录结构3.架构图Spider（爬虫）：负责处理所有Responses，从中分析提取数据，获取Item字段需要的数据，并将需要跟进的URL提交给引擎，再次进入RabbitMQEngine（引擎）：框架核心，负责Spider、RabbitMQ、Downloader、ItemPipeline中间的通讯，信号、数据传递等RabbitMQ（消息队列）：负责接受引擎发送过来的Request请求，并按照指定优先级存入消息队列Downloader（下载器）：负责下载RabbitMQ中所有Requests请求，并将其获取到的Responses交还给Engine，由引擎交给Spider来处理ItemPipeline（管道）：负责处理Spider中获取到的Item，并进行进行后期处理（详细分析、过滤、存储等）的地方Downloader Middlewares（下载中间件）：介于Scrapy引擎和下载器之间的中间件，主要是处理Scrapy引擎与下载器之间的请求及响应Spider Middlewares（Spider中间件）：介于Scrapy引擎和爬虫之间的中间件，主要工作是处理蜘蛛的响应输入和请求输出4.运行流程Spider将所有Requests发送给Engine引擎把Url封装成一个请求(Request)传给RabbitMQ，RabbitMQ将所有的请求按照优先级保存下来下载器把资源下载下来，并封装成应答包(Response)交给Spider，如果请求失败则重新放入到RabbitMQ中爬虫解析Response若是解析出实体（Item）,则交给实体管道进行进一步的处理若是解析出的是链接（Url）,则把Url交给引擎，封装成一个请求(Request)传给RabbitMQ等待下载5.监控截图6.Todo生产结束后自动消费Spiders里可以指定ItemPipeline请求失败重新返回到消息队列支持Middlewares与Scarpy兼容可以在Spider里指定RabbitMQ队列Log日志需要整理脚本采集信息以及日志存入MongoDB使用平台化来开发爬虫对当前项目下所有爬虫进行控制
Package: scrapy-rabbitmq
## A RabbitMQ Scheduler for Scrapy Framework.Scrapy-rabbitmq is a tool that lets you feed and queue URLs from RabbitMQ via Scrapy spiders, using the [Scrapy framework](http://doc.scrapy.org/en/latest/index.html).Inpsired by and modled after [scrapy-redis](https://github.com/darkrho/scrapy-redis).## InstallationUsing pip, type in your command-line prompt```pip install scrapy-rabbitmq```Or clone the repo and inside the scrapy-rabbitmq directory, type```python setup.py install```## Usage### Step 1: In your scrapy settings, add the following config values:```# Enables scheduling storing requests queue in rabbitmq.SCHEDULER = "scrapy_rabbitmq.scheduler.Scheduler"# Don't cleanup rabbitmq queues, allows to pause/resume crawls.SCHEDULER_PERSIST = True# Schedule requests using a priority queue. (default)SCHEDULER_QUEUE_CLASS = 'scrapy_rabbitmq.queue.SpiderQueue'# RabbitMQ Queue to use to store requestsRABBITMQ_QUEUE_NAME = 'scrapy_queue'# Provide host and port to RabbitMQ daemonRABBITMQ_CONNECTION_PARAMETERS = {'host': 'localhost', 'port': 6666}# Store scraped item in rabbitmq for post-processing.ITEM_PIPELINES = {'scrapy_rabbitmq.pipelines.RabbitMQPipeline': 1}```### Step 2: Add RabbitMQMixin to Spider.#### Example: multidomain_spider.py```from scrapy.contrib.spiders import CrawlSpiderfrom scrapy_rabbitmq.spiders import RabbitMQMixinclass MultiDomainSpider(RabbitMQMixin, CrawlSpider):name = 'multidomain'def parse(self, response):# parse all the thingspass```### Step 3: Run spider using [scrapy client](http://doc.scrapy.org/en/1.0/topics/shell.html)```scrapy runspider multidomain_spider.py```### Step 4: Push URLs to RabbitMQ#### Example: push_web_page_to_queue.py```#!/usr/bin/env pythonimport pikaimport settingsconnection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))channel = connection.channel()channel.basic_publish(exchange='',routing_key=settings.RABBITMQ_QUEUE_NAME,body='</html>raw html contents<a href="http://twitter.com/roycehaynes">extract url</a></html>')connection.close()```## Contributing and ForkingSee [Contributing Guidlines](CONTRIBUTING.MD)## ReleasesSee the [changelog](CHANGELOG.md) for release details.| Version | Release Date || :-----: | :----------: || 0.1.0 | 2014-11-14 || 0.1.1 | 2015-07-02 |## Copyright & LicenseCopyright (c) 2015 Royce Haynes - Released under The MIT License.
Package: scrapy-rabbitmq-publisher
A RabbitMQ Item Publisher for Scrapy Framework.Scrapy-RabbitMQ-Publisher allows you to save publish items to RabbitMQ.InstallationUsing pip, type in your command-line promptpip install scrapy-rabbitmq-publisherOr clone the repo and inside thescrapy-rabbitmq-publisherdirectory, typepython setup.py installUsageAdd following code to your scrapy settings.RABBITMQ_HOST="localhost"RABBITMQ_PORT=5672RABBITMQ_USER="guest"RABBITMQ_PASSWORD="guest"RABBITMQ_VIRTUAL_HOST="/"RABBITMQ_EXCHANGE="scrapy"RABBITMQ_ROUTING_KEY="item"RABBITMQ_QUEUE="item"ITEM_PIPELINES={"scrapy_rabbitmq_publisher.pipelines.RabbitMQItemPublisherPipeline":1,}Copyright & LicenseCopyright (c) 2019 Artem Rys - Released under The MIT License.
Package: scrapy-random-fake-ua
scrapy-random-fake-uaScrapy random User-Agent middlewareInstallationThe simplest way is to install it viapip:pip install scrapy-random-fake-uaConfigurationsettings.pyDOWNLOADER_MIDDLEWARES = {
    'scrapy_random_fake_ua.middleware.RandomUserAgentMiddleware': 400,
}
Package: scrapy-random-ua
Does your scrapy spider get identified and blocked by servers because
you use the default user-agent or a generic one?Use thisrandom_useragentmodule and set a random user-agent for
every request.InstallingInstalling it is pretty simple.pipinstallgit+https://github.com/cleocn/scrapy-random-useragent.gitUsageIn yoursettings.pyfile, update theDOWNLOADER_MIDDLEWARESvariable like this.DOWNLOADER_MIDDLEWARES={'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware':None,'random_useragent.RandomUserAgentMiddleware':400}This disables the defaultUserAgentMiddlewareand enables theRandomUserAgentMiddleware.Now all the requests from your crawler will have a random user-agent.
Package: scrapy-random-useragent
Does your scrapy spider get identified and blocked by servers because
you use the default user-agent or a generic one?Use thisrandom_useragentmodule and set a random user-agent for
every request. You are limited only by the number of different
user-agents you set in a text file.InstallingInstalling it is pretty simple.pipinstallscrapy-random-useragentUsageIn yoursettings.pyfile, update theDOWNLOADER_MIDDLEWARESvariable like this.DOWNLOADER_MIDDLEWARES={'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware':None,'random_useragent.RandomUserAgentMiddleware':400}This disables the defaultUserAgentMiddlewareand enables theRandomUserAgentMiddleware.Then, create a new variableUSER_AGENT_LISTwith the path to your
text file which has the list of all user-agents
(one user-agent per line).USER_AGENT_LIST="/path/to/useragents.txt"Now all the requests from your crawler will have a random user-agent
picked from the text file.
Package: scrapy-random-useragent-pro
scrapy-random-useragent-proRandom UserAgent middleware for Scrapy.You can chose your preferred ua in request.meta[desktop,mobile,bot].Installpipinstallscrapy-random-useragent-proUsagesettings.pyRANDOM_UA_ENABLED=TrueRANDOM_UA_DEFAULT_TYPE='mobile'# always change user-agentRANDOM_UA_OVERWRITE=FalseDOWNLOADER_MIDDLEWARES={'scrapy_random_useragent_pro.middleware.RandomUserAgentMiddleware':100,'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware':None,# important}In your codeyieldRequest('http://scrapytest.org/',meta={'ua':'mobile'})
Package: scrapy-redirect
scrapy-redirectrestricts authorized HTTP redirections to the website start_urlsWhy?If theScrapyREDIRECT_ENABLEDconfig key is set toFalseand a request to the homepage of the crawled website returns a3XX status code, the crawl will stop immediatly, as the redirection will not be followed.scrapy-redirectwill force Scrapy to tolerate redirections coming from thestart_urlsurls, in the case  whereREDIRECT_ENABLED = False, to avoid this particular problem.Installation$pipinstallscrapy-redirectConfigurationInstallscrapy-redirectin your Scrapy middlewares by adding the following key/value pair in theSPIDER_MIDDLEWARESsettings key (insettings.py):SPIDER_MIDDLEWARES={...'scrapyredirect.HomepageRedirectMiddleware':575,...}Note that it is important for the middleware order value to be inferior to 600 (thedefault valueof the'scrapy.contrib.downloadermiddleware.redirect.RedirectMiddleware'middleware), as it must be executed before Scrapy blocks the redirection.NB: ifREDIRECT_ENABLED = True,scrapy-redirectdoes nothing.Licensescrapy-redirectis published under the MIT License.
Package: scrapy-redis-bf
Scrapy-Redis-BloomFilter这是个scrapy_redis的布隆过滤器版本，与https://github.com/Python3WebSpider/ScrapyRedisBloomFilter不同的是，该项目使用redis里的布隆过滤器，而不是使用bit来实现必要条件需要redis加载了布隆过滤器的插件，默认安装的redis是没有加载的
具体请看：https://redis.io/docs/stack/bloom/quick_start/安装使用pip:pip install scrapy-redis-bf使用在scrapy项目里的settings.py添加如下设置:SCHEDULER="scrapy_redis_bf.scheduler.Scheduler"DUPEFILTER_CLASS="scrapy_redis_bf.dupefilter.RFPDupeFilter"# 默认是通过spider的name来创建redis keySCHEDULER_DUPEFILTER_ATTR="name"# 格式：redis://[:password@]host[:port][/database][?[timeout=timeout[d|h|m|s|ms|us|ns]][&database=database]]REDIS_URL='redis://localhost:6379'# 错误率BLOOMFILTER_ERRORRATE=0.001# 去重量BLOOMFILTER_CAPACITY=10000测试下载该项目，然后运行里面的test spider即可Githubhttps://github.com/kanadeblisst/scrapy_redis_bf
Package: scrapy-redis-cf
Scrapy-Redis-CuckooFilter这是个scrapy_redis的布谷鸟过滤器版本，项目和https://github.com/kanadeblisst/scrapy_redis_bf基本一样必要条件需要redis加载了布隆过滤器的插件，默认安装的redis是没有加载的
具体请看：https://redis.io/docs/stack/bloom/quick_start/安装使用pip:pip install scrapy-redis-cf使用在scrapy项目里的settings.py添加如下设置:SCHEDULER="scrapy_redis_cf.scheduler.Scheduler"DUPEFILTER_CLASS="scrapy_redis_cf.dupefilter.RFPDupeFilter"# 格式：redis://[:password@]host[:port][/database][?[timeout=timeout[d|h|m|s|ms|us|ns]][&database=database]]REDIS_URL='redis://localhost:6379'# 桶的大小，桶越大错误率越高, 但过滤器填充率越高# 等于1时错误率最小为0.78%，等于3时为2.35%, 等于2时官网没说我也不知道# https://redis.io/commands/cf.reserve/CUCKOOFILTER_BUCKETSIZE=2# 去重数量 2 ** 18 ,# 注意：实际容量要小于这个，因为布谷鸟过滤器很难完全填满CUCKOOFILTER_CAPACITY_BIT=18# 支持扩展容量的次数, 扩展次数越多，错误率越高CUCKOOFILLTER_EXPANSION=2测试下载该项目，然后运行里面的test spider即可Githubhttps://github.com/kanadeblisst/scrapy_redis_cf
Package: scrapy-redis-sentinel
scrapy-redis 集群版本项目基于原项目scrapy-redis进行修改，修改内容如下：添加了Redis哨兵连接支持添加了Redis集群连接支持添加了Bloomfilter去重安装pipinstallscrapy-redis-sentinel--user配置示例原版本 scrapy-redis 的所有配置都支持, 优先级：哨兵模式 > 集群模式 > 单机模式# ----------------------------------------Bloomfilter 配置-------------------------------------# 使用的哈希函数数，默认为 6BLOOMFILTER_HASH_NUMBER=6# Bloomfilter 使用的 Redis 内存位，30 表示 2 ^ 30 = 128MB，默认为 30   (2 ^ 22 = 1MB 可去重 130W URL)BLOOMFILTER_BIT=30# 是否开启去重调试模式 默认为 False 关闭DUPEFILTER_DEBUG=False# ----------------------------------------Redis 单机模式-------------------------------------# Redis 单机地址REDIS_HOST="172.25.2.25"REDIS_PORT=6379# REDIS 单机模式配置参数REDIS_PARAMS={"password":"password","db":0}# ----------------------------------------Redis 哨兵模式-------------------------------------# Redis 哨兵地址REDIS_SENTINELS=[('172.25.2.25',26379),('172.25.2.26',26379),('172.25.2.27',26379)]# REDIS_SENTINEL_PARAMS 哨兵模式配置参数。REDIS_SENTINEL_PARAMS={"service_name":"mymaster","password":"password","db":0}# ----------------------------------------Redis 集群模式-------------------------------------# Redis 集群地址REDIS_STARTUP_NODES=[{"host":"172.25.2.25","port":"6379"},{"host":"172.25.2.26","port":"6379"},{"host":"172.25.2.27","port":"6379"},]# REDIS_CLUSTER_PARAMS 集群模式配置参数REDIS_CLUSTER_PARAMS={"password":"password"}# ----------------------------------------Scrapy 其他参数-------------------------------------# 在 redis 中保持 scrapy-redis 用到的各个队列，从而允许暂停和暂停后恢复，也就是不清理 redis queuesSCHEDULER_PERSIST=True# 调度队列SCHEDULER="scrapy_redis_sentinel.scheduler.Scheduler"# 基础去重DUPEFILTER_CLASS="scrapy_redis_sentinel.dupefilter.RedisDupeFilter"# BloomFilter# DUPEFILTER_CLASS = "scrapy_redis_sentinel.dupefilter.RedisBloomFilter"# 启用基于 Redis 统计信息STATS_CLASS="scrapy_redis_sentinel.stats.RedisStatsCollector"# 指定排序爬取地址时使用的队列# 默认的 按优先级排序( Scrapy 默认)，由 sorted set 实现的一种非 FIFO、LIFO 方式。# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderPriorityQueue'# 可选的 按先进先出排序（FIFO）# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderStack'# 可选的 按后进先出排序（LIFO）# SCHEDULER_QUEUE_CLASS = 'scrapy_redis_sentinel.queue.SpiderStack'注：当使用集群时单机不生效spiders 使用修改 RedisSpider 引入方式原版本scrapy-redis使用方式fromscrapy_redis.spidersimportRedisSpiderclassSpider(RedisSpider):...scrapy-redis-sentinel使用方式fromscrapy_redis_sentinel.spidersimportRedisSpiderclassSpider(RedisSpider):...
Package: scrapy-requests
scrapy-requestsScrapy middleware to asynchronously handle javascript pages using requests-html.requests-html uses pyppeteer to load javascript pages, and handles user-agent specification for you.
Using requests-html is very intuitive and simple.Check out their documentation.RequirementsPython >= 3.6Scrapy >= 2.0requests-htmlInstallationpip install scrapy-requestsConfigurationMake twisted use Asyncio event loop
And add RequestsMiddleware to the downloader middlewaresettings.pyTWISTED_REACTOR='twisted.internet.asyncioreactor.AsyncioSelectorReactor'DOWNLOADER_MIDDLEWARES={'scrapy_requests.RequestsMiddleware':800}UsageUse scrapy_requests.HtmlRequest instead of scrapy.Requestfromscrapy_requestsimportHtmlRequestyieldHtmlRequest(url=url,callback=self.parse)The requests will be handled by requests_html, and the request will add an additional meta varialblepagecontaining the HTML object.defparse(self,response):page=response.request.meta['page']page.html.render()Additional settingsIf you would like the page to be rendered by pyppeteer - passTrueto therenderkey paramater.yieldHtmlRequest(url=url,callback=self.parse,render=True)You could choose a more speific functionality for the HTML object.For example -
You could set up a sleep timer before loading the page, and js script execution when loading the page - doing it this way:script="document.body.querySelector('.btn').click();"yieldHtmlRequest(url=url,callback=self.parse,render=True,options={'sleep':2,'script':script})You could pass default settings to requests-html session - specifying header, proxies, auth settings etc...
You do this by specifying an addtional variable insettings.pyDEFAULT_SCRAPY_REQUESTS_SETTINGS={'verify':False,# Verifying SSL certificates'mock_browser':True,# Mock browser user-agent'browser_args':['--no-sandbox','--proxy-server=x.x.x.x:xxxx'],}NotesPlease star this repo if you found it useful.Feel free to contribute and propose issues & additional features.License is MIT.
Package: scrapy-requests-downloader
scrapy-requests-downloaderScrapy middleware with requests support as an alternative to Twisted calls.Dependencies :globe_with_meridians:Python 3.7Scrapy 2.4.0requests 2.27.1Installation :inbox_tray:This is a python package hosted on pypi, so to install simply run the following command:pip install scrapy-requests-downloaderSettingsREQUESTS_ENABLEDWhether the requests middle is used to process any request (defaults to false).Meta field to enable/disable this per request is:requests_enabledUsage example :eyes:In order to use this plugin simply add the following settings and substitute your variables:DOWNLOADER_MIDDLEWARES={"requestsmiddleware.middleware.RequestsDownloaderMiddleware":632}In order to use it as a downloader you can add the following to your settings:REQUESTS_ENABLED=TrueThis will make every request use requests for a response.License :memo:The project is available under theMIT License.