Saravanan666/pypi-org huggingface dataset:
Package: scrapy-block-inspector
OverviewScrapy is a great framework for web crawling. This package provides a spider
middleware to inspect the spider blocked or not in a highly customized way.RequirementsTests on Python 2.7 and Python 3.5, but it should work on other version higher
then Python 3.3Tests on Linux, but it’s a pure python module, it should work on other
platforms with official python supported, e.g. Windows, Mac OSX, BSDInstallationThe quick way:pip install scrapy-block-inspectorOr put this middleware just beside the scrapy project.DocumentationBlock Inspector in spider middleware, insettings.py, for example:# -----------------------------------------------------------------------------
# USER AGENT
# -----------------------------------------------------------------------------

SPIDER_MIDDLEWARES.update({
    'scrapy_block_inspector.spidermiddlewares.block_inspector.BlockInspector': 500,
})
BLOCK_INSPECTOR = 'scrapy_project.spiders.spider.inspect_block'
BLOCK_SIGNALS = ['scrapy_rotated_proxy.signals.proxy_block']
BLOCK_SIGNALS_DEFERRED = ['scrapy_httpcache.signals.response_block']
RECYCLE_BLOCK_REQUEST = 'scrapy_project.utils.recycle_block_request'This middleware will add a new stats in the stats collector, named
‘block_inspector/block’.Settings ReferenceBLOCK_INSPECTORA function in the spidermiddleware to inspect block, if blocked this function
will returnTrue, otherwise returnFalse.The input of this function is theresponse.BLOCK_SIGNALSWhen a block inspected, this spidermiddleware can send a signal to thesignal managerof the crawler to let other parts (middlewares, extensions, stats,
etc.) to execute relative operations.This should be a list.BLOCK_SIGNALS_DEFERREDIf the signal is connected to a function or method which will return a deferred
object, this signal should be put in this setting.This should be a list.RECYCLE_BLOCK_REQUESTA function to recycle the blocked request. Sometimes the block request need to
recycle after some further treatment, like to remove proxy related key in
request.meta, etc.Note: in this middleware ‘dont_filter=True’ will be added automatically.The input of this function is the request.Build-in Functions To Inspect Blockinspect_block_google_recaptchaThis is a function to check google recaptcha block.To use this inspector, in settings:BLOCK_INSPECTOR = 'scrapy_block_inspector.utils.inspect_block_google_recaptcha.inspect_block'NOTEPlease note: in scrapy, the exception raised by the methodprocess_spider_inputwill be sent torequest.err_backfirst if there is
err_back defined. So please make sure the exceptionBlockExceptiondefined
by this middleware can be raised in err_back function to trigger off the methodprocess_spider_exceptioncorrectly.TODO
Package: scrapy-bloomerfiler
处理若干个pdf水印事项项目PYPI地址https://pypi.org/project/scrapy_bloomerfiler/安装:pip install scrapy-bloomerfilterscrapy 管道中添加配置ITEM_PIPELINES = {'scrapy_bloomerfiler.bloomerfilerpipeline': 400},scrapy.cfg添加配置REDIS_HOST / REDIS_PORT / REDIS_DB / REDIS_PASSWORD测试环境
[redis_cfg_dev]
REDIS_HOST = ***
REDIS_PORT = ***
REDIS_DB = ***
REDIS_PASSWORD= ***

正式环境
[redis_cfg_prod]
REDIS_HOST = ***
REDIS_PORT = ***
REDIS_DB = ***
REDIS_PASSWORD= ***scrapy settings 中添加IF_PROD 是否为正式环境配置 eg : True
Data_Size 数据体量 eg: 1000*10000/百万级/千万级/千万 
Aim_Set  除重依据字段 Aim_Set <dict>  eg: {"title","all_json"}环境变量中参数(可选)IF_PROD 是否为正式环境配置 eg : True
Package: scrapy-boilerplate
scrapy-boilerplateis a small set of utilities forScrapyto simplify
writing low-complexity spiders that are very common in small and one-off projects.It requiresScrapy(>= 0.16)and has been tested usingpython 2.7.
Additionally,PyQueryis required to run the scripts in theexamplesdirectory.NoteThe code is experimental, includes some magic under the hood and might be
hard to debug. If you are new toScrapy, don’t use this code unless
you are ready to debug errors that nobody have seen before.Usage GuideItemsStandard item definition:fromscrapy.itemimportItem,FieldclassBaseItem(Item):url=Field()crawled=Field()classUserItem(BaseItem):name=Field()about=Field()location=Field()classStoryItem(BaseItem):title=Field()body=Field()user=Field()Becomes:fromscrapy_boilerplateimportNewItemBaseItem=NewItem('url crawled')UserItem=NewItem('name about location',base_cls=BaseItem)StoryItem=NewItem('title body user',base_cls=BaseItem)BaseSpiderStandard spider definition:fromscrapy.spiderimportBaseSpiderclassMySpider(BaseSpider):name='my_spider'start_urls=['http://example.com/latest']defparse(self,response):# do stuffBecomes:fromscrapy_boilerplateimportNewSpiderMySpider=NewSpider('my_spider')@MySpider.scrape('http://example.com/latest')defparse(spider,response):# do stuffCrawlSpiderStandard crawl-spider definition:fromscrapy.contrib.spidersimportCrawlSpider,Rulefromscrapy.contrib.linkextractors.sgmlimportSgmlLinkExtractorclassMySpider(CrawlSpider):name='my_spider'start_urls=['http://example.com']rules=(Rule(SgmlLinkExtractor('category\.php'),follow=True),Rule(SgmlLinkExtractor('item\.php'),callback='parse_item'),)defparse_item(self,response):# do stuffBecomes:fromscrapy_boilerplateimportNewCrawlSpiderMySpider=NewCrawlSpider('my_spider')MySpider.follow('category\.php')@MySpider.rule('item\.php')defparse_item(spider,response):# do stuffRunning HelpersSingle-spider running script:# file: my-spider.py# imports omitted ...classMySpider(BaseSpider):# spider code ...if__name__=='__main__':fromscrapy_boilerplateimportrun_spidercustom_settings={# ...}spider=MySpider()run_spider(spider,custom_settings)Multi-spider script with standard crawl command line options:# file: my-crawler.py# imports omitted ...classMySpider(BaseSpider):name='my_spider'# spider code ...classOtherSpider(CrawlSpider):name='other_spider'# spider code ...if__name__=='__main__':fromscrapy_boilerplateimportrun_crawler,SpiderManagercustom_settings={# ...}SpiderManager.register(MySpider)SpiderManager.register(OtherSpider)run_crawler(custom_settings)NoteSee theexamplesdirectory for working code examples.
Package: scrapy-botproxy
BotProxy downloader middleware for Scrapy.OverviewBotProxy is an IP Rotating HTTP Proxy. Plug BotProxy into your web
scraping application and it will automatically route your requests
through one of our outgoing proxy servers. Multiple locations, fresh IP
addresses every day. Typical integrations take less than 5 minutes into
any script or application.Please note, in order to use this service you need to have active
subscription. You can get one atBotProxy
website.InstallationFrom PyPIpip install scrapy-botproxyFrom GitHubpip install -e git+https://github.com/botproxy/scrapy-botproxy@master#egg=scrapy_botproxyUsagesettings.py:DOWNLOADER_MIDDLEWARES = {
    'scrapy_botproxy.BotProxyMiddleware': 100,
}

BOTPROXY_USER = 'proxy_username'
BOTPROXY_PASSWORD = 'proxy_password'
BOTPROXY_LOCATION = 'us-ny'  # optional
BOTPROXY_COUNTRY = 'US'  # optionalYou can also control botproxy behavior using scrapyrequest.metaproperty. Setbotproxy_disabletoTrueto temorary bypass proxy
for current request. Setlocationorcountryto route request
through selected location or any location in the specific country.
Package: scrapy-box
简单好用的scrapy插件盒Download middlewares代理下载中间件# 普通代理urlPROXY_ADDR='your proxy API'DOWNLOADER_MIDDLEWARES={'scrapy_box.RandomProxyDownloaderMiddleware':740,}# splash代理url, the number must under 723PROXY_ADDR='your proxy API'DOWNLOADER_MIDDLEWARES={'scrapy_box.SplashProxyDownloaderMiddleware':160,'scrapy_box.SplashRetryProxyMiddleware':162,}错误重定向重试的下载中间件# 需要对重定向的url,使用原url进行重试的url片段ERROR_REDIRECT_URL_SNIPPET=['redirect','retry']# 注意，数字必须大于600DOWNLOADER_MIDDLEWARES={'scrapy_box.ErrorRedirectMiddleware':601,}当响应中存在某些字符串的时候，进行重试的中间件# 需要判断的字符串，如果响应中包含其中任何一个字符串，将重试ERROR_RESPONSE_SNIPPET=['Are your a robot?','访问过于频繁，请稍后再试']DOWNLOADER_MIDDLEWARES={'scrapy_box.ErrorResponseMiddleware':540,}Pipelinesmongo批量插入数据库 pipelineMONGO_ADDR=['mongo_cluster_addr1','mongo_cluster_addr2']MONGO_USERNAME='root'MONGO_PASSWORD='xxx'MONGO_INSERT_SIZE=100MONGO_DB='xxx'MONGO_COLLECTION='xxxx'ITEM_PIPELINES={'scrapy_box.MongoBatchInsertPipeline':300,}mysql异步插入数据库 pipelineMYSQL_HOST='192.168.0.43'MYSQL_USERNAME='root'MYSQL_PASSWORD='xxx'MYSQL_DB='xxx'MYSQL_CHARSET='utf8'MYSQL_TABLE='xxx'ITEM_PIPELINES={'scrapy_box.MysqlInsertPipeline':300,}
Package: scrapy-cloudflare-middleware
[![PyPI](https://img.shields.io/pypi/v/scrapy_cloudflare_middleware.svg)](https://pypi.python.org/pypi/scrapy_cloudflare_middleware)## Scrapy "CloudFlare" middlewareA Scrapy Middleware to bypass the CloudFlare's anti-bot protection, based on [cloudflare-scrape](https://github.com/Anorov/cloudflare-scrape).### Installation```pip install scrapy_cloudflare_middleware```### UsageAdd the middleware to your `DOWNLOADER_MIDDLEWARES` settings```pythonDOWNLOADER_MIDDLEWARES = {'scraping_hub.middlewares.CloudFlareMiddleware': 560}```The priority (560) is important, because we want this middleware to kick in just before the scrapy built-in `RetryMiddleware`.Done. Happy scraping !
Package: scrapy-colorlog
scrapy-colorlogColor log output support forScrapy.Installationpipinstallscrapy-colorlogConfigurationAdd the following to your Scrapy project settings module:importscrapy_colorlogscrapy_colorlog.install()SettingsCOLORLOG_FORMATDefault:("%(light_black)s%(asctime)s%(reset)s""%(light_black)s[%(name)s]%(reset)s""%(log_color)s%(levelname)s%(reset)s%(light_black)s:%(reset)s""%(message)s")String for formatting log messages. Refer to thecolorlogpackage
documentation for available escape codes and parameters to the format string.See also:LOG_FORMAT.COLORLOG_DATEFORMATDefault:"%Y-%m-%d %H:%M:%S"String for formatting date/time, expansion of the%(asctime)splaceholder inCOLORLOG_FORMAT.See also:LOG_DATEFORMAT.COLORLOG_COLORS{"DEBUG":"blue","INFO":"cyan","WARNING":"yellow","ERROR":"red","CRITICAL":"purple",}A mapping of record level names to color names. Refer to thecolorlogpackage documentation for details.COLORLOG_SECONDARY_COLORSDefault:NoneA mapping of names to log_colors style mappings, defining additional colors that
can be used in format strings. Refer to thecolorlogpackage documentation
for details.COLORLOG_RESETDefault:TrueImplicitly adds a color reset code to the message output, unless the output
already ends with one.COLORLOG_NO_COLORDefault:FalseDisable color output.See also:NO_COLORenvironment variable.COLORLOG_FORCE_COLORDefault:FalseEnable color output. Takes precedence overCOLORLOG_NO_COLOR.See also:FORCE_COLORenvironment variable.Environment variablesNO_COLORDisable color output. SeeNO_COLORfor details.FORCE_COLOREnable color output. Takes precedence overNO_COLOR.
Package: scrapy-contrib-bigexporters
Overviewscrapy-contrib-bigexporters provides additional exporters for the web crawling and scraping framework Scrapy (https://scrapy.org).The following big data formats are supported:Avro:https://avro.apache.org/Parquet:https://parquet.apache.org/Orc:https://orc.apache.orgRequirementsPython 3.6+Scrapy 2.4+Works on Linux, Windows, macOS, BSDParquet export requires fastparquet 0.4.1+Avro export requires fastavro 1.1.0ORC export requires pyorc 0.4.0+InstallThe quick way (pip):pip install scrapy-contrib-bigexportersAlternatively, you can install it fromconda-forge:conda install -c conda-forge scrapy-contrib-bigexportersDepending on which format you want to use you need to install one or more of the following libraries.Avro:pip install fastavroORC:pip install pyorcParquet:pip install fastparquetAdditional libraries may be needed for specific compression algorithms. See “Use”.UseUse of the library is simple. Install it with your Scrapy project as described above.You only need to configure the exporter in the Scrapy settings, run your scraper and the data will be exported into your desired format. There is no development needed.See here for configuring the exporter in settings:AvroParquetORCSourceThe source is available at:Codeberg (a non-commercial European hosted Git for Open Source):https://codeberg.org/ZuInnoTe/scrapy-contrib-bigexportersGithub (an US hosted commercial Git platform):https://github.com/ZuInnoTe/scrapy-contrib-bigexporters
Package: scrapy-count-filter
Scrapy-count-filterTwo Downloader Middlewares that allows aScrapy Spiderto stop requests after a number of pages, or items are scraped.
There is a similar functionality in theCloseSpider extensionthat stops spiders after a number of pages, items, or errors, but this middleware allows defining counters per domain, and define them as spider arguments instead of project settings.InstallThis project requiresPython 3.6+andpip. Using avirtual environmentis strongly encouraged.$pipinstallscrapy-count-filterUsageFor the middlewares to be enabled, they must be added in the projectsettings.py:DOWNLOADER_MIDDLEWARES = {
    # maybe other Downloader Middlewares ...
    # it's suggested to have the Count Filters after all the default middlewares
    'scrapy_count_filter.middleware.GlobalCountFilterMiddleware': 995,
    'scrapy_count_filter.middleware.HostsCountFilterMiddleware': 996,
}You can use one, or the other, or both middlewares.The counter limits must be defined in the spider instance, in aspider.count_limitsdict.The possible fields are:page_countanditem_count- are used by the GlobalCountFilterMiddleware to stop the spider, if the number of requests, or items scraped is larger than the value providedpage_host_countanditem_host_count- are used by the HostsCountFilterMiddleware to start ignoring requests, if the number of requests, or items scrapedper hostis larger than the value providedAll field values must be integers.Note that the Spider stops whenany of the countersoverflow.Example when the count of requests, and items scraped are active:fromscrapy.spidersimportSpiderclassMySpider(Spider):count_limits={"page_count":99,"item_count":10}LicenseBSD3© Cristi Constantin.
Package: scrapy-crawlbase-middleware
Crawlbase API middleware for ScrapyProcessesScrapyrequests usingCrawlbaseservices either with Normal or Javascript tokensInstallingChoose a way of installing:Clone the repository inside your Scrapy project and run the following:pythonsetup.pyinstallOr usePyPiPython package manager.pip install scrapy-crawlbase-middlewareThen in your Scrapysettings.pyadd the following lines:# Activate the middlewareCRAWLBASE_ENABLED=True# The Crawlbase API token you wish to use, either normal of javascript tokenCRAWLBASE_TOKEN='your token'# Enable the middlewareDOWNLOADER_MIDDLEWARES={'scrapy_crawlbase.CrawlbaseMiddleware':610}UsageUse the scrapy_crawlbase.CrawlbaseRequest instead of the scrapy built-in Request.
The scrapy_crawlbase.CrawlbaseRequest accepts additional arguments, used in Proxy Crawl API:fromscrapy_crawlbaseimportCrawlbaseRequestclassExampleScraper(Spider):defstart_requests(self):yieldCrawlbaseRequest("http://target-url",callback=self.parse_resultdevice='desktop',country='US',page_wait=1000,ajax_wait=True,dont_filter=True)The target url will be replaced with proxy crawl url and parameters will be encoded into the url by the middleware automatically.If you have questions or need help using the library, please open an issue orcontact us.Copyright 2023 Crawlbase
Package: scrapy-crawlera
scrapy-crawlera provides easy use ofCrawlerawith Scrapy.RequirementsPython 2.7 or Python 3.4+ScrapyInstallationYou can install scrapy-crawlera using pip:pip install scrapy-crawleraDocumentationDocumentation is available online athttps://scrapy-crawlera.readthedocs.io/and in thedocsdirectory.
Package: scrapy-crawl-once
scrapy-crawl-onceThis package provides aScrapymiddleware which allows to avoid re-crawling
pages which were already downloaded in previous crawls.License is MIT.Installationpip install scrapy-crawl-onceUsageTo enable it, modify your settings.py:SPIDER_MIDDLEWARES = {
    # ...
    'scrapy_crawl_once.CrawlOnceMiddleware': 100,
    # ...
}

DOWNLOADER_MIDDLEWARES = {
    # ...
    'scrapy_crawl_once.CrawlOnceMiddleware': 50,
    # ...
}By default it does nothing. To avoid crawling a particular page
multiple times setrequest.meta['crawl_once']= True. When a response
is received and a callback is successful, the fingerprint of such request
is stored to a database. When spider schedules a new request middleware
first checks if its fingerprint is in the database, and drops the request
if it is there.Otherrequest.metakeys:crawl_once_value- a value to store in DB. By default, timestamp
is stored.crawl_once_key- request unique id; by default request_fingerprint
is used.SettingsCRAWL_ONCE_ENABLED- set it to False to disable middleware.
Default is True.CRAWL_ONCE_PATH- a path to a folder with crawled requests database.
By default.scrapy/crawl_once/path inside a project dir is used;
this folder contains<spider_name>.sqlitefiles with databases of
seen requests.CRAWL_ONCE_DEFAULT- default value forcrawl_oncemeta key
(False by default). When True, all requests are handled by
this middleware unless disabled explicitly usingrequest.meta['crawl_once']= False.Alternativeshttps://github.com/scrapy-plugins/scrapy-deltafetchis a similar package; it
does almost the same. Differences:scrapy-deltafetch chooses whether to discard a request or not based on
yielded items; scrapy-crawl-once uses an explicitrequest.meta['crawl_once']flag.scrapy-deltafetch uses bsddb3, scrapy-crawl-once uses sqlite.Another alternative is a built-inScrapy HTTP cache. Differences:scrapy cache stores all pages on disc, scrapy-crawl-once only keeps request
fingerprints;scrapy cache allows a more fine grained invalidation consistent with how
browsers work;with scrapy cache all pages are still processed (though not all pages are
downloaded).Contributingsource code:https://github.com/TeamHG-Memex/scrapy-crawl-oncebug tracker:https://github.com/TeamHG-Memex/scrapy-crawl-once/issuesTo run tests, installtoxand runtoxfrom the source checkout.CHANGES0.1.1 (2017-03-04)new'crawl_once/initial'value in scrapy stats - it contains the
initial size (number of records) of crawl_once database.0.1 (2017-03-03)Initial release.
Package: scrapy-custom-delay
Scrapy-Custom-DelayScrapy-Custom-Delayis a package that lets you set different delay for different website, using theScrapyframework.Install$ pip install scrapy-custom-delayUsageStep 1: Use the following config values in your scrapy settings:Enable the AutoThrottle extension.AUTOTHROTTLE_ENABLED=TrueEnable the Custom Delay Throttle by adding it toEXTENSIONS.EXTENSIONS={'scrapy.extensions.throttle.AutoThrottle':None,'scrapy_domain_delay.extensions.CustomDelayThrottle':300,}Add{'regex': 'download delay (in seconds)'}to theDOMAIN_DELAYS.something like:# set up custom delays per domain# if two or more regexes satisfy, the first one will be usedDOMAIN_DELAYS={r'^images.google.com*$':1.0,r'*github*':0.5,}
Package: scrapy-custom-proxy-pool
A Scrapy middleware to manage a custom proxy pool and randomly choose a proxy for each request.Installationpipinstallscrapy-custom-proxy-poolUsageIn settings.py:DOWNLOADER_MIDDLEWARES={# ...'scrapy_custom_proxy_pool.middlewares.ProxyPoolMiddleware':610,'scrapy_custom_proxy_pool.middlewares.BanDetectionMiddleware':620,# ...}PROXY_POOL_ENABLED=TruePROXY_POOL_SIZE=10PROXY_POOL_COLLECTOR='myproject.collectors.MyCollector'PROXY_POOL_COLLECTOR_ARGS={}PROXY_POOL_REFRESH_INTERVAL=60PROXY_POOL_PAGE_RETRY_TIMES=5PROXY_POOL_TRY_WITH_HOST=TrueSettingsPROXY_POOL_ENABLED-Enable/disableproxypool(default:False)PROXY_POOL_SIZE-Maxproxiesinpool(default:10)PROXY_POOL_COLLECTOR-Customproxycollectorpath(required)PROXY_POOL_COLLECTOR_ARGS-Collectorinitializationarguments(default:{})PROXY_POOL_REFRESH_INTERVAL-Proxyrefreshintervalinsecs(default:60)PROXY_POOL_PAGE_RETRY_TIMES-Maxretrytimesperpage(default:5)PROXY_POOL_TRY_WITH_HOST-Tryrequestswithoutproxy(default:True)CustomizationBan Detection PolicyYou can customize the ban detection policy:# myproject/policy.pyfromscrapy_custom_proxy_pool.policyimportBanDetectionPolicyclassMyPolicy(BanDetectionPolicy):# override response_is_ban and exception_is_ban# settings.pyPROXY_POOL_BAN_POLICY='myproject.policy.MyPolicy'Proxy CollectorYou need to implement a custom proxy collector by subclassing ProxyCollector:fromscrapy_custom_proxy_pool.collectorsimportProxyCollectorclassMyCollector(ProxyCollector):deffetch_proxies(self,count):# return list of `Proxy` objectsLicenseMIT
Package: scrapycutter
Template padrão de crawler para projetos da B2WPré-requisitosPython >= 3.6scrapycutter (pip install scrapycutter)Acesso ao nosso GitlabComo usarDepois de ter todo o necessário visto acima, então basta apenas executar o comandoscrapycutter revolutionque o comando nos permitirá fazer as configurações básicas para o projeto RevolutionOBS: O template foi criado de forma que seja possível adicionar novas funcionalidades ou templates de projetos facilmente, para nossa comodidade.Caso prefira, também temos outras duas maneiras de criar o template, a partir do código fonte:Primeiramente lembrando que é necessário estar com a VPN  ativa no momento que realizar o comando acima para que funcione corretamente.Fazer clone para sua máquina permitindo que seja possível executar esse comando localmente:git clone git@gitlab.internal.b2w.io:team/a-tech/crawler-template.gitcookiecutter crawler-templateTambém podemos usar a forma mais prática, que é referenciando esse repositório da seguinte forma:cookiecutter git+http://gitlab.internal.b2w.io/team/a-tech/crawler-template.gitBasta apenas inserir esse comando referenciando esse repositório de template, digitar o usuário e senha do gitlab da empresa (já que funciona como se fosse umgit clone) que as opções serão mostradas para criar o template do projeto da forma que bem entender;OBS:Por motivos de segurança o arquivo B2W.json não é gerado junto com o template, é o único arquivo necessário de ser adicionado assim que o template é gerado.ToDoInvestigar como gerar o template diretamente do nosso repositório  privado do Gitlab, atualmente isso ainda não é possível;(Resolvido)Melhorar dinamismo de como o template é gerado, talvez adicionando novos campos que possam ser configurados;Fazer com que se torne um pacote Python, para que seja ainda mais simples de executar.(Feito)
Package: scrapyd-api
scrapyd APIGithub:https://github.com/mouday/scrapyd-apiGitee:https://gitee.com/mouday/scrapyd-apiPypi:https://pypi.org/project/scrapyd-api安装pipinstallscrapyd-api使用示例# -*- coding: utf-8 -*-frompprintimportpprintfromscrapyd_apiimportScrapydClientclient=ScrapydClient()pprint(client.daemon_status())"""{'finished': 67,'node_name': 'localhost','pending': 0,'running': 0,'status': 'ok','total': 67}"""简介说明，基于scrapyd 1.2.1进行调用，如果版本差异大，可能会出现异常ScrapydAPI对原有的Scrapyd api进行原样返回，有利于二次开发接口文档：https://scrapyd.readthedocs.io/en/stable/api.htmlclassScrapydAPI:add_versioncanceldelete_projectdelete_versionlist_jobslist_projectslist_spiderslist_versionsscheduledaemon_statusScrapydClient类继承自 ScrapydAPI，对其进行了扩展和加强classScrapydClient(ScrapydAPI):# 加强的数据接口daemon_status# 增加了返回参数 totaladd_version# 添加version 默认值为当前时间戳 10位list_spiders# 返回值：列表+字符串 改为 列表+字典list_projects# 返回值：列表+字符串 改为 列表+字典list_versions# 返回值：列表+字符串 改为 列表+字典# 扩展的数据接口job_status# 查询任务状态list_versions_format# 格式化版本号为日期时间格式 '%Y-%m-%d %H:%M:%S'list_jobs_merge# 合并后的任务列表cancel_all_project_job# 取消所有项目下的任务cancel_all_job# 取消指定项目下的任务# 扩展的日志接口logs# 获取日志-项目列表project_logs# 获取日志-爬虫列表spider_logs# 获取日志-任务列表job_log# 获取job日志