Saravanan666/pypi-org huggingface dataset:
Package: scrapy-mattermostbot
scrapy-mattermostbotAScrapyextension for sending notification to Mattermost channels.Installation$pipinstallscrapy-mattermostbotSettingsInsettings.py:EXTENSIONS={'scrapyslackbot.extensions.Mattermost':500,}MATTERMOST_WEBHOOK_URL=<mattermostwebhookurl>MATTERMOST_WEBHOOK_URLMattermost Webhook URLgenerated bymattermost.Follow the instructions ofMattermost Developer Guides,DevelopmentWe usePipenvas the package and dependency manager. Please make sure you have installed pipenv at first.$pipenvshellDemoLicenseThe project is licensed under the MIT License.
Package: scrapy-memcached-cache
Scrapy Memcached CacheMemcached HTTP cache storage backend forScrapy.InstallInstall the package withpipenv installscrapy-memcached-cacheorpip installmemcached-cache.UsageIn your Scrapysettings.py:Enable HTTP cache withHTTPCACHE_ENABLED = TrueSet the cache expiration (in seconds), for example, withHTTPCACHE_EXPIRATION_SECS = 600SetMemcachedCacheStorageas your cache storage withHTTPCACHE_STORAGE = 'scrapy_memcached_cache.MemcachedCacheStorage'Set the location where your Memcached is running, for example, withMEMCACHED_LOCATION=localhost:11211Done 🎉ContributingPlease, format your code withBlack.
Package: scrapy-middlewares
scrapy-middlewaresscrapy middlewaresInstallation 安装pipinstallscrapy_middlewaresrequestshttpxaiohttpUsage example 使用示例'DOWNLOADER_MIDDLEWARES':{'scrapy_middlewares.req_sync_middleware.ModuleReqMiddleware':601},# module_name:# 	'httpx', 'requests'yieldModuleRequest(url=url,callback=self.parse,module_name='requests')'DOWNLOADER_MIDDLEWARES':{'scrapy_middlewares.req_async_middleware.ModuleAsyncReqMiddleware':601},# module_name:# 	'httpx_async', 'aiohttp'yieldModuleRequest(url=url,callback=self.parse,module_name='httpx_async')
Package: scrapy-mock
A Pytest plugin to record Scrapy responses and use them as testing fixtures.
The motivation here is to decrease the friction to write unit tests for Scrapy spiders.Installpip install scrapy-mockUsage@pytest.mark.parametrize(
    ["url", "expected"],
    ("http://foobar.com/item/1",  {"name": "item1", "url": "/item/1"}),
])
def test_parse(spider, response, expected):
    result = next(spider.parse(response))
    assert result == expected
Package: scrapy-mongodb-queue
# Scrapy MongoDB Queue
MongoDB-based components for scrapy that allows distributed crawling# Available Scrapy components:
* Scheduler
* Duplication FilterInstallationFrompypi:$ pip install scrapy-mongodb-queueFromgithub:$ git clone https://github.com/jbinfo/scrapy-mongodb-queue.git
$ cd scrapy-mongodb-queue
$ python setup.py installUsageEnable the components in yoursettings.py:# Enables scheduling storing requests queue in redis.SCHEDULER="scrapy_mongodb_queue.scheduler.Scheduler"# Don't cleanup mongodb queues, allows to pause/resume crawls.MONGODB_QUEUE_PERSIST=True# Specify the host and port to use when connecting to Redis (optional).MONGODB_SERVER='localhost'MONGODB_PORT=27017MONGODB_DB="my_db"# MongoDB collection nameMONGODB_QUEUE_NAME="my_queue"AuthorThis project is maintained by Lhassan Baazzi ([GitHub](https://github.com/jbinfo) | [Twitter](https://twitter.com/baazzilhassan) | [LinkedIn](https://ma.linkedin.com/pub/lhassan-baazzi/49/606/a70))
Package: scrapy-mosquitera
How can I scrape items off a site from the last five days?—Scrapy UserThat question started the development ofscrapy-mosquitera, a tool to help
you restrict crawling and scraping scope usingmatchers.Matchers are simple Python functions that return the validity of an element
under certain restrictions.The first goal in the project was date matching, but you can create your own
matcher for your own crawling and scraping needs.How it worksIn the case where the dates are available in the URLs, you will just use
the matcher function directly in your code:from scrapy_mosquitera.matchers import date_matches

 date = scrape_date_from_url(url)

 if date_matches(data=date, after='5 days ago'):
    yield Request(url=url, callback=self.parse_item)To handle the case when the date is only available at the time when you scrape
the items,scrapy-mosquiteraprovides aPaginationMixinto control the
crawl according to the dates scraped.Head on to the remaining of thedocumentationfor more details.InstallationThe quick way:pip install scrapy-mosquitera
Package: scrapy-multifeedexporter
ThisScrapyextension exports scraped items of
different types to multiple feeds. By default each item gets its own
feed.Installation$pipinstallscrapy-multifeedexporterConfigurationYou’ll have to switch the defaultFeedExporterwithMultiFeedExporterby adding the following lines to thesettings.pyfile of your spider:frommultifeedexporterimportMultiFeedExporterEXTENSIONS={'scrapy.contrib.feedexport.FeedExporter':None,'multifeedexporter.MultiFeedExporter':500,}# Automatically configure available item names from your moduleMULTIFEEDEXPORTER_ITEMS=MultiFeedExporter.get_bot_items(BOT_NAME)UsageWhen callingscrapy crawlyou need to use the%(item_name)splaceholder in the output file/URI name. The following calls toscrapy crawldemonstrate the placeholder:$scrapycrawl-o"spider_name_%(item_name)s.csv"-tcsvspider_name$scrapycrawl-o"ftp://foo:bar@example.com/spider_name_%(item_name)s.csv"-tcsvspider_nameIf you omit the placeholder, all items will be placed in one file.Licensescrapy-multifeedexporter is published under MIT license
Package: scrapy-mysql-pipeline
Python 3.6scrapy-mysql-pipelineAsynchronous mysqlScrapyitem
pipelineInstallationpipinstallscrapy-mysql-pipelineConfigurationAdd pipelineITEM_PIPELINES={'scrapy_mysql_pipeline.MySQLPipeline':300,}Default values:MYSQL_HOST='localhost'MYSQL_PORT=3306MYSQL_USER=NoneMYSQL_PASSWORD=''MYSQL_DB=NoneMYSQL_TABLE=NoneMYSQL_UPSERT=FalseMYSQL_RETRIES=3MYSQL_CLOSE_ON_ERROR=TrueMYSQL_CHARSET='utf8'MYSQL_USER,MYSQL_PASSWORD,MYSQL_DBandMYSQL_TABLE,
variables must be set in settings.py
Package: scrapy-new
scrapy-command-newA package providing code generation command for scrapy CLI.The project is a WIP, so expect major changes and additions (latter, mostly).
Master branch is to be considered as always ready to use, with major changes/features introduced in feature branches.This is a part of a bigger project -Scrapy Boilerplate.The command works with a specific scrapy project structure (not the default one). Rationale for this is describedhere.UsageThis is a scrapy command to generate class files and automatically add imports to respective module's__init__files. It can be used as follows:scrapy new spider SampleSpiderThe first argument (spider) is a type of class file to be generated, and can be one of the following:commandextensionitemmiddlewaremodelpipelinespiderThe second argument is class name.Also forpipelineandspiderclass an option--rabbitcan be used to add RabbitMQ connection code to generated source.Option--itemwith valueCLASSNAMEis supported for generating pipelines, which adds an import and type-check for a provided item class to the resulting code.Option--settingsis also supported for pipelines, extension, middlewares and spider middlewares. It has an optional integer valuePRIORITYthat adds specified priority. If only-sis used, settings file will besettings.py.(experimental) Option--fileis used for specifying settings file name (or class). You can use spider file for adding newly generated class to spiders'custom_settingsproperty. If you enumerate file names (or class names) using,(like-f SomeSpider,AnotherSpider) - script will add generated class to custom_settings of each file. If only-fis used, will be used default priority (300).Option--terminalwill output 'custom_settings' code to terminal.Option--customcan be used for custom template folder path. Template names should be like{}.py.mako. Option will enable usage of TEMPLATES_MODULE setting from projects` settings.py. If this setting is not defined, will cause exception.InstallationThis command is included in theScrapy Boilerplateout of the box. If you want to install it manually, you can get it from PyPi:pip install scrapy-newPlease notethat this package won't work with default Scrapy project structure, it requires a specific custom one, as describedhere.
Package: scrapy-nimble
Scrapy Nimble Middlewarescrapy-nimbleis aScrapy Downloader Middlewarethat helps to integrateScrapywithNimble Web API.InstallYou can installscrapy-nimbleas a regular Python package fromPyPIusing:pipinstallscrapy-nimbleConfigurationIf you don't have it yet,open an accountwithNimble.Provide your credentials and enable the middleware through Scrapy settings.# settings.pyNIMBLE_ENABLED=TrueNIMBLE_USERNAME="username"NIMBLE_PASSWORD="password"Add the downloader middleware to yourDOWNLOADER_MIDDLEWARESScrapy setting.# settings.pyDOWNLOADER_MIDDLEWARES={"scrapy_nimble.middlewares.NimbleWebApiMiddleware":570,}If you havescrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddlewareenabled
(it is enabled by default inDOWNLOADER_MIDDLEWARES_BASEsetting with default order equal to 590), configurescrapy-nimblemiddlewarebeforeit.Basic UsageOnce the downloader middleware is properly configured, every request goes through the Nimble's Web API.
There is no need to change anything in your spider's code.Real-time URL requestscrapy-nimbleusesNimble Web APIwith Real-time URL
requests. In addition to the default GET request for a specific URL, this API provides some extra options
that allow you to execute geolocated requests, render dynamic content, among others.Right now the followingrequest optionscan be used. Check the documentation for usage and the valid values that can be provided.
If the option is not given, the default value from Web API will be used.methodcountrylocaleheaderscookiesrenderrender_optionsAdd the options you want to be used inside themetakey of your request, appendingnimble_to the
option name such as:# Inside your spideryieldscrapy.Request("https://nimbleway.com",meta={"nimble_country":"DE","nimble_locale":"uk","nimble_render":True,})DevelopmentWe suggest the use ofpyenvto manage your Python version and create an isolated
environment where you can safely develop. After installing it, you can prepare the environment using the following
commands:$pyenvvirtualenv3.11.6myvenv
$pyenvactivatemyvenv
$python-mpipinstall-e.To keep a standard in code formatting and do some linter checks, we usepre-commit hooks.Installpre-commitpackage and install the project hooks using:$pre-commitinstallNow you are ready to start development.
Package: scrapy-notifications
A Scrapy extension to be used, for example, withHTTP Callback Tasksof Celery project.InstallationInstall via pip:pip install scrapy-notificationsTested withPython 2.7+Scrapy 0.16SetupBasically you need configureSPIDER_NOTIFICATION_ENABLED,SPIDER_NOTIFICATION_URLand add thescrapy_notifications.extensions.SpiderNotificationextension to your
Scrapy projectsettings.py.Example:EXTENSIONS = {
    'scrapy_notifications.extensions.SpiderNotification': 500,
}
SPIDER_NOTIFICATION_ENABLED = True
SPIDER_NOTIFICATION_URL = 'http://app.example.com/spider-task/'By default HTTP requests are sent with the spider name attribute as an URL
parameter.
But optionally you can also specify a list of spider attributes passed as
URL parameters onSPIDER_NOTIFICATION_ATTRS.:SPIDER_NOTIFICATION_ATTRS = ['name', 'jobid']UsageFor example crawling with thedmozspider used in theScrapy tutorial:scrapy crawl -a jobid=fe648c57f8 dmozwill generate a HTTP request like this:http://app.example.com/spider-task/?name=dmoz&jobid=fe648c57f8ContributingWant to contribute? Great! Bug reports and code and documentation patches are greatly appretiated.
Please file bugs and send pull requests using theissue tracker.
Package: scrapy-pagestorage
A scrapy extension to store requests and responses information in storage service.InstallationYou can install scrapy-pagestorage using pip:pip install scrapy-pagestorageYou can then enable the middleware in yoursettings.py:SPIDER_MIDDLEWARES = {
    ...
    'scrapy_pagestorage.PageStorageMiddleware': 900
}How to use itEnable extension throughsettings.py:PAGE_STORAGE_ENABLED = True
PAGE_STORAGE_ON_ERROR_ENABLED = TrueConfigure the exension throughsettings.py:PAGE_STORAGE_MODE = "VERSIONED_CACHE"
PAGE_STORAGE_LIMIT = 100
PAGE_STORAGE_ON_ERROR_LIMIT = 100
PAGE_STORAGE_TRIM_HTML = TrueThe extension is auto-enabled for Portia spiders (SHUB_SPIDER_TYPE=portia).SettingsPAGE_STORAGE_MODEDefault:NoneA string which specifies if the extension will store information using cache store or
versioned cache store (setPAGE_STORAGE_MODE=”VERSIONED_CACHE”to use versioned one).PAGE_STORAGE_LIMITAn integer to set a limit of visited pages amount to store.PAGE_STORAGE_ON_ERROR_LIMITAn integer to set a limit for page errors amount to store.PAGE_STORAGE_TRIM_HTMLDefault:FalseRemove whitespace from the start and end of the HTML to reduce file size.
Package: scrapypi
scrapypi is used to get pypi package information using python.scrapypi can be used to get package version, name, licence, author, summary and package download statistics.Insallationpython -m pip install scrapypiUsageTypescrapypiin terminal or command prompt to scrapypi file for getting downloads statistics. scrapypi can also be used in code.$scrapypiBelow are examples and functions and classes in scrapypi for data fetching.infostatsversioninfoGet package information. You can either get information using package name or package pypi url.fromscrapypiimportinfo,stats,licence# returns a dictionary of information.info(<package_name>)values returned by info() is listed below:nameauthorlicencesummaryprevious_versionlatest versiondownloadsstatsGet package full statistics, including download values everyday of package upload, date and number of downloads.fromscrapypiimportstatsstatistics=stats(<package_name>)# get all daily download counts of package in the year.statistics.dataset()# get total number of downloads.statistics.get_total()versionYou can also get package name and version using the version function.fromscrapypiimportversion# returns package name and version in a tuple. (name, version)version(<package_name>)Use the version function if you need to get those information fast. But i recommend using the info function.mainThe main function is to get a statics view of packages and also compare the statistics of multiple packages. Use themain()function to run the scrapypi script for statistics fetcher. Fetching multiple packages needs to be seperated with a comma (,)Package Name(s): package_one, package_two
Package: scrapy-plus
Scrapy+Scrapy扩展工具包。为《从0学爬虫专栏》提供，详细的使用方法请到专栏内参考。$ pip install scrapy_plusScrapy+提供以下的内容过滤器Redis 去重过滤器Redis 布隆去重过滤器中间件自登录中间件花瓣网专用中间件Chrome通用中间件Splash渲染中间件Tor中间件随机UA中间件随机代理中间件管道MongoDB数据存储管道可支持阿里云的OSS图片管道SQL存储端输入/输出处理器蜘蛛BookSpiderNeteaseSpiderTaobaoSpider
Package: scrapy-po
WarningCurrent status is “experimental”.scrapy-poimplements Page Object pattern for Scrapy.License is BSD 3-clause.Installationpip install scrapy-poscrapy-po requires Python >= 3.6 and Scrapy 1.7+.UsageTODO. For now, check spiders in “example” folder.ContributingSource code:https://github.com/scrapinghub/scrapy-poIssue tracker:https://github.com/scrapinghub/scrapy-po/issuesUsetoxto run tests with different Python versions:toxThe command above also runs type checks; we use mypy.Changes0.0.1 (2019-08-28)Initial release.
Package: scrapy-poet
scrapy-poetis theweb-poetPage Object pattern implementation for Scrapy.scrapy-poetallows to write spiders where extraction logic is separated from the crawling one.
Withscrapy-poetis possible to make a single spider that supports many sites with
different layouts.Read thedocumentationfor more information.License is BSD 3-clause.Documentation:https://scrapy-poet.readthedocs.ioSource code:https://github.com/scrapinghub/scrapy-poetIssue tracker:https://github.com/scrapinghub/scrapy-poet/issuesQuick StartInstallationpip install scrapy-poetRequiresPython 3.8+andScrapy >= 2.6.0.Usage in a Scrapy ProjectAdd the following inside Scrapy’ssettings.pyfile:DOWNLOADER_MIDDLEWARES={"scrapy_poet.InjectionMiddleware":543,"scrapy.downloadermiddlewares.stats.DownloaderStats":None,"scrapy_poet.DownloaderStatsMiddleware":850,}SPIDER_MIDDLEWARES={"scrapy_poet.RetryMiddleware":275,}REQUEST_FINGERPRINTER_CLASS="scrapy_poet.ScrapyPoetRequestFingerprinter"DevelopingSetup your local Python environment via:pip install -r requirements-dev.txtpre-commit installNow everytime you perform agit commit, these tools will run against the
staged files:blackisortflake8You can also directly invokepre-commit run –all-filesortox -e lintersto run them without performing a commit.
Package: scrapy-prometheus
Scrapy stats collector that exports scrapy stats as prometheus metrics through pushgateway service.InstallationVia pip:pip install scrapy_prometheusUsageTo start using, addscrapy_prometheus.PrometheusStatsCollectorto STATS_CLASS setting:STATS_CLASS='scrapy_prometheus.PrometheusStatsCollector'This stats collector works exactly like the vanilla one (because it subclasses it), but also
creates prometheus metrics and pushes them to pushgateway service on spider close signal.It supports two metric types:CounterandGauge. Stat metric type is determined by operation used on
the stat:stats.inc_valuewill create aCountermetric, while other methods,stats.set_value,stats.max_value,stats.min_value, will createGauge.All metrics will have aspiderlabel attached with spider name.Stat value must be eitherintorfloat.Note, that trying to perform action on a metric, that is not supposed to be used with this
action (set_value on Counter or inc_value on Gauge) will producescrapy_prometheus.InvalidMetricTypeerror. To suppress it, setPROMETHEUS_SUPPRESS_TYPE_CHECKto True.If you want to create custom metrics, you can access your spider’s CollectorRegistry by usingstats.get_registry(spider).Available settings# Prometheus pushgateway hostPROMETHEUS_PUSHGATEWAY='localhost:9091'# default# Metric name prefixPROMETHEUS_METRIC_PREFIX='scrapy_prometheus'# default# Timeout for pushing metrics to pushgatewayPROMETHEUS_PUSH_TIMEOUT=5# default# Method to use when pushing metrics# Read https://github.com/prometheus/pushgateway#put-methodPROMETHEUS_PUSH_METHOD='POST'# default# Do not raise scrapy_prometheus.InvalidMetricType when stat is accessed as different type metric.# For example, doing stats.inc_value('foo', 1) and then stats.set_value('foo', 2) will raise an error,# Because metric of type Counter was already created for stat foo.PROMETHEUS_SUPPRESS_TYPE_CHECK=False# job label value, applied to all metrics.PROMETHEUS_JOB='scrapy'# default# grouping label dict, applied to all metrics.# by default it is an instance key with hostname value.PROMETHEUS_GROUPING_KEY={'instance':<hostname>}How metrics are createdMetric name is build fromPROMETHEUS_METRIC_PREFIXand stat name, where all/are replaced with_.For example:statfoo: 67whill produce metricscrapy_prometheus_foo{instance="...",job="scrapy",spider="..."}67statfoo/bar: 67whill produce metricscrapy_prometheus_foo_bar{instance="...",job="scrapy",spider="..."}67statfoo/bar/baz: 67whill produce metricscrapy_prometheus_foo_bar_baz{instance="...",job="scrapy",spider="..."}67
Package: scrapy-proxies-tool
Random proxy middleware for Scrapy (http://scrapy.org/)base onhttps://github.com/aivarsk/scrapy-proxies, support  load proxies fromhttps://github.com/qiyeboy/IPProxyPoolProcesses Scrapy requests using a random proxy from list to avoid IP ban and
improve crawling speed.Get your proxy list from sites likehttp://www.hidemyass.com/(copy-paste into text file
and reformat tohttp://host:portformat)InstallThe quick way:pipinstallscrapy-proxies-toolOr checkout the source and runpythonsetup.pyinstallsettings.py# Retry many times since proxies often failRETRY_TIMES=10# Retry on most error codes since proxies fail for different reasonsRETRY_HTTP_CODES=[500,503,504,400,403,404,408]DOWNLOADER_MIDDLEWARES={'scrapy.downloadermiddlewares.retry.RetryMiddleware':90,'scrapy_proxies.RandomProxy':100,'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':110,}PROXY_SETTINGS={# Proxy list containing entries like# http://host1:port# http://username:password@host2:port# http://host3:port# ...# if PROXY_SETTINGS[from_proxies_server] = True , proxy_list is server address (ref https://github.com/qiyeboy/IPProxyPool and https://github.com/awolfly9/IPProxyTool )# Only support http(ref https://github.com/qiyeboy/IPProxyPool#%E5%8F%82%E6%95%B0)# list : ['http://localhost:8000?protocol=0'],'list':['/path/to/proxy/list.txt'],# disable proxy settings and  use real ip when all proxies are unusable'use_real_when_empty':False,'from_proxies_server':False,# If proxy mode is 2 uncomment this sentence :# 'custom_proxy': "http://host1:port",# Proxy mode# 0 = Every requests have different proxy# 1 = Take only one proxy from the list and assign it to every requests# 2 = Put a custom proxy to use in the settings'mode':0}For older versions of Scrapy (before 1.0.0) you have to usescrapy.contrib.downloadermiddleware.retry.RetryMiddlewareandscrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddlewaremiddlewares instead.Your spiderIn each callback ensure that proxy /really/ returned your target page by
checking for site logo or some other significant element.
If not - retry request with dont_filter=Trueifnothxs.select('//get/site/logo'):yieldRequest(url=response.url,dont_filter=True)
Package: scrapy-proxycrawl-middleware
DEPRECATION NOTICE:warning:IMPORTANT:This package is no longer maintained or supported. For the latest updates, please use our new package atscrapy-crawlbase-middleware.ProxyCrawl API middleware for ScrapyProcessesScrapyrequests usingProxyCrawlservices either with Normal or Javascript tokensInstallingChoose a way of installing:Clone the repository inside your Scrapy project and run the following:pythonsetup.pyinstallOr usePyPiPython package manager.pip install scrapy-proxycrawl-middlewareThen in your Scrapysettings.pyadd the following lines:# Activate the middlewarePROXYCRAWL_ENABLED=True# The ProxyCrawl API token you wish to use, either normal of javascript tokenPROXYCRAWL_TOKEN='your token'# Enable the middlewareDOWNLOADER_MIDDLEWARES={'scrapy_proxycrawl.ProxyCrawlMiddleware':610}UsageUse the scrapy_proxycrawl.ProxyCrawlRequest instead of the scrapy built-in Request.
The scrapy_proxycrawl.ProxyCrawlRequest accepts additional arguments, used in Proxy Crawl API:fromscrapy_proxycrawlimportProxyCrawlRequestclassExampleScraper(Spider):defstart_requests(self):yieldProxyCrawlRequest("http://target-url",callback=self.parse_resultdevice='desktop',country='US',page_wait=1000,ajax_wait=True,dont_filter=True)The target url will be replaced with proxy crawl url and parameters will be encoded into the url by the middleware automatically.If you have questions or need help using the library, please open an issue orcontact us.Copyright 2023 ProxyCrawl
Package: scrapy-proxyland-middleware
scrapy-proxyland-middlewareThis middleware lets you useProxylandfor every request you process with Scrapy.SettingsYou need to specify your package credentials for Proxyland in your settings.py or settings object for this middleware to take effect.PROXYLAND={'username':'package_username','password':'package_password'}You need to also enable ScrapyProxylandMiddleware as well as Scrapy's HttpProxyMiddleware.DOWNLOADER_MIDDLEWARES={'scrapy_proxyland_middleware.ScrapyProxylandMiddleware':350,'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':400,}
Package: scrapyproxyport
Proxy Portintegration for Scrapy.
AboutProxy PortPrerequisitesTo use this package you will need a free API key. Get your API keyhere.
Detailed instructionshere.Installation$pipinstallscrapyproxyportGetting StartedYou need to assign an API key.
This can be done either through an environment variable$exportPROXY_PORT_API_KEY=<API_KEY>or inside settings.py.# inside <your_project>/settings.pyPROXY_PORT_API_KEY='<API_KEY>'DOWNLOADER_MIDDLEWARES={# Add middleware with order number right before CookiesMiddleware'scrapyproxyport.middlewares.ProxyMiddleware':898,'scrapy.downloadermiddlewares.cookies.CookiesMiddleware':899,}DOWNLOAD_TIMEOUT=10RETRY_TIMES=20DUPEFILTER_CLASS='scrapy.dupefilters.BaseDupeFilter'Example Scrapy projectscrapy-proxyport-example