Saravanan666/pypi-org huggingface dataset:
Package: scrapy-sqs-exporter
scrapy-sqs-exporterThis is an extension toScrapyto allow exporting of scraped items to an Amazon SQS instance.SetupAfter installing the package, the two classes defined in the library need to be added to the relevant
sections of the settings file:FEED_EXPORTERS = {
  'sqs': 'sqsfeedexport.SQSExporter'
}

FEED_STORAGES = {
  'sqs': 'sqsfeedexport.SQSFeedStorage'
}TheFEED_STORAGESsection uses a URL prefixed withsqsto differentiate it from other URI based storage
options.In the environment we also need to define some keys:AWS_DEFAULT_REGION=eu-central-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
FEED_URI=sqs://foo
FEED_FORMAT=sqsTheAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYare the AWS credentials to be used, andAWS_DEFAULT_REGIONis the region to default to for the SQS instance.FEED_URIis the name of the AWS SQS instance in theAWS_DEFAULT_REGIONregion for example:AWS_DEFAULT_REGION=us-east-1
FEED_URI=sqs://bar
FEED_FORMAT=sqswould refer to a queue namebarin theus-east-1region.Finally, theFEED_FORMAToption makes the Scrapy spiders use the SQSExporter class.
Package: scrapy-sqspipeline
Scrapy SQS PipelineEach time this pipeline receives an item, it sends it to SQS.RequirementsPython 3+Scrapy 1.1+boto3Installationpip3installscrapy-sqspipelineConfigurationAdd the following lines to your Scrapy project settings.ITEM_PIPELINES={'sqspipeline.SQSPipeline':100,}# Either `SQSPIPELINE_QUEUE_URL` or `SQSPIPELINE_QUEUE_NAME` is required.SQSPIPELINE_QUEUE_URL='https://sqs.ap-northeast-1.amazonaws.com/xxxxxxxxxx/scrapy-sqspipeline'# SQSPIPELINE_QUEUE_NAME=''Use AWS CLI'saws configurecommand to set up credentials. Alternatively, you can use Scrapy's settingsAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY.
Package: scrapy-ssdb-spider
scrapy-ssdb-spider对着 scrapy-redis 照葫芦画瓢的作品基于 ssdb 队列的 scrapy 分布式解决方案依赖说明Python 3.6（测试环境）SSDB 1.9.7scrapypyssdb使用说明shell:git clone https://github.com/PickledFish/scrapy-ssdb-spider
python3 setup.py install或者pip install scrapy-ssdb-spider在scrapy项目中:# settings# ssdb服务SSDB_HOST='127.0.0.1'SSDB_PORT=8888# ssdb密码，可选配置#SSDB_PWD = 'your password'# 配置调度器SCHEDULER='scrapy_ssdb_spider.scheduler.Scheduler'# 配置去重类DUPEFILTER_CLASS='scrapy_ssdb_spider.dupefilter.SSDBDupeFilter'# 配置调度队列键(可选)#SCHEDULER_QUEUE_KEY = ''# 配置调度队列类(可选)#SCHEDULER_QUEUE_CLASS = ''# 配置去重队列键#SCHEDULER_DUPEFILTER_KEY = ''# 下面两个配置，如果我先启动了A爬虫，过了半小时启动B爬虫？# 队列被清空了？？？？？我没搞懂，反正scrapy-redis有这个功能，我也搞一个，默认不清空队列# 配置在爬虫开始前清空去重及调度队列（布尔类型）#SCHEDULER_OPEN_CLEAR_QUEUE =# 配置在爬虫结束后清空去重及调度队列（布尔类型）#SCHEDULER_CLOSE_CLEAR_QUEUE =# 编写爬虫fromscrapy_ssdb_spider.spidersimportSsdbSpiderclassTestSpider(SsdbSpider):# 配置种子队列键ssdb_key='start_key'defparse(self,response):pass一切都和scrapy_redis那么像，即使是代码，都很像相信聪明如你，一定没问题的，欢迎提意见差异虽然代码都是参照scrapy-redis写的，但是有些功能并未实现:基于 ssdb 的 Pipeline 没有实现没有爬虫结束或爬虫开始清除队列的配置忘了
Package: scrapy-state
scrapy-stateScrapy state between jobÂ runs. Recreating spider's state from settings
Seehttps://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9dSee CHANGELOG.md for detail description.Getting HelpQuickStartpython3-mpipinstall-Uscrapy-stateInstalling from Githubpython3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-state/archive/master.zipOptionally installing tests requirements.python3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-state/archive/master.zip#egg=alex-ber-utils[tests]Or explicitly:wgethttps://github.com/alex-ber/scrapy-state/archive/master.zip-Omaster.zip;unzipmaster.zip;rmmaster.zipAnd then installing from source (see below).Installing from sourcepython3-mpipinstall.# only installs "required"python3-mpipinstall.[tests]# installs dependencies for testsAlternatively you install install from requirements file:python3-mpipinstall-rreq.txt# only installs "required" (relaxed)python3-mpipinstall-rrequirements.txt# only installs "required"python3-mpipinstall-rrequirements-tests.txt# installs dependencies for testsFrom the directory with setup.pypython3setup.pytest#run all testsorpytestInstalling new versionSeehttps://docs.python.org/3.1/distutils/uploading.htmlpython3setup.pysdistuploadRequirementsscrapy-state requires the following modules.Python 3.6+ChangelogScrapy state between jobÂ runs. Recreating spider's state from settings.
Seehttps://medium.com/@alex_ber/scrapy-state-between-job-runs-b880c7b34a9dAll notable changes to this project will be documented in this file.#https://pypi.org/manage/project/scrapy-state/releases/[Unrelased][0.0.1] - 02/07/2020AddedSpiderSettingsStateclass inalexber.spiderstate.statemodule.
Inspired byscrapy.extensions.spiderstateSee [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.
Package: scrapy-statsd-middleware
Usagepip install scrapy-statsd-middlewareDOWNLOADER_MIDDLEWARES={'statsd_middleware.StatsdMiddleware':543,}SPIDER_MIDDLEWARES={'statsd_middleware.StatsdMiddleware':543,}There’s also a few settings that you can use:STATSD_HOSTNAME - Defaults to the current machine’s hostnameSTATSD_PREFIX - Defaults to “hostname.spider-name.”STATSD_HOST_IP - Defaults to “0.0.0.0”This will increment statsd with the following:
* requests (spider_reqs_issued)
* response (spider_resps_received)
* errors (error_KeyError, where KeyError is whatever the error name is)
* items processed (processed_Product, where Product is whatever the item class name is)Example ImplementationAn example implementation of this middleware is in /example
It includes a docker-compose file that describes how to use this middleware with statsd & graphiteExample Installation & UsageBuild the docker imagesdocker-compose buildStart the statsd containerdocker-compose up -dRun the example spider:docker-compose -f ./example/docker-compose.yml run spider bash -c “cd ./opt/scrapy/dirbot/ && scrapy crawl dmoz”You can see a live graphite dashboard athttp://0.0.0.0/dashboardYou should see stats show up under something like “stats.Z-MacBook-Pro.local.dmoz.spider_reqs_issued”DevelopmentYou can run the tests viamake test
Package: scrapy-status-mailer
Send an email when a crawler finishes or breaks.InstallThe quick way:pip install scrapy-status-mailerOr install from GitHub:pip install git+git://github.com/light4/scrapy-status-mailer.git@masterOr checkout the source and run:python setup.py installsettings.pysettings for send email163qqgmailSTATUSMAILER_RECIPIENTS = []
STATUSMAILER_COMPRESSION = 'gzip'
# STATUSMAILER_COMPRESSION = None

MAIL_FROM = ''
MAIL_HOST = 'smtp.qq.com'
MAIL_PORT = 465
MAIL_USER = ''
MAIL_PASS = ''
MAIL_TLS = True
MAIL_SSL = True

EXTENSIONS = {
    'scrapy_status_mailer.StatusMailer': 80,
}

# all bots send email by default
# BOTS_SEND_EMAIL = ['first_bot', 'second_bot']Thanksstackoverflow
Package: scrapy-sticky-meta-params
scrapy-sticky-meta-paramsA Scrapy spider middleware that forwards meta params through subsequent requests.What does it do?This middleware simplify the process of carrying information through requests and responses on spiders.Without the middlewareclass SampleSpider(Spider):
    name = "without_middleware"
    start_urls = ["https://www.example.com"]

    def parse(self, response):
        for param in range(5):
            yield Request(
                "https://www.example.com/next",
                meta={"param": param},
                callback=self.parse_2
            )

    def parse_2(self, response):
        # Get important information from response
        info = response.xpath("//info/text()").get("info")
        # We need to get the param from meta and forward it again in this request
        param = response.meta["param"]
        yield Request(
            "https://www.example.com/next",
            meta={"info": info, "param": param},
            callback=self.parse_3
        )

    def parse_3(self, response):
        yield {
            "param": response.meta["param"],  # The value that we've extracted in the first callback
            "info": response.meta["info"]
        }With the middlewareclass SampleSpider(Spider):
    name = "with_middleware"
    start_urls = ["https://www.example.com"]
    sticky_meta_keys = ["param"]  # Will always forward the meta param "param"

    def parse(self, response):
        for param in range(5):
            yield Request(
                "https://www.example.com/next",
                meta={"param": param},
                callback=self.parse_2
            )

    def parse_2(self, response):
        # Get important information from response
        info = response.xpath("//info/text()").get("info")
        # We don"t need to get the "param" value from meta and resend it.
        yield Request(
            "https://www.example.com/next",
            meta={"info": info},
            callback=self.parse_3
        )

    def parse_3(self, response):
        yield {
            "param": response.meta["param"],  # The value that we've extracted in the first callback
            "info": response.meta["info"]
        }Awesome, how to use it?To enable the middleware you need to add it to your projects'sSPIDER_MIDDLEWARESsetting insettings.py.SPIDER_MIDDLEWARES = {
    'scrapy_sticky_meta_params.middleware.StickyMetaParamsMiddleware': 550,
}This middleware needs to be enabled per spider, to do this you need to add the following attribute on your spider:sticky_meta_keys = []You need to fill this list with every key that you want to be forwarded to subsequent requests.
Package: scrapy-tdd
scrapy_tddHelpers and examples to build Scrapy Crawlers in a test driven way.Motivation / Why should I develop Scrapy Crawlers using TDD?The develop - test cycle goes down to a few seconds and so it allows you to get a properly
working scraper up much fasterWhen bugs are discovered in “the wild” with real data, new example files, a test and a fix can be created and tested
much fasterIt allows for fast refactoring without breaking anything - which results in much cleaner scraper codeIt just feels right when you are used to be doing TDDWhat’s the difference to Scrapy’s Spiders Contracts?Scrapy has its own builtin testing feature namedSpiders ContractsI tried to use them for some time, but decided to build real unit tests in a unit test framework like py.test because
of these shortcomings:its philosophy is geared towards testing against contracts (thus the name) that by nature are more broad and less
specific concepts. Testing for exact field contents in items can be done, but is difficult and fragileits documentation and basic set of features is a bit thinit mixes implementation code with contract descriptions which is only usable when there are few and simple contractsInstallationpip install scrapy_tddQuick Start Examplesdef describe_fancy_spider():to_test = MySpider().from_crawler(get_crawler())def describe_parse_suggested_terms():resp = response_from(“Result_JSON_Widget.txt”)
results = to_test.parse(resp)def should_get_item():item = results
assert item[0][“lorem”] == ‘ipsum’
assert item[0][“iterem”] == “ipsem”Full Documentation… coming soon …Missing / next stepsMocking Request-Response pairsHow to contribute… coming soon …
Package: scrapy-time-machine
scrapy-time-machineRun your spider with a previously crawled request chain.Installpip install scrapy-time-machineWhy?Lets say your spider crawls some page everyday and after some time you notice that an important information was added and you want to start saving it.You may modify your spider and extract this information from now on, but what if you want the historical value of this data, since it was first introduced to the site?With this extension you can save a snapshot of the site at every run to be used in the future (as long as you don't change the request chain).EnablingTo enable this middlware, add this information to your projects'ssettings.py:DOWNLOADER_MIDDLEWARES = {
    "scrapy_time_machine.timemachine.TimeMachineMiddleware": 901
}

TIME_MACHINE_ENABLED = True
TIME_MACHINE_STORAGE = "scrapy_time_machine.storages.DbmTimeMachineStorage"UsingStore a snapshot of the current state of the sitescrapy crawl sample -s TIME_MACHINE_SNAPSHOT=true -s TIME_MACHINE_URI="/tmp/%(name)s-%(time)s.db"This will save a snapshot at/tmp/sample-YYYY-MM-DDThh-mm-ss.dbRetrieve a snapshot from a previously saved state of the sitescrapy crawl sample -s TIME_MACHINE_RETRIEVE=true -s TIME_MACHINE_URI=/tmp/sample-YYYY-MM-DDThh-mm-ss.dbIf no change was made to the spider between the current version and the version that produced the snapshot, the extracted items should be the same.Sample projectThere is a sample Scrapy project available at theexamplesdirectory.
Package: scrapy-tor-downloader
scrapy-tor-downloaderScrapy middleware with TOR support for more robust scrapers or anonymous scraping.Dependencies :globe_with_meridians:Python 3.7Scrapy 2.4.0torpy 1.1.6Installation :inbox_tray:This is a python package hosted on pypi, so to install simply run the following command:pip install scrapy-tor-downloaderSettingsTOR_PROXY_ENABLEDWhether TOR is used to proxy any request (defaults to false).Meta field to enable/disable this per request is:tor_proxy_enabledTOR_FALLBACK_ENABLEDWhether TOR is used when a request fails as a fallback (defaults to true).Meta field to enable/disable this per request is:tor_fallback_enabledTOR2WEB_PROXYWhether ator2webproxy is used for onion address. The value of this setting is the domain for the proxy.Meta field to add this per request is:tor2web_proxytor_reset_sessionWhether to reset the TOR session before processing the request. This field only exists in the meta on the request astor_reset_sessionand is a boolean.Usage example :eyes:In order to use this plugin simply add the following settings and substitute your variables:DOWNLOADER_MIDDLEWARES={"tormiddleware.middleware.TORDownloaderMiddleware":631}This will immediately allow you begin using TOR as a fallback when one of your requests fail. In order to use it as a proxy you can add the following to your settings:TOR_PROXY_ENABLED=TrueThis will make every request hit TOR for a response. If you have turned the proxy on the TOR fallback is ignored, however if it is off the fallback is still on by default, which means if a request returns an error it will be tried again on TOR. In order to turn this off add the following to your settings:TOR_FALLBACK_ENABLED=FalseIf you want to make use oftor2webproxies for onion addresses, you can add it to the settings like so:TOR2WEB_PROXY="https://onion.moe"License :memo:The project is available under theMIT License.
Package: scrapy-tor-proxy-rotation
scrapy-tor-proxy-rotationThe purpose of this module is to allow rotation of IPs toScrapyvia Tor.InstallationSimple way to install, viapip:pipinstallscrapy-tor-proxy-rotationConfiguring TorYou need to configureTor. First, install it:sudoapt-getinstalltorStop its execution to perform configuration:sudoservicetorstopOpen your configuration file as root, available at/etc/tor/torrc, for example using nano:sudonano/etc/tor/torrcInsert the lines below and save:ControlPort 9051
CookieAuthentication 0Restart Tor:sudoservicetorstartYou can check your machine's IP and compare it with Tor's by doing the following:To see your machine's IP:curlhttp://icanhazip.com/To see Tor's IP:torifycurlhttp://icanhazip.com/Tor proxies are not supported by Scrapy. To get around this problem, it is necessary to use an intermediary, in this casePrivoxy.The Tor proxy server is by default at 127.0.0.1:9050Installing and configuringPrivoxy:Install:sudoaptinstallprivoxyStop its execution:sudoserviceprivoxystopConfigure it to use TOr, open its configuration file:sudonano/etc/privoxy/configAdd the following lines:forward-socks5t/127.0.0.1:9050.Start it up:service privoxy startBy default, privoxy will run at the address 127.0.0.1:8118Test:torifycurlhttp://icanhazip.com/curl-x127.0.0.1:8118http://icanhazip.com/The IP shown in the two steps above must be the same.How to useAfter you have made these settings, you can now integrate Tor with Scrapy.Configure the middleware in your project's configuration file (settings.py):DOWNLOADER_MIDDLEWARES={...,'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':110,'tor_ip_rotator.middlewares.TorProxyMiddleware':100}Enable the use of extension:TOR_IPROTATOR_ENABLED=TrueTOR_IPROTATOR_CHANGE_AFTER=#number of requests made on the same Tor's IP addressBy default, an IP can be reused after 10 other uses. This value can be changed by the variable TOR_IPROTATOR_ALLOW_REUSE_IP_AFTER, as below:TOR_IPROTATOR_ALLOW_REUSE_IP_AFTER=0#another integer valueA number too large for TOR_IPROTATOR_ALLOW_REUSE_IP_AFTER may make it slower to retrieve a new IP for use or not find one at all. If the value is 0, there will be no record of used IPs.
Package: scrapy-tor-proxy-rotator
Scrapy Tor Proxy RotationThis module allows Scrapy to rotate Tor IPs.InstallSimple install, viapip:pipinstallscrapy-tor-proxy-rotationConfig TorTo configureTor. First, install :sudoapt-getinstalltorStop its execution to make configurations:sudoservicetorstopOpen your configuration file as root, available in/etc/tor/torrc, for example, using nano:sudonano/etc/tor/torrcPlace the lines below and save:ControlPort 9051
CookieAuthentication 0Restart Tor:sudoservicetorstartIt is possible to verify the IP of your machine and compare it as Tor in the following way:To see your IP:curlhttp://icanhazip.com/To see the ip of Tor:torifycurlhttp://icanhazip.com/For Scrapy it is necessary to use an intermediary, in this case orPrivoxy.Tor Default Proxy Server: 127.0.0.1:9050Install and ConfigPrivoxy:Install:sudoaptinstallprivoxyStop the service:sudoserviceprivoxystopOpen the config file:sudonano/etc/privoxy/configAdd the following lines:forward-socks5t/127.0.0.1:9050.Start the service:service privoxy startTest:torifycurlhttp://icanhazip.com/curl-x127.0.0.1:8118http://icanhazip.com/UseAfter performing these configurations, it is possible to integrate Tor with Scrapy.Configure the middleware in your settings file (settings.py):DOWNLOADER_MIDDLEWARES={...,'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':110,'tor_ip_rotator.middlewares.TorProxyMiddleware':100}Add those in your custom_settings in your spider or in (settings.py) if you want to use them on all spiders from the project:TOR_IPROTATOR_ENABLED=TrueTOR_IPROTATOR_CHANGE_AFTER=#número de requisições feitas em um mesmo endereço IPBy default, an IP can be reused after 10 other uses. This value can be altered by the variable TOR_IPROTATOR_ALLOW_REUSE_IP_AFTER, as below:TOR_IPROTATOR_ALLOW_REUSE_IP_AFTER=#A large number can also make it slower to retrieve a new IP to use or find. If the value is 0, there will be no record of used IPs.
Package: scrapy-twostage
Have you ever written a web scraper, only to find out after
a long time that there’s some extra data on the pages you
should’ve been scraping all along?Or a change on a website means your scraper stops working,
and you lose days or weeks of data until you can find the
time to fix it?This library aims to solve this problem by splitting aScrapyscraper up into two asynchronous stages:Download stage- The website is crawled, and the pages to
be scraped are downloaded and saved to disk.Extract stage- The pages to be scraped are loaded from disk.
The desired data is extracted from the pages and exported (e.g. to
a file or database).The crawler logic for the download stage should be kept as simple
as possible. It would typically open a known URL and perform very
simple actions such as clicking a “next page” button or submitting
a search query. This reduces the risk of the downloader breaking if
there are minor changes made to the website.And since all of the raw data is being saved, if you ever decide to
change your extractor logic, you can simply re-run the extractor on
all of the data that has been downloaded.InstallationDownloading and installing from PyPITo install usingpip:$ pip install scrapy-twostageOr to install usingeasy_install:$ easy_install scrapy-twostageDownloading and installing from sourceDownload the latest version ofscrapy-twostagefromhttp://pypi.python.org/pypi/scrapy-twostage/.You can install it by doing the following:$ tar xvfz scrapy-twostage-0.0.0.tar.gz
$ cd scrapy-twostage-0.0.0
# python setup.py install # as rootUsing the development versionYou can clone the git repository by doing the following:$ git clone git://github.com/acordiner/scrapy-twostage.gitUsing scrapy-twostageComing soon…Bug trackerIf you have any suggestions, bug reports or annoyances please report them
athttp://github.com/acordiner/scrapy-twostage/issues/LicenseThis software is licensed under theGPL v2 License. See theLICENSEfile in the top distribution directory for the full license text.
Package: scrapy-tw-rental-house
TW Rental House Utility for ScrapyThis package is built for crawling Taiwanese rental house related website usingScrapy.
As behaviour of crawlers may differ from their goal, scale, and pipeline, this package provides only minimum feature set, which allow developer to list and decode a rental house web page into structured data, without knowing too much about detail HTML and API structure of each website. In addition, this package is also designed for extensibility, which allow developers to insert customized callback, manipulate data, and integrate with existing crawler structure.Although this package provide the ability to crawl rental house website, it's developer's responsibility to ensure crawling mechanism and usage of data. Please be friendly to target website, such as consider usingDOWNLOAD_DELAYorAUTO_THROTTLINGto prevent bulk requesting.RequirementPython 3.10+Installationpoetryaddscrapy-tw-rental-houseBasic UsageThis package currently support591. Each rental house website is a Scrapy Spider class. You can either crawl entire website using default setting , which will take couple days, or customize the behaviour base on your need.The most basic usage would be creating a new Spider class that inherit Rental591Spider:fromscrapy_twrh.spiders.rental591importRental591SpiderclassMyAwesomeSpider(Rental591Spider):name='awesome'And than start crawling byscrapycrawlawesomePlease seeexamplefor detail usage.ItemsAll spiders populates 2 type of Scrapy items:GenericHouseItemandRawHouseItem.GenericHouseItemcontains normalized data field, spirders from different website will decode their data and fit into this schema in best effort.RawHouseItemcontains unnormalized data field, which keep original and structured data in best effort.Note that both item are super set of schema. It developer's responsibility to check which field is provided when receiving an item.
For example, inRental591Spider, for a single rental house, Scrapy will get:1xRawHouseItem+ 1xGenericHouseItemduring listing all houses, which provide only minimun data field forGenericHouseItem1xRawHouseItem+ 1xGenericHouseItemduring retrieving house detail.HandlersAll spiders in this package provide the following handlers:start_list, similiar tostart_requestsin Scrapy, control how crawler issue search/list request to find all rental houses.parse_list, similiar toparsein Scrapy, control how crawler handles response fromstart_listand generate request for detail house info page.parse_detail, control how crawler parse detail page.All spiders implements their own default handler, say,default_start_list,default_parse_list, anddefault_parse_detail, and can be overwrite during__init__. Please seeexamplefor how to control spider behavior using handlers.