Saravanan666/pypi-org huggingface dataset:
Package: scrapy-rss-exporter
Generate an RSS feed using theScrapyframework.Table of ContentsInstallationUsageFeed ItemsGlobal ExporterPer Spider ExporterInstallationInstallscrapy-rss-exporterusingpip:pipinstallscrapy-rss-exporteror usingsetuptools:pythonsetup.pyinstallUsageFeed ItemsThe most convenient way to use the exporter is to return the objects ofRssItemclass from your spiders. This class derives fromscrapy.Item, so it will work with other exporters as well.You will need to set the following keys:fromscrapy_rss_exporter.itemsimportRssItem,Enclosurerss_item=RssItem()rss_item['title']='Item title'rss_item['link']='Item url'rss_item['guid']='Item ID'rss_item['description']='Item Description'rss_item['pub_date']=Nonerss_item['enclosure']=[Enclosure(url=img,type='image/jpeg')]Thepub_datefield should contain a date in theRFC882format. If you useNone, the system will insert the current date
in the appropriate format. Theenclosurefield is optional and should
contain a (possibly empty) list ofEnclosureobjects.Global ExporterTo set the exporter up globally, you need to declare it in theFEED_EXPORTERSdictionary in thesettings.pyfile:FEED_EXPORTERS={'rss':'scrapy_rss_exporter.exporters.RssItemExporter'}You can then use it as aFEED_FORMATand specify the output file in theFEED_URI:FEED_FORMAT='rss'FEED_URI='s3://my-feeds/my-feed.rss'Note:Bear in mind that, if you use a local file as output,scrapywill append to an existing file resulting with an invalid RSS code. You should,
therefore, make sure to delete any existing output file before running the
spider. Thes3storage does not have this problem becausescrapyuploads are using theS3 PutObjectmethod.scrapydoes not seem to allow to push any configuration option to an
exporter. Therefore, if you want to customize the feed title and other metadata,
you need to create a subclass and update theFEED_EXPORTERSdictionary
with the new class name:classMyRssExporter(RssItemExporter):def__init__(self,*args,**kwargs):kwargs['title']='My RSS'kwargs['link']='https://www.mywebsite.com'kwargs['description']='My RSS Items'super(MyRssExporter,self).__init__(*args,**kwargs)Per Spider ExporterYou can, of course, specify a different exporter with different settings for
each spider. Just use thecustom_settingsfield to override the global
configuration fields:classMySpider(scrapy.Spider):name="my"start_urls=['https://www.mywebsite.com']custom_settings={'FEED_EXPORTERS':{'rss':'project.spiders.my_spider.MyExporter'},'FEED_FORMAT':'rss','FEED_URI':'s3://my-feeds/my-feed.rss',}defparse(self,response):pass
Package: scrapyrt
ScrapyRT (Scrapy realtime)Add HTTP API for yourScrapyproject in minutes.You send a request to ScrapyRT with spider name and URL, and in response, you get items collected by a spider
visiting this URL.All Scrapy project components (e.g. middleware, pipelines, extensions) are supportedYou run Scrapyrt in Scrapy project directory. It starts HTTP server allowing you to schedule spiders and get spider output in JSON.Quickstart1. install>pipinstallscrapyrt2. switch to Scrapy project (e.g. quotesbot project)>cdmy/project_path/is/quotesbot3. launch ScrapyRT>scrapyrt4. run your spiders>curl"localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"5. run more complex query, e.g. specify callback for Scrapy request and zipcode argument for spider>curl--data'{"request": {"url": "http://quotes.toscrape.com/page/2/", "callback":"some_callback"}, "spider_name": "toscrape-css", "crawl_args": {"zipcode":"14000"}}'http://localhost:9080/crawl.json-vScrapyrt will look forscrapy.cfgfile to determine your project settings,
and will raise error if it won’t find one.  Note that you need to have all
your project requirements installed.NoteProject is not a replacement forScrapydorScrapy Cloudor other infrastructure to run long running crawlsNot suitable for long running spiders, good for spiders that will fetch one response from some website and return items quicklyDocumentationDocumentation is available on readthedocs.SupportOpen source support is provided here in Github. Pleasecreate a question
issue(ie. issue with “question” label).Commercial support is also available byZyte.LicenseScrapyRT is offered underBSD 3-Clause license.DevelopmentDevelopment taking place onGithub.
Package: scrapyrt-with-params
ScrapyRT (Scrapy realtime)Add HTTP API for yourScrapyproject in minutes.You send a request to ScrapyRT with spider name and URL, and in response, you get items collected by a spider
visiting this URL.All Scrapy project components (e.g. middleware, pipelines, extensions) are supportedYou run Scrapyrt in Scrapy project directory. It starts HTTP server allowing you to schedule spiders and get spider output in JSON.Quickstart1. install>pipinstallscrapyrt2. switch to Scrapy project (e.g. quotesbot project)>cdmy/project_path/is/quotesbot3. launch ScrapyRT>scrapyrt4. run your spiders>curl"localhost:9080/crawl.json?spider_name=toscrape-css&url=http://quotes.toscrape.com/"5. run more complex query, e.g. specify callback for Scrapy request and zipcode argument for spider>curl--data'{"request": {"url": "http://quotes.toscrape.com/page/2/", "callback":"some_callback"}, "spider_name": "toscrape-css", "crawl_args": {"zipcode":"14000"}}'http://localhost:9080/crawl.json-vScrapyrt will look forscrapy.cfgfile to determine your project settings,
and will raise error if it won’t find one.  Note that you need to have all
your project requirements installed.NoteProject is not a replacement forScrapydorScrapy Cloudor other infrastructure to run long running crawlsNot suitable for long running spiders, good for spiders that will fetch one response from some website and return items quicklyDocumentationDocumentation is available on readthedocs.SupportOpen source support is provided here in Github. Pleasecreate a question
issue(ie. issue with “question” label).Commercial support is also available byZyte.LicenseScrapyRT is offered underBSD 3-Clause license.DevelopmentDevelopment taking place onGithub.
Package: scrapyrwiki
A collection of helpers for running scrapers built withScrapyinScraperWikiLaunch scraper without scrapy CLIExample:fromscrapy.confimportsettingsfromscrapyrwikiimportrun_spiderdefmain():run_spider(MySpider(),settings)if__name__=='__main__':main()Save produced data to ScraperWikiJust add “scrapyrwiki.pipelines.ScraperWikiPipeline” to ITEM_PIPELINESExample:fromscrapy.confimportsettingsfromscrapyrwikiimportrun_spiderdefscraperwiki():options={'SW_SAVE_BUFFER':5,'SW_UNIQUE_KEYS':{"MyItem":['url']},'ITEM_PIPELINES':['scrapyrwiki.pipelines.ScraperWikiPipeline'],}settings.overrides.update(options)run_spider(MySpider(),settings)if__name__=='scraper':scraperwiki()Check spider contracts in CIJust launch spider with run_testsExample:fromscrapyrwikiimportrun_testsfromscrapy.confimportsettingsrun_tests(MySpider(),"output.xml",settings)Note: For testing the HTTP cache is used. In the directory where the script is
launched there must be a scrapy.cfg (needed by Scrapy to identify that’s a scraper
directory) and a .scrapy directory with the HTTP cache db.The output is in XUnit format, tested onJenkinsLog scraper errors to SentryInstallscrapy-sentryand set the
environment variable SENTRY_DSN with the Sentry key. Scrapyrwiki will handle
everything for you.
Package: scrapy-s3logstorage
Scrapy S3 Log StorageDEPRECATEDThis package is deprecated. Please usescrapy-logexportinstead.DescriptionA Scrapy extension to upload log files to S3.If you're already exporting your feeds to S3 this extension uses many of the same settings. After adding this extension to yourEXTENSIONSsetting, just set theLOG_FILE,S3_LOG_BUCKETand an optionalS3_LOG_ACLsettings and you're good to go.InstallationYou can install scrapy-s3logstorage using pip:pip install scrapy-s3logstorageConfigurationThis extension still requires that a local log file is written. Once scrapy's engine has stopped, the extension will upload the log file to S3 and optionally delete the local file.Enable the extension by adding it to yoursettings.py:from environs import Env

    env = Env()  
    env.read_env() 

    EXTENSIONS = {
        'scrapy_s3logstorage.extension.S3LogStorage': 0,
    }

    LOG_FILE = 'scrapy.log' # Must be a local file
    S3_LOG_BUCKET = 'my-bucket' # Bucket name to store logs
    S3_LOG_DELETE_LOCAL = True # Delete local log file after upload, defaults to False

    # If AWS CLI is configured, and you're using the same credentials the following settings are optional
    AWS_ACCESS_KEY_ID = env("AWS_ACCESS_KEY_ID")
    AWS_SECRET_ACCESS_KEY = env("AWS_SECRET_ACCESS_KEY")
    AWS_SESSION_TOKEN = env("AWS_SESSION_TOKEN") # if required
    AWS_ENDPOINT_URL = None  # or your endpoint URL

    # S3_LOG_ACL takes priority over FEED_STORAGE_S3_ACL.
    # If S3_LOG_ACL is not set, FEED_STORAGE_S3_ACL will be used.
    # Setting one or both of these settings is optional.

    S3_LOG_ACL = ''  # or other S3 ACL
    FEED_STORAGE_S3_ACL = '' # or other S3 ACL
Package: scrapy-save-as-pdf
Pipeline to Download PDF or Save page as PDF for scrapy itemInstallationInstallscrapy-save-as-pdfusingpip:pip install scrapy-save-as-pdfConfiguration(Optionally)if you want to useWEBDRIVER_HUB_URL, you can usedockerto setup one like this:dockerrun-d-p4444:4444-v/dev/shm:/dev/shmselenium/standalone-chrome:4.0.0-alpha-7-20201119thenWEBDRIVER_HUB_URLvalue ishttp://docker_host_ip:4444/wd/huband we often debug on local host, so we usehttp://127.0.0.1:4444/wd/hubAdd thesettings.pyof your Scrapy project like this:PROXY=""CHROME_DRIVER_PATH='/snap/bin/chromium.chromedriver'PDF_SAVE_PATH="./pdfs"PDF_SAVE_AS_PDF=FalsePDF_DOWNLOAD_TIMEOUT=60PDF_PRINT_OPTIONS={'landscape':False,'displayHeaderFooter':False,'printBackground':True,'preferCSSPageSize':True,}WEBDRIVER_HUB_URL='http://127.0.0.1:4444/wd/hub'If bothWEBDRIVER_HUB_URLandCHROME_DRIVER_PATHare set, we useWEBDRIVER_HUB_URL.Enable the pipeline by adding it toITEM_PIPELINESin yoursettings.pyfile and changing priority:ITEM_PIPELINES={'scrapy_save_as_pdf.pipelines.SaveAsPdfPipeline':-1,}The order should before your persist pipeline such as save to database and after your preprocess pipeline.In the demo scrapy project, I put theSaveToQiniuPipelineafter this plugin to persist pdf to the cloud.Usageset thepdf_urland/orurlfield in your yielded itemimportscrapyclassMySpider(scrapy.Spider):start_urls=["http://example.com",]defstart_requests(self):forurlinself.start_urls:yieldscrapy.Request(url,self.parse)defparse(self,response):yield{"url":"http://example.com/cate1/page1.html","pdf_url":"http://example.com/cate1/page1.pdf",}yield{"url":"http://example.com/cate1/page2.html","pdf_url":"http://example.com/cate1/page2.pdf",}thepdf_urlfield will be populated with the downloaded pdf file location, ifpdf_urlfield has old value then move it toorigin_pdf_urlfield, you can handle them in your next pipeline.Getting helpPlease use github issueContributingPRs are always welcomed.Changes0.1.0 (2020-12-25)Initial release
Package: scrapy-save-statistics
Save statistics to mongo for analytics.InstallThe quick way:pip install scrapy-save-statisticsOr install from GitHub:pip install git+git://github.com/light4/scrapy-save-statistics.git@masterOr checkout the source and run:python setup.py installsettings.pyMongodb settings for save statistics, need astatisticsdatabase.MONGO_HOST = "127.0.0.1"
MONGO_PORT = 27017
MONGO_DB = "myspider"
MONGO_STATISTICS = "statistics"

EXTENSIONS = {
    'scrapy_save_statistics.SaveStatistics': 100,
}SpiderSpider must havestatisticsattributes and contains spider_url.
We’ll save that info to mongodb.class TestSpider(scrapy.Spider):
    name = "test"

    def __init__(self, name=None, **kwargs):
        super(TestSpider, self).__init__(name=name, **kwargs)
        self.statistics = []

    def parse(self, response):
        crawl_info = {'spider_url': spider_url,
                      'expected_crawl_num': expected_crawl_num,
                      'pages': total_page}
        self.statistics.append(crawl_info)
Package: scrapy-save-to-qiniu
Pipeline to save scrapy item field attachment to qiniuInstallationInstall scrapy-save-to-qiniu using pip::$ pip install scrapy-save-to-qiniuConfigurationAdd thesettings.pyof your Scrapy project like this:QINIU_AK=''QINIU_SK=''QINIU_BUCKET=''QINIU_DOMAIN=''QINIU_DEL_SRC=TrueQINIU_FIELDS=['pdf_url',]Enable the pipeline by adding it toITEM_PIPELINESin yoursettings.pyfile and changing HttpCompressionMiddleware
priority:ITEM_PIPELINES={'scrapy_save_to_qiniu.pipelines.SaveToQiniuPipeline':-1,}The order should before your persist pipeline such as save to database and after your preprocess pipeline.Usageno need to change your codeGetting helpPlease use github issueContributingPRs are always welcomed.Changes0.1.0 (2020-12-26)Initial release
Package: scrapy-scraper
Scrapy ScraperWeb crawler and scraper based on Scrapy and Playwright's headless browser.To use the headless browser specify-poption. Browsers, unlike other standard web request libraries, have the ability to render JavaScript encoded HTML content.To automatically download and beautify all JavaScript files, including minified ones, specify-dir downloadsoption - wheredownloadsis your desired output directory.Resources:scrapy.org(official)playwright.dev(official)scrapy/scrapy(GitHub)scrapy-plugins/scrapy-playwright(GitHub)Tested on Kali Linux v2023.3 (64-bit).Made for educational purposes. I hope it will help!Table of ContentsHow to InstallHow to Build and Install ManuallyHow to RunUsageHow to Installpip3install--upgradescrapy-scraper

playwrightinstallchromiumHow to Build and Install ManuallyRun the following commands:gitclonehttps://github.com/ivan-sincek/scrapy-scraper&&cdscrapy-scraper

python3-mpipinstall--upgradebuild

python3-mbuild

python3-mpipinstalldist/scrapy-scraper-1.4-py3-none-any.whl

playwrightinstallchromiumHow to RunRestricted (auto throttling and domain whitelisting is on):scrapy-scraper -u https://example.com/home -o results.txt -a random -dir js -lUnrestricted (auto throttling and domain whitelisting is off):scrapy-scraper -u https://example.com/home -o results.txt -a random -dir js -at off -w off -lUsageScrapy Scraper v1.4 ( github.com/ivan-sincek/scrapy-scraper )

Usage:   scrapy-scraper -u urls                     -o out         [-dir directory]
Example: scrapy-scraper -u https://example.com/home -o results.txt [-dir downloads]

DESCRIPTION
    Crawl and scrape websites
URLS
    File with URLs or a single URL to start crawling and scraping from
    -u, --urls = urls.txt | https://example.com/home | etc.
WHITELIST
    File with whitelisted domains to limit the crawling scope
    Specify 'off' to disable domain whitelisting
    Default: domains extracted from URLs
    -w, --whitelist = whitelist.txt | off | etc.
LINKS
    Include all [3rd party] links and sources in the output file
    -l, --links
PLAYWRIGHT
    Use Playwright's headless browser
    -p, --playwright
CONCURRENT REQUESTS
    Number of concurrent requests
    Default: 30
    -cr, --concurrent-requests = 15 | 45 | etc.
CONCURRENT REQUESTS PER DOMAIN
    Number of concurrent requests per domain
    Default: 10
    -crd, --concurrent-requests-domain = 5 | 15 | etc.
AUTO THROTTLE
    Auto throttle crawling speed
    Specify value lesser than 1 to decrease the speed
    Specify value greater than 1 to increase the speed
    Specify 'off' to disable auto throttling
    Default: 1
    -at, --auto-throttle = 0.5 | 1.5 | off | etc.
RECURSION
    Recursion depth limit
    Specify '0' for no limit
    Default: 1
    -r, --recursion = 0 | 2 | 4 | etc.
USER AGENT
    User agent to use
    Default: Scrapy Scraper/1.4
    -a, --user-agent = curl/3.30.1 | random | etc.
PROXY
    Web proxy to use
    -x, --proxy = http://127.0.0.1:8080 | etc.
DIRECTORY
    Output directory
    All extracted JavaScript files will be saved in this directory
    -dir, --directory = downloads | etc.
OUT
    Output file
    -o, --out = results.txt | etc.
Package: scrapy-scraperapi-middleware
scrapy-scraperapi-middlewareThis middleware lets you useScraper APIfor every request you process with Scrapy.You could instead set theproxyfield of your request's meta attribute tohttp://scraperapi.your=options:your_key@proxy-server.scraperapi.com:8001and enableHttpProxyMiddleware.This middleware's only purpose is to achieve this in a more convenient manner.Installation$ pip install scrapy-scraperapi-middlewareSettingsYou need to specify your key for Scraper API in your settings.py or settings object.SCRAPERAPI_KEY='your_key'You also need to enable ScrapyScraperAPIMiddleware as well as Scrapy's HttpProxyMiddleware.DOWNLOADER_MIDDLEWARES={'scrapy_scraperapi_middleware.ScrapyScraperAPIMiddleware':350,'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware':400,}Additional optionsScraper API supports additional options as documentedhere.
You could use these options by adding them to your settings as dictionary.SCRAPERAPI_OPTIONS={'render':'true','country_code':'us'}Affiliate linkIf this example is helpful to you and you do not yet have a subscription to Scraper API, consider usingmy affiliate linkif you plan on getting one. Be aware that all other links to ScraperAPI in this Readme are also affiliate links.
Package: scrapy-scrapingbee
Scrapy ScrapingBee MiddlewareIntegrate Scrapy with ScrapingBee API to use headless browsers for JavaScript and proxy rotation. Requires to create an account onscrapingbee.comto get an API key.Installationpip install scrapy-scrapingbeeConfigurationAdd yourSCRAPINGBEE_API_KEYand theScrapingBeeMiddlewareto your project settings.py. Don't forget to setCONCURRENT_REQUESTSaccording to yourScrapingBee plan.SCRAPINGBEE_API_KEY='REPLACE-WITH-YOUR-API-KEY'DOWNLOADER_MIDDLEWARES={'scrapy_scrapingbee.ScrapingBeeMiddleware':725,}CONCURRENT_REQUESTS=1UsageInherit your spiders fromScrapingBeeSpiderand yield aScrapingBeeRequest.ScrapingBeeSpider overrides the default logger to hide your API key in the Scrapy logs.Below you can see an example from the spider inhttpbin.py.fromscrapy_scrapingbeeimportScrapingBeeSpider,ScrapingBeeRequestJS_SNIPPET='window.scrollTo(0, document.body.scrollHeight);'classHttpbinSpider(ScrapingBeeSpider):name='httpbin'start_urls=['https://httpbin.org',]defstart_requests(self):forurlinself.start_urls:yieldScrapingBeeRequest(url,params={# 'render_js': False,# 'block_ads': True,# 'block_resources': False,# 'js_snippet': JS_SNIPPET,# 'premium_proxy': True,# 'country_code': 'fr',# 'return_page_source': True,# 'wait': 3000,# 'wait_for': '#swagger-ui',},headers={# 'Accept-Language': 'En-US',},cookies={# 'name_1': 'value_1',})defparse(self,response):...You can passScrapingBee parametersin the params argument of a ScrapingBeeRequest. Headers and cookies are passed like a normal Scrapy Request. ScrapingBeeRequest formats all parameters, headers and cookies to the format expected by the ScrapingBee API.ExamplesAdd your API key tosettings.py.To run the examples you need to clone this repository. In your terminal, go toexamples/httpbin/httpbinand run the example spider with:scrapycrawlhttpbin
Package: scrapy-scraping-link
Scrapy ScrapingLink MiddlewareAcknowledgementsThanks toarimbrandScrapingBee, this is adaptation of their work.Installationpip install scrapy-scraping-linkConfigurationAdd yourScrapingLink_API_KEYand theScrapingLinkMiddlewareto your project settings.py. Don't forget to setCONCURRENT_REQUESTSaccording to yourScrapingLink plan.SCRAPINGLINK_API_KEY='REPLACE-WITH-YOUR-API-KEY'DOWNLOADER_MIDDLEWARES={'scrapy_scraping_link.ScrapingLinkMiddleware':700,}CONCURRENT_REQUESTS=1UsageInherit your spiders fromScrapingLinkSpiderand yield aScrapingLinkRequest.Below you can see an example from the spider inparascrapear.py.fromscrapyimportSpiderfromscrapy_scraping_linkimportScrapingLinkSpider,ScrapingLinkRequestclassParascrapearSpider(Spider):name='parascrapear'allowed_domains=['parascrapear.com']start_urls=['http://parascrapear.com/']defparse(self,response):print('Parseando '+response.url)next_urls=response.css('a::attr(href)').getall()fornext_urlinnext_urls:ifnext_urlisnotNone:yieldScrapingLinkRequest(response.urljoin(next_url))sentences=response.css('q::text').getall()forsentenceinsentences:print(sentence)You can passScrapingLink parametersin the params argument of a ScrapingLinkRequest. Headers and cookies are passed like a normal Scrapy Request. ScrapingLinkRequests formats all parameters, headers and cookies to the format expected by the API.ExamplesAdd your API key tosettings.py.To run the examples you need to clone this repository. In your terminal, go toexamples/parascrapear/parascrapearand run the example spider with:scrapyrunspiderparascrapear.pyCustomer SupportSimply reach out to us viaTelegram Groupor or write us anemail.Sign up for our free planto get a free API key loaded with 100 free credits. No credit card required!
Package: scrapyscript
Embed Scrapy jobs directly in your codeWhat is Scrapyscript?Scrapyscript is a Python library you can use to runScrapyspiders directly from your code. Scrapy is a great framework to use for scraping projects, but sometimes you don't need the whole framework, and just want to run a small spider from a script or aCeleryjob. That's where Scrapyscript comes in.With Scrapyscript, you can:wrap regular ScrapySpidersin aJobload theJob(s)in aProcessorcallprocessor.run()to execute them... returning all results when the last job completes.Let's see an example.importscrapyfromscrapyscriptimportJob,Processorprocessor=Processor(settings=None)classPythonSpider(scrapy.spiders.Spider):name="myspider"defstart_requests(self):yieldscrapy.Request(self.url)defparse(self,response):data=response.xpath("//title/text()").extract_first()return{'title':data}job=Job(PythonSpider,url="http://www.python.org")results=processor.run(job)print(results)[{"title":"Welcome to Python.org"}]See theexamplesdirectory for more, including a completeCeleryexample.InstallpipinstallscrapyscriptRequirementsLinux or MacOSPython 3.8+Scrapy 2.5+APIJob (spider, *args, **kwargs)A single request to call a spider, optionally passing in *args or **kwargs, which will be passed through to the spider constructor at runtime.# url will be available as self.url inside MySpider at runtimemyjob=Job(MySpider,url='http://www.github.com')Processor (settings=None)Create a multiprocessing reactor for running spiders. Optionally provide ascrapy.settings.Settingsobject to configure the Scrapy runtime.settings=scrapy.settings.Settings(values={'LOG_LEVEL':'WARNING'})processor=Processor(settings=settings)Processor.run(jobs)Start the Scrapy engine, and execute one or more jobs. Blocks and returns consolidated results in a single list.jobscan be a single instance ofJob, or a list.results=processor.run(myjob)orresults=processor.run([myjob1,myjob2,...])A word about Spider outputsAs per thescrapy docs, aSpidermust return an iterable ofRequestand/ordictorItemobjects.Requests will be consumed by Scrapy inside theJob.dictorscrapy.Itemobjects will be queued
and output together when all spiders are finished.Due to the way billiard handles communication between processes, eachdictorItemmust be
pickle-able using pickle protocol 0.It's generally best to outputdictobjects from your Spider.ContributingUpdates, additional features or bug fixes are always welcome.SetupInstallPoetrygit clone git@github.com:jschnurr/scrapyscript.gitpoetry installTestsmake testormake toxVersion HistorySeeCHANGELOG.mdLicenseThe MIT License (MIT). See LICENCE file for details.