Saravanan666/pypi-org huggingface dataset:
Package: scrapspyder
ScrapSpyderScrapSpyderis a Python package that allows you to scrape and download files from a website.InstallationTo installScrapSpyder, simply run:pip install scrapspyderUsageTo useScrapSpyder, create an instance of theScraperclass and pass in the URL of the website you want to scrape. You can also specify the destination folder where the downloaded files will be saved, the depth of the crawl, whether to perform a deep scan for assets, and more. Here's an example:fromscraperimportScrapersc=Scraper(source="https://www.example.com/",dest="example",depth=1,deep_scan=True,verbose=True,patterns_to_trim=["some-pattern"])sc.download()This will crawl the website and download all the files to a folder called example.RequirementsThis package requires the following Python libraries:requestsbeautifulsoup4ContributingIf you find a bug or would like to contribute to the development ofScrapSpyder, please open an issue or pull request onGitHub repository.LicenseScrapSpyderis licensed under the MIT License. SeeLICENSEfor more information.
Package: scrap-stackoverflow
Scrap Stack OverflowThe project is to scrap stack overflow website.UsageThis libary can be used when the key interest is to retrieve information from stackoverflow questions page.The package has enums for each of the available stack overflow questions tabs and scrapes 50 questions per pageSample codesfromscrapsoimportStackTab,ScrapStackOverflow#get questions from ACTIVE TABst=ScrapStackOverflow(tab=StackTab.ACTIVE)#pagestart option is defaulted to 1, but pagination can be supplieddata=st.scrap(100000,pagestart=2001)#to return the scrapped urlst.scrapped_url
Package: scraptty
### A Python Based Ptt Crawler which integrates with sql databases and crawlers ###scra-ptt-y is a web crawler which integrates SQL databases__asyncio provides infrastructure for writing single-threaded concurrent code using coroutines, multiplexing I/O access over sockets and other resources, running network clients and servers, and other related primitives.__Python version requirement: python 3.4You could try this command.` $> ./scrapttyd-bGossip-s`
Package: scrap-utils
Scrap UtilsThis is small package that contains some code regularly repeated when scraping.To installpipinstallscrap-utilsSample codeimportscrap_utilsassuresponse=su.requests_get("https://python.org")len(response.text)It has the following functions:load_json(filepath,encoding=None,errors=None,parse_float=None,parse_int=None,parse_constant=None)dump_json(data,filepath,encoding=None,errors=None,indent=4,skipkeys=False,ensure_ascii=False,separators=None,sort_keys=False)to_csv(dataset,filepath,mode="a",encoding=None,errors=None,newline='',header=True,dialect='excel',**fmtparams)requests_get(url,trials=0,sleep_time=30,max_try=math.inf,**requests_kwargs)requests_post(url,trials=0,sleep_time=30,max_try=math.inf,**requests_kwargs)To-do list I'm considering:remove print statementsadd unittestsoup_get()driver_get()start_firefox()read_csv()Feel free to add your contributionhere
Package: scrapwiki
scrapwikiA python package for scrap Wikipedia page. Fetch title, paragraph or entire page htmlpip install scrapwikiWikipedia page examplehttps://en.wikipedia.org/wiki/CatCode examplefrom scrapwiki import read_url

page = read_url("https://en.wikipedia.org/wiki/Cat")
page.fetch()print(page)#WikipediaPage :	 Cat#URL :	 https://en.wikipedia.org/wiki/Catprint(page.paragraph)#Return all paragraph <p></p>print(page.soup)#Return entire html
Package: scrapy-accessory
Scrapy AccessoryIntroductionUseful accessory utilities forScrapy.Containing:middlewareitem pipelinefeed exporter storage backendInstallationpip install scrapy-accessoryUsageMiddlewareRandomUserAgentDownloadMiddlewareAdd random user-agent to requests.In settings.py add# USER_AGENT_LIST_FILE = 'path-to-files'
USER_AGENT_LIST = [
    'Mozilla/5.0(compatible;MSIE9.0;WindowsNT6.1;Trident/5.0',
    'Mozilla/5.0(WindowsNT6.1;rv:2.0.1)Gecko/20100101Firefox/4.0.1',
]

DOWNLOADER_MIDDLEWARES = {
    'scrapy_accessory.middlewares.RandomUserAgentDownloadMiddleware': 200,
}You can use eitherUSER_AGENT_LIST_FILEorUSER_AGENT_LISTto configure user-agents.USER_AGENT_LIST_FILEpoints to a text file containing one user-agent per line.USER_AGENT_LISTis a list or tuple of user-agents.ProxyDownloadMiddlewareAdd http or https proxy for requests.In settings.py addPROXY_ENABLED = True  # True to use proxy, default is False
# PROXY_HOST = 'localhost:8080'  # default static proxy, format: <ip>:<port>, default empty
PROXY_CACHE = 'redis://localhost:6379/0'  # cache for proxy, use redis://<host>:<port>/<db> to use redis cache, default dict in memory
PROXY_TTL = 30 # proxy cache ttl in seconds, default 30s
CHANGE_PROXY_STATUS = [429]  # a list of status codes that force to change proxy if received, default [429]Default is a static proxy configured in settings.py, you can add dynamic proxy from API or others.
Just need to extend theProxyDownloadMiddlewareclass and implement thegenerate_proxymethod.Example:class DynamicProxyDownloadMiddleware(ProxyDownloadMiddleware):

    api = 'http://api-to-get-proxy-ip'

    def generate_proxy(self):
        res = requests.get(self.api)
        if res.status_code < 300:
            return res.text  # return format <ip>:<port>
        return NoneFeed exporter storage backendObsFeedStorageFeed exporter storage backend for huawei cloud OBS.Install obs sdk firstpip install esdk-obs-pythonConfigure in settings.pyFEED_STORAGES={'obs':'scrapy_accessory.feedexporter.ObsFeedStorage',}HUAWEI_ACCESS_KEY_ID='<your access key id>'HUAWEI_SECRET_ACCESS_KEY='<your secret access key>'HUAWEI_OBS_ENDPOINT='<your obs bucket endpoint> ex: https://obs.cn-north-4.myhuaweicloud.com'Output to OBS by obs schema-o obs://<bucket>/<key>OssFeedStorageFeed exporter storage backend for ali cloud OSS.Install oss sdk firstpip install oss2Configure in settings.pyFEED_STORAGES={'oss':'scrapy_accessory.feedexporter.OssFeedStorage',}ALI_ACCESS_KEY_ID='<your access key id>'ALI_SECRET_ACCESS_KEY='<your secret access key>'ALI_OSS_ENDPOINT='<your oss bucket endpoint> ex: https://oss-cn-beijing.aliyuncs.com'Output to OSS by oss schema-o oss://<bucket>/<key>Item PipelineRedisListPipelineExport items to redis list.Install redis first.pip install redisConfigure in settings.pyREDIS_CONNECTION_URL='redis://localhost:6379/0'# requiredREDIS_DEFAULT_QUEUE='test'# use spider's queue attribute to override itREDIS_MAX_RETRY=5# default 5Addscrapy_accessory.pipelines.RedisListPipelineto yourITEM_PIPELINESsettings.ITEM_PIPELINES={'scrapy_accessory.pipelines.RedisListPipeline':1}
Package: scrapy-agentfive-middleware
This package provides a ScrapyDownloader Middlewareto interact with theagentfive API.RequirementsPython 3.5+Scrapy 1.6+Installationpip install scrapy-agentfive-middlewareConfigurationEnable theAgentfiveMiddlewarevia theDOWNLOADER_MIDDLEWARESsetting:DOWNLOADER_MIDDLEWARES = {
    "agentfive_middleware.AgentfiveMiddleware": 585,
}Please note that the middleware needs to be placed before the built-inHttpCompressionMiddlewaremiddleware (which has a priority of 590), otherwise incoming responses will be compressed and the
agentfive middleware won't be able to handle them.SettingsAGENTFIVE_KEY(typestr)API key to be used to authenticate against the agentfive API.AGENTFIVE_API_URL(Typestr, default"https://api.agentfive.cn/v1")The endpoint of a agentfive API.AGENTFIVE_DEFAULT_ARGS(typedict, default{})Default values to be sent to the agentfive API. For instance, set to{"profile": "mobile"}to set all requests with a mobile profile.UsageIf the middleware is enabled, by default all requests will be redirected to the specified
agentfive API endpoint, and append necessary params which agentfive API expected.For example:scrapy.Request(url="https://httpbin.org/anything")will be set to agentfive API and let agentfive to fetch the url.Additional argumentsAdditional arguments could be specified under theagentfiveRequest.metakey. For instance:Request(url="https://example.org",meta={"crawlera_fetch":{"render":True,"wait_ms":5000}},)for more information on agentfive API parameter, please refer toagentfive document.Skipping requestsYou can instruct the middleware to skip a specific request by setting theagentfive.skipRequest.metakey:Request(url="https://example.org",meta={"agentfive":{"skip":True}},)
Package: scrapy-aiohttp
scrapy-aiohttpThis library simply integratesaiohttpwithScrapy,
addressing challenges encountered when dealing with websites imposing
restrictions on standard Scrapy requests. Specifically, it simply
resolves the 403 Forbidden issue often encountered when utilizing
Scrapy's built-in requests.InstallationTo installscrapy-aiohttp, use the following command:$ pip install scrapy-aiohttpConfigurationAdd theaiohttp serveraddress tosettings.pyof your Scrapy project like this:AIOHTTP_SERVER_URL="http://localhost:8080/"Enable thescrapy-aiohttpmiddleware by adding it toDOWNLOADER_MIDDLEWARESin yoursettings.pyfile:DOWNLOAD_HANDLERS={"scrapy_aiohttp.AiohttpMiddleware":651,}Set up aiohttp request headers configuration tosettings.pylike this:fromscrapy_aiohttp.utilsimportDEFAULT_AIOHTTP_REQUEST_HEADERS_CONFIGAIOHTTP_REQUEST_HEADERS_CONFIG=DEFAULT_AIOHTTP_REQUEST_HEADERS_CONFIGTypedict[str, str | Callable[[aiohttp.web.Request], str] | None]TheAIOHTTP_REQUEST_HEADERS_CONFIGserves as an interface for inheriting headers from a Scrapy request and reusing
them to create an aiohttp request.DEFAULT_AIOHTTP_REQUEST_HEADERS_CONFIGis initialized with the values given below in the example.Here's how you can customizeAIOHTTP_REQUEST_HEADERS_CONFIGusing the following guidelines:If the header value is aCallableobject (a function), it gets executed with the HTTP request
object (aiohttp.web.Request)
as an argument during header construction. Result of the executed function becomes the header value.{"Host":lambdarequest:urlparse(request.match_info.get("url")).hostname}If the header value is astr, it serves as a static value for the header.{"Content-Type":"text/html"}If the header value is set toNone, it implies that the header should be inherited from the request headers. In
other words, the server will use the same value for this header as it receives in the incoming request.{"User-Agent":None}Note: Headers missing inAIOHTTP_REQUEST_HEADERS_CONFIGwill not be appliedto the aiohttp request!Ensure that all necessary headers are defined to meet your specific requirements.UsageThe easiest way to send requests withaiohttpis to usescrapy_aiohttp.AiohttpRequest.
You can also use regularscrapy.Requestand the"aiohttp"Request meta key:fromscrapyimportSpider,Requestfromscrapy_aiohttpimportAiohttpRequestclassExampleSpider(Spider):name="example"defstart_requests(self):# use case: scrapy_aiohttp.AiohttpRequestyieldAiohttpRequest(url="https://example.com",callback=self.parse)# use case: scrapy.Request with meta keyyieldRequest(url="https://example.com",callback=self.parse,meta={"aiohttp":True},)defparse(self,response,**kwargs):return{"url":response.url}
Package: scrapy-ajax-utils
scrapy_ajax_utilsutils for ajax in scrapy project. includes selenium, splash.UsageFor seleniumDefine the selenium webdriver name, executable path and is headless insettings.py# Default: chrome# Only support chrome & firefox.SELENIUM_DRIVER_NAME='chrome'# Default: NoneSELENIUM_DRIVER_PATH=None# Default: TrueSELENIUM_HEADLESS=True# Default: 30SELENIUM_DRIVER_PAGE_LOAD_TIMEOUT=30# Set the min/max concurrent drivers to download page.# Default: 3# SELENIUM_MIN_DRIVERS = 5# Default: 5# SELENIUM_MAX_DRIVERS = 10Use in your spider:importscrapyfromscrapy_ajax_utilsimportselenium_support,SeleniumRequest@selenium_supportclassMySpider(scrapy.Spider):start_urls=['https://www.baidu.com']defstart_requests(self):forurlinself.start_urls:yieldSeleniumRequest(url)For splashThe default splash url ishttp://127.0.0.1:8050, if not right, DO NOT use thesplash_supportfunction.importscrapyfromscrapy_ajax_utilsimportsplash_support,SplashRequest@splash_supportclassMySpider(scrapy.Spider):start_urls=['https://www.baidu.com']defstart_requests(self):forurlinself.start_urls:yieldSplashRequest(url)
Package: scrapy-algolia-exporter
[![PyPI](https://img.shields.io/pypi/v/scrapy_algolia_exporter.svg)](https://pypi.python.org/pypi/scrapy_algolia_exporter) [![Build Status](https://travis-ci.org/clemfromspace/scrapy-algolia-exporter.svg?branch=master)](https://travis-ci.org/clemfromspace/scrapy-algolia-exporter)## Scrapy Algolia ExporterDirectly populate a given **Algolia** index from a scrapy spider.### Usage:warning: **Your crawled items must contains a unique `ObjectID` key.** :warning:Add the mandatory scrapy settings to configure the **Algolia** API:```pythonALGOLIA_API_ID='my_algolia_api_id'ALGOLIA_API_KEY='my_algolia_api_key'ALGOLIA_INDEX_NAME='my_algolia_index_name'```Add the `AlgoliaItemPipeline` in the `ITEM_PIPELINES` scrapy setting:```pythonITEM_PIPELINES = {'scrapy_algolia_exporter.pipelines.AlgoliaItemPipeline': 10}```The `ALGOLIA_ITEM_BULK_NBR` setting control how many items will be send to Algolia at the same time.If not provided, the items will be send by group of `100`.### Install```pip install scrapy_algolia_exporter```
Package: scrapy-arweave
Welcome to Scrapy-ArweaveScrapy is a popular open-source and collaborative python framework for extracting the data you need from websites. scrapy-arweave provides scrapy pipelines and feed exports to store items intoArweave.🏠HomepageInstallpipinstallscrapy-arweaveExamplesscrapy-arweave-examplescrapy-arweave-doubanUsageInstall scrapy-arweave and some additional requirements.pipinstallscrapy-arweaveIt has some requirements that must be installed as well:Debian/Ubuntusudoapt-getinstalllibmagic1Windowspipinstallpython-magic-binOSXWhen using Homebrew:brew install libmagicWhen using macports:port install fileAdd 'scrapy-arweave.pipelines.ImagesPipeline' and/or 'scrapy-arweave.pipelines.FilesPipeline' to ITEM_PIPELINES setting in your Scrapy project if you need to store images or other files to Arweave.
For Images Pipeline, use:ITEM_PIPELINES={'scrapy_arweave.pipelines.ImagesPipeline':1}For Files Pipeline, use:ITEM_PIPELINES={'scrapy_arweave.pipelines.FilesPipeline':1}The advantage of using the ImagesPipeline for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size.Or You can also use both the Files and Images Pipeline at the same time.ITEM_PIPELINES={'scrapy_arweave.pipelines.ImagesPipeline':0,'scrapy_arweave.pipelines.FilesPipeline':1}If you are using the ImagesPipeline make sure to install the pillow package. The Images Pipeline requires Pillow 7.1.0 or greater. It is used for thumbnailing and normalizing images to JPEG/RGB format.pipinstallpillowThen, configure the target storage setting to a valid value that will be used for storing the downloaded images. Otherwise the pipeline will remain disabled, even if you include it in the ITEM_PIPELINES setting.Add store path of files or images for Web3Storage, LightHouse, Moralis, Pinata or Estuary as required.# For ImagesPipelineIMAGES_STORE='ar://images'# For FilesPipelineFILES_STORE='ar://files'For more info regarding ImagesPipeline and FilesPipline.See hereFor Feed storage to store the output of scraping as json, csv, json, jsonlines, jsonl, jl, csv, xml, marshal, pickle etc set FEED_STORAGES as following for the desired output format:fromscrapy_arweave.feedexportimportget_feed_storagesFEED_STORAGES=get_feed_storages()Then set WALLET_JWK and GATEWAY_URL. And, set FEEDS as following to finally store the scraped data.WALLET_JWK="<WALLET_JWK>"# It can be wallet jwk file path or jwk data itselfGATEWAY_URL="https://arweave.net"FEEDS={'ar://house.json':{"format":"json"},}See more on FEEDShereNow perform the scrapping as you would normally.Author👤Pawan PaudelGithub:@pawanpaudel93🤝 ContributingContributions, issues and feature requests are welcome!Feel free to checkissues page.Show your supportGive a ⭐️ if this project helped you!Copyright © 2023Pawan Paudel.
Package: scrapy-athlinks
scrapy-athlinks: web scraper for race results hosted on AthlinksIntroductionscrapy-athlinksprovides theRaceSpiderclass.This spider crawls through all results pages from a race hosted on athlinks.com,
building and following links to each athlete's individual results page, where it
collects their split data. It also collects some metadata about the race itself.By default, the spider returns one race metadata object (RaceItem), and oneAthleteItemper participant.
EachAthleteItemconsists of some basic athlete info and a list ofRaceSplitItemcontaining data from each split they recorded.How to use this packageOption 1: In python scriptsScrapy can be operated entirely from python scripts.See the scrapy documentation for more info.InstallationThe package is available onPyPiand can be installed withpip:pipinstallscrapy-athlinksExample usageAn demo script is included in this repo.fromscrapy.crawlerimportCrawlerProcessfromscrapy_athlinksimportRaceSpider,AthleteItem,RaceItemsettings={'FEEDS':{# Athlete data. Inside this file will be a list of dicts containing# data about each athlete's race and splits.'athletes.json':{'format':'json','overwrite':True,'item_classes':[AthleteItem],},# Race metadata. Inside this file will be a list with a single dict# containing info about the race itself.'metadata.json':{'format':'json','overwrite':True,'item_classes':[RaceItem],},}}process=CrawlerProcess(settings=settings)# Crawl results for the 2022 Leadville Trail 100 Runprocess.crawl(RaceSpider,'https://www.athlinks.com/event/33913/results/Event/1018673/')process.start()Option 2: Command lineAlternatively, you may clone this repo for use like a typical Scrapy project
that you might create on your own.Installationgitclonehttps://github.com/aaron-schroeder/athlinks-scraper-scrapycdathlinks-scraper-scrapy
pipinstall-rrequirements.txtExample usageRun aRaceSpider:cdscrapy_athlinks
scrapycrawlrace-aurl=https://www.athlinks.com/event/33913/results/Event/1018673-Oout.jsonDependenciesAll that is required isScrapy(and its dependencies).Testingmake testLicenseThis project is licensed under the MIT License. SeeLICENSEfile for details.ContactYou can get in touch with me at the following places:Website:trailzealot.comLinkedIn:linkedin.com/in/aarondschroederGitHub:github.com/aaron-schroeder
Package: scrapy-beautifulsoup
scrapy-beautifulsoupSimple Scrapy middleware to process non-well-formed HTML with BeautifulSoupInstallationThe package is on PyPI and can be installed withpip:pip install scrapy-beautifulsoupConfigurationAdd the middleware toDOWNLOADER_MIDDLEWARESdictionary setting:DOWNLOADER_MIDDLEWARES = {
    'scrapy_beautifulsoup.middleware.BeautifulSoupMiddleware': 400
}By default,BeautifulSoupwould use the built-inhtml.parserparser. To change it, set theBEAUTIFULSOUP_PARSERsetting:BEAUTIFULSOUP_PARSER = "html5lib"  # or BEAUTIFULSOUP_PARSER = "lxml"html5libis anextremely lenientparser and, if the target HTML is seriously broken, you might consider being it your first choice.
Note:html5libhas to be installed in this case:pip install html5libMotivationBeautifulSoupitself with the help of anunderlying parser of choicedoes a pretty good job of handling non-well-formed or broken HTML.
In some cases, it makes sense to pipe the HTML throughBeautifulSoupto “fix” it.
Package: scrapy-bigquery
scrapy-bigqueryA Big Query pipeline to store items intoGoogle BigQuery.Dependencies :globe_with_meridians:Python 3.7Scrapy 2.4.0Google Cloud Bigquery 2.23.2Bigquery Schema Generator 1.4Installation :inbox_tray:This is a python package hosted on pypi, so to install simply run the following command:pip install scrapy-bigquerySettingsBIGQUERY_DATASET (Required)The name of the bigquery dataset to post to.BIGQUERY_TABLE (Required)The name of the bigquery table in the dataset to post to.BIGQUERY_SERVICE_ACCOUNT (Required)The base64'd JSON of theGoogle Service Accountused to authenticate with Google BigQuery. You can generate it from a service account like so:cat service-account.json | jq . -c | base64BIGQUERY_ADD_SCRAPED_TIME (Optional)Whether to add the time the item was scraped to the item when posting it to BigQuery. This will add current datetime to the columnscraped_timein the BigQuery table.BIGQUERY_ADD_SCRAPER_NAME (Optional)Whether to add the name of the scraper to the item when posting it to BigQuery. This will add the scrapers name to the columnscraperin the BigQuery table.BIGQUERY_ADD_SCRAPER_SESSION (Optional)Whether to add the session ID of the scraper to the item when posting it to BigQuery. This will add the scrapers session ID to the columnscraper_session_idin the BigQuery table.BIGQUERY_ITEM_BATCH (Optional)The number of items to batch process when inserting into BigQuery. The higher this number the faster the pipeline will process items.BIGQUERY_FIELDS_TO_SAVE (Optional)A list of item fields to save to BigQuery. If this is not set, all fields of an item will be saved.Usage example :eyes:In order to use this plugin simply add the following settings and substitute your variables:BIGQUERY_DATASET = "my-dataset"
BIGQUERY_TABLE = "my-table"
BIGQUERY_SERVICE_ACCOUNT = "eyJ0eX=="
ITEM_PIPELINES = {
    "bigquerypipeline.pipelines.BigQueryPipeline": 301
}
BIGQUERY_FIELDS_TO_SAVE = ["name", "age"] # Optional. This will only save the name and age fields of an item to BigQuery.The pipeline will attempt to create a dataset/table if none exist by inferring the type from the dictionaries it processes, however be aware that this can be flaky (especially if you have nulls in the dictionary), so it is recommended you create the table prior to running.If you want to specify a table for a specific item, you can add the keys "BIGQUERY_DATASET" and "BIGQUERY_TABLE" to the item you pass back to the pipeline. This will override where the item is posted, allowing you to handle more than one item type in a scraper. The keys/values here will not be part of the final item in the table.License :memo:The project is available under theMIT License.