Saravanan666/pypi-org huggingface dataset:
Package: scrapemeagain
ScrapeMeAgainScrapeMeAgain is a Python 3 powered web scraper. It uses multithreading (ThreadPoolExecutor) and multiprocessing to get the work done quicker and stores collected data in anSQLitedatabase.Installationpip install scrapemeagainSystem requirementsTorin combination withPrivoxyare used for anonymity (i.e. regular IP address changes).UsingDockerandDocker Composeis the preferred (and easier) way to
use ScrapeMeAgain.You have to manually install and setupTorandPrivoxyon your system if not usingDocker. For further information about installation and configuration refer to:A step-by-step guide how to use Python with Tor and PrivoxyCrawling anonymously with Tor in Python(alternative link (Gist))UsageYou have to provide your own database table description and an actual scraper class which must follow theBaseScraperinterface. Seeexamples/examplescraperfor more details.DockerizedWith Docker it is possible to use multiple Tor IPs at the same time and, unless you abuse it, scrape data faster.The easiest way to start is to duplicateexamples/examplescraperand then update, rename, expand, etc. your scraper and related classes.Your scraper must defineconfig.pyandmain_dockerized.py. These two names are hardcoded throughout the codebase.scrapemeagain-compose.pydynamically creates adocker-compose.ymlwhich orchestrates scraper instances. The idea is that the first scraper (scp1) is amasterscraper and hence is the host for a couple of helper services which communicate over HTTP (seedockerized/apps).Get Docker host Ipip addr show docker0NOTEYour Docker interface name may be different fromdocker0.Runexamplesiteon Docker host IPpython3 examples/examplescraper/examplesite/app.py 172.17.0.1NOTEYour Docker host IP may be different from172.17.0.1.Startdocker-composescrapemeagain-compose.py -s $(pwd)/examples/examplescraper -c tests.integration.fake_config | docker-compose -f - upNOTEA special config file path is provided:-c tests.integration.fake_config. This isrequired only for test/demo purposes. You don't have to provide specific config for a real/production scraper.LocalRunexamplesitepython3 examples/examplescraper/examplesite/app.pyStartexamplescraperpython3 examples/examplescraper/main.pyNOTEYou may need to update yourPYTHONPATH, e.g.export PYTHONPATH=$PYTHONPATH:$(pwd)/examples.DevelopmentTo simplify running integration tests with latest changes:replaceimage: dusanmadar/scrapemeagain:x.y.zwithimage: scp:latestin thescrapemeagain/dockerized/docker-compose.ymltemplateand make sure to rebuild the image locally before running tests, e.g.docker build . -t scp:latest; python -m unittest discover -p test_integration.pyLegacyThe Python 2.7 version of ScrapeMeAgain, which also provides geocoding capabilities, is available under thelegacybranch and is no longer maintained.
Package: scrapemed
ScrapeMedData Scraping for PubMed Central⭐Used by Duke Universityto power medical generative AI research.⭐ Enables pythonic object-oriented access to amassive amount of research data. PMC constitutes over 14% ofThe Pile.⭐ Natural language Paper querying and Paper embedding, powered via LangChain and ChromaDB⭐ Easy to integrate with pandas for data science workflowsInstallationAvailable on PyPI! Simplypip install scrapemed.Feature ListScraping API for PubMed Central (PMC) ✅Data Validation ✅Markup Language Cleaning ✅Processes all PMC XML intoPaperobjects ✅Dataset building functionality (paperSets) ✅Semantic paper vectorization withChromaDB✅Natural languagePaperquerying ✅Integration withpandas✅paperSetvisualization ✅Direct Search for Papers by PMCID on PMC ✅Advanced Term Search for Papers on PMC ✅IntroductionScrapeMed is designed to make large-scale data science projects relying on PubMed Central (PMC) easy. The raw XML that can be downloaded from PMC is inconsistent and messy, and ScrapeMed aims to solve that problem at scale. ScrapeMed downloads, validates, cleans, and parses data from nearly all PMC articles intoPaperobjects which can then be used to build datasets (paperSets), or investigated in detail for literature reviews.Beyond the heavy-lifting performed behind the scenes  by ScrapeMed to standardize data scraped from PMC, a number of features are included to make data science and literature review work easier. A few are listed below:Papers can be queried with natural language [.query()], or simply chunked and embedded for storage in a vector DB [.vectorize()].Papers can also be converted to pandas Series easily [.to_relational()] for data science workflows.paperSets can be visualized [.visualize()], or converted to pandas DataFrames [.to_relational()].paperSets can be generated not only via a list of PMCIDs, but also via a search term using PMCadvanced search[.from_search()].Useful for advanced users:TextSections andTextParagraphs found within.abstractand.bodyattributes ofPaperobjects contain not only text [.text], but also text with attached reference data [.text_with_refs]. Reference data includes tables, figures, and citations. These are processed into DataFrames and data dicts and can be found within the.ref_mapattribute of aPaperobject. Simply decode references based on their MHTML index. ie. an MHTML tag of "MHTML::dataref::14" found in aTextSectionof paperpcorresponds to the table, fig, or citation atp.ref_map[14].DocumentationThedocsare hosted on Read The Docs!SponsorshipPackage sponsored by Daceflow.ai!If you'd like to sponsor a feature or donate to the project, reach out to me atdanielfrees@g.ucla.edu.
Package: scrapemyst
ScrapeMystScrapeMyst is a Python class for sending web requests with customizable headers and user-agents. It provides methods for sending both GET and POST requests with customizable headers, user-agents, and optional proxy support. The class also includes features for adding random sleep intervals before making requests to simulate human-like behavior.InstallationYou can install ScrapeMyst using pip:pipinstallscrapemystUsage# Import the ScrapeMyst classfromscrapemystimportScrapeMyst# Create an instance of ScrapeMyst, optionally providing a list of proxy URLsscrapemyst=ScrapeMyst(proxies=["127.0.0.1:8000","example.com:8080"])# OR# Import the ScrapeMyst object directly if you are not providing list of proxy urlsfromscrapemystimportscrapemyst# Example of sending a GET requesturl_get="https://www.example.com"get_response=scrapemyst.send_get(url_get,params={"param1":"value1"},sleep=True,referer="https://www.referer.com")ifget_response['success']:print(f"GET Request Successful. Status Code:{get_response['status_code']}")# Access the response object if needed: get_response['data']# Example of sending a POST request with form dataurl_post_form="https://www.example.com/post"form_data={"key1":"value1","key2":"value2"}post_response_form=scrapemyst.send_post(url_post_form,data=form_data,sleep=True,referer="https://www.referer.com")ifpost_response_form['success']:print(f"POST Request (Form Data) Successful. Status Code:{post_response_form['status_code']}")# Access the response object if needed: post_response_form['data']# Example of sending a POST request with JSON dataurl_post_json="https://www.example.com/api"json_data={"key1":"value1","key2":"value2"}post_response_json=scrapemyst.send_post(url_post_json,json=json_data,sleep=True,referer="https://www.referer.com")ifpost_response_json['success']:print(f"POST Request (JSON Data) Successful. Status Code:{post_response_json['status_code']}")# Access the response object if needed: post_response_json['data']CustomizationYou can customize the headers, user-agents, and other settings by updating the ScrapeMyst instance. For example:# Update headerscustom_headers={"User-Agent":"CustomUserAgent","Accept-Language":"en-US","Custom-Header":"CustomValue"}scrapemyst.update_headers(custom_headers)# Send a request with updated headersresponse_custom=scrapemyst.send_get('https://example.com')print("Custom Response:",response_custom)ContributingIf you'd like to contribute to ScrapeMyst, feel free to submit pull requests, report issues, or suggest improvements.LicenseThis project is licensed under theMIT License- see theLICENSEfile for details.
Package: scrap-engine
Scrap_engineBy lxgrlxgr@protonmail.comInstallationManually:# python setup.py installVia pip (This will just install the latest stable release):# python -m pip install scrap_engineFrom the AUR (When using Arch Linux):$buildaur-Spython-scrap_engine-git# You can use what ever AUR-helper you wantTo run the examples install python and the pynput module via pip.See the project onpypiUsageSeeexamplesandDocumentation. For further documentation seepydoc scrap_engine.pyExamples and testsThe examples inexamplesare made to show how scrap_engine works in a simple and commented way by, in case of scrap_test, building a little game. For more examples you can look attestswhich also contain explanatory programs.They can be ran by either installing scrap_engine like in the installation section or by moving thescrap_engine.pyfile in their directories, and then just executing them.Another source to understand the functionality of scrap_engine is the small snake implementationscrape, that was firstly designed to be a tutorial but was later split out due to it's size. This may also help to implement a proper game using scrap_engine.NotesThe examples and tests where all tested on Arch Linux x86 and Windows 10.
Package: scrapepy
vidstreamUnder construction! Not ready for use yet! Currently experimenting and planning!Developed by Florian Dedov from NeuralNine (c) 2020Examples of How To Use (Buggy Alpha Version)Creating A ServerfromvidstreamimportStreamingServerserver=StreamingServer('127.0.0.1',9999)server.start_server()# Other Code# When You Are Doneserver.stop_server()Creating A ClientfromvidstreamimportCameraClientfromvidstreamimportVideoClientfromvidstreamimportScreenShareClient# Choose Oneclient1=CameraClient('127.0.0.1',9999)client2=VideoClient('127.0.0.1',9999,'video.mp4')client3=ScreenShareClient('127.0.0.1',9999)client1.start_stream()client2.start_stream()client3.start_stream()Check out:https://www.youtube.com/c/NeuralNine
Package: scrape_quora
A Python module to fetch and parse data from Quora user profiles.InstallationInstall using pip:pip install scrape_quoraUsagefrom scrape_quora import Scrape_Quoraname = Scrape_Quora.get_name(‘Hansika-Hewamalage’)FeaturesCurrently implementedRetrieveProfile NameURLProfile Picture LinkNo of QuestionsNo of AnswersNo of FollowersNo of FollowingNo of EditsTodoRetrieveFacebook Profile LinkTwitter LinkWordpress LinkLinkedIn Link
Package: scraper
Minimalist Python DOM ScraperDescriptionThis module is an easy to use HTML/XML scraper. It supports both XPath and Regular
Expression retrieval.Once you have a file you want to extract information from, you can extract
multiple pieces of information with a simple function call.You should obtain the files you want to scrape by your own ways.Installationpip install scraperUsageScrape using xpath:import scraper
import requests

content = requests.get('https://github.com/explore').content

conf = {'trending-repos' : {'xpath' : '//ol/li/h3/a[2]/@href'}}

scraper.scrapes(content, conf)

>>> {'trending-repos': ['/jamescryer/grumble.js', '/dominictarr/JSON.sh', '/JamieLottering/DropKick', '/harvesthq/chosen', '/velvia/ScalaStorm']}Scrape using regexp:import scraper
import requests

content = requests.get('http://wiki.nomasnumeros900.com/Air_Liquide').content

conf = {
        'numbers':
            {'regexp': '91[\s\d]+',
             'transf': [lambda x: x.strip()],
             'encoding': 'utf-8'}
        }

scraper.scrapes(content, conf)

>>> {'numbers': [u'915 029 300', u'915 029 560', u'915 029 330', u'91']}
Package: scraper1830
1830 Game Log Scraper:The goal of this repository is to build a package to scrape game data for 1830 games from the 18xx.games api.Installation Instructions:conda env create --file=environment.ymlImplemented Features:An attribute recording the list of players in the game in initial turn order.Methods that record the distribution of privates, the player with priority in stock round 1 and the final player scores.Upcoming Features:Here are some features that are planned to be added to this scraper:A table recording the player scores at the end of each stock round and operating round, along with their share counts.A dictionary that records when a private is sold in to a company or closes and how much it is sold for, or for the B&O private, when the private closes.Graphical representations of the data.Representations of the data in a 2d pandas array.Planned Applications of the Scraper:There are two main applications in mind:Build a package that allows users to input game ids and obtain tables and graphs representing their game history.Scrape a large number of finished 1830 games to build a model that predicts player win probabilities from the results of the private auction.
Package: scraper2-hj3415
scraper2_hj3415 PackageThis is a simple example package. You can useGithub-flavored Markdownto write your content.새로운 scrapy 프로젝트를 만들 때는 프로젝트 가장 외부폴더명과 그 안쪽 폴더명이 달라야한다. 안그러면 setting파일을 찾을수 없다는 에러가 발생한다.
(ex - nfscrapy/nfs(프로젝트명은 nfs로 하고 외부 폴더명을 nfscrapy로 변경))Tutorials - How to packaging
Package: scrapera
Table of ContentsAbout The ProjectGetting StartedPrerequisitesInstallationUsageContributingSponsorsLicenseContactAcknowledgementsAbout The ProjectScrapera is a completelyChromedriver freepackage that provides access to a variety of scraper scripts for most commonly used machine learning and data science domains. Scrapera directly and asynchronously scrapes from public API endpoints, thereby removing the heavy browser overhead which makes Scrapera extremely fast and robust to DOM changes. Currently, Scrapera supports the following crawlers:ImagesDuckDuckGo ScraperGIPHY ScraperTumblr ScraperTextAmazon Product ReviewsVoice of America News ScraperScroll News ScraperIMDB Reviews ScraperReddit ScraperAudioYoutube Playlist ScraperVideosVimeo ScraperYoutube ScraperMiscellaneousYahoo Stocks ScraperThe main aim of this package is to cluster common scraping tasks so as to make it more convenient for ML researchers and engineers to focus on their models rather than worrying about the data collection processDISCLAIMER: Owner or Contributors do not take any responsibility for misuse of data obtained through Scrapera. Contact the owner if copyright terms are violated due to any module provided by Scrapera.PrerequisitesPrerequisites can be installed separately through therequirements.txtfile as belowpipinstall-rrequirements.txtInstallationScrapera is built with Python 3 and can bepipinstalled directlypipinstallscraperaAlternatively, if you wish to install the latest version directly through GitHub then runpipinstallgit+https://github.com/DarshanDeshpande/Scrapera.gitUsageTo use any sub-module, you just need to import, instantiate and executefromscrapera.video.vimeoimportVimeoScraperscraper=VimeoScraper()scraper.scrape('https://vimeo.com/191955190','540p')For more examples, please refer to the individual test folders in respective modulesContributingScrapera welcomes any and all contributions and scraper requests. Please raise an issue if the scraper fails at any instance. Feel free to fork the repository and add your own scrapers to help the community!For more guidelines, refer toCONTRIBUTINGLicenseDistributed under the MIT License. SeeLICENSEfor more information.SponsorsContactFeel free to reach out for any issues or requests related to ScraperaDarshan Deshpande (Owner) -Email|LinkedInAcknowledgementsPyTube
Package: scrapere
Scraping libaryThis is a simple toolbox for scraping info from websites which includes methods for generating headers with a random user agent picked from a list of more than 24k user agents, a method for waiting a random time between 1 and 3 seconds, a progress bar for it to look cool ;) and a couple of methods that will help you retreive email addresses and spanish phone numbers from a given url.As this package may be useful for a bunch of legitimate and non-legitimate use cases, I'm not responsible of whatever you do with it.Version 0.3 features:Improves the email regex.Adds a method to get random browser user agentsVersion 0.3 issues:There are no tests...scrap_phones() only works for spanish phone numbersAs this is my first python package, I might not mantain it, so better try finding another one...
Package: scrapereads
scrape-goodreadsPython package to scrape data from goodreads.com website. Authors, Books, and Quotes can be extracted.Project made during a Deep Learning project for poem generation using GPT2 model.InstallationInstallscrapereadspackage fromPyPi:pip install scrapereadsOr fromGitHub:git clone https://github.com/arthurdjn/scrape-goodreads
cd scrape-goodreads
pip install .Getting StartedGoodReads APIYou can search forAuthor,BookorQuotefrom the API:fromscrapereadsimportGoodReads# Connect to the APIgoodreads=GoodReads()# Search for an author, from it's ID.AUTHOR_ID=3389author=goodreads.search_author(AUTHOR_ID)# Search for a bookBOOK_ID=3048970book=goodreads.search_book(AUTHOR_ID,BOOK_ID)# Look for the 10 first books (set it to ``top_k=None`` to turn it off)books=goodreads.search_books(AUTHOR_ID,top_k=10)# ...Or quotesquotes=goodreads.search_quotes(AUTHOR_ID,top_k=5)Quotes are made of a text, but optional information can be added (like number of likes, tags,
reference etc.)quotes=goodreads.search_quotes(AUTHOR_ID,top_k=5)forquoteinquotes:print(quote)print()Output:"Books are a uniquely portable magic."- Stephen King, from "On Writing: A Memoir Of The Craft"Likes: 16225, Tags: books, magic, reading"If you don't have time to read, you don't have the time (or the tools) to write. Simple as that."- Stephen KingLikes: 12565, Tags: reading, writing"Get busy living or get busy dying."- Stephen King, from "Different Seasons"Likes: 9014, Tags: life"Books are the perfect entertainment: no commercials, no batteries, hours of enjoyment for each dollar spent. What I wonder is why everybody doesn't carry a book around for those inevitable dead spots in life."- Stephen KingLikes: 8667, Tags: books"When his life was ruined, his family killed, his farm destroyed, Job knelt down on the ground and yelled up to the heavens, "Why god? Why me?" and the thundering voice of God answered, There's just something about you that pisses me off."- Stephen King, from "Storm Of The Century"Likes: 7686, Tags: god, humor, religionStructureThe package is divided as follows:AuthorBook, inherits from AuthorQuote, inherits from BookRetrieve dataOnce you have one of these objects, you can also access data directly through their methods:author=goodreads.search_author(AUTHOR_ID)books=author.get_books()quotes=author.get_quotes()# Idem from an bookbook=goodreads.search_book(AUTHOR_ID,BOOK_ID)quotes=book.get_quotes()In addition, you can retrieve parent objects from children:author=goodreads.search_author(AUTHOR_ID)quotes=author.get_quotes(top_k=10)quote=quotes[0]# Access to parent classesbook=quote.get_book()author=quote.get_author()You can scrape for description, links and other details information:author=goodreads.search_author(AUTHOR_ID)info=author.get_info()# description of the author (genre, description, references etc.)You can finally retrieve similar authors from an author with:author=goodreads.search_author(AUTHOR_ID)authors=author.get_similar_authors(top_k=5)Save and exportYou can save data in a JSON format (and encode it to ASCII if you want).author=goodreads.search_author(AUTHOR_ID)author_data=author.to_json(encode='ascii')# Idem for book and quote
Package: scraperex
Web scraper using dynamic proxy and user agent.DescriptionScraperex is simple and easy to use web scraper for retreiving data from request andavoiding HTTP 503 error(usually emerges when server is watching for bots/crawlers/requests while regular scraping).Pakage is generating random user-agent headers usingfake-useragent, and a list of proxy servers that is used while maiking requests.InstallationpipinstallscraperexDependenciesrequestsfake-useragentUsageimportscraperexScrape textual responseBasic usage requires one paramether of type dictionary which containsurlto the web resource, andregexwhich will extract the data from the response.config={'my_scraping':{'method':'GET','url':'https://www.resource.for/scraping/1','params':{perPage:5},'regex':r'my_regular_expression'}}result=scraperex.find(config)Scrape json responseIf you set configurationjsonoptionTruethenregexoption will be ignored and(requests) response.json()will be invoked.config={'my_scraping':{'method':'GET','url':'https://www.resource.for/scraping/2','params':{perPage:5},'json':True}}result=scraperex.find(config)Note: If proxy server fails, next one from the list will be used, while proxy list is not exhausted or limit is not touched. You can set limitation by sendingattemptsparameter. By default attempts are set to 3.result=scraperex.find(config,attempts=1)ConfigConfigconfigmust be of type dictionary which must contain at least one item used for scraping (as you can guess the amount of the requests will equal at least to the items amount).Note: You also can build a tree of configurations and the same structure will be in your result.config={'item_A':{...},'item_B':{'item_C':{..},'item_D':{..},}}Structured textual ( regex ) resultsYou can rescrape response content as many times as you wish by passing toregexproperty dictionary instead of string (also it is useful if you want to structure your results, scraping will result in the same structure you defined).'item_B':{'url':'https://www.resource.for/scraping/2','regex':{'structure_item':{'child_structure_item_1':r'my_regular_expression'}'structure_item_1':r'my_regular_expression_1'}}Note: Constructive criticism is always awaited, please share your thoughts with me:GitHub.
Package: scraper-factory
Scraping library to retrieve data from useful pages, such as Amazon wishlistsAPIThe API to use the library, scrape data and manage spiders is the following:scrape(SPIDER_NAME, URL): scrapes the givenURLusing the spider referenced onSPIDER_NAME.spiders(): list all spiders found by the library.Custom SpidersUsing custom spiders is possible, as long as they:They must be implemented as a class, and inherit fromBaseSpider.The spider file need to be either onscraper_factory/spiders, or in a custom location, as long as the environment variable$SPIDER_PATHis set to the directory where the spider is located.Usage example>>>importscraper_factoryasSF>>>SF.scrape('amazon-wishlist','https://www.amazon.com/hz/wishlist/ls/24XY9873RPAYN')[{'id':'I1MZVK8RDPYK8P','title':'AmazonBasics Heavy Weight Ruled Lined Index Cards, White, 3x5 Inch Card, 100-Count - AMZ63500','byline':None,'price':None,'link':'https://www.amazon.com/dp/B06XSRLP51/','img':'https://images-na.ssl-images-amazon.com/images/I/71i7LVTzpsL._SS135_.jpg'},{'id':'I14TUJ6TADACU5','title':"Women's Walking Shoes Sock Sneakers - Mesh Slip On Air Cushion Lady Girls Modern Jazz Dance Easy Shoes Platform Loafers",'byline':None,'price':None,'link':'https://www.amazon.com/dp/B07MWCDJ9X/','img':'https://images-na.ssl-images-amazon.com/images/I/61sHA7o-bxL._SS135_.jpg'},{'id':'I3C97JA2JR06PN','title':'Tenergy Redigrill\xa0Smoke-Less Infrared Grill, Indoor Grill, Heating\xa0Electric Tabletop Grill, Non-Stick Easy to Clean\xa0BBQ Grill, for Party/Home, ETL Certified','byline':None,'price':'$179.99','link':'https://www.amazon.com/dp/B07BZ412HT/','img':'https://images-na.ssl-images-amazon.com/images/I/41uGvSPg-ML._SS135_.jpg'},{'id':'I1C7RJI2H0VWZ7','title':'Shelf Liners for Wire Shelf Liner Set of 4 - Graphite (14-Inch-by-36-Inch)','byline':None,'price':'$29.99','link':'https://www.amazon.com/dp/B01N9V4A9A/','img':'https://images-na.ssl-images-amazon.com/images/I/71Lg6J7sGHL._SS135_.jpg'},...]InstallationLatest release through PyPI:$pipinstallscraper_factoryDevelopment version:$gitclonegit@github.com:machinia/scraper-factory.git$cdscraper_factory$pipinstall-e.