Saravanan666/pypi-org huggingface dataset:
Package: scrape-facebook
Use localinstallpackagepythonsetup.pyinstallInstallpipinstallscrape-facebook
orpip3installscrape-facebookUsage>>>fromscrape_facebookimportget_posts>>>forpostinget_posts('nintendo',pages=1):...print(post['text'][:50])...ThefinalstepontheroadtotheSuperSmashBrosWe’reheadedtoPAXEast3/28-3/31withnewgamesOptional parameterspages: how many pages of posts to request, usually the first page has 2 posts and the rest 4. Default is 10.timeout: how many seconds to wait before timing out. Default is 5.sleep: how many seconds to sleep between each request. Default is 0.credentials: tuple of user and password to login before requesting the posts. Default isNone.Post example{'post_id':'2257188721032235','text':'Don’t let this diminutive version of the Hero of Time fool you, ''Young Link is just as heroic as his fully grown version! Young Link ''joins the Super Smash Bros. series of amiibo figures!','time':datetime.datetime(2019,4,29,12,0,1),'image':'https://scontent.flim16-1.fna.fbcdn.net''/v/t1.0-0/cp0/e15/q65/p320x320''/58680860_2257182054366235_1985558733786185728_n.jpg''?_nc_cat=1&_nc_ht=scontent.flim16-1.fna''&oh=31b0ba32ec7886e95a5478c479ba1d38&oe=5D6CDEE4','likes':2036,'comments':214,'shares':0,'post_url':'https://m.facebook.com/story.php''?story_fbid=2257188721032235&id=119240841493711','link':'https://bit.ly/something'}NotesThere is no guarantee that every field will be extracted (they might beNone).Shares doesn't seem to work at the moment.
Package: scrape_files
scrape_filesis a tool to help scrape things online to your local machine.
Currently, it supports scraping and converting htmls to well-formatted markdowns for easy reading as well as scraping and downloading images of various formats in a web page.Scraping htmls to your local machineThe html parsing logic is similar to a browser's easyread extension's, which trims off all the unnecessary decorations from a web page, only keeping the title and the article content. The main difference is that the file is downloaded and converted as a pretty formatted markdown.Also support scraping links under the<p>tag in the current page concurrently.Terminal usage:scrapehtml<url># specify a url for scrapingscrapehtml<url>-d# specify a directory name for saving files in current folderscrapehtml<url>-l# specify a level: 1 by default for the current page; 2 for links in the current pageScraping images to your local machineImages are scraped and downloaded concurrently. Supported formats: jpg, png, gif, svg, jpeg, webp; defaults to all supported formats.Terminal usage:scrapeimage<url># specify a url for scrapingscrapeimage<url>-d# specify a diretory name for saving files in current folderscrapeimage<url>-f# specify image formats separated with spaceInstallationpipinstallscrape_files
Package: scrapeGG
A python library to pull information on Profile and MatchHistory from
op.gg (last verified 06/05/2020).Note: scrapes using English, only tested on NA server at the moment.Requirementsfrom selenium import webdriverfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECfrom selenium.webdriver.support.wait import WebDriverWaitimport reimport osfrom bs4 import BeautifulSoupInstallationpip install scrapeGG==0.0.4orgit clone https://github.com/emily-yu/animelyrics.git
cd animelyrics
python setup.pyUsageset up chromedriver, if not already exists check if already exists usingchromedriver--versioninstall chromedriverbrew cask install chromedriverfind pathwhich chromedriverimport and create an instance, such that ‘API’ is the summonerName you
want to search for.scrapeGG(string summonerName)from scrapeGG import scrapeGG
init = scrapeGG('API')Functions(see scrape_test.py for usage examples) (see snapshots/06-06-2020.json
for example output)scrapeGG.profile.getProfile() - [CONSTRUCTOR] - returns profile().recently_played_with().top_played_champions() - todo.rank(string game_type= [“Ranked Solo”, “Total”, “Ranked Flex 5v5”)
- todoscrapeGG.match.getMatch(int count) - returns match().getMatchSequence(int count) - returns
arrscrapeGG.match.game_player_names(is_win).self_stats().recent.player_stats(string summonerName) - summonerName can be any
of the players in the game.overview() - unstable*.build() - unstable**selenium has issues with click events, which occasionally
raisesselenium.common.exceptions.ElementClickInterceptedException: Message: element click interceptedexample usagefrom scrapeGG import scrapeGG

init = scrapeGG('API')

# test match api class
recent = init.getMatch(1)
player_names = recent.game_player_names(True)

profile = init.getProfile()
ranksolo_rank = profile.rank('Ranked Solo')changelog0.0.2 - first release on PyPI
Package: scrapeghost
scrapeghostscrapeghostis an experimental library for scraping websites using OpenAI's GPT.Source:https://github.com/jamesturk/scrapeghostDocumentation:https://jamesturk.github.io/scrapeghost/Issues:https://github.com/jamesturk/scrapeghost/issuesUse at your own risk. This library makes considerably expensive calls ($0.36 for a GPT-4 call on a moderately sized page.) Cost estimates are based on theOpenAI pricing pageand not guaranteed to be accurate.FeaturesThe purpose of this library is to provide a convenient interface for exploring web scraping with GPT.While the bulk of the work is done by the GPT model,scrapeghostprovides a number of features to make it easier to use.Python-based schema definition- Define the shape of the data you want to extract as any Python object, with as much or little detail as you want.PreprocessingHTML cleaning- Remove unnecessary HTML to reduce the size and cost of API requests.CSS and XPath selectors- Pre-filter HTML by writing a single CSS or XPath selector.Auto-splitting- Optionally split the HTML into multiple calls to the model, allowing for larger pages to be scraped.PostprocessingJSON validation- Ensure that the response is valid JSON.  (With the option to kick it back to GPT for fixes if it's not.)Schema validation- Go a step further, use apydanticschema to validate the response.Hallucination check- Does the data in the response truly exist on the page?Cost ControlsScrapers keep running totals of how many tokens have been sent and received, so costs can be tracked.Support for automatic fallbacks (e.g. use cost-saving GPT-3.5-Turbo by default, fall back to GPT-4 if needed.)Allows setting a budget and stops the scraper if the budget is exceeded.
Package: scrape-glosbe-dict
scrape-glosbe-dictScrape a glosbe dictInstall itpipinstallscrape-glosbe-dict# pip install git+https://github.com/ffreemt/scrape-glosbe-dict# poetry add git+https://github.com/ffreemt/scrape-glosbe-dict# git clone https://github.com/ffreemt/scrape-glosbe-dict && cd scrape-glosbe-dictUse itscrape-glosbe-dicthead-word-file# default english-chinese# or python -m scrape_glosbe_dict head-word-file# scrape-glosbe-dict head-word-file -f de  # german-chinesehead word file formt: one word/phrase per line, empty lines will be ignored.output will be saved to a tsv file.Docspython-mscrape_glosbe_dict--helpUsage:python-mscrape_glosbe_dict[OPTIONS]head-word-file

Arguments:head-word-fileHeadwordfile,oneword/phraseperline,eachwillbeusedtofetchcorrespondingdefinitonsfromhttps://glosbe.com/.[required]Options:-f,--from-langTEXTSourcelanguage,checkhttps://glosbe.com/forvalidvalue,e.g.https://glosbe.com/en/zhimpliesfrom_lang='en'.[default:en]-t,--to-langTEXTTargetlanguage,checkhttps://glosbe.com/forvalidvalue,e.g.https://glosbe.com/en/zhimpliesto_lang='zh'.[default:zh]-v,--verboseShowoutputintheprocess.-V,--versionShowversioninfoandexit.--helpShowthismessageandexit.MiscellanyA retry mechanism (via pypitenacity) is built-in to fetch info from glosbe. Refer to the source file for details.Local cache (via pypijoblib) is used so that you can interrupt anytime and continue later.Scraping is often frowned upon and sometimes can result in your IP being banned from the website. Use this package at your own discretion.
Package: scrapegoat
SCRAPEGOATScrape data in one shot.Scrapegoatis a python library that can be used to scrape the websites from internet based on the relevance of the given topic irrespective of language usingNatural Language Processing. It can be mainly used fornon-English languageto get accurate and relevant scraped text.ConceptInitially the data is scraped from a website  and  processed ( to remove English words if the data required is in other language). TheBERTmodel is feed with processed data and topic  to compute thecosine similarityof the given topic with each word of the scraped data then mean of cosine similarity scores of is computed. If the mean is greater than threshold then scraped data is generated as output. Also there is a section where we are usingAdaptive threshold.BERT ModelBERT, which stands forBidirectional Encoder Representations from Transformers, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. The BERT framework waspre-trained using text from Wikipedia. The transformer is the part of the model that gives BERT its increased capacity for understanding context and ambiguity in language. The transformer does this by processing any given word in relation to all other words in a sentence, rather than processing them one at a time. By looking at all surrounding words, the Transformer allows the BERT model to understand the full context of the word, and therefore better understand searcher intent.Cosine SimilarityCosine similarityis one of the metrics to measure the text-similarity between two documents irrespective of their size in Natural language Processing. A word can be represented in the vector form, therefore the text documents are represented in n-dimensional vector space.If the Cosine similarity score is 1, it means two vectors have the same orientation. The value closer to 0 indicates that the two documents have less similarity. The Cosine similarity of two documents will range from 0 to 1.Multi ProcessingThe multiprocessing module allows the programmer to fully leverage multiple processors on a given machine.The basic ideology of Multi-Processing is that if you have an algorithm that can be divided into different workers (small processors/cores), then you can speed up the program.Machines nowadays come with 4,6,8 and 16 cores, therefore parts of the code can be deployed in parallel.Using ScrapegoatThe examples/test.py file contains thesefromscrapegoat.mainimportgetLinkDatafromscrapegoat.mainimportgenerateDataif__name__=="__main__":# scrape one link and get the relevence scoretopic=" cricket"language='kn'url="https://vijaykarnataka.com/sports/cricket/news/ind-vs-eng-brian-lara-congratulates-jasprit-bumrah-for-breaking-his-world-record-in-test-cricket/articleshow/92628545.cms"text,score=getLinkData(url,topic,language=language)print(score)# scrape and download datatopic=" cricket"language='hi'generateData(topic,language,n_links=20)
Package: scrapegraphai
🕷️ ScrapeGraphAI: You Only Scrape OnceScrapeGraphAI is aweb scrapingpython library based on LangChain which uses LLM and direct graph logic to create scraping pipelines.
Just say which information you want to extract and the library will do it for you!🚀 Quick installThe reference page for Scrapegraph-ai is avaible on the official page of pypy:pypi.pipinstallscrapegraphai🔍 DemoOfficial streamlit demo:Is it possible to try also the colab versionTry out ScrapeGraphAI in your browser:📖 DocumentationThe documentation for ScrapeGraphAI can be foundhere.
Behind this there is also the docusaurus documentationhere).Setup the api keysFollow the procedure on the following link to setup your OpenAI API key:link.💻 UsageCase 1: Extracting information using a promptYou can use theSmartScraperclass to extract information from a website using a prompt.TheSmartScraperclass is a direct graph implementation that uses the most common nodes present in a web scraping pipeline. For more information, please see thedocumentation.fromscrapegraphai.graphsimportSmartScraperOPENAI_API_KEY="YOUR_API_KEY"llm_config={"api_key":OPENAI_API_KEY,"model_name":"gpt-3.5-turbo",}smart_scraper=SmartScraper("List me all the titles and project descriptions","https://perinim.github.io/projects/",llm_config)answer=smart_scraper.run()print(answer)The output will be a dictionary with the extracted information, for example:{'titles':['Rotary Pendulum RL'],'descriptions':['Open Source project aimed at controlling a real life rotary pendulum using RL algorithms']}🤝 ContributingScrapegraph-ai isMIT LICENSED.Contributions are welcome! Please check out the todos below, and feel free to open a pull request.For more information, please see thecontributing guidelines.Join our Discord server to discuss with us improvements and give us suggestions!You can also follow all the updates on linkedin!ContributorsAuthorsContact InfoMarco VinciguerraMarco PeriniLorenzo Padoan📜 LicenseScrapeGraphAI is licensed under the Apache 2.0 License. See theLICENSEfile for more information.AcknowledgementsWe would like to thank all the contributors to the project and the open-source community for their support.ScrapeGraphAI is meant to be used for data exploration and research purposes only. We are not responsible for any misuse of the library.
Package: scrape-html
Generate HTML in string from URL# One Single Page Websites also work# Get html from pagefromscraping.scraperimportPageSourcespage=PageSources('https://...')print(page.get_current_html())# save data in a directory call web_datafromscraping.scraperimportPageSourcespage=PageSources('https://...')page.get_current_html()page.save()# page.save(directory='web_page') default#Multiple linkfromscraping.scraperimportPageSourceslista=['https://...','https://...']page=PageSources()page.get_multiple_html(lista)When create a file it'll get name of hostPage and amount of file in your directory, like:
-> web_data
    -hostPage_1.html
    -hostPage_2.html
    -hostPage_3.html
    ...# getting csv filefromscraping.scraperimportPageSourcespage=PageSources()dict_data=[{'name':'a','page':'b'},{'name':'a','page':'b'},{'name':'a','page':'b'}]page.save_csv(dict_data)# def save_csv(self, dict_data, outfile = 'output.csv', open_file = 'w'):# Using proxyfromscraping.scraperimportPageSources"""PROXY = "158.69.25.178:32769" # IP:PORT or HOST:PORT"""page=PageSources('https://andycode.ga',headless=False,proxy='158.69.25.178:32769')page.get_current_html()page.save()It need a Google Chrome DriverTo check the version you have of Google Chrome, you can do it from the browser information and in the "Help" section:Open a window in the browser.Go to the three points in the upper right.Choose the "Help" option from the drop-down menu.Tap on "Google Chrome Information"Go tohttps://chromedriver.chromium.org/downloadsselect your version, system and downloadIt will be a file like this:Copy and paste in your root project
Package: scrape-it
Scrape_itScrape_it is a tool for extracting valueble information from the website of interest. Save your time on reading and crawling through the website and leave it for Scrape_it!InstallationScrape_it is avalable on Pypi, you can install it using pip:pipinstallscrape-itInstall the lastest version:pipinstallgit+https://github.com/erelin6613/Scrape_itScrape_it objectAs a baseline Scrape_it relies on the model (dictionary) which could be customized although specific methods should be defined too.Currently the object's base-line model is set up to scrape contact information, address, social media links, links for website's policies pages and if posiible condence its' texts.To initialize the object specify the url as a string. For more precision provide some more details if known:country: 'us' for United States of America, 'gb' for Great Britain/United Kingdom, 'au' for Australiageo_key: API key for address verification, test is set up to work withthisAPImethod: 'requests' for usual get request or 'webdriver' for request capable of rendering JavaScript code and dynamically changing webpagesUsageInitialize Scrape_it object (find an example in run.py)fromscrape_itimportScrape_it

withopen('/home/val/Downloads/geo-key.txt','r')askey:geo_key=key.read()scrape_it=Scrape_it(url='https://www.all-wall.com',country='us',geo_key=geo_key,method='webdriver')scrape_it.scrape()The output will like this:Scrapinghttps://www.all-wall.com...
url:https://www.all-wall.com
country:us
category:None
company_name:All
contact_link:None
phones:{'+18009290927'}address:6561WPostRd
state:NV
county:Clark
city:LasVegas
street:WPostRd
housenumber:6561postalcode:89118district:SpringValley
email:None
facebook:https://www.facebook.com/AllWallEquipment
instagram:https://www.instagram.com/allwall_inc/
linkedin:None
pinterest:None
twitter:https://twitter.com/AllWall_Inc
youtube:https://www.youtube.com/channel/UCsNTFJvx3Wi8D3I92pYVZSg
faq_link:None
privacy_link:https://www.iubenda.com/privacy-policy/569672
return_link:None
shipping_link:None
terms_link:None
warranty_link:None
faq_text:None
privacy_text:None
return_text:None
shipping_text:None
terms_text:None
warranty_text:NoneContributingThe Scrape_it is by no means a perfect package and can be improved for sure. If you have any ideas, issues or would like to improve code or documentation please feel free to open issue or pull request. It is my honor to be at help if I can.FAQQ: The object returns the emplty dictionary. What do I do?A: It could be the case the tools used did not find anything though it is certainly an exception rather than a rule. What you can try though: use 'webdriver' method to ensure JavaScript is rendered too, try specify the country, use proxy/VPN in case the website might block requests from your locationQ: Should I pass a root link or any would work?A: Yes, for now at least. Scrape_it will scrape some information still but it relies on finding additional links to scrape the most information possible and I did not set the pipeline to process non-root links yet (I am working on it)
Package: scrapeit-cloud
Scrapeit Cloud Python SDKScrapeit Cloud- Web Scraping API with Proxy Rotation. Get valuable data at scale in HTML format from any website without need for a proxy.Interface to callScrapeit Cloud APIeasily from Python.Installpip install scrapeit-cloudUsageSignup to Scrapeit Cloud toget your API keyand some free credits to get started.>>> from scrapeit_cloud import ScrapeitCloudClient
>>>
>>> client = ScrapeitCloudClient(api_key='INSERT_YOUR_API_KEY_HERE')
>>>
>>> response = client.scrape(
    params={
        "url": "https://example.com"
    }
)
>>>
>>> response.text
'{"status":"ok","scrapingResult":{"content":"<!DOCTYPE html><html><head>\\n    <title>Example Domain</title>\\n\\n    <meta charset=\\"utf-8\\">\\n    <meta http-equiv=\\"Content-type\\" content=\\"text/html; charset=utf-8\\">\\n    <meta name=\\"viewport\\" content=\\"width=device-width, initial-scale=1\\">\\n    <style type=\\"text/css\\">\\n    body {\\n        background-color: #f0f0f2;\\n        margin: 0;\\n        padding: 0;\\n        font-family: -apple-system, system-ui, BlinkMacSystemFont, \\"Segoe UI\\", \\"Open Sans\\", \\"Helvetica Neue\\", Helvetica, Arial, sans-serif;\\n        \\n    }\\n    div {\\n        width: 600px;\\n        margin: 5em auto;\\n        padding: 2em;\\n        background-color: #fdfdff;\\n        border-radius: 0.5em;\\n        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);\\n    }\\n    a:link, a:visited {\\n        color: #38488f;\\n        text-decoration: none;\\n    }\\n    @media (max-width: 700px) {\\n        div {\\n            margin: 0 auto;\\n            width: auto;\\n        }\\n    }\\n    </style>    \\n</head>\\n\\n<body>\\n<div>\\n    <h1>Example Domain</h1>\\n    <p>This domain is for use in illustrative examples in documents. You may use this\\n    domain in literature without prior coordination or asking for permission.</p>\\n    <p><a href=\\"https://www.iana.org/domains/example\\">More information...</a></p>\\n</div>\\n\\n\\n</body></html>","headers":{"accept-ranges":"bytes","age":"477265","cache-control":"max-age=604800","content-encoding":"gzip","content-length":"648","content-type":"text/html; charset=UTF-8","date":"Thu, 11 Aug 2022 11:56:20 GMT","etag":"\\"3147526947\\"","expires":"Thu, 18 Aug 2022 11:56:20 GMT","last-modified":"Thu, 17 Oct 2019 07:18:26 GMT","server":"ECS (nyb/1D17)","vary":"Accept-Encoding","x-cache":"HIT"},"cookies":[]}}'
>>>Scrapeit Cloud supports various parameters to execute a custom JavaScript script, use a premium proxy from a specific geolocation and more.You can find all the supported parameters onScrapeit Cloud documentation.
Package: scrapejs
Javascript frameworks are all the rage now. Unfortunately that means some of my favourite tools, wget and curl, are no longer up to the task. Loading up a page on the browser then opening up source and then copying it is just too cumbersome.I initially thought of creating an extension to help with the situation. Unfortunately an extension would never have the seamless feel of wget or curl for the unfortunate ones like me who are in a relationship with the terminal. Also an extension cannot be used in those quick and dirty bash/perl/python scripts.Hencepyscrapeand its http siblingpyrun. Hope it helps you navigate the unwieldy world of javascript rendering :) .## Installationpip install pyscrapeORClonehttps://github.com/animeshkundu/pyscrapepip install -r requirements.txtpython setup.py install## Test
1. pyscrapehttp://www.google.co.in/2. pyrun -p 1234; curl localhost:1234/scrape?url=http://www.google.co.in/Improvements are welcome.
Package: scrapekit
scrapekitModular scraping convenience framework.Convenience Methods:scrapekit.common.get_user_agent(os, browser): Returns a random User-Agent string.Can filter by OS and browserProxy Provider Module List:IP BurgerInstallationpipinstallscrapekitUsage ExamplesSimple proxified sessionimportscrapekitsession=scrapekit.ipburger.make_session('MyIPBurgerUsername")res=session.get('https://icanhazip.com')print(res.status_code,res.text)# 200 89.46.62.37Proxified session with random Windows Firefox User-Agent:importscrapekituser_agent=scrapekit.common.get_user_agent(os='Windows',browser='Firefox')session=scrapekit.ipburger.make_session('MyIPBurgerUsername',headers={'User-Agent':user_agent})
Package: scrapelib
scrapelibis a library for making requests to less-than-reliable websites.Source:https://github.com/jamesturk/scrapelibDocumentation:https://jamesturk.github.io/scrapelib/Issues:https://github.com/jamesturk/scrapelib/issuesFeaturesscrapeliboriginated as part of theOpen Statesproject to scrape the websites of all 50 state legislatures and as a result
was therefore designed with features desirable when dealing with sites that
have intermittent errors or require rate-limiting.Advantages of using scrapelib over using requests as-is:HTTP(S) and FTP requests via an identical APIsupport for simple caching with pluggable cache backendshighly-configurable request throttingconfigurable retries for non-permanent site failuresAll of the power of the suberbrequestslibrary.Installationscrapelibis onPyPI, and can be installed via any standard package management tool:poetry add scrapelibor:pip install scrapelibExample Usageimportscrapelibs=scrapelib.Scraper(requests_per_minute=10)# Grab Google front pages.get('http://google.com')# Will be throttled to 10 HTTP requests per minutewhileTrue:s.get('http://example.com')