Package: pyscraping
PyScrapingPyScraping is a universal web-scraping util for Python, built with simplicity in mind.InstallationStart to do the installation.pipinstallpyscrapingExampleAll scraping functionality can be accessed either as a function call or a property call. For example, the title can be accessed in two ways:frompyscraping.PyScrapingimportPyScrapingpyscraping=PyScraping("https://google.com")print(pyscraping.title())Documentation<head>tagsGet Website TitleScraping the title from a website is simple.pyscraping.title()Get Meta CharsetTo access the defined charset, you can use the following method:pyscraping.charset()Get Meta ViewportIn some cases, such as the viewport and the meta keywords, the string is representing an array and will be provided as such:pyscraping.viewport()If you need to access the original "viewport"-string, you can useviewportString:pyscraping.viewportString()Get Canonical URLThe canonical URL, if given, can be accessed as shown in the example below:pyscraping.canonical()Get Meta Content-TypeTo access the content type you can use the following functionality:pyscraping.contentType()Get Meta CSFR TokenThe CSFR token method assumes that the token is stored in a meta tag with the name "csrf-token". This is the default for Laravel. You can access it using the following code:pyscraping.csrfToken()Get Meta Author, Description and ImageThe following example shows the extraction of three attributes:the Meta Author,the Meta Description andthe Meta Image URLpyscraping.author()pyscraping.description()pyscraping.image()Get Meta KeywordsThe keywords meta-tag is naturally an array and will be split for your convenience:pyscraping.keywords()Alternatively, you can access the original keyword string:pyscraping.keywordString()Get Meta Open-Graph (OG) DataFetching open-graph data can be done:og:site_nameog:typeog:titleog:descriptionog:urlog:image# Examplepyscraping.openGraph("og:title")# Allpyscraping.openGraph()Get Meta Twitter CardParsing the Twitter Card works similarly:twitter:cardtwitter:titletwitter:descriptiontwitter:urltwitter:image# Examplepyscraping.twitterCard("twitter:title")# Allpyscraping.twitterCard()<body>tagsGet Headings by LevelThere might be cases, in which all headings of a particular level should be retrieved. The example below shows how to do so:pyscraping.h1()pyscraping.h2()pyscraping.h3()pyscraping.h4()pyscraping.h5()pyscraping.h6()Get all ParagraphsThe following example will return a list of all paragraphs (<p>-tags) on the website:pyscraping.p()Get Unordered ListsThe following example will return a list of all list (<ul>-tags) on the website:pyscraping.ul()Get Ordered ListsThe following example will return a list of all list (<ol>-tags) on the website:pyscraping.ol()Get all Image URLsThe following example parses a web-page for images and returns absolute image URLs as an array.pyscraping.images()Get all Images with DetailsIf you are in need of more details the following requests allows you to access attributes of the image tag:pyscraping.imagesDetails()Get all Link ListThe following example parses a web-page for any links and returns an array of absolute URLs:pyscraping.links()Get all Links with DetailsIf you are in need of more details you can access these in a similar way as on the images. Below is an example to access the detailed data of the first link on the page:pyscraping.linksDetails()Custom xPath SelectorsThe following examples of custom selectors should be seen as a starting point for any custom information you need to scrape.pyscraping.filter(element,attribute)Examplepyscraping.filter('div','class="container"')DonatePaypal :Support meContact meContact me via email:rioagungpurnomo.ak@gmail.com, I'm waiting for your input or suggestions.