Package: samssimplescraper
Simple ScraperExplore the docs »View Demo·Report Bug·Request FeatureTable of ContentsAbout The ProjectBuilt WithGetting StartedPrerequisitesInstallationUsageRoadmapContributingLicenseContactAbout The ProjectWrote this program to scrape some sitemaps and the following sitemap links on multiple servers. In order to save time it was pip packaged for easy repeated use.(back to top)Built WithPythonBeautifulsoup(back to top)Getting StartedFollow the installation instructions. The docstrings have detailed explainations for use.PrerequisitesThis program uses python 3.8Installationpipinstall package and use as needed.Installpippackagepipinstallsamssimplescraper==0.1.3(back to top)UsageThe package has two modules.sitemapscraperis used to scrape sitemaps and can also scrape further levels of sub-sitemaps The methods will return lists of the scraped links that can be used to scrape the wanted links.scraperis used to scrape the the list that is returned from the sitemapscraper or a user-made list of links. There is also a method that returns a status check of how many links have been scraped of the total.(back to top)RoadmapFind sitemap for the site you are looking to scrape. Some tips can be found here:how-to-find-your-sitemaphttps://writemaps.com/blog/how-to-find-your-sitemap/Scrape sitemap:fromsamssimplescraperimportLinksRetriever# instantiate LinksRetriever with the sitemap you wish to scrapelinks_retriever=LinksRetriever(url='https://www.example.com/sitemap_index.xml')# get a list of the link using .get_sitemap_links method, can also add filtermainpage_links=links_retriever.get_sitemap_links(tag='loc')# if website has more layers use the method to get the links on those pagesfinal_links=links_retriever.get_next_links(links=mainpage_links,tag='loc')Note:If you are not going to continue scraping in the same script then be sure to save your list using pickle:importpickle# the data folder is automatically created when LinksRetriever is instantiatedwithopen('./data/pickled_lists/sitemap_links_list.pkl','wb')asfp:pickle.dump(final_links,fp)Now you can scrape the list of links that theLinksRetrievermodule has produced for you. The files will be saved in thedata/scraped_htmlfolder.fromsamssimplescraperimportScraper# pass the list of links and for naming purposes the root_urlScraper.get_html(link_list=final_links,root_url='https://www.example.com/)See theopen issuesfor a full list of proposed features (and known issues).(back to top)ContributingContributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make aregreatly appreciated.If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement".
Don't forget to give the project a star! Thanks again!Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull Request(back to top)LicenseDistributed under the MIT License. SeeLICENSE.txtfor more information.(back to top)ContactSamuel Adams McGuire -samuelmcguire@engineer.comPypi Link:https://pypi.org/project/samssimplescraper/0.1.3/Linkedin:LinkedInProject Link:https://github.com/SamuelAdamsMcGuire/simplescraper(back to top)