Package: as-scraper
as-scraperPython library for scraping using SeleniumIf you are looking for the library implemented inside airflow, go tohttps://github.com/Avila-Systems/as-scraper-airflow.InstallationTheas-scraperlibrary uses Geckodriver (Firefox) for scraping with the Selenium library.
In order to use it, you need to have an Geckodriver dependency. Check theselenium documentationfor details about how to install the Firefox browser driver.UsageCreating a simple scraperLets say that we want to scrapyellowpages.com. Our target data would be the popular cities that we can find in thesitemapurl.Our output data will have two columns:nameof the city andurlwhich is linked to the city. For example, forHouston, we would want the following output:nameurlHoustonhttps://www.yellowpages.com/houston-txDeclaring our Scraper ClassSo first we create a scraper that extends from the Scraper class, and define theCOLUMNSvariable to['name', 'url'].Create thescrapers/yellowpages.pyfile and type the following code into it:fromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']Deciding wether to load javascript or notNow, there are two execution options when running scrapers. We can eitherload javascriptwhich uses theSeleniumlibrary, or not load javascript and use therequestslibrary for http requests.For this example, let's go ahead and use theSeleniumlibrary. To configure this, simply add the following variable to your scraper:fromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']LOAD_JAVASCRIPT=TrueDefining thescrape_handlerAnd the magic comes in the next step. We will define thescrape_handlermethod in our class, which will have the responsibility to scrape a given url and extract the data from it.All scrapers must define thescrape_handlermethod.fromtypingimportOptionalfromselenium.webdriverimportFirefoxfromselenium.webdriver.common.byimportByimportpandasaspdfromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']LOAD_JAVASCRIPT=Truedefscrape_handler(self,url:str,html:Optional[str]=None,driver:Optional[Firefox]=None,**kwargs)->pd.DataFrame:rows=[]div_tag=driver.find_element(By.CLASS_NAME,"row-content")div_tag=div_tag.find_element(By.CLASS_NAME,"row")section_tags=div_tag.find_elements(By.TAG_NAME,"section")forsection_taginsection_tags:a_tags=section_tag.find_elements(By.TAG_NAME,"a")fora_tagina_tags:city_name=a_tag.textcity_url=a_tag.get_attribute("href")rows.append({"name":city_name,"url":city_url})df=pd.DataFrame(rows,columns=self.COLUMNS)returndfExecutionFinally, to execute the scraper you must call the *executemethod.fromtypingimportOptionalfromselenium.webdriverimportFirefoxfromselenium.webdriver.common.byimportByimportpandasaspdfromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']LOAD_JAVASCRIPT=Truedefscrape_handler(self,url:str,html:Optional[str]=None,driver:Optional[Firefox]=None,**kwargs)->pd.DataFrame:rows=[]div_tag=driver.find_element(By.CLASS_NAME,"row-content")div_tag=div_tag.find_element(By.CLASS_NAME,"row")section_tags=div_tag.find_elements(By.TAG_NAME,"section")forsection_taginsection_tags:a_tags=section_tag.find_elements(By.TAG_NAME,"a")fora_tagina_tags:city_name=a_tag.textcity_url=a_tag.get_attribute("href")rows.append({"name":city_name,"url":city_url})df=pd.DataFrame(rows,columns=self.COLUMNS)returndfif__name__=='__main__':urls=['https://www.yellowpages.com/sitemap']scraper=YellowPagesScraper(urls)results,errors=scraper.execute()print(results)print(errors)Now go ahead and runpython scrapers/yellowpages.py. Have fun!