Package: as-scraper-airflow
as-scraper-airflowPython library for scraping inside Airflow.InstallationTheas-scraperlibrary uses Geckodriver (Firefox) for scraping with the Selenium library.
In order to use it, you need to have an airflow image having the Geckodriver dependency.We have theas-airflowDocker image for you to have airflow ready with the Geckodriver dependency.To use this library follow the next steps:1. Download thedocker-compose.ymlfile from the Airflow docs.Airflow provides thedocker-compose.ymlfile you need for this library.You can directly copy thedocker-compose.ymlfile fromhereor run the following command to download it:curl-LfO'https://airflow.apache.org/docs/apache-airflow/2.3.4/docker-compose.yaml'2. Modify thedocker-compose.ymlfile to use theas-airflowimage.There are two ways of configuring the required docker image for this library.Option a. Create a Dockerfile that extends from thealmiavicas/as-airflowimage.To do this, simply go into thedocker-compose.ymlfile, comment theimageline and uncomment thebuildtag:...version:'3'x-airflow-common:&airflow-common# In order to add custom dependencies or upgrade provider packages you can use your extended image.# Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml# and uncomment the "build" line below, Then run `docker-compose build` to build the images.# image: ${AIRFLOW_IMAGE_NAME:-apache/airflow:2.3.4}build:....Then create yourDockerfileand copy and paste the following lines:FROMalmiavicas/as-airflow:2.2.3RUNpipinstall--no-cache-diras-scraper-airflowOption b. Modify thedocker-compose.yamlto install the library.To do this, go to thedocker-compose.ymlfile and make the following changes:...version:'3'x-airflow-common:&airflow-common# In order to add custom dependencies or upgrade provider packages you can use your extended image.# Comment the image line, place your Dockerfile in the directory where you placed the docker-compose.yaml# and uncomment the "build" line below, Then run `docker-compose build` to build the images.image:${AIRFLOW_IMAGE_NAME:-almiavicas/as-airflow:2.2.3}# build: .environment:..._PIP_ADDITIONAL_REQUIREMENTS:${_PIP_ADDITIONAL_REQUIREMENTS:-as-scraper-airflow}And that's it! You can now start using the as-scraper library.UsageIf you are starting a new Airflow project, before running your containers you need to run the following command to configure volumes:mkdirdags/logs/plugins/You can now rundocker-compose upand you'll have your Airflow environment up & running.Creating a simple scraperLets say that we want to scrapyellowpages.com. Our target data would be the popular cities that we can find in thesitemapurl.Our output data will have two columns:nameof the city andurlwhich is linked to the city. For example, forHouston, we would want the following output:nameurlHoustonhttps://www.yellowpages.com/houston-txDeclaring our Scraper ClassSo first we create a scraper that extends from the Scraper class, and define theCOLUMNSvariable to['name', 'url'].Create thedags/scrapers/yellowpages.pyfile and type the following code into it:fromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']Deciding wether to load javascript or notNow, there are two execution options when running scrapers. We can eitherload javascriptwhich uses theSeleniumlibrary, or not load javascript and use therequestslibrary for http requests.For this example, let's go ahead and use theSeleniumlibrary. To configure this, simply add the following variable to your scraper:fromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']LOAD_JAVASCRIPT=TrueDefining thescrape_handlerAnd the magic comes in the next step. We will define thescrape_handlermethod in our class, which will have the responsibility to scrape a given url and extract the data from it.All scrapers must define thescrape_handlermethod.fromtypingimportOptionalfromselenium.webdriverimportFirefoxfromselenium.webdriver.common.byimportByimportpandasaspdfromas_scraper.scraperimportScraperclassYellowPagesScraper(Scraper):COLUMNS=['name','url']LOAD_JAVASCRIPT=Truedefscrape_handler(self,url:str,html:Optional[str]=None,driver:Optional[Firefox]=None,**kwargs)->pd.DataFrame:rows=[]div_tag=driver.find_element(By.CLASS_NAME,"row-content")div_tag=div_tag.find_element(By.CLASS_NAME,"row")section_tags=div_tag.find_elements(By.TAG_NAME,"section")forsection_taginsection_tags:a_tags=section_tag.find_elements(By.TAG_NAME,"a")fora_tagina_tags:city_name=a_tag.textcity_url=a_tag.get_attribute("href")rows.append({"name":city_name,"url":city_url})df=pd.DataFrame(rows,columns=self.COLUMNS)returndfCreating the DAG.Now we want to create a DAG that will trigger the scraper. For that we will use theScraperToLogsOperator.As we mentioned before, the target url for our scraper is thehttps://www.yellowpages.com/sitemap. In the Dag definition file we will define the url that we want to scrape.There are other ways of specifying urls based on a discovery strategy. However, for this example it's not required.Create thedags/yellowpages.pyfile and copy the following content into it:fromdatetimeimportdatetime,timedeltafromairflow.modelsimportDAGfromscrapers.yellowpagesimportYellowPagesScraperfromas_scraper_airflow.operatorsimportScraperToLogsOperatorwithDAG(dag_id="yellow_pages",catchup=False,default_args={'depends_on_past':False,'email':['airflow@example.com'],'email_on_failure':False,'email_on_retry':False,'retries':1,'retry_delay':timedelta(minutes=5),},description="A simple Scraper DAG",schedule_interval=timedelta(days=1),start_date=datetime(2022,8,4),catchup=False,)asdag:t1=ScraperToLogsOperator(scraper_cls=YellowPagesScraper,urls=['https://www.yellowpages.com/sitemap'],task_id='scrape',save_errors=True,)And that's it! Head to the airflow webserver to run your DAG!