Package: harambe-sdk
🦍 Harambe Web extraction SDK 🦍HarambeHarambe is the extraction SDK for Reworkd. It provides a simple interface
for interacting with the web. It provides a unified interface and runtime
for both manual and automatically created web extractorsSetup and InstallationFolder StructureExample ScraperDetail Only ScraperListing ScraperRunning a ScraperSubmitting a PRSetup and InstallationTo install Harambe, clone the repository and install the requirements.
All requirements are managed viapoetry.gitclonehttps://github.com/reworkd/harambe.git
poetryinstallFolder StructureThescrapersfolder contains all the scrapers. Theharambefolder
contains the SDK and utility functions.Example ScraperGenerally scrapers come in two types,listingand detailscrapers. Listing
scrapers are used to collect a list of items to scrape. Detail scrapers
are used to scrape the details of a single item.If all the items that you want to scrape are available on a single page,
then you can use a detail scraper to scrape all the items. If the
items are spread across multiple pages, then you will need to use a
listing scraper to collect the items and then use a detail scraper to
scrape the details of each item.ALL scrapers must be decorated withSDK.scraper. This decorator
registers the scraper with the SDK and provides the SDK with the
necessary information to run the scraper.Detail Only ScraperShown below is an example detail scraper. Thecontextparameter is
used to pass data from the listing scraper to the detail scraper.
In this example, thecontextparameter is used to pass the phoneimportasynciofromtypingimportAnyfromplaywright.async_apiimportPagefromharambeimportSDKfromharambeimportPlaywrightUtilsasPuSELECTORS={"last_page":"","list_view":"//div[@class='et_pb_blurb_content']","name":"//h4/*[self::span or self::a]","fax":">Fax.*?strong>(.*?)<br>",# etc...}# Annotation registers the scraper with the SDK@SDK.scraper(domain="https://apprhs.org/our-locations/",stage="detail")asyncdefscrape(sdk:SDK,url:str,*args:Any,**kwargs:Any)->None:page:Page=sdk.pageawaitpage.goto(url)locations=awaitpage.locator(SELECTORS["list_view"]).all()forlocationinlocations:# Save the data to the database or fileawaitsdk.save_data({"name":awaitPu.get_text(location,SELECTORS["name"]),"fax":awaitPu.parse_by_regex(location,SELECTORS["fax"]),# etc...})if__name__=="__main__":asyncio.run(SDK.run(scrape,"https://apprhs.org/our-locations/"))Listing ScraperShown below is an example listing scraper. UseSDK.enqueueto to add
urls that will need be scraped by the detail scraper. Thecontextparameter is used to pass data from the listing scraper to the detail
scraper.importasynciofromtypingimportAnyfromplaywright.async_apiimportPagefromharambeimportSDKSELECTORS={}@SDK.scraper(domain="https://example.org",stage="listing")asyncdefscrape(sdk:SDK,url:str,*args:Any,**kwargs:Any)->None:page:Page=sdk.pageawaitpage.goto(url)forurlin["https://example.org/1","https://example.org/2","https://example.org/3",]:# Imagine these are locatorsawaitsdk.enqueue(url,context={"phone":"123-456-7890",# Some data from the listing page that we want to pass to the detail page, (optional)"foo":"bar","baz":"qux",},)@SDK.scraper(domain="https://example.org",stage="detail")asyncdefscrape_detail(sdk:SDK,url:str,context:Any)->None:page:Page=sdk.pageawaitpage.goto(url)# Grab all properties from the contextdetail={**context}detail["fax"]="123-456-7890"# Some data grabbed from the detail pagedetail["type"]="Hospital"# Some data grabbed from the detail pageawaitsdk.save_data(detail)# Save the data to the databaseif__name__=="__main__":asyncio.run(SDK.run(scrape,"https://navicenthealth.org/locations"))asyncio.run(SDK.run_from_file(scrape_detail))Running a ScraperYou can use poetry to run a scraper. Theruncommand takes the
scraper function and the url to scrape. Therun_from_filecommand
takes the scraper function and the path to the file containing the
urls to scrape.poetryrunpythonpoetryrunpythonscrapers/medical/apprhs.pySubmitting a PRBefore submitting a PR, please run the following commands to ensure
that your code is formatted correctly../format.shHappy extraction! 🦍