Package: ezscrape
LicenseVersionTravis CICoverageWheelImplementationStatusDownloadsSupported versionsezscrapeezscrape provides a boilerplate for simple scraping tasks.It provide generic access to scraping functionality without exposing the user directly the the underlying libraries used (e.g. requests, selenium) when using the scraping functionality.
The used scraper is chosen based on the specified config parameters and will prefer the most flexibe / least resource intensive ones if possible.The exceptions of the underlying modules are handled and converted into the status of the result object.Setup RequirementsSetup Chrome and WebdriverFor some websites selenium will be used for scraping (e.g. requests doesn't support javascript)For that purpose the following environment variables need to be set otherwise an exception will be raised when the code is using selenium.The Chrome ExecutableCHROME_EXEC_PATHThe Chrome Webdriver ExecutableCHROME_WEBDRIVER_PATHUsageThe basic concept of a simple scrape isCreate the Scrape Config with a UrlOptional - set additional parameters on the scrape configScrape with the given ConfigCheck the Return Object if the Scrape was succesfullGet the HTML from the Return ObjectScrape a simple HTML Pageimport ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import ScrapeStatus

result = scraper.scrape_url(ScrapeConfig('http://www.website.com'))

if result.status == ScrapeStatus.SUCCESS:
    html = result.first_page.html
else:
    print(result.error_msg)Scrape a Page with Multiple Pagesimport ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import WaitForXpathElem
from ezscrape.scraping.core import ScrapeStatus

config = ScrapeConfig('http://www.website.com')
# Add contition to wait until Element with "title='id'" is loaded"
config.wait_for_elem_list.append(WaitForXpathElem(R'''//a[@title='id']'''))

result = scraper.scrape_url(config)

for page in result:
    if page.status == ScrapeStatus.SUCCESS:
        html = page.html
    else:
        print(result.error_msg)Scrape a Page and wait until an Element is Loadedimport ezscrape.scraping.scraper as scraper
from ezscrape.scraping.core import ScrapeConfig
from ezscrape.scraping.core import WaitForXpathElem
from ezscrape.scraping.core import ScrapeStatus


config = ScrapeConfig('http://www.website.com')
# Add contition to wait until Element with "title='id'" is loaded"
config.wait_for_elem_list.append(WaitForXpathElem(R'''//a[@title='id']'''))

result = scraper.scrape_url(config)

if result.status == ScrapeStatus.SUCCESS:
    html = result.first_page.html
else:
    print(result.error_msg)Scrape Configezscrape.scraping.core.ScrapeConfigThe url is specified when creating the object.from ezscrape.scraping.core import ScrapeConfig

config = ScrapeConfig('http://some-url.com')Additional parameters can be specifiedOptionPurposeTypeDefaultExample Use CaseScrapeConfig.urlThe URL used for the requeststrN/ARequired for all RequestsScrapeConfig.request_timeoutThe timeout in seconds of the requestlong15Wait longer before timeout in a slow Network environment.ScrapeConfig.page_load_waitTime ti wait until a page is loaded completely before it times outint5.0Specify a longer time if the page loads dynamic elements slowlyScrapeConfig.proxy_httpHTTP Proxy to usestrN/ASend the request through an HTTP proxy (Proxy needs to support the Target protocol i.e. HTTP/HTTPS)ScrapeConfig.proxy_httpsHTTPS Proxy to usestrN/ASend the request through an HTTPS proxy (Proxy needs to support the Target protocol i.e. HTTP/HTTPS)ScrapeConfig.useragentCustom Useragent to usestrInternally ChosenUser want to scrape with a custom UseragentScrapeConfig.max_pagesMaximum Pages to collect if "next_button" specifiesint15User only wants to return 3 Pages max even if more pages availableScrapeConfig.next_buttonAdd a button element that needs to be loaded and clicked for ultiple pagesezscrape.scraping.core.WaitForPageElemor one of the subtypes e.g.ezscrape.scraping.core.WaitForXpathElemN/AUser wants to return multiple pages if the next page links are generated with JavascriptScrapeConfig.wait_for_elem_listA list of Elements that need to be loaded on the page before returning the scrape resultList ofezscrape.scraping.core.WaitForPageElemor one of the subtypes e.g.ezscrape.scraping.core.WaitForXpathElemN/AUser is interested in multiple elements of a Javascript/Ajax page and needs to wait for all to load completely.Scrape StatusThe following statuses are supported in ezscrape.scraping.core.ScrapeStatusStatusMeaningScrapeStatus.SUCCESSScrape SuccesfullScrapeStatus.TIMEOUTA timeout error occuredScrapeStatus.PROXY_ERRORA proxy error occuredScrapeStatus.ERRORA generic error occuredFor non Success cases, additional error details are given in theScrapeResultobjectScrape ResultThe following attributes are available in ezscrape.scraping.core.ScrapeResultsAttributePurposeTypeScrapeResult.urlThe url ScrapedstrScrapeResult.caller_ipThe caller IP.This is not set for all cases. But where it is it should be reliable e.g. if Scraped through proxy, the proxy IP should be shown)strScrapeResult.statusThe overall status of the Scrapeezscrape.scraping.core.ScrapeStatusScrapeResult.error_msgThe error message if the result is not SUCCESSstrrequest_time_msThe combined scrape time of all pages scrapedfloatfirst_pageThe ScrapePage scraped (first if multiple pages)ezscrape.scraping.core.ScrapePageScrape PageThe following attributes are available in ezscrape.scraping.core.ScrapePageAttributePurposeTypehtmlThe HTML content scrapedstrrequest_time_msthe scrape duration for this pagefloatstatusThe scrape status for this pageScrapePage doesn't have it's own error message. For details check ScrapeResult.error_msgezscrape.scraping.core.ScrapeStatusContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.LicenseGPLv3[1]: # Scrape Result
[2]: # Scrape Result