Package: thatscraper
thatscraperScrap more, write less.thatscraperis a selenium adapter.Selenium automates browsers. That's it! What you do with that power is entirely up to you.fromSelenium official webpageSelenium is a simple API to write functional/acceptance tests using Selenium WebDriver. Through Selenium Python API you can access all functionalities of Selenium WebDriver in an intuitive way.fromSelenium with pythonSince there are so many websites full of javascript code, scrapping pages using static methods, as those found on Beautifulsoup, gets harder or impossible. With Selenium WebDriver you can get around with it.However it is very common to make sure the desired element is expected with certain conditions. For that you have to add 'waits' contexts, like:fromseleniumimportwebdriverfromselenium.webdriver.common.byimportByfromselenium.webdriver.support.uiimportWebDriverWaitfromselenium.webdriver.supportimportexpected_conditionsasECdriver=webdriver.Firefox()driver.get("http://somedomain/url_that_delays_loading")try:# the waiting context with the selection of element by idelement=WebDriverWait(driver,10).until(EC.presence_of_element_located((By.ID,"myDynamicElement")))finally:driver.quit()For a single purpose web scraping single script that means some couple of lines like that. However, for a larger testing or scraping project, that can become anoying. With usingthatscraperabove script becomes:importthatscraperastscrawler=ts.Crawler().goto("http://somedomain/url_that_delays_loading")element=crawler.element_id("myDynamicElement")crawler.quit()It is very important to quit the webdriver to avoid memory leakage or overcrowding. Always callquit()method when job is done . However,Crawlercomes with a decorator that make sure the webdriver quits in case of any exception.Installation$pipinstallthatscraperUsagethatscrapercan be used to perform basic actions on webpages, such as clicking buttons, dropdown menu, press keyboard keys, send text or filling forms. It is also suitable to extract data.Instances ofthatscraper.Crawlerare used to navigate pages, perform actions and select elements.Run the webdriver:importtimeimportthatscrapercrawler=thatscraper.Crawler()# open pagecrawler.goto("https://phptravels.com/demo/")# wait long enough so you can check the resulttime.sleep(5)# always quit the drivercrawler.quit()Alternatively, you can crawl pages withou opening browser graphics:crawler=thatscraper.Crawler(headless=True)Choosing the webdriverBy defaultthatscrapermake use of FireFox webdriver (geckodriver), however other drivers can be selected. But make sure you have the one of your choosing, and itspath is added to your enviroment variables. For Linux users, download the werdriver and put it in /usr/bin or /usr/local/bin (windows user,check this outin order to see how to do that in your system).Here's a list of suported browser drivers:BrowserDownload linkChromehttps://sites.google.com/chromium.org/driver/Edgehttps://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/FireFoxhttps://github.com/mozilla/geckodriver/releasesSafarihttps://webkit.org/blog/6900/webdriver-support-in-safari-10/To use other driver, pass it's browser name toCrawlerclass:importtimeimportthatscraperastscrawler=ts.Crawler(browser='chrome')# extract data fromcrawler.goto("https://www.techlistic.com/p/demo-selenium-practice.html")# wait long enough so you can check the resulttime.sleep(5)# always quit the drivercrawler.quit()Elements their childrenElements can be selected with one of four methods:element(value, by): selects an element based on given attributeby, with valuevalue. A list of attributes is given bythatscraper.ATTR_SELECTOR.keys().elements(value, by): selects all elements based on given attributeby, with valuevalue.child_of(element, value, by): selects an element child ofelement(WebElement) based on given attributeby, with valuevalue.children_of: selects all elements child ofelementbased on given attributeby, with valuevalue.Consider the following section of a page:<divclass="form"><inputtype="text"name="first_name"class="first_name input mb1"placeholder="First Name"><inputtype="text"name="last_name"class="last_name input mb1"placeholder="Last Name"><inputtype="text"name="business_name"class="business_name input mb1"placeholder="Business Name">...</div>In order to make sure you select the input tags from that div with class="from", and not another input that the page may contain, first you can select that div, and then select its children:form_element=crawler.element("form","class name")fields=crawler.children_of(form_element,"input","tag name")Sending keysSending inputs or keys to and element is performed one of two methods:send(key, value, by): sendkeyto an element based on given attributeby, with valuevalue.send(key, element): sendkeyto previously selectedelement.Consider the same abov section of a page, and the selected elementsfield. Sending a string to element<input type="text" name="first_name" class="first_name input mb1" placeholder="First Name">can be:crawler.send_to_element(fields[0],"Vagner Bessa")Sending keyboard keys works the same way. Checkthatscraper.Keyorselenium.webdriver.common.keys.Keysfor valid keys.ContributingInterested in contributing? Check out the contributing guidelines. Please note that this project is released with a Code of Conduct. By contributing to this project, you agree to abide by its terms.TODOsFix loggers.Fixrun_script: the 'wait' holds execution. Change the expected condition?Remove unnecessary dependencies.Test all methods (current coverage: 67%).Implement data extraction modules and classes.Link or adapt to a database handler.Implement an API boilerplate builder. That API is to serve data extracted from extractor modules and classes.Addget_attributemethod.Addwaitmethod.Addswitch_to_framemethod.Licensethatscraperwas created by Vagner Bessa. It is licensed under the terms of the MIT license.Creditsthatscraperwas created withcookiecutterand thepy-pkgs-cookiecuttertemplate.