Package: screpe
ScrepeHigh-level Python web scraping.(Crepes not included)InstallationUsingpipThe Python package installer makes it easy to installscrepe.pip install screpeUsinggitOtherwise, clone this repository to your local machine with git, then install
with Python.git clone https://github.com/shanedrabing/screpe.gitcd screpepython setup.py installYou can also simply downloadscrepe.pyand place it in your working
directory.Getting StartedInitializing ScrepeImport the module in Python, and initialize aScrepeobject.fromscrepeimportScrepe# do we want the scraper to remember previous responses?scr=Screpe(is_caching=True)All methods in this module live on theScrepeclass, so there is no need to
import anything else!Requests and BeautifulSoupIf you are familiar with web scraping in Python, then you have probably used
therequestsandbs4packages before. There are a couple of static methods
that Screpe provides to make their usage even easier!# a webpage we want to scrapeurl="https://www.wikipedia.org"# returns None if status code is not 200html=Screpe.get(url)# can handle None as input, parses the HTML with `lxml`soup=Screpe.cook(html)# check to make sure we have a soup object, otherwise see bs4ifsoupisnotNone:print(soup.select_one("h1"))We can marry these two functions with the instance methodScrepe.dine.
Remember that we have thescrobject from the section above.# get and cooksoup=scr.dine(url)Responses fromScrepe.dinecan be cached and adhere to rate-limiting (see
next sections).Downloading a Webpage or a FileCommonly, we just want to download an image, webpage, generic file, etc. Let us
see how to do this with Screpe!# locator to file we want, local path to where we want iturl="https://www.python.org/static/img/python-logo.png"fpath="logo.png"# let us use our object to download the filescr.download(url,fpath)Note that the URL can be pretty much any filetype as the response is saved in
binary, just make sure you get the filetype right.Downloading an HTML TableSometimes there is a nice HTML table on a webpage that we want as more
interoperable format. Thepandaspackage can do this easily, and we take
advantage of that with Screpe.# this webpage contains a table that we want to downloadurl="https://www.multpl.com/cpi/table/by-year"# we save the tables as a CSV filefpath="table.csv"# the `which` parameter decides what table to savescr.download_table(url,fpath,which=0)SeleniumOne of the most challenging tasks in web scraping is to deal with dynamic pages
that require a web browser to work properly. Thankfully, theseleniumpackage
is pretty good at this. Screpe removes headaches surrounding Selenium.# the homepage of Wikipedia has a search boxurl="https://www.wikipedia.org"# let us open the page in a webdriverscr.open(url)# we can click on the input boxscr.click("input#searchInput")# ...enter a search termscr.send_keys("Selenium")# ...and hit return to initiate the searchscr.bide(lambda:scr.send_enter())# note that the `Screpe.bide` function takes a function as input, checks what# page it is on, calls the function, and waits for the next page to load# we can use bs4 once the next page loads!soup=scr.source()Caching does not apply to the Selenium-related functions, it is a stateful activity and we cannot simply load an old webdriver state.Asynchronous RequestsScrepe usesconcurrent.futuresto spawn a bunch of threads that can work
simulatanously to retrieve webpages.# a collection of URLsurls=["https://www.wikipedia.org/wiki/Dog","https://www.wikipedia.org/wiki/Cat","https://www.wikipedia.org/wiki/Sheep"]# we want soup objects for allsoups=scr.dine_many(urls)Rate-LimitingIf sites are sensitive to how often you can request, consider setting yourScrepeobject to halt before sending another request.# we give the function a duration, but can find that from a raterate_per_second=2duration_in_seconds=1/rate_per_second# inform your scraper to not surpass the request intervalscr.halt_duration(duration_in_seconds)Note that cached responses do not adhere to the rate limit. After all, we
already have the reponse!CachingSometimes, we have to request many pages. So that we do not waste bandwidth, or
a rate limit, we can use cached reponses. Note that caching is on by default,
turn it off if you want real-time responses.# turn caching onscr.cache_on()# ...or turn it offscr.cache_off()We can save and load the cache between sessions for even more greatness!# where shall we save the cache? (binary file)fpath="cahce.bin"# save the cachescr.cache_save(fpath)# load the cachescr.cache_load(fpath)# clear the cachescr.cache_clear()LicenseMIT LicenseCopyright (c) 2022Shane Drabing