Package: keyscraper
keyscraper Package DocumentationThis library provides various functions which simplifies webpage scraping.There are three modules in this package.utils-basic utilitiesstaticscraper-used to scrape raw html datadynamicscraper-used to scrape html data rendered byJavaScriptTo install this package, type in command prompt:pip install keyscraper[1] Basic Utilities(1-A)TimeName- Generating a file name composed of the current time:TimeName(mode = "default")argumentoptionaldefaultavailablemodeyesTimeName.MODE_KEYWINDTimeName.MODE_KEYWIND, TimeName.MODE_DATETIME, "default"self.get_name(basename = "", extension = "", f_datetime = None)argumentoptionaldefaultavailablebasenameyes""[ string type ]extensionyes""[ string type ]f_datetimeno[ string type ]There are two available modes: "keywind" and "datetime". By default, "keywind" is used.In mode "keywind", the date is formatted asD-{month}{day}{year}where{month}consists of asingle character,{day}is a2-digitnumber ranging from01 to 31and{year}is a4-digitnumber such as2000.Jan.Feb.Mar.Apr.MayJun.Jul.Aug.Sep.Oct.Nov.Dec.ifmaMjJAsondFor example, onDecember 7th of 2000,D-d072000will be the resulting date string.In mode "keywind", the time is formatted asT-{hour}{minute}{second}where{hour}consists of a2-digitnumber ranging from00 to 23, both{minute}and{second}are a2-digitnumber ranging from00 to 59.For example, at05:43:07 PM., the resulting time string will beT-174307.For example, at01:23:45 AM.onApril 26th, 1986, the resulting string will be{basename}_D-a261986_T-012345{extension}.In mode "datetime", the programmer must pass astrftimestring. The complete documentation to datetime formatting is linkedhere.(1-A-1) Example of using TimeName (mode:keywind).from keyscraper.utils import TimeName
mode = TimeName.MODE_KEYWIND # or TimeName.MODE_DATETIME
name = "images"
extension = ".jpg"
timename = TimeName(mode).get_name(name, extension)
print(timename) # "images_D-d072000_T-012345.jpg"(1-A-2) Example of using TimeName (mode:datetime).from keyscraper.utils import TimeName
mode = TimeName.MODE_DATETIME # or TimeName.MODE_KEYWIND
format_string = "_%y%m%d-%H%M%S"
name = "images"
extension = ".jpg"
timename = TimeName(mode).get_name(name, extension, format_string)
print(timename) # "images_001207-012345.jpg"(1-B)FileName- Dividing a filename into folder, file and extension:FileName(filename, mode = "default")argumentoptionaldefaultavailablefilenameno[ string type ]modeyesFileName.MODE_FORWARDSLASHFileName.MODE_FORWARDSLASH, FileName.MODE_BACKWARDSLASHself.__getitem__(key = "all")argumentoptionaldefaultavailablekeyno"all""all", "folder", "name", "extension"(1-B-1) Example of using FileNamefrom keyscraper.utils import FileName
mode = FileName.MODE_FORWARDSLASH
filename = "C:/Users/VIN/Desktop/utils.py"
name_object = FileName(filename)
full_name = name_object["all"]
file_name = name_object["name"]
folder_name = name_object["folder"]
extension = name_object["extension"]
print(full_name) # "C:/Users/VIN/Desktop/utils.py"
print(folder_name) # "C:/Users/VIN/Desktop/"
print(file_name) # "utils"
print(extension) # ".py"(1-C)FileRetrieve- Downloading a file from a direct URL:FileRetrieve(directlink, filename = None, buffer = 4096, progress_bar = False, overwrite = None)argumentoptionaldefaultavailabledirectlinkno[ string type ]filenameyes[ string type ]bufferyes4096[ integer (>0) type ]progress_baryesFalseTrue, FalseoverwriteyesNoneNone, True, FalseIf overwrite isNone, the programmer will be asked to enter(Y/N)on each download.self.simple_retrieve()Calling this function will download the file from the target URL and save it to disk with the provided filename.(1-C-1) Example of using FileRetrievefrom keyscraper.utils import FileRetrieve
url = "http://www.lenna.org/len_top.jpg"
filename = "lenna.jpg"
progress_bar = True
overwrite = True
downloader = FileRetrieve(url, filename = filename, progress_bar = progress_bar, overwrite = overwrite)
downloader.simple_retrieve()(1-D)ImageGrabber- Downloading an image from a direct URL:ImageGrabber(filename, progressBar = False, url_timeout = None)argumentoptionaldefaultavailablefilenameno[ string type ]progressBaryesFalseTrue, Falseurl_timeoutyes600[ integer (>0) type ]The URL request will be open for a maximum ofurl_timeoutseconds.self.retrieve(directlink, overwrite = None, timeout = None)argumentoptionaldefaultavailabledirectlinkno[ string type ]overwriteyesNoneNone, True, FalsetimeoutyesNoneNone, [ integer (>0) type ]If the image hasn't finished downloading intimeoutseconds, the process will terminate.If overwrite isNone, the programmer will be asked to enter(Y/N)on each download.(1-D-1) Example of using ImageGrabberfrom keyscraper.utils import ImageGrabber
url = "http://www.lenna.org/len_top.jpg"
filename = "lenna.jpg"
progressBar = True
url_timeout = 60
downloader = ImageGrabber(filename, progressBar = progressBar, url_timeout = url_timeout)
downloader.retrieve(url, overwrite = True, timeout = 15)[2] Static Scraper(2-A)SSFormat- Defining the node attributes to scrape:SSFormat(element_type, **kwargs)argumentoptionaldefaultavailableelement_typeno[ string type ]search_typeyesNoneNone, [ string type ]search_clueyesNoneNone, [ string type ]multipleyesFalseTrue, FalseextractyesNoneNone, [ function (1-arg) type ]formatyesNoneNone, [ function (1-arg) type ]nicknameyesNoneNone, [ string type ]filteryesNoneNone, [ function (1-arg) type ]keepyesTrueTrue, Falseself.__getitem__(key)argumentoptionaldefaultavailablekeyno"element_type", "search_type", "search_clue", "multiple", "extract", "format", "nickname", "filter", "keep"self.get_value(key)argumentoptionaldefaultavailablekeyno"element_type", "search_type", "search_clue", "multiple", "extract", "format", "nickname", "filter", "keep"(2-B)SSInfo- Defining information needed for scraping:SSInfo(f_site, f_page, f_item, f_attr)argumentoptionaldefaultavailablef_siteno[ string type ]f_pageno[ string type ]f_itemno[ SSFormat type ]f_attrno[ list-SSFormat type ]self.__getitem__(key)argumentoptionaldefaultavailablekeyno"f_site", "f_page", "f_item", "f_attr"self.format_page(page)argumentoptionaldefaultavailablepageno[ integer/string type ]Iff_pageis not an empty string, page is put intof_pageinside curly braces. For instance, iff_page="page-{}.html"andpage=5, this function will return"page-5.html". On the contrary, iff_page="", the function will return"".(2-C)StaticScraper- Scraping a static webpage:StaticScraper(info, filename = None, mode = "default", timesleep = 0, **kwargs)argumentoptionaldefaultavailableinfono[ SSInfo type ]filenameyesNoneNone, [ string type ]modeyesStaticScraper.MODE_FILEStaticScraper.MODE_FILE, StaticScraper.MODE_READtimesleepyes0[ integer/float (>=0) type ]bufferyes100[ integer (>0) type ]self.scrape(start = 1, pages = 1)argumentoptionaldefaultavailablestartyes1[ integer (>0) type ]pagesyes1[ integer (>0) type ](2-C-1) Example of using StaticScraperfrom keyscraper.staticscraper import SSFormat, SSInfo, StaticScraper
f_site = "http://books.toscrape.com/"
f_page = "catalogue/page-{}.html"
f_item = SSFormat(element_type = "li", search_type = "class_", search_clue = "col-xs-6 col-sm-4 col-md-3 col-lg-3", multiple = True)
price = SSFormat(element_type = "p", search_type = "class_", search_clue = "price_color", extract = "text", nickname = "price")
url = SSFormat(element_type = "a", extract = "href", nickname = "link")
f_attr = [ price, url ]
info = SSInfo(f_site, f_page, f_item, f_attr)
scraper = StaticScraper(info)
scraper.scrape(start = 1, pages = 15)[3] Dynamic Scraper(3-A)DSFormat- Defining the node attributes to scrape:DSFormat(element_type, **kwargs)argumentoptionaldefaultavailablexpathno[ string type ]relativeyesFalseTrue, FalsemultipleyesFalseTrue, FalseextractyesNoneNone, [ function (1-arg) type ]formatyesNoneNone, [ function (1-arg) type ]filteryesNoneNone, [ function (1-arg) type ]retryyesNoneNone, [ function (1-arg) type ]callbackyesNoneNone, [ function (1-arg) type ]nicknameyesNoneNone, [ string type ]keepyesTrueTrue, FalseclickyesFalseTrue, FalseIn dynamic scraper, the path to each item/attribute must be provided as x-path.If the xpath of an attribute is relative to the item (parent),relativemust be set to True.To scrape multiple items,multiplemust be set to True.If we want to extract thehrefattribute from theatag, we should setextractto"href".If we want to format a particular attribute before saving it to file, we should define a function and pass it to the argumentformat. The following is an example:from keyscraper.dynamicscraper import DSFormat
def strip_spaces(attribute):
    return attribute.strip(" ")
DSFormat(format = strip_spaces)If we want to filter out items whose attributes don't satisfy a certain condition, we should define a function and pass it to the argumentfilter. The following is an example:from keyscraper.dynamicscraper import DSFormat
def filter_prices(price):
    price = float(price)
    return (price <= 50) # True to keep the item
DSFormat(filter = filter_prices)In cases where we must wait for a specific item to render, we should define a function and pass it to the argumentretry. If this function returns True, the item is saved; otherwise, we wait for it to change. The following is an example:from keyscraper.dynamicscraper import DSFormat
def retry(attribute):
  return (attribute[:4] == "data") # keep trying until False
DSFormat(retry = retry)InMODE_READ, we may want to add the scraped data to a list; therefore, we should define a function and pass it to the argumentcallback. The following is an example:from keyscraper.dynamicscraper import DSFormat
scraped = []
def callback(attribute):
    global scraped
    scraped.append(attribute)
    return attribute
DSFormat(callback = callback)In the csv file, we can assign a custom column name for each attribute. It can be done by passing a string to the argumentnickname.In cases where some attributes aren't needed further on, we can setkeepto False so the column will be dropped when saving to csv file.If the item/attribute must be clicked before the desired data is available,clickshould be set to True.self.__getitem__(key)argumentoptionaldefaultavailablekeyno"xpath", "relative", "multiple", "extract", "format", "filter", "retry", "callback", "nickname", "keep", "click"(3-B)DSInfo- Defining information needed for scraping:DSInfo(f_site, f_page, f_item, f_attr)argumentoptionaldefaultavailablef_siteno[ string type ]f_pageno[ string type ]f_itemno[ DSFormat type ]f_attrno[ list-DSFormat type ]self.__getitem__(key)argumentoptionaldefaultavailablekeyno"f_site", "f_page", "f_item", "f_attr"self.format_page(page)argumentoptionaldefaultavailablepageno[ integer/string type ]Iff_pageis not an empty string, page is put intof_pageinside curly braces. For instance, iff_page="page-{}.html"andpage=5, this function will return"page-5.html". On the contrary, iff_page="", the function will return"".(3-C)DriverOptions- Defining driver:DriverOptions(mode = "default", path = None, window = True)argumentoptionaldefaultavailablemodeyesDriverOptions.MODE_CHROME"default", DriverOptions.MODE_CHROME, DriverOptions.MODE_FIREFOXpathyesNoneNone, [ string type ]windowyesTrueTrue, FalseIn order to use dynamic scraper, a (browser) driver must be provided. As ofFebruary 6th of 2022,Google ChromeandMozilla Firefoxare supported.The (file) path to the driver (executable) must be provided. By default, the program will search for the driver in thesame folderas it's run in or folders stored in thePATH environment variable.To download a driver for Google Chrome, visithere.To download a driver for Mozilla Firefox, visithere.To hide the browser, setwindowto False.(3-D)DynamicScraper- Scraping:DynamicScraper(info, driveroptions, mode = "default", filename = None, timesleep = 0, buttonPath = None, itemWait = 1, **kwargs)argumentoptionaldefaultavailableinfono[ DSInfo type ]driveroptionsno[ DriverOptions type ]modeyesDynamicScraper.MODE_READ"default", DynamicScraper.MODE_FILE, DynamicScraper.MODE_READfilenameyesNoneNone, [ string type ]timesleepyes0[ integer (>=0) type ]buttonPathyesNoneNone, [ string type ]itemWaityes1[ integer/float (>=0) type ]bufferyes100[ integer (>0) type ]There are two modes available for dynamic scraper,MODE_FILEwill save the scrape result in acsvfile, whileMODE_READwill simply scrape the webpage and the data can be accessed incallback.In MODE_FILE, a filename should be provided. By default, atime namewill be generated for the csv file.To slow down the scraping time, an integer can be passed to thetimesleepargument. The scraping of two consecutive pages will be separated by at leasttimesleepseconds.In cases where a load-more button exists on a single page, the x-path to that button can be provided to the argument buttonPath.If each item must be clicked to render its content, a number can be passed to the argumentitemWait. Two consecutive item clicks will be separated by at leastitemWaitseconds.In MODE_FILE, if we want to save the scrape result once every other 10 items, we should setbufferto 10.self.scrape(start = 1, pages = 1, perPage = None)argumentoptionaldefaultavailablestartyes1[ integer (>0) type ]pagesyes1[ integer (>0) type ]perPageyesNoneNone, [ integer (>0) type ]The dynamic scraper will scrapepagespages onward from pagestart.In cases where there are too many items on each page, we can setperPageto 50 to scrape just 50 items per page.(3-D-1) Example of using DynamicScraperfrom keyscraper.dynamicscraper import DSFormat, DSInfo, DriverOptions, DynamicScraper
f_site = "https://www.ebay.com/sch/"
f_page = "i.html?_nkw=cpu&_pgn={}"
f_item = DSFormat(xpath = "(//li[contains(@class, 's-item s-item__pl-on-bottom s-item--watch-at-corner')])", multiple = True)
price = DSFormat(xpath = "//span[contains(@class, 's-item__price')]", relative = True, extract = "innerHTML", nickname = "price")
url = DSFormat(xpath = "//a[contains(@class, 's-item__link')]", relative = True, extract = "href", nickname = "url")
f_attr = [ price, url ]
driveroptions = DriverOptions(path = "./chromedriver.exe")
info = DSInfo(f_site, f_page, f_item, f_attr)
scraper = DynamicScraper(info, driveroptions, mode = DynamicScraper.MODE_FILE)
scraper.scrape(start = 1, pages = 2, perPage = 5)