Package: webmage
DescriptionA wrapper around requests, BeautifulSoup, and Selenium (Chrome) to facilitate web scraping.Installationpip install webmageBasic UsageImport WebSpellfrom webmage import WebSpellInitializing the WebSpell classThe WebSpell class returns either a StaticSpell or DynamicSpell depending on which method you select: static or dynamic. Choosing static will use the requests module while dynamic will use a selenium webdriver. Currently Chrome is the only supported browser.spell = WebSpell('https://javascriptorian.com')url: A URL.driver_path (optional): A filepath for a chromedriver. If omitted, the class will automatically download one into your cache.browser (optional): A string to choose which browser you want to use. Options includechrome|undetected_chrome|firefoxghost (optional): A boolean for making the chromedriver headless.Changing from one URL to anotherWhen you initialize the WebSpell class, you provide a URL. You probably need to change the url if you are web scraping.spell.change_url('https://wordcruncher.com')url = A string of an HTTP(S) request.Closing the webdriverYou can close the webdriver browser by using the close method.spell.close()Selecting an elementThe select method returns a StaticRune or DynamicRune of the first element found on the webpage. It takes a css selector as its only argument. XPath is not implemented yet.rune = spell.select('a')css_selector: A string of a css selector.Selecting multiple elementsThe selectAll method returns a list of StaticRune or DynamicRune elements.runes = spell.selectAll('a')css_selector: A string of a css selector.Seeing all attributes from elementAll element attributes are added to rune.attributes.attributes = rune.attributesGetting tag attributes from elementYou can use the dictionary-like format to get an attribute from an HTML tag. Unlike BeautifulSoup, webmage returns None instead of KeyError if no attribute is found.url = rune['href']Getting the text from elementtext = rune.textGetting the innerHTML or outerHTML from elementinner = rune.innerHTML
outer = rune.outerHTMLClicking on an elementThe click method lets you click on an element on the page. It will click on the first element it finds. It takes a css selector as its only argument.spell.click(css_selector='button')css_selector: A string of a css selector.Clicking on multiple elementsThe clickAll method lets you click on multiple elements. Generally, it's good to wait between clicks, so there's an optional wait_interval argument you can pass to this method.spell.clickAll(css_selector='button', wait_interval=2)css_selector: A string of a css selector.wait_interval (optional): A float or integer of the amount of seconds to wait in between clicks.Scrolling AbilitiesWebSpell has 4 different scrolling methods depending on the nature of the website.Limited ScrollThe limited scroll method will scroll down to the bottom of the page X amount of times. It always scrolls down to the bottom of the page immediately.spell.scroll(wait_interval, scroll_count, scroll_css_selector="document.scrollingElement", callback=None, verbose=True)wait_interval: A float or integer of the amount of seconds to wait in between scrolls.scroll_count: An integer of how many times you want to scroll down the page.scroll_css_selector (optional): A string of a css selector. If the website has a custom scrolling element, you must specify the CSS selector for the css selector. Defaults to the normal scrolling element.callback (optional): A callback function to call after each scroll. The callback function must have one argument that contains the spell object.verbose (optional): A boolean for whether you want it to dynamically print how many times it's scrolled down the page.Delicate Limited ScrollThe delicate limited scroll will scroll down X amount of pixels X amount of times. It scrolls down a specific amount of pixels, allowing you to grab content from the webpage that only is available if the page is scrolled down gradually.spell.delicate_scroll(wait_interval, scroll_count, scroll_pixel_length=500, scroll_css_selector="document.scrollingElement", callback=None, verbose=True)wait_interval: A float or integer of the amount of seconds to wait in between scrolls.scroll_count: An integer of how many times you want to scroll down the page.scroll_pixel_length (optional): A float or integer of the amount of pixels you want to scroll down for each scroll. Defaults to 500 pixelsscroll_css_selector (optional): A string of a css selector. If the website has a custom scrolling element, you must specify the CSS selector for the css selector. Defaults to the normal scrolling element.callback (optional): A callback function to call after each scroll. The callback function must have one argument that contains the spell object.verbose (optional): A boolean for whether you want it to dynamically print how many times it's scrolled down the page.Infinite ScrollThe infinite scroll is a special ability programmed into webmage. It will scroll down the page for as long as there is no more content added to the page.spell.infinite_scroll(wait_interval, scroll_css_selector="document.scrollingElement", callback=None, verbose=True)wait_interval: A float or integer of the amount of seconds to wait in between scrolls.scroll_pixel_length: A float or integer of the amount of pixels you want to scroll down for each scroll.scroll_css_selector (otional): A string of a css selector. If the website has a custom scrolling element, you must specify the CSS selector for the css selector. Defaults to the normal scrolling element.callback (optional): A callback function to call after each scroll. The callback function must have one argument that contains the spell object.verbose (optional): A boolean for whether you want it to dynamically print how many times it's scrolled down the page.Delicate Infinite Scrolldelicate_infinite_scroll(wait_interval, scroll_pixel_length=500, scroll_css_selector="document.scrollingElement", callback=None, verbose=True)wait_interval: A float or integer of the amount of seconds to wait in between scrolls.
scroll_pixel_length: A float or integer of the amount of pixels you want to scroll down for each scroll.
scroll_css_selector: A string of a css selector. If the website has a custom scrolling element, you must specify the CSS selector for the css selector. Defaults to the normal scrolling element.
callback: A callback function to call after each scroll. The callback function must have one argument that contains the spell object.
verbose: A boolean for whether you want it to dynamically print how many times it's scrolled down the page.Destroying/Removing ElementsYou may want to remove elements from the page, especially if you don't need that content or you are scrolling on a very long page. Below are a few methods that help you remove elements from a pageDestroy all tags on an HTML page.spell.destroyAll('div')Iterate through each tag and then destroy it.divs = spell.selectAll('div')
for div in divs:
    div.destroy()Select the a tag and then destroy the first of another tag.body = spell.select('div.body')
body.destroy('sup')Select a tag and then destroy all of another tag within the main tag.body = spell.select('div.body')
body.destroyAll('sup')Other AbilitiesPause ScraperSame as time.sleep()spell.wait(5)time_interval: A float or integer of the amount of seconds to pause your scraper.Get URL SlugThis returns the last part of the URL. It removes any hashes (#...) or queries (?...&...)spell.get_slug()Get Network LogAnother special ability of WebSpell. This returns a list of the network log. Useful for getting data that is only found in the network requests (like image or video URLs). It's important to note that the network log wipes the log each time you call this function, so you must save the data to a variable if you intend on getting the network log multiple times.log = spell.network_log()Executing JavaScript codeThe cast_js method will execute JavaScript code on the web browser.spell.cast_js('window.open("window.open("https://google.com", "_blank");')Opening a New Tab TemporarilySometimes you need to open an image, video, tweet, etc. in a different tab, and you don't want to lose your progress on your main page. You can use the cast_in_discrete_tab method to open a new tab and do something there.spell.cast_in_discrete_tab(url, callback=False, payload)url: A string of the URL that you want to open in a new tab
callback: A callback function that you want to run after opening the tab.
payload: An object (e.g. dictionary or list) that you want available within the callback function.Taking a ScreenshotUse the take_screenshot method if you want to take a picture of an image.spell.take_screenshot(css_selector='img')css_selector: A string of a CSS selector.