Web data extraction in its multitudes using Python
By: Art Myftiu
Tags: Python, Tech, Scraping, Selenium

Selenium

If you prefer to rely on simple intuitive syntax and derivatives, then Selenium is always a sound call.

Granted, this simplicity comes at the price of speed, as Selenium along with Chromedriver (which I will be using in the example) engages a browser to achieve its goal.

Without further ado:

from selenium import webdriver

from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

import codetiming



t = codetiming.Timer(name="execTime")

t.start()



staticURL = 'https://steamdb.info/app/440/graphs/'



peakxPath = '/html/body/div[1]/div[1]/div[3]/div/div[2]/div[4]/ul/li[3]'



options = webdriver.ChromeOptions()

strategy = DesiredCapabilities().CHROME

strategy["pageLoadStrategy"] = "eager"



# Experimental options for not loading images to speed up selenium chrome_prefs = {}

options.experimental_options["prefs"] = chrome_prefs

chrome_prefs["profile.default_content_settings"] = {"images": 2}

chrome_prefs["profile.managed_default_content_settings"] = {"images": 2}





driver = webdriver.Chrome(options=options, desired_capabilities=strategy)



driver.get(staticURL)

peakPlayersElement = driver.find_element_by_xpath(peakxPath).text.split()



print(peakPlayersElement[0])

t.stop()

driver.quit()

Which gives us the number of players at peak within 24h and the script execution time.

86,919

Elapsed time: 1.7432 seconds

So it took Selenium 1.7432 seconds to give us the value we needed, the peak number of concurrent players for the game Team Fortress 2.

I’d also like to note:

In line 14 by setting the page strategy to ‘eager’ it tells Selenium to proceed looking for our values without loading the entire page.

Lines 19, 20 instruct a set of additional preferences to disable loading images on a page, since the value we needed is a string.

These aren’t options I would use however, if the objective was to render the HTML completely or perform actions which would require images.

I also stopped timing it before it closes the driver instance since it has already provided us with the information we needed.

Selectolax

Now this one really doesn’t disappoint, if you need to scale your project to work at larger capacities at once than Selectolax is a solid choice.

An additional user agent was needed in this case, since the page wasn’t too friendly towards Python requests. The code is quite straightforward with the difference being that a CSS selector was used to locate the element.

import requests

from selectolax.parser import HTMLParser

import codetiming



t = codetiming.Timer(name="execTime")

t.start()



headers = {'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25'}



staticURL = 'https://steamdb.info/app/440/graphs/'

selector = '#graphs > ul > li:nth-child(3) > strong'



response = requests.get(staticURL, headers=headers)

pageHtml = response.text

tree = HTMLParser(pageHtml)



print(tree.css_first(selector).text())

t.stop()

Which outputs:

86,919

Elapsed time: 0.3130 seconds

That is quite remarkable timing as we can see, it has fetched the page, rendered it and parsed our value at 0.3130 seconds. An entire second and a half faster then Selenium and then some.

BeautifulSoup

This tends to be the most popular choice when it comes to web scraping in the Python world, the code is almost a carbon copy of the Selectolax code with a few adjustments to the HTML parsing engine.

import requests

from bs4 import BeautifulSoup

import codetiming

import lxml



t = codetiming.Timer(name="execTime")

t.start()



headers = {'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25'}



staticURL = 'https://steamdb.info/app/440/graphs/'

selector = '#graphs > ul > li:nth-child(3) > strong'



response = requests.get(staticURL, headers=headers)

pageHtml = response.text



soup = BeautifulSoup(pageHtml, 'lxml')

SoupSelected = soup.select_one(selector).text



print(SoupSelected)



t.stop()

Giving us

86,919

Elapsed time: 0.5421 seconds

About twice as slow as Selectolax and three times as fast as Selenium. Not bad.

Grab

Why don’t you grab yourself some snacks and a drink because this one is almost going to be that exciting. Comes with a plethora of features including custom spider building, possibly intended for easier scaling of your web data extraction projects. Now lets put it to the test.

import time

import requests

from grab import Grab

import codetiming



t = codetiming.Timer(name="execTime")

t.start()



staticURL = 'https://steamdb.info/app/440/graphs/'

peakxPath = '/html/body/div[1]/div[1]/div[3]/div/div[2]/div[4]/ul/li[3]'



g = Grab()

g.setup(headers={'User-Agent': 'Mozilla/5.0 (iPad; CPU OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5355d Safari/8536.25'})



g.go(staticURL)



ourText = g.doc(peakxPath).text().split()

print(ourText[0])



t.stop()

Which resulted in

86,919

Elapsed time: 1.5842 seconds

About as fast as Selenium, and kind of disappointing since this particular library uses pyCurl which is a Python implementation of cURL. I was expecting Grab to be faster than Selenium at least.

However we’ll make an exception since Grab is so feature packed…

The Verdict

Selectolax is by far the most swift approach to getting the information you need, even if it is a little one-dimensional.

If you need something that can handle multiple-stage scraping (i.e. logging in, clicking around, then retrieving data) then you’re probably best off with something like Selenium or Grab.

Personally I find myself falling back on Selenium just because of it’s ease of use and its ability to interact with the web page in different ways.

Some honorable mentions would be: Ruia, Scrapy

Thanks for reading this article! Hopefully you can make some use of it in the future.