Three Ways to Web Scrape a Page With Python…
Tags: Python Pandas, Data Science, Python Programming, Web Scraping, Python String Methods

1. Pandas

This is the no-brainer. When I find a table on a website begging to be scrapped, my first attempt is always to throw pandas at it.

To practice, we are going to use the table page on webscraper.io, a site for testing your webscraping scripts. Here is a screenshot of the data we will be scraping. In this example we will be trying to grab the second table with idex 4–6.

To scrape a page with pandas, simply read the url into a pandas data frame:

import pandas as pd

df = pd.read_html(url)

print(df[1].head()) url = ' https://webscraper.io/test-sites/table s'df = pd.read_html(url)print(df[1].head())

And just like that, we have scraped the data we wanted. Note that when you scrape a web page with pandas, the tables are brought back as data frames in a list. Thus we need to iterate through df to find the correct table. Looking at the web page, we can make a logical guess that it will be df[1].

Web scraping really doesn’t get easier than that, unfortunately not every table you find on the web will be able to be scraped with pandas. On to the next level…

2. Requests and BeautifulSoup:

In my mind, this is where any Pythonista should start when learning to scrape websites. There are several more sophisticated alternatives when you are ready to crawl web sites, but for starting out this is a great library. Start by installing Requests and BS4 if you haven’t already:

pip install requests bs4

Simple enough, now we use requests to obtain the pages html code and BeautifulSoup out of bs4 to parse the page:

import requests

from bs4 import BeautifulSoup

page = requests.get(url).text

soup = BeautifulSoup(page, 'html.parser') url = ' https://webscraper.io/test-sites/table s'page = requests.get(url).textsoup = BeautifulSoup(page, 'html.parser')

Now we have options. We can take a look at the html by simply calling soup or we can take a look at our page and inspect the tags of elements of interest.

Looks like we need to pull <table>, <tr>, and finally <td> to get at our data. Lets get <table class=”table table-bordered”> first:

# get the tables first

tbls = soup.find_all('table', {'class': 'table table-bordered'})

Now if we look at the length of tbls we find that it is 4. We have obtained all 4 tables on the page in a few lines of code. Lets get the data out of it next:

tr = tbls[1].find_all('tr')

This code simply grabs all the data in the first <tr> tag which includes all of our ‘table header’ data. Now we can use list comprehension to get all the text from our table to include the column names.

data = [d.text for d tr]

Printing “data”, we see that we now have a list of strings.

Lets drop off the first and last “

” for each string and then split on all the other Line Feed characters (

):

data = [d[1:-1].split('

') for d in data]

Now make a data frame:

df = pd.DataFrame(data[1:], columns=data[0])

3. Simplify and Remove Dependencies

Lets make this as independent as possible now. No imports at all, only core python modules. We will scrape this page in the following steps:

Source the page: note that this wont work with JS dynamic pages Copy the data you want out Use string manipulation to get your data

View the page source of the website you want to scrape

Here is the data we want

Store it in a string variable and then manipulate that string to pull out the data. This can be tedious, but it allows you to get at data that can be very difficult to scrape otherwise. In this case it’s straightforward:

Set you string variable

Lets make it easier to work with by identifying and then matching our tags that will be easiest to manipulate. I am going to use the table row <tr> tag to pull out the data. Lets get rid of the </tr> closing tags:

data = data.replace('</tr>', '<tr>')

If we look at string now, we will see that there are a lot of tab and line breaks. Lets get rid of those as well. We could use REGEX here, but no imports…

data = data.replace('

\t\t\t\t', '')

Now lets get after that data. Note that all our data is inside either the <th> or <td> tags. Lets make it easier to work with that:

data = data.replace('<th>','<td>').replace('</th>','<td>')

data = data.replace('</td>','<td>') # now try spliting on '<td>'

data = ddata.split('<td>')

Looks like we need 1 to -1. Lets get that and lets drop all the garbage:

data_clean = [d for d in data[1:-1] if d != '' and '

' not in d]

Now we have all our data in a list. Lets finish this! Our data is currently in a list and we need it in either an array or in a list of lists. We could import numpy and use it to convert the shape like this…

# Using Numpy - super easy, but requires another import import numpy as np

data_array = np.array(rows).reshape(3,4)

But, again, we aren’t importing anything in this method so lets just use zip…

# zipping rows to itself in groups of 4 data_grouped = list(zip(*(iter(data_clean),) * 4))

And finally we export this to a flat file:

with open('scraped_page_with_strings_only.csv', 'w+') as f:

for row in data_grouped:

for x in row:

f.write(str(x)+',')

f.write('

')

Now I know you are doubting if this actually worked, so lets check.

A little cleaning and…