All Pandas read_html() you should know for scraping data from HTML tables
By: B. Chen
Tags: Python, Web Scraping, Machine Learning, Data Science, Pandas

4. Parsing date columns with parse_dates

The date column gets read as an object data type. To read the date column correctly, we can use the argument parse_dates to specify a list of date columns.

>>> dfs = pd.read_html(html_string, parse_dates=['date'])

>>> dfs[0].info() RangeIndex: 3 entries, 0 to 2

Data columns (total 5 columns):

# Column Non-Null Count Dtype

--- ------ -------------- -----

0 date 3 non-null datetime64[ns]

1 name 3 non-null object

2 year 3 non-null int64

3 cost 3 non-null float64

4 region 3 non-null object

dtypes: datetime64[ns](1), float64(1), int64(1), object(2)

memory usage: 248.0+ bytes

5. Explicitly typecast with converters

By default, columns that are numerical are cast to numeric types, for example, the year and cost columns we have seen. But not all numerical text data are necessary to be numeric types, for example, an ID column that all values have leading zero.

ID = 0001

In addition, sometimes you may want to explicitly typecast to ensure dtype integrity. For these requirements, we can do explicitly typecast with the argument converters :

dfs = pd.read_html(html_string, converters={

'ID': str,

'year': int,

'cost': float,

})

explicitly typecast with Pandas read_html() (Image by Author)

6. MultiIndex, header, and index column

By default, <th> or <td> elements located within a <thead> are used to form the column index, if multiple rows are contained within <thead> then a Multiindex is created.

Below is an example of an HTML table with multiple rows in <thead> .

html_string = """

<table>

<thead>

<tr>

<th colspan="5">Year 2020</th>

</tr>

<tr>

<th>date</th>

<th>name</th>

<th>year</th>

<th>cost</th>

<th>region</th>

</tr>

</thead>

<tbody>

<tr>

<td>2020-01-01</td>

<td>Jenny</td>

<td>1998</td>

<td>1.2</td>

<td>South</td>

</tr>

<tr>

<td>2020-01-02</td>

<td>Alice</td>

<td>1992</td>

<td>-1.34</td>

<td>East</td>

</tr>

</tbody>

</table>

"""

Multi-head table (image by author)

It creates MultiIndex because there are multiple rows in <thead>

dfs = pd.read_html(html_string)

dfs[0]

multiindex with Pandas read_html() (image by author)

Specify a header row:

dfs = pd.read_html(html_string, header=1)

dfs[0]

specify header with Pandas read_html() (image by author)

Specify an index column:

dfs = pd.read_html(html_string, header=1, index_col=0)

dfs[0]

specify index column with Pandas read_html() (image by author)

7. Matching a table with match

The argument match takes a string or a regular expression. The value defaults to .+ (match any non-empty string) and will return all tables.

Let’s see how this works with the help of an example.

html_string = """

<table id="report">

<caption>2020 report</caption>

<thead>

<tr>

<th>date</th>

<th>name</th>

</tr>

</thead>

<tbody>

<tr>

<td>2020-01-01</td>

<td>Jenny</td>

</tr>

<tr>

<td>2020-01-02</td>

<td>Alice</td>

</tr>

</tbody>

</table> <table>

<caption>Average income</caption>

<thead>

<tr>

<th>name</th>

<th>income</th>

</tr>

</thead>

<tbody>

<tr>

<td>Tom</td>

<td>200</td>

</tr>

<tr>

<td>James</td>

<td>300</td>

</tr>

</tbody>

</table>

"""

To read a table that contains the specific text:

# text in caption

dfs = pd.read_html(html_string, match='2020 report') # text in table cell

dfs = pd.read_html(html_string, match='James')

8. Filtering tables with attrs

The argument attrs takes a dictionary of any valid HTML tag attributes for filtering tables. For example,

dfs = pd.read_html(html_string, attrs={'id': 'report'})

id is a valid HTML tag attribute.

9. Working with missing values

By default, all empty strings are treated as missing values and read as NaN .

Here is an example of an HTML table with some empty strings in the <td> cell.

html_string = """

<table>

<tr>

<th>date</th>

<th>name</th>

<th>year</th>

<th>cost</th>

<th>region</th>

</tr>

<tr>

<td>2020-01-01</td>

<td>Jenny</td>

<td>1998</td>

<td>1.2</td>

<td>South</td>

</tr>

<tr>

<td>2020-01-02</td>

<td>Alice</td>

<td>1992</td>

<td></td>

<td>East</td>

</tr>

<tr>

<td>2020-01-03</td>

<td>Tomas</td>

<td>1982</td>

<td></td>

<td>South</td>

</tr>

</table>

"""

To read it with default settings

dfs = pd.read_html(html_string)

dfs[0]

Pandas read_html() working with missing values (image by author)

To keep these empty strings, we can set the argument keep_default_na to False

dfs = pd.read_html(html_string, keep_default_na=False)

image by author

Sometimes, you may have other character representations for missing values. If we know what kind of characters used as missing values in the table, we can handle them using na_values parameter:

dfs = pd.read_html(html_string, na_values=['?', '&'])

Pandas read_html() working with missing values (image by author)

When the DataFrame is already created, we can use pandas replace() function to handle these values:

df_clean = dfs[0].replace({ "?": np.nan, "&": np.nan })

Conclusion

Pandas read_html() function is a quick and convenient way for scraping data from HTML tables.

I hope this article will help you to save time in scrapping data from HTML tables. I recommend you to check out the documentation for the read_html() API and to know about other things you can do.

Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.

You may be interested in some of my other Pandas articles:

More tutorials can be found on my Github