All Pandas read_html() you should know for scraping data from HTML tables | by B. Chen | Towards Data Science

Web scraping is the process of collecting and parsing data from the web. The Python community has come up with some pretty powerful web scrapping tools. Among them, Pandas read_html() is a quick and convenient way for scraping data from HTML tables.
In this article, you’ll learn Pandas read_html() to deal with the following common problems and should help you get started with web scraping.
Reading tables from a stringReading tables from a URLReading tables from a fileParsing date columns with parse_datesExplicitly typecast with convertersMultiIndex, header, and index columnMatching a table with matchFiltering tables with attrsWorking with missing values
Reading tables from a string
Reading tables from a URL
Reading tables from a file
Parsing date columns with parse_dates
Explicitly typecast with converters
MultiIndex, header, and index column
Matching a table with match
Filtering tables with attrs
Working with missing values
Please check out Notebook for the source code.
In the first example, we are going to read an HTML table from a string.
html_string = """<table>  <thead>    <tr>      <th>date</th>      <th>name</th>      <th>year</th>      <th>cost</th>      <th>region</th>    </tr>  </thead>  <tbody>    <tr>      <td>2020-01-01</td>      <td>Jenny</td>      <td>1998</td>      <td>0.2</td>      <td>South</td>    </tr>    <tr>      <td>2020-01-02</td>      <td>Alice</td>      <td>1992</td>      <td>-1.34</td>      <td>East</td>    </tr>    <tr>      <td>2020-01-03</td>      <td>Tomas</td>      <td>1982</td>      <td>1.00023</td>      <td>South</td>    </tr>  </tbody></table>"""
To read tables from a string:
dfs = pd.read_html(html_string)
Now, the result we get is not a Pandas DataFrame but a Python list. If we use the type() function we can see that:
>>> type(dfs)list
If we want to get the table, we can use the index to access it:
dfs[0]
The result looks great. Let’s take a look at the data types with dfs[0].info(). By default, columns that are numerical are cast to numeric types, for example, the year and cost columns have been cast to int64 and float64 respectively.
>>> df[0].info()RangeIndex: 3 entries, 0 to 2Data columns (total 5 columns): #   Column  Non-Null Count  Dtype  ---  ------  --------------  -----   0   date    3 non-null      object  1   name    3 non-null      object  2   year    3 non-null      int64   3   cost    3 non-null      float64 4   region  3 non-null      object dtypes: float64(1), int64(1), object(3)memory usage: 248.0+ bytes
Pandas read_html() accepts a URL. Let’s see how this works with the help of an example.
URL = 'https://en.wikipedia.org/wiki/London'dfs = pd.read_html(URL)
Same as reading from a string, it returns a list of DataFrames. If we run len(dfs) we can see that 31 tables available from the given URL.
>>> print(f'Total tables: {len(dfs)}')31
And below is an example of dfs[6]
Pandas read_html() accepts a file. Let’s see how this works with the help of an example.
file_path = 'html_string.txt'with open(file_path, 'r') as f:    dfs = pd.read_html(f.read())dfs[0]
Note: The following tutorials will be reading data from a string as the web page content is unlikely to be the same over time.
The date column gets read as an object data type. To read the date column correctly, we can use the argument parse_dates to specify a list of date columns.
>>> dfs = pd.read_html(html_string, parse_dates=['date'])>>> dfs[0].info()RangeIndex: 3 entries, 0 to 2Data columns (total 5 columns): #   Column  Non-Null Count  Dtype         ---  ------  --------------  -----          0   date    3 non-null      datetime64[ns] 1   name    3 non-null      object         2   year    3 non-null      int64          3   cost    3 non-null      float64        4   region  3 non-null      object        dtypes: datetime64[ns](1), float64(1), int64(1), object(2)memory usage: 248.0+ bytes
By default, columns that are numerical are cast to numeric types, for example, the year and cost columns we have seen. But not all numerical text data are necessary to be numeric types, for example, an ID column that all values have leading zero.
ID = 0001
In addition, sometimes you may want to explicitly typecast to ensure dtype integrity. For these requirements, we can do explicitly typecast with the argument converters :
dfs = pd.read_html(html_string, converters={    'ID': str,    'year': int,    'cost': float,})
By default, <th> or <td> elements located within a <thead> are used to form the column index, if multiple rows are contained within <thead> then a Multiindex is created.
Below is an example of an HTML table with multiple rows in <thead>.
html_string = """<table>  <thead>    <tr>      <th colspan="5">Year 2020</th>    </tr>    <tr>      <th>date</th>      <th>name</th>      <th>year</th>      <th>cost</th>      <th>region</th>    </tr>  </thead>  <tbody>    <tr>      <td>2020-01-01</td>      <td>Jenny</td>      <td>1998</td>      <td>1.2</td>      <td>South</td>    </tr>    <tr>      <td>2020-01-02</td>      <td>Alice</td>      <td>1992</td>      <td>-1.34</td>      <td>East</td>    </tr>  </tbody></table>"""
It creates MultiIndex because there are multiple rows in <thead>
dfs = pd.read_html(html_string)dfs[0]
Specify a header row:
dfs = pd.read_html(html_string, header=1)dfs[0]
Specify an index column:
dfs = pd.read_html(html_string, header=1, index_col=0)dfs[0]
The argument match takes a string or a regular expression. The value defaults to .+ (match any non-empty string) and will return all tables.
Let’s see how this works with the help of an example.
html_string = """<table id="report">  <caption>2020 report</caption>  <thead>    <tr>      <th>date</th>      <th>name</th>    </tr>  </thead>  <tbody>    <tr>      <td>2020-01-01</td>      <td>Jenny</td>    </tr>    <tr>      <td>2020-01-02</td>      <td>Alice</td>    </tr>  </tbody></table><table>  <caption>Average income</caption>  <thead>    <tr>      <th>name</th>      <th>income</th>    </tr>  </thead>  <tbody>    <tr>      <td>Tom</td>      <td>200</td>    </tr>    <tr>      <td>James</td>      <td>300</td>    </tr>  </tbody></table>"""
To read a table that contains the specific text:
# text in captiondfs = pd.read_html(html_string, match='2020 report')# text in table celldfs = pd.read_html(html_string, match='James')
The argument attrs takes a dictionary of any valid HTML tag attributes for filtering tables. For example,
dfs = pd.read_html(html_string, attrs={'id': 'report'})
id is a valid HTML tag attribute.
By default, all empty strings are treated as missing values and read as NaN .
Here is an example of an HTML table with some empty strings in the <td> cell.
html_string = """<table>  <tr>    <th>date</th>    <th>name</th>    <th>year</th>    <th>cost</th>    <th>region</th>  </tr>  <tr>    <td>2020-01-01</td>    <td>Jenny</td>    <td>1998</td>    <td>1.2</td>    <td>South</td>  </tr>  <tr>    <td>2020-01-02</td>    <td>Alice</td>    <td>1992</td>    <td></td>    <td>East</td>  </tr>  <tr>    <td>2020-01-03</td>    <td>Tomas</td>    <td>1982</td>    <td></td>    <td>South</td>  </tr></table>"""
To read it with default settings
dfs = pd.read_html(html_string)dfs[0]
To keep these empty strings, we can set the argument keep_default_na to False
dfs = pd.read_html(html_string, keep_default_na=False)
Sometimes, you may have other character representations for missing values. If we know what kind of characters used as missing values in the table, we can handle them using na_values parameter:
dfs = pd.read_html(html_string, na_values=['?', '&'])
When the DataFrame is already created, we can use pandas replace() function to handle these values:
df_clean = dfs[0].replace({ "?": np.nan, "&": np.nan })
Pandas read_html() function is a quick and convenient way for scraping data from HTML tables.
I hope this article will help you to save time in scrapping data from HTML tables. I recommend you to check out the documentation for the read_html() API and to know about other things you can do.
Thanks for reading. Please check out the notebook for the source code and stay tuned if you are interested in the practical aspect of machine learning.
Using Pandas method chaining to improve code readability
How to do a Custom Sort on Pandas DataFrame
All the Pandas shift() you should know for data analysis
When to use Pandas transform() function
Pandas concat() tricks you should know
Difference between apply() and transform() in Pandas
All the Pandas merge() you should know
Working with datetime in Pandas DataFrame
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More tutorials can be found on my Github