Repository: pandas-dev/pandas

Issue:
read_html omits a column when reading a wikipedia table
https://en.wikipedia.org/wiki/List_of_countries_by_road_network_size
#Copy/paste the table HTML in a file
df=pd.read_html(table)
df=pd.DataFrame(df[0])
#The result is shown in the image. The Density column values are missing.
![Screen Shot 2566-02-25 at 16 49 16](https://user-images.githubusercontent.com/472428/221350621-b993d5ec-adb5-4f7a-b3e7-ffb61e898f59.png)



Discussion:
Hi, thanks for your report.

Can you please provide a minimal and reproducible example? You can define your html table as
```
data = """Put your html table here
"""
pd.read_html(StringIO(data))
```

Also, please provide your pandas versions and dependencies, e.g. fill out the issue template
#here's a few first lines of the table in question. Pandas 1.3.5. I'm running it on Google Colab.
import pandas as pd
from io import StringIO;
roadtable = """<table border="1" class="wikitable sortable static-row-numbers plainrowheaders srn-white-background" style="text-align:right;">
<tbody><tr class="static-row-header nowrap" style="text-align:center;vertical-align:bottom;vertical-align:top">
<th>Country
</th>
<th>Total<br/><style data-mw-deduplicate="TemplateStyles:r886047488">.mw-parser-output .nobold{font-weight:normal}</style><span class="nobold"><span style="font-size:85%;">(km)</span></span>
</th>
<th>Density<br/><link href="mw-data:TemplateStyles:r886047488" rel="mw-deduplicated-inline-style"/><span class="nobold"><span style="font-size:85%;">(km/100 km<sup>2</sup>)</span></span>
</th>
<th colspan="2">Paved<br/><link href="mw-data:TemplateStyles:r886047488" rel="mw-deduplicated-inline-style"/><span class="nobold"><span style="font-size:85%;">(km)</span></span>
</th>
<th colspan="2">Unpaved<br/><link href="mw-data:TemplateStyles:r886047488" rel="mw-deduplicated-inline-style"/><span class="nobold"><span style="font-size:85%;">(km)</span></span>
</th>
<th colspan="2"><a href="/wiki/Controlled-access_highway" title="Controlled-access highway">Controlled-access</a> <br/><link href="mw-data:TemplateStyles:r886047488" rel="mw-deduplicated-inline-style"/><span class="nobold"><span style="font-size:85%;">(km)</span></span>
</th>
<th>Source<br/>&amp; Year
</th></tr>
<tr class="static-row-header">
<th style="background-position:center;line-height:70%;padding-top:0"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center"> </th>
<th style="background-position:center">
</th></tr>
<tr class="static-row-header" style="font-weight:bold;">
<th scope="row" style="text-align:left"> <span class="flagicon" style="padding-left:25px;"> </span><b>World</b>
</th>
<td style="text-align:right">64,285,009</td>
<td style="text-align:right;font-size:inherit"><span data-sort-value="7,001,472,121,509,672,000♠" style="display:none"></span>47</td>
<td class="table-na" colspan="2" data-sort-value="" style="background: #ececec; color: #2C2C2C; vertical-align: middle; text-align: center;">—</td>
<td class="table-na" colspan="2" data-sort-value="" style="background: #ececec; color: #2C2C2C; vertical-align: middle; text-align: center;">—</td>
<td class="table-na" colspan="2" data-sort-value="" style="background: #ececec; color: #2C2C2C; vertical-align: middle; text-align: center;">—</td>
<td>2021
</td></tr>
<tr>
<th style="text-align:left"><span class="flagicon" style="display:inline-block;width:25px;text-align:left"><img alt="" class="thumbborder" data-file-height="650" data-file-width="1235" decoding="async" height="12" src="//upload.wikimedia.org/wikipedia/en/thumb/a/a4/Flag_of_the_United_States.svg/23px-Flag_of_the_United_States.svg.png" srcset="//upload.wikimedia.org/wikipedia/en/thumb/a/a4/Flag_of_the_United_States.svg/35px-Flag_of_the_United_States.svg.png 1.5x, //upload.wikimedia.org/wikipedia/en/thumb/a/a4/Flag_of_the_United_States.svg/46px-Flag_of_the_United_States.svg.png 2x" width="23"/></span> <a class="mw-redirect" href="/wiki/Roads_in_the_United_States" title="Roads in the United States">United States *</a>
</th>
<td style="text-align:right">6,803,479</td>
<td style="text-align:right;font-size:inherit"><span data-sort-value="7,001,691,866,086,610,000♠" style="display:none"></span>69</td>
<td style="text-align:right">4,304,715</td>
<td style="text-align:right;font-size:small"><span data-sort-value="7,001,632,722,611,475,000♠" style="display:none"></span>63%</td>
<td style="text-align:right">2,581,895</td>
<td style="text-align:right;font-size:small"><span data-sort-value="7,001,379,496,284,180,000♠" style="display:none"></span>38%</td>
<td style="text-align:right">95,932</td>
<td style="text-align:right;font-size:small"><span data-sort-value="7,000,141,004,330,285,000♠" style="display:none"></span>1.4%</td>
<td><sup class="reference" id="cite_ref-3"><a href="#cite_note-3">[3]</a></sup> 2019
</td></tr>
</tbody></table>"""
df = pd.read_html(StringIO(roadtable))[0]

Can you try on the newest pandas version? 2.0.0rc0
Unfortunately the issue is there with 2.0.0rc0
 pd.__version__
'2.0.0rc0'
>>> print(df.columns)
Index(['Country', 'Total .mw-parser-output .nobold{font-weight:normal}(km)',
       'Density (km/100 km2)', 'Paved (km)', 'Paved (km).1', 'Unpaved (km)',
       'Unpaved (km).1', 'Controlled-access (km)', 'Controlled-access (km).1',
       'Source & Year'],
      dtype='object')
>>> print(df['Density (km/100 km2)'])
0   NaN
1   NaN
2   NaN
Name: Density (km/100 km2), dtype: float64

Good, now please reduce everything from your html table that is not necessary to reproduce. The example should be __minimal__
Hi, it seems the problem is related to `style="display:none"`. When one of elements in table cell has this style attribute, pandas returns `NaN`, while it should only skip this element. Below reproducible example:

```
html_table = """
    <table>
  <tr>
    <th>Col 1</th>
    <th>Col 2</th>
    <th>Col 3</th>
  </tr>
  <tr>
    <td>1</td>
    <td>2</td>
    <td>3</td>
  </tr>
  <tr>
    <td><span style="display:none"></span>4</td>
    <td>5</td>
    <td>6</td>
  </tr>
</table>
"""

pd.read_html(html_table)[0]

Out[1]:
   Col 1  Col 2  Col 3
0    1.0      2      3
1    NaN      5      6

```

I can try to fix this problem.
take

Patch:
diff --git a/doc/source/whatsnew/v2.1.0.rst b/doc/source/whatsnew/v2.1.0.rst
index a37bdf11c66bc..ee11777cfbe0c 100644
--- a/doc/source/whatsnew/v2.1.0.rst
+++ b/doc/source/whatsnew/v2.1.0.rst
@@ -196,6 +196,7 @@ MultiIndex
 
 I/O
 ^^^
+- Bug in :func:`read_html`, tail texts were removed together with elements containing ``display:none`` style (:issue:`51629`)
 - :meth:`DataFrame.to_orc` now raising ``ValueError`` when non-default :class:`Index` is given (:issue:`51828`)
 -
 
diff --git a/pandas/io/html.py b/pandas/io/html.py
index 70cdedfd21ada..ce95c2be8581f 100644
--- a/pandas/io/html.py
+++ b/pandas/io/html.py
@@ -732,7 +732,7 @@ def _parse_tables(self, doc, match, kwargs):
                 # attribute and iterate them to check for display:none
                 for elem in table.xpath(".//*[@style]"):
                     if "display:none" in elem.attrib.get("style", "").replace(" ", ""):
-                        elem.getparent().remove(elem)
+                        elem.drop_tree()
 
         if not tables:
             raise ValueError(f"No tables found matching regex {repr(pattern)}")


Test patch:
diff --git a/pandas/tests/io/test_html.py b/pandas/tests/io/test_html.py
index 03f1bcb13d077..252f028e0dffc 100644
--- a/pandas/tests/io/test_html.py
+++ b/pandas/tests/io/test_html.py
@@ -1238,6 +1238,28 @@ def test_displayed_only(self, displayed_only, exp0, exp1):
         else:
             assert len(dfs) == 1  # Should not parse hidden table
 
+    @pytest.mark.parametrize("displayed_only", [True, False])
+    def test_displayed_only_with_many_elements(self, displayed_only):
+        html_table = """
+        <table>
+            <tr>
+                <th>A</th>
+                <th>B</th>
+            </tr>
+            <tr>
+                <td>1</td>
+                <td>2</td>
+            </tr>
+            <tr>
+                <td><span style="display:none"></span>4</td>
+                <td>5</td>
+            </tr>
+        </table>
+        """
+        result = read_html(html_table, displayed_only=displayed_only)[0]
+        expected = DataFrame({"A": [1, 4], "B": [2, 5]})
+        tm.assert_frame_equal(result, expected)
+
     @pytest.mark.filterwarnings(
         "ignore:You provided Unicode markup but also provided a value for "
         "from_encoding.*:UserWarning"