Repository: matthewwithanm/python-markdownify
Pull request #161 (2024-12-07 08:03:22)

Add a new configuration option to control tabler header row inference
For the instance that a table does not contain a header defined by `<thead>` or `<th>` , markdownify will use the first row of the table as header fallback by default.  

However, sometimes I expect the first row to be parsed as a part of the tbody, for example:  

```html
<table>
    <tbody>
        <tr>
            <td>John</td>
            <td>123-456-7890</td>
            <td>john@example.com</td>
        </tr>
        <tr>
            <td>Bob</td>
            <td>987-654-3210</td>
            <td>bob@example.com</td>
        </tr>
    </tbody>
</table>
```

I expect it to be converted as follows:  

```markdown
|     |     |     |
| --- | --- | --- |
| John | 123-456-7890 | john@example.com |
| Bob | 987-654-3210 | bob@example.com |
```

I think I may not be the only one who has encountered this problem, so I added a new option `table_header_fallback` to control this behavior.  

`table_header_fallback` is set to `True` by default, and the html above will be converted to:  

```markdown
| John | 123-456-7890 | john@example.com |
| --- | --- | --- |
| Bob | 987-654-3210 | bob@example.com |
```

If I set `table_header_fallback=False`, the result will be:  
```markdown 
|  |  |  |
| --- | --- | --- |
| John | 123-456-7890 | john@example.com |
| Bob | 987-654-3210 | bob@example.com |
```  

I've added this option to the ArgumentParser in `main.py` (alongwith another `--escape-misc` that may be forgotten), and I also created corresponding test cases.  

Hope this will help, thank you for your work!  

chrispy-snps: @SomeBottle - thanks for the pull request! I was not aware of this behavior, and I would want your proposed behavior in our own application. Thanks for catching this and proposing a solution.

I have some suggestions:

1. Could you consider renaming `table_header_fallback` to `table_infer_header`?
2. Could you move the missing `--escape-misc` argument to a separate pull request?

@AlexVonB, @matthewwithanm - my preference would be to *not* infer `<th>` header rows by default so that round-trips between HTML and Markdown are symmetrical. What do you think?
matthewwithanm: > @AlexVonB, @matthewwithanm - my preference would be to _not_ infer `<th>` header rows by default so that round-trips between HTML and Markdown are symmetrical. What do you think?

yeah, that behavior is surprising to me

SomeBottle: Thank you for your response! I’ve renamed the option and removed `--escape-misc` from argument parser in the recent commit.  

Should I change the default value to `False` ? Looking forward to your reply.  
chrispy-snps: @SomeBottle - yes, please change the default to *not* infer header rows from data rows unless specified by the user.
SomeBottle: OK, I just set the default value of the option to `False` and changed corresponding test cases and docs.  
chrispy-snps: @SomeBottle - thanks!

Looking through your code, I see this:

```python
        is_headrow = (
            all([cell.name == 'th' for cell in cells])
            or (el.parent.name == 'thead'
                # avoid multiple tr in thead             # <----
                and len(el.parent.find_all('tr')) == 1)  # <----
        )
```

What is the thinking behind the check for multiple `<tr>` elements? Is the thinking that if a `<thead>` contains multiple `<tr>` rows, they are probably truly data rows instead of header rows?
SomeBottle: > @SomeBottle - thanks!
> 
> Looking through your code, I see this:
> 
> ```python
>         is_headrow = (
>             all([cell.name == 'th' for cell in cells])
>             or (el.parent.name == 'thead'
>                 # avoid multiple tr in thead             # <----
>                 and len(el.parent.find_all('tr')) == 1)  # <----
>         )
> ```
> 
> What is the thinking behind the check for multiple `<tr>` elements? Is the thinking that if a `<thead>` contains multiple `<tr>` rows, they are probably truly data rows instead of header rows?

Yes, considering the possibility of such an unusual situation, which is actually an invalid header for markdown. To keep these rows, I think it should be treated as a case without a header.  
chrispy-snps: @SomeBottle - could you rebase/merge this branch to the latest state of `develop`? Thanks!
SomeBottle: @chrispy-snps OK, I've rebased the branch.

Patch:
diff --git a/README.rst b/README.rst
index af87f51..34ed7e0 100644
--- a/README.rst
+++ b/README.rst
@@ -139,6 +139,11 @@ keep_inline_images_in
   that should be allowed to contain inline images, for example ``['td']``.
   Defaults to an empty list.
 
+table_infer_header
+  Controls handling of tables with no header row (as indicated by ``<thead>``
+  or ``<th>``). When set to ``True``, the first body row is used as the header row.
+  Defaults to ``False``, which leaves the header row empty.
+
 wrap, wrap_width
   If ``wrap`` is set to ``True``, all text paragraphs are wrapped at
   ``wrap_width`` characters. Defaults to ``False`` and ``80``.
diff --git a/markdownify/__init__.py b/markdownify/__init__.py
index ac53077..2360210 100644
--- a/markdownify/__init__.py
+++ b/markdownify/__init__.py
@@ -102,6 +102,7 @@ class MarkdownConverter(object):
         strong_em_symbol = ASTERISK
         sub_symbol = ''
         sup_symbol = ''
+        table_infer_header = False
         wrap = False
         wrap_width = 80
 
@@ -518,13 +519,24 @@ class MarkdownConverter(object):
         cells = el.find_all(['td', 'th'])
         is_headrow = (
             all([cell.name == 'th' for cell in cells])
-            or (not el.previous_sibling and not el.parent.name == 'tbody')
+            or (el.parent.name == 'thead'
+                # avoid multiple tr in thead
+                and len(el.parent.find_all('tr')) == 1)
+        )
+        is_head_row_missing = (
+            (not el.previous_sibling and not el.parent.name == 'tbody')
             or (not el.previous_sibling and el.parent.name == 'tbody' and len(el.parent.parent.find_all(['thead'])) < 1)
         )
         overline = ''
         underline = ''
-        if is_headrow and not el.previous_sibling:
-            # first row and is headline: print headline underline
+        if ((is_headrow
+             or (is_head_row_missing
+                 and self.options['table_infer_header']))
+                and not el.previous_sibling):
+            # first row and:
+            # - is headline or
+            # - headline is missing and header inference is enabled
+            # print headline underline
             full_colspan = 0
             for cell in cells:
                 if 'colspan' in cell.attrs and cell['colspan'].isdigit():
@@ -532,13 +544,16 @@ class MarkdownConverter(object):
                 else:
                     full_colspan += 1
             underline += '| ' + ' | '.join(['---'] * full_colspan) + ' |' + '\n'
-        elif (not el.previous_sibling
-              and (el.parent.name == 'table'
-                   or (el.parent.name == 'tbody'
-                       and not el.parent.previous_sibling))):
+        elif ((is_head_row_missing
+               and not self.options['table_infer_header'])
+              or (not el.previous_sibling
+                  and (el.parent.name == 'table'
+                       or (el.parent.name == 'tbody'
+                           and not el.parent.previous_sibling)))):
+            # headline is missing and header inference is disabled or:
             # first row, not headline, and:
-            # - the parent is table or
-            # - the parent is tbody at the beginning of a table.
+            #  - the parent is table or
+            #  - the parent is tbody at the beginning of a table.
             # print empty headline above this row
             overline += '| ' + ' | '.join([''] * len(cells)) + ' |' + '\n'
             overline += '| ' + ' | '.join(['---'] * len(cells)) + ' |' + '\n'
diff --git a/markdownify/main.py b/markdownify/main.py
index 4e1c874..432efb5 100644
--- a/markdownify/main.py
+++ b/markdownify/main.py
@@ -61,6 +61,10 @@ def main(argv=sys.argv[1:]):
                         "should be converted to markdown images instead, this option can "
                         "be set to a list of parent tags that should be allowed to "
                         "contain inline images.")
+    parser.add_argument('--table-infer-header', dest='table_infer_header',
+                        action='store_true',
+                        help="When a table has no header row (as indicated by '<thead>' "
+                        "or '<th>'), use the first body row as the header row.")
     parser.add_argument('-w', '--wrap', action='store_true',
                         help="Wrap all text paragraphs at --wrap-width characters.")
     parser.add_argument('--wrap-width', type=int, default=80)


Test patch:
diff --git a/tests/test_tables.py b/tests/test_tables.py
index dcf9ad7..da4bf53 100644
--- a/tests/test_tables.py
+++ b/tests/test_tables.py
@@ -141,6 +141,33 @@ table_head_body_missing_head = """<table>
     </tbody>
 </table>"""
 
+table_head_body_multiple_head = """<table>
+    <thead>
+        <tr>
+            <td>Creator</td>
+            <td>Editor</td>
+            <td>Server</td>
+        </tr>
+        <tr>
+            <td>Operator</td>
+            <td>Manager</td>
+            <td>Engineer</td>
+        </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td>Bob</td>
+            <td>Oliver</td>
+            <td>Tom</td>
+        </tr>
+        <tr>
+            <td>Thomas</td>
+            <td>Lucas</td>
+            <td>Ethan</td>
+        </tr>
+    </tbody>
+</table>"""
+
 table_missing_text = """<table>
     <thead>
         <tr>
@@ -245,10 +272,28 @@ def test_table():
     assert md(table_with_linebreaks) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith Jackson | 50 |\n| Eve | Jackson Smith | 94 |\n\n'
     assert md(table_with_header_column) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
     assert md(table_head_body) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_head_body_multiple_head) == '\n\n|  |  |  |\n| --- | --- | --- |\n| Creator | Editor | Server |\n| Operator | Manager | Engineer |\n| Bob | Oliver | Tom |\n| Thomas | Lucas | Ethan |\n\n'
     assert md(table_head_body_missing_head) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
     assert md(table_missing_text) == '\n\n|  | Lastname | Age |\n| --- | --- | --- |\n| Jill |  | 50 |\n| Eve | Jackson | 94 |\n\n'
-    assert md(table_missing_head) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
-    assert md(table_body) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
-    assert md(table_with_caption) == 'TEXT\n\nCaption\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n\n'
+    assert md(table_missing_head) == '\n\n|  |  |  |\n| --- | --- | --- |\n| Firstname | Lastname | Age |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_body) == '\n\n|  |  |  |\n| --- | --- | --- |\n| Firstname | Lastname | Age |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_caption) == 'TEXT\n\nCaption\n\n|  |  |  |\n| --- | --- | --- |\n| Firstname | Lastname | Age |\n\n'
     assert md(table_with_colspan) == '\n\n| Name | | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
     assert md(table_with_undefined_colspan) == '\n\n| Name | Age |\n| --- | --- |\n| Jill | Smith |\n\n'
+
+
+def test_table_infer_header():
+    assert md(table, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_html_content, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| **Jill** | *Smith* | [50](#) |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_paragraphs, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_linebreaks, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith Jackson | 50 |\n| Eve | Jackson Smith | 94 |\n\n'
+    assert md(table_with_header_column, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_head_body, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_head_body_multiple_head, table_infer_header=True) == '\n\n| Creator | Editor | Server |\n| --- | --- | --- |\n| Operator | Manager | Engineer |\n| Bob | Oliver | Tom |\n| Thomas | Lucas | Ethan |\n\n'
+    assert md(table_head_body_missing_head, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_missing_text, table_infer_header=True) == '\n\n|  | Lastname | Age |\n| --- | --- | --- |\n| Jill |  | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_missing_head, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_body, table_infer_header=True) == '\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_caption, table_infer_header=True) == 'TEXT\n\nCaption\n\n| Firstname | Lastname | Age |\n| --- | --- | --- |\n\n'
+    assert md(table_with_colspan, table_infer_header=True) == '\n\n| Name | | Age |\n| --- | --- | --- |\n| Jill | Smith | 50 |\n| Eve | Jackson | 94 |\n\n'
+    assert md(table_with_undefined_colspan, table_infer_header=True) == '\n\n| Name | Age |\n| --- | --- |\n| Jill | Smith |\n\n'