Repository: Python-Markdown/markdown

Issue:
3.8 - Layout issues
## Introduction
The recent release of 3.8 has resulted in some layout issues for me and I am attempting to determine if I need to change my generated pages or if this is a problem with this release. If this is a "me" problem, then I have 2 years worth of docs (1,000+) to review and change before I can switch to 3.8.

I have narrowed down one issue and attached is a simple docker-compose project with three services - 3.7, 3.8 and 3.8-fix - that highlights the issue and a potential change I need to make, albeit that the results are not identical to 3.7.

The issue identified concerns the layout/format of multiple `div` tags. One example being:

```
<div style="display: inline-flex" markdown>
  <div class="circle"></div>AAAAA<div class="circle"></div>BBBBB<div class="circle"></div><span>CCCCC</span>
</div>
```

## Docker 
The are three docker projects, each contain 4 files, a `Dockerfile`, a python `requirements.txt`, an mkdocs configuration file and a sample `index.md`. 

The 3.7 and 3.8 projects differ only in pinning the version of markdown and 3.8-fix is the same as 3.8 but with a few minor changes to `index.md` to mitigate and attempt to resolve the layout issues.

[markdown-3.8-problem.tar.gz](https://github.com/user-attachments/files/19754989/markdown-3.8-problem.tar.gz)

Should be a simple case of extracting and running:

```console
docker compose up --build
```

and then browse to http://localhost:8037/ http://localhost:8038/ and http://localhost:8039/

## Screenshots

![Image](https://github.com/user-attachments/assets/c3c541ad-d101-4bcc-ae2c-74f7d7556b86)

![Image](https://github.com/user-attachments/assets/80336892-9d1b-4915-92b3-cef2c4af8104)

![Image](https://github.com/user-attachments/assets/e31269ac-1073-4550-adca-4cb16f545154)

Hints from issue comments:
@facelessuser: I won't be running your whole documentation, but I will run the specfic example you gave.

I see you are using the `markdown` attribute so it appears you are using `md_in_html` (correct me if I'm wrong). From what I see, you were not using `md_in_html` as intended and relying on a buggy and broken output. In both Python Markdown 3.7 and Python Markdown 3.8, the output isn't great due to how you are using `md_in_html`.

So here is your example using Python Markdown 3.7:

```py
import markdown

text = '''
<div style="display: inline-flex" markdown>
  <div class="circle"></div>AAAAA<div class="circle"></div>BBBBB<div class="circle"></div><span>CCCCC</span>
</div>
'''

html = markdown.markdown(text, extensions=['md_in_html'])
print(html)
```

```html
<div style="display: inline-flex">
<div class="circle"></div>
<p>AAAAA<div class="circle"></p>
</div>
<p>BBBBB
<div class="circle"></div></p>
<p><span>CCCCC</span>
</div></p>
```

Notice that it produces invalid HTML.


Now let's run it in Python Markdown 3.8:

```html
<div style="display: inline-flex">
<div class="circle"></div>
</div>
<p>BBBBB
<div class="circle"></div></p>
<p><span>CCCCC</span></p>
</div>
```

Output is still not valid and yes, I do see some content getting dropped which should be fixed, but if we use `md_in_html` as it is intended to be used, problems go away.

```py
import markdown

text = '''
<div style="display: inline-flex" markdown>
<div class="circle"></div>
AAAAA
<div class="circle"></div>
BBBBB
<div class="circle"></div>
<span>CCCCC</span>
</div>
'''

html = markdown.markdown(text, extensions=['md_in_html'])
print(html)
```

In both Python Markdown 3.7 and 3.8 we get valid, expected output.

```html
<div style="display: inline-flex">
<div class="circle"></div>
<p>AAAAA</p>
<div class="circle"></div>
<p>BBBBB</p>
<div class="circle"></div>
<p><span>CCCCC</span></p>
</div>
```

So, yes, there is an edge case to fix in Python Markdown 3.8 as it should not drop content, but the HTML will still come out invalid if you continue approaching your `md_in_html` as you currently are. You should have each Markdown block starting at the begining of their own line.

---

One of the changes in `md_in_html` was to try and get it parse inside a `markdown` HTML tag more like it does outside, but there still seems to be some edge cases when people use it in a way not fully intended. Neither Python Markdown 3.7 or 3.8 are outputting things great, but because some content is being dropped in 3.8, I agree this is a little worse and needs to be addressed.



@facelessuser: If I had to guess, I think the first `<div>` under the `markdown` element isn't having its tail checked for content.@facelessuser: I think I found the issue@facelessuser: I have a PR up which should now output the following for your example. Output is still invalid, we just aren't dropping any content anymore. I would recommend you update your content to use `md_in_html` as intended as I suggested earlier to get valid HTML output.

```html
<div style="display: inline-flex">
<div class="circle"></div>
<p>AAAAA<div class="circle"></p>
</div>
<p>BBBBB
<div class="circle"></div></p>
<p><span>CCCCC</span>
</div></p>
```@tgit-hub: Possible not a full example, I carved out quite a lot in making the much smaller example that reflected what I saw happening - for which the `md_in_html` was in use.

Pleased you found something though and will be happy to test locally once released.@facelessuser: You can test the PR branch now if you like.@tgit-hub: Using the sample project I created, just tested with the following in the `requirements.txt`:

```
markdown@git+https://github.com/Python-Markdown/markdown@90bfcca9ba3ca79b21d911b027ad00cff352f7b4
```

and the output between 3.7 and 3.8 is structurally the same.

So, thank you for that 👍 @facelessuser: @tgit-hub Thanks for trying it out! That helps us ensure we aren't missing anything else.@tgit-hub: I've done a complete rebuild of the main site and that appears to be OK now - albeit only a cursory glance at a couple of the "problem" pages. Will have a chance to go through more tomorrow.

Will also take a look at linting the output as well, just to see how much "bad" html is being generated and correct where appropriate.

Again, thank you for the prompt change.

Solution patch:
diff --git a/docs/changelog.md b/docs/changelog.md
index 930199f..7c96a6f 100644
--- a/docs/changelog.md
+++ b/docs/changelog.md
@@ -9,6 +9,12 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 See the [Contributing Guide](contributing.md) for details.
 
+## [Unreleased]
+
+### Fixed
+
+* Fixed dropped content in `md_in_html` (#1526).
+
 ## [3.8.0] - 2025-04-09
 
 ### Changed
diff --git a/markdown/extensions/md_in_html.py b/markdown/extensions/md_in_html.py
index d1fbd7a..ba73c94 100644
--- a/markdown/extensions/md_in_html.py
+++ b/markdown/extensions/md_in_html.py
@@ -387,12 +387,16 @@ class MarkdownInHtmlProcessor(BlockProcessor):
             element = self.parser.md.htmlStash.rawHtmlBlocks[index]
             if isinstance(element, etree.Element):
                 # We have a matched element. Process it.
-                blocks.pop(0)
+                block = blocks.pop(0)
                 parent.append(element)
                 self.parse_element_content(element)
                 # Cleanup stash. Replace element with empty string to avoid confusing postprocessor.
                 self.parser.md.htmlStash.rawHtmlBlocks.pop(index)
                 self.parser.md.htmlStash.rawHtmlBlocks.insert(index, '')
+                content = block[m.end(0):]
+                # Ensure the rest of the content gets handled
+                if content:
+                    blocks.insert(0, content)
                 # Confirm the match to the `blockparser`.
                 return True
         # No match found.


Test patch:
diff --git a/tests/test_syntax/extensions/test_md_in_html.py b/tests/test_syntax/extensions/test_md_in_html.py
index 1bdca39..5ef860d 100644
--- a/tests/test_syntax/extensions/test_md_in_html.py
+++ b/tests/test_syntax/extensions/test_md_in_html.py
@@ -1517,6 +1517,27 @@ class TestMdInHTML(TestCase):
             extensions=['md_in_html']
         )
 
+    def test_trailing_content_after_tag_in_md_block(self):
+
+        # It should be noted that this is not the way `md_in_html` is intended to be used.
+        # What we are specifically testing is an edge case where content was previously lost.
+        # Lost content should not happen.
+        self.assertMarkdownRenders(
+            self.dedent(
+                """
+                <div markdown>
+                <div class="circle"></div>AAAAA<div class="circle"></div>
+                </div>
+                """
+            ),
+            '<div>\n'
+            '<div class="circle"></div>\n'
+            '<p>AAAAA<div class="circle"></p>\n'
+            '</div>\n'
+            '</div>',
+            extensions=['md_in_html']
+        )
+
 
 def load_tests(loader, tests, pattern):
     """ Ensure `TestHTMLBlocks` doesn't get run twice by excluding it here. """