Repository: Python-Markdown/markdown (python)

Issue:
PI (<? ?>) is escaped in error with md_in_html
In any of the following source documents:

```
<div markdown=1>

<?php print("foo"); >

</div>
```

```
<div markdown=1>
<?php print("foo"); >
</div>
```

```
<div markdown=1><?php print("foo"); ></div>
```

We get the same output:

```html
<div>
<p>&lt;?php print("foo"); &gt;</p>
</div>
```

Which is correct except for the fact the the opening and closing brackets of the processing instruction (PI) are escaped. We should be getting:

```html
<div>
<p><?php print("foo"); ></p>
</div>
```

The problem is related to the fact that the extension builds an etree object of raw HTML when `markdown=1`. However, the PI (`<?php print("foo"); >`) is simply passed to the etree object as text data (to `treebuilder.data`). Upon serialization, it is escaped. The obvious fix would be to pass the content of the PI to `treebuilder.pi`, except that [xml.etree.ElementTree.TreeBuilder.pi][1] was only added in PY3.8. What do we do for PY3.6 and PY3.7?

[1]: https://docs.python.org/3/library/xml.etree.elementtree.html#xml.etree.ElementTree.TreeBuilder.pi

Pull request:
Properly parse processing instructions in md_in_html
Empty tags do not have a `mardkown` attribute set on them. Therefore,
there is no need to check the mdstack to determine behavior. If we
are in any md_in_html state (regardless of block, span, etc) the
behavior is the same. Fixes #1070.

Patch:
diff --git a/docs/change_log/index.md b/docs/change_log/index.md
index bce97da..7edb2b9 100644
--- a/docs/change_log/index.md
+++ b/docs/change_log/index.md
@@ -5,6 +5,7 @@ Python-Markdown Change Log
 
 Under development: version 3.3.4 (a bug-fix release).
 
+* Properly parse processing instructions in md_in_html (#1070).
 * Properly parse code spans in md_in_html (#1069).
 
 Oct 25, 2020: version 3.3.3 (a bug-fix release).
diff --git a/markdown/extensions/md_in_html.py b/markdown/extensions/md_in_html.py
index b8848ef..6d2a0e7 100644
--- a/markdown/extensions/md_in_html.py
+++ b/markdown/extensions/md_in_html.py
@@ -204,10 +204,7 @@ class HTMLExtractorExtra(HTMLExtractor):
             if self.at_line_start() and is_block:
                 self.handle_data('\n' + self.md.htmlStash.store(data) + '\n\n')
             else:
-                if self.mdstate and self.mdstate[-1] == "off":
-                    self.handle_data(self.md.htmlStash.store(data))
-                else:
-                    self.handle_data(data)
+                self.handle_data(self.md.htmlStash.store(data))
 
 
 class HtmlBlockPreprocessor(Preprocessor):


Test patch:
diff --git a/tests/test_syntax/extensions/test_md_in_html.py b/tests/test_syntax/extensions/test_md_in_html.py
index 7786b80..8655607 100644
--- a/tests/test_syntax/extensions/test_md_in_html.py
+++ b/tests/test_syntax/extensions/test_md_in_html.py
@@ -634,6 +634,56 @@ class TestMdInHTML(TestCase):
             )
         )
 
+    def test_md1_PI_oneliner(self):
+        self.assertMarkdownRenders(
+            '<div markdown="1"><?php print("foo"); ?></div>',
+            self.dedent(
+                """
+                <div>
+                <?php print("foo"); ?>
+                </div>
+                """
+            )
+        )
+
+    def test_md1_PI_multiline(self):
+        self.assertMarkdownRenders(
+            self.dedent(
+                """
+                <div markdown="1">
+                <?php print("foo"); ?>
+                </div>
+                """
+            ),
+            self.dedent(
+                """
+                <div>
+                <?php print("foo"); ?>
+                </div>
+                """
+            )
+        )
+
+    def test_md1_PI_blank_lines(self):
+        self.assertMarkdownRenders(
+            self.dedent(
+                """
+                <div markdown="1">
+
+                <?php print("foo"); ?>
+
+                </div>
+                """
+            ),
+            self.dedent(
+                """
+                <div>
+                <?php print("foo"); ?>
+                </div>
+                """
+            )
+        )
+
     def test_md_span_paragraph(self):
         self.assertMarkdownRenders(
             '<p markdown="span">*foo*</p>',