Repository: zulip/zulip

Issue:
Block quotes (>) and fenced code blocks don't interact nicely
When you try to block-quote a code block, it.. really doesn't work right. The best way of explaining this is with images:

(Apologies for the wide screenshots)
![](https://hydraz.semi.works/img/91a18.png)

Source for the message:
![](https://hydraz.semi.works/img/4fe77.png)

I expected something more like Pandoc's rendering, where the whole code block is block-quoted:
![](https://hydraz.semi.works/img/95884.png)
```html
<blockquote>
	<div class="sourceCode" id="cb1">
		<pre class="sourceCode haskell">
			<code class="sourceCode haskell">
				<span id="cb1-1">
					<a href="#cb1-1"></a>
					<span class="ot">foo ::</span>
					<span class="dt">Int</span>
					<span class="ot">-&gt;</span>
					<span class="dt">Int</span>
				</span>
				<span id="cb1-2">
					<a href="#cb1-2"></a>foo x 
					<span class="ot">=</span> x 
					<span class="op">+</span>
					<span class="dv">5</span>
				</span>
			</code>
		</pre>
	</div>
</blockquote>
```




Discussion:
@aero31aero this is a duplicate, right?  
I don't think so. Are you thinking of #12446?
Ahh, I think so.  I'm not sure the syntax proposed here makes sense; we might want to use the syntax in #12446 for this purpose.
Hello @zulip/server-markdown, @zulip/server-message-view members, this issue was labeled with the "area: markdown", "area: message view" labels, so you may want to check it out!

<!-- areaLabelAddition -->
~~I can't see the images here anymore, but I think end-quote markers in #13001 would fix this.~~

Edit: no, this is a separate issue, but the commonmark implementation doesn't agree with the proposed syntax here:

```
<blockquote>

<pre>
    <code class="language-js"></code>
  </pre>
</blockquote>
<p>
  let y = 1; console.log(y);
</p>

<pre>
  <code></code>
</pre>
```

for 

~~~
> ```js
let y = 1;
console.log(y);
```
~~~
It’s not a bug that we don’t support the incorrect syntax where you only quote the first line of the code block, but it is a bug that we don’t support the CommonMark syntax where you quote every line of the code block:

```md
> ```haskell
> foo :: Int -> Int
> foo x = x + 5
> ```
```

Resolving patch:
diff --git a/zerver/lib/bugdown/__init__.py b/zerver/lib/bugdown/__init__.py
--- a/zerver/lib/bugdown/__init__.py
+++ b/zerver/lib/bugdown/__init__.py
@@ -1814,6 +1814,51 @@ def get_sub_registry(r: markdown.util.Registry, keys: List[str]) -> markdown.uti
 DEFAULT_BUGDOWN_KEY = -1
 ZEPHYR_MIRROR_BUGDOWN_KEY = -2
 
+class NormalizeWhitespace(markdown.preprocessors.NormalizeWhitespace):
+    """ Upstream NormalizeWhitespace strays away from its stated goal
+    by also stripping the characters that the HtmlStash uses to mark the
+    start/end of the stashed text. This means that we cannot run the upstream
+    preprocessor after we have already stashed some Html blocks.
+
+    Our NormalizeWhitespace is exactly the same as upstream except for one line
+    marked below. """
+
+    def run(self, lines: List[str]) -> List[str]:
+        source = '\n'.join(lines)
+        # source = source.replace(util.STX, "").replace(util.ETX, "") # Zulip change: we comment this line.
+        source = source.replace("\r\n", "\n").replace("\r", "\n") + "\n\n"
+        source = source.expandtabs(self.md.tab_length)
+        source = re.sub(r'(?<=\n) +\n', '\n', source)
+        return source.split('\n')
+
+class BlockParser(markdown.blockprocessors.BlockParser):
+    """ Upstream doesn't support running preprocessors when parsing blocks, which
+    is needed for issues like https://github.com/zulip/zulip/issues/12800. This
+    subclass retains the default upstream behavior for all blocks except blockquotes.
+    """
+    def __init__(self, md: markdown.Markdown) -> None:
+        super().__init__(md)
+        self.custom_normalize_whitespace = NormalizeWhitespace(self.md)
+
+    def run_preprocessors(self, block: str) -> str:
+        # Due to the restriction described in NormalizeWhitespace, we need to
+        # run our subclassed NormalizeWhitespace preprocessor instead of the
+        # upstream version when we run the preprocessors inside a block.
+        lines = block.split("\n")
+        whitespace = self.md.preprocessors.__getitem__('normalize_whitespace')
+        for prep in self.md.preprocessors:
+            if prep == whitespace:
+                lines = self.custom_normalize_whitespace.run(lines)
+            else:
+                lines = prep.run(lines)
+        return '\n'.join(lines)
+
+    def parseChunk(self, parent: Element, text: str) -> None:
+        if parent.tag == 'blockquote':
+            # Rerun preprocessors, mainly to detect fenced_code inside quotes.
+            text = self.run_preprocessors(text)
+        super().parseChunk(parent, text)
+
 class Bugdown(markdown.Markdown):
     def __init__(self, *args: Any, **kwargs: Union[bool, int, List[Any]]) -> None:
         # define default configs
@@ -1864,7 +1909,7 @@ def build_block_parser(self) -> markdown.util.Registry:
         # olist - replaced by ours
         # ulist - replaced by ours
         # quote - replaced by ours
-        parser = markdown.blockprocessors.BlockParser(self)
+        parser = BlockParser(self)
         parser.blockprocessors.register(markdown.blockprocessors.EmptyBlockProcessor(parser), 'empty', 95)
         parser.blockprocessors.register(ListIndentProcessor(parser), 'indent', 90)
         if not self.getConfig('code_block_processor_disabled'):
diff --git a/zerver/lib/bugdown/fenced_code.py b/zerver/lib/bugdown/fenced_code.py
--- a/zerver/lib/bugdown/fenced_code.py
+++ b/zerver/lib/bugdown/fenced_code.py
@@ -293,7 +293,6 @@ def run(self, lines: Iterable[str]) -> List[str]:
 
         handler = OuterHandler(processor, output, self.run_content_validators)
         self.push(handler)
-
         for line in lines:
             self.handlers[-1].handle_line(line)
 


Test patch:
diff --git a/zerver/tests/fixtures/markdown_test_cases.json b/zerver/tests/fixtures/markdown_test_cases.json
--- a/zerver/tests/fixtures/markdown_test_cases.json
+++ b/zerver/tests/fixtures/markdown_test_cases.json
@@ -94,6 +94,12 @@
       "input": ">\n>\ntext",
       "expected_output": "<blockquote>\n<p>text</p>\n</blockquote>"
     },
+    {
+      "name": "blockquote_with_fenced_code",
+      "input": "> ```\n> multilne\n> codeblock\n> ```",
+      "expected_output": "<blockquote>\n<div class=\"codehilite\"><pre><span></span>multilne\ncodeblock\n</pre></div>\n\n\n</blockquote>",
+      "marked_expected_output": "<blockquote>\n<div class=\"codehilite\"><pre>multilne\ncodeblock\n</pre></div>\n\n\n</blockquote>"
+    },
     {
       "name": "fenced_quote_with_hashtag",
       "input": "```quote\n# line 1\n#line 2\n```",