Repository: zulip/zulip

Issue:
Show a pretty-printed diff of the full strings in bugdown tests' errors
(Make sure you've read zulip.readthedocs.io/en/latest/markdown.html to understand this issue).

# Description
Right now, when a test in the strings checked in `zerver/fixtures/markdown_test_cases.json` doesn't match what it should, `test-js-with-node` (aka the frontend tests) raises an `AssertionError`, with a small diff of the conflicting strings:

```
assert.js:85
  throw new assert.AssertionError({
  ^
AssertionError: 'foo' == 'bar'
    at /srv/zulip/frontend_tests/node_tests/echo.js:156:14
    ...
```

However, when the strings are longer it's hard to identify the difference between them because they aren't readably organized. 

The same issue exists with the backend markdown tests as well, with the added problem that if they're long enough, they get truncated (which can potentially skip the difference between both strings).

```
AssertionError: '<p><span class="katex-display"><span class="katex"><span class
="katex-mathml"><math><semantics><mrow><msub><mi mathvariant="nor == '<p><span 
class="katex-display"><span class="katex"><span class="katex-mathml"><math>
<semantics><mrow><msub><mi mathvariant="nor
```

# Additional information
The truncation problem in the backend Bugdown probably depends on the `maxDiff` setting; we may be able to fix it in general by updating the Bugdown test classes.

For the pretty-printing HTML diff, it's not clear what is required, but it might be worth looking at how Markdown processors write their own unit tests (for `marked.js` and `python-markdown` to see if they have a good solution for this that we can just import)


Discussion:
Hello @zulip/server-testing members, this issue was labeled with the **area: testing-infrastructure** label, so you may want to check it out!
Hello @zulip/server-markdown members, this issue was labeled with the **area: markdown** label, so you may want to check it out!
Edited the description a bit :)
@PentaHiggs since you've been working on markdown, this might be an issue worth working on.  I imagine done well, it'd make life a lot easier for reading things like the `katex` output :).  I'd look for a third-party HTML diff printer (actually, I think we're using `lxml` for this in `zerver/lib/html_diff.py` -- maybe worth looking at that?).  

Also CC @HarshitOnGitHub who wrote that `html_diff.py` code.
Thanks for pointing it out to me!  I've been thinking of doing something about that; the `katex` output diffs are more or less unreadable sometimes.
@zulipbot claim
Hello @PentaHiggs, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 14 days.

You can reclaim this issue or claim any other issue by commenting `@zulipbot claim` on that issue.

Thanks for your contributions, and hope to see you again soon!

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -1,3 +1,3 @@
 ZULIP_VERSION = "1.7.1+git"
 
-PROVISION_VERSION = '14.1'
+PROVISION_VERSION = '14.2'
diff --git a/zerver/lib/mdiff.py b/zerver/lib/mdiff.py
new file mode 100755
--- /dev/null
+++ b/zerver/lib/mdiff.py
@@ -0,0 +1,15 @@
+import os
+import subprocess
+import logging
+import difflib
+
+def diff_strings(output: str, expected_output: str) -> str:
+
+    mdiff_path = "frontend_tests/zjsunit/mdiff.js"
+    if not os.path.isfile(mdiff_path):  # nocoverage
+        logging.error("Cannot find mdiff for markdown diff rendering")
+        return None
+
+    command = ['node', mdiff_path, output, expected_output]
+    diff = subprocess.check_output(command).decode('utf-8')
+    return diff


Test patch:
diff --git a/frontend_tests/node_tests/markdown.js b/frontend_tests/node_tests/markdown.js
--- a/frontend_tests/node_tests/markdown.js
+++ b/frontend_tests/node_tests/markdown.js
@@ -1,6 +1,7 @@
 /*global Dict */
 var path = zrequire('path', 'path');
 var fs = zrequire('fs', 'fs');
+
 zrequire('hash_util');
 zrequire('katex', 'node_modules/katex/dist/katex.min.js');
 zrequire('marked', 'third/marked/lib/marked');
@@ -172,18 +173,19 @@ var bugdown_data = JSON.parse(fs.readFileSync(path.join(__dirname, '../../zerver
 
 (function test_marked_shared() {
     var tests = bugdown_data.regular_tests;
+
     tests.forEach(function (test) {
         var message = {raw_content: test.input};
         markdown.apply_markdown(message);
         var output = message.content;
 
         if (test.marked_expected_output) {
-            assert.notEqual(test.expected_output, output);
-            assert.equal(test.marked_expected_output, output);
+            global.bugdown_assert.notEqual(test.expected_output, output);
+            global.bugdown_assert.equal(test.marked_expected_output, output);
         } else if (test.backend_only_rendering) {
             assert.equal(markdown.contains_backend_only_syntax(test.input), true);
         } else {
-            assert.equal(test.expected_output, output);
+            global.bugdown_assert.equal(test.expected_output, output);
         }
     });
 }());
diff --git a/frontend_tests/zjsunit/bugdown_assert.js b/frontend_tests/zjsunit/bugdown_assert.js
new file mode 100644
--- /dev/null
+++ b/frontend_tests/zjsunit/bugdown_assert.js
@@ -0,0 +1,184 @@
+/**
+ * bugdown_assert.js
+ *
+ * Used to determine whether two Markdown HTML strings are semantically
+ * equivalent. Differs from the naive string-comparison approach in that
+ * differently typed but equivalent HTML fragments, such as '<p>&quot;</p>'
+ * and '<p>\"</p>', and '<span attr1="a" attr2="b"></span>' and
+ * '<span attr2="a" attr1="b"></span>', are still considered equal.
+ *
+ * The exported method equal() serves as a drop-in replacement for
+ * assert.equal().  Likewise, the exported method notEqual() replaces
+ * assert.notEqual().
+ *
+ * There is a default _output_formatter used to create the
+ * AssertionError error message; this function can be overriden using
+ * the exported setFormatter() function below.
+ *
+ * The HTML passed to the _output_formatter is not the original HTML, but
+ * rather a serialized version of a DOM element generated from the original
+ * HTML.  This makes it easier to spot relevant differences.
+ */
+
+const jsdom = require('jsdom');
+const _ = require('underscore');
+
+const mdiff = require('./mdiff.js');
+
+// Module-level global instance of MarkdownComparer, initialized when needed
+let _markdownComparerInstance = null;
+
+class MarkdownComparer {
+    constructor(output_formatter) {
+        this._output_formatter = output_formatter || function (actual, expected) {
+            return ["Actual and expected output do not match.",
+                    actual,
+                    "!=",
+                    expected,
+                ].join('\n');
+        };
+        this._document = jsdom.jsdom();
+    }
+
+    setFormatter(output_formatter) {
+        this._output_formatter = output_formatter || this._output_formatter;
+    }
+
+    _htmlToElement(html, id) {
+        const template = this._document.createElement('template');
+        const id_node = this._document.createAttribute('id');
+        id_node.value = id;
+        template.setAttributeNode(id_node);
+        template.innerHTML = html;
+        return template;
+    }
+
+    _haveEqualContents(node1, node2) {
+        if (node1.content.childNodes.length !== node2.content.childNodes.length) {
+            return false;
+        }
+        return _.reduce(
+            _.zip(node1.content.childNodes, node2.content.childNodes),
+            (prev, nodePair) => { return prev && nodePair[0].isEqualNode(nodePair[1]); },
+            true
+        );
+    }
+
+    _reorderAttributes(node) {
+        // Sorts every attribute in every element by name.  Ensures consistent diff HTML output
+
+        const attributeList = [];
+        _.forEach(node.attributes, (attr) => {
+            attributeList.push(attr);
+        });
+
+        // If put in above forEach loop, causes issues (possible nodes.attribute invalidation?)
+        attributeList.forEach((attr) => {node.removeAttribute(attr.name);});
+
+        attributeList.sort((a, b) => {
+            const name_a = a.name;
+            const name_b = b.name;
+            if (name_a < name_b) {
+                return -1;
+            } else if (name_a > name_b) {
+                return 1;
+            }
+            return 0;
+        });
+
+        // Put them back in, in order
+        attributeList.forEach((attribute) => {
+            node.setAttribute(attribute.name, attribute.value);
+        });
+
+        if (node.hasChildNodes()) {
+            _.forEach(node.children, (childNode) => {
+                this._reorderAttributes(childNode);
+            });
+        }
+        if (node.content && node.content.hasChildNodes()) {
+            _.forEach(node.content.children, (childNode) => {
+                this._reorderAttributes(childNode);
+            });
+        }
+        return node;
+    }
+
+    _compare(actual_markdown, expected_markdown) {
+        const ID_ACTUAL = "0";
+        const ID_EXPECTED = "1";
+
+        const element_actual = this._htmlToElement(actual_markdown, ID_ACTUAL);
+        const element_expected = this._htmlToElement(expected_markdown, ID_EXPECTED);
+
+        let are_equivalent = false;
+        let html = {};
+
+        are_equivalent = this._haveEqualContents(element_actual, element_expected);
+        if (!are_equivalent) {
+            html = {
+                actual : this._reorderAttributes(element_actual).innerHTML,
+                expected : this._reorderAttributes(element_expected).innerHTML,
+            };
+        }
+
+        element_actual.remove();
+        element_expected.remove();
+
+        return { are_equivalent, html };
+    }
+
+    assertEqual(actual, expected, message) {
+        const comparison_results = this._compare(actual, expected);
+
+        if (comparison_results.are_equivalent === false) {
+            throw new assert.AssertionError({
+                message : message || this._output_formatter(
+                    comparison_results.html.actual,
+                    comparison_results.html.expected
+                ),
+            });
+        }
+    }
+
+    assertNotEqual(actual, expected, message) {
+        const comparison_results = this._compare(actual, expected);
+
+        if (comparison_results.are_equivalent) {
+            throw new assert.AssertionError({
+                message : message || [
+                    "actual and expected output produce semantially identical HTML",
+                    actual,
+                    "==",
+                    expected,
+                ].join('\n'),
+            });
+        }
+    }
+}
+
+function returnComparer() {
+    if (!_markdownComparerInstance) {
+        _markdownComparerInstance = new MarkdownComparer((actual, expected) => {
+            return [
+                "Actual and expected output do not match.  Showing diff",
+                mdiff.diff_strings(actual, expected),
+            ].join('\n');
+        });
+    }
+    return _markdownComparerInstance;
+}
+
+module.exports = {
+    equal(expected, actual, message) {
+        returnComparer().assertEqual(actual, expected, message);
+    },
+
+    notEqual(expected, actual, message) {
+        returnComparer().assertNotEqual(actual, expected, message);
+    },
+
+    setFormatter(output_formatter) {
+        returnComparer().setFormatter(output_formatter);
+    },
+};
diff --git a/frontend_tests/zjsunit/index.js b/frontend_tests/zjsunit/index.js
--- a/frontend_tests/zjsunit/index.js
+++ b/frontend_tests/zjsunit/index.js
@@ -54,6 +54,9 @@ module.prototype.hot = {
     accept: noop,
 };
 
+// Set up bugdown comparison helper
+global.bugdown_assert = require('./bugdown_assert.js');
+
 output.start_writing();
 
 files.forEach(function (file) {
diff --git a/frontend_tests/zjsunit/mdiff.js b/frontend_tests/zjsunit/mdiff.js
new file mode 100644
--- /dev/null
+++ b/frontend_tests/zjsunit/mdiff.js
@@ -0,0 +1,146 @@
+/**
+ * mdiff.js
+ *
+ * Used to produce colorful and informative diffs for comparison of generated
+ * Markdown.  Unlike the built-in diffs used in python or node.js assert libraries,
+ * is actually designed to be effective for long, single-line comparisons.
+ *
+ * Based on diffing library difflib, a js port of the python library.
+ *
+ * The sole exported function diff_strings(string_0, string_1) returns a pretty-printed
+ * unicode string containing their diff.
+ */
+
+const _ = require('underscore');
+const difflib = require('difflib');
+
+function apply_color(input_string, changes) {
+    let previous_index = 0;
+    let processed_string = input_string.slice(0,2);
+    input_string = input_string.slice(2);
+
+    const formatter = {
+        delete : (string) => { return "\u001b[31m" + string + "\u001b[0m"; },
+        insert : (string) => { return "\u001b[32m" + string + "\u001b[0m"; },
+        replace : (string) => { return "\u001b[33m" + string + "\u001b[0m"; },
+    };
+    changes.forEach((change) => {
+        if (formatter.hasOwnProperty(change.tag)) {
+            processed_string += input_string.slice(previous_index, change.beginning_index);
+            processed_string += formatter[change.tag](
+                input_string.slice(change.beginning_index, change.ending_index)
+            );
+            previous_index = change.ending_index;
+        }
+    });
+
+    processed_string += input_string.slice(previous_index);
+    return processed_string;
+}
+
+/**
+ * The library difflib produces diffs that look as follows:
+ *
+ * - <p>upgrade! yes</p>
+ * ?    ^^     -
+ * + <p>downgrade yes.</p>
+ * ?    ^^^^         +
+ *
+ * The purpose of this function is to facilitate converting these diffs into
+ * colored versions, where the question-mark lines are removed, replaced with
+ * directions to add appropriate color to the lines that they annotate.
+ */
+function parse_questionmark_line(questionmark_line) {
+    let current_sequence = "";  // Either "^", "-", "+", or ""
+    let beginning_index = 0;
+    let index = 0;
+
+    const changes_list = [];
+    const aliases = {
+        "^" : "replace",
+        "+" : "insert",
+        "-" : "delete",
+    };
+    const add_change = () => {
+        if (current_sequence) {
+            changes_list.push({
+                tag : aliases[current_sequence],
+                beginning_index,
+                ending_index : index,
+            });
+            current_sequence = "";
+        }
+    };
+
+    questionmark_line = questionmark_line.slice(2).trimRight("\n");
+
+    for (const character of questionmark_line) {
+        if (aliases.hasOwnProperty(character)) {
+            if (current_sequence !== character) {
+                add_change();
+                current_sequence = character;
+                beginning_index = index;
+            }
+        } else {
+            add_change();
+        }
+        index += 1;
+    }
+
+    // In case we have a "change" involving the last character on a line
+    // e.g. a string such as "? ^^  -- ++++"
+    add_change();
+
+    return changes_list;
+}
+
+function diff_strings(string_0, string_1) {
+    let output_lines = [];
+    let ndiff_output = "";
+    let changes_list = [];
+
+    ndiff_output = difflib.ndiff(string_0.split("\n"), string_1.split("\n"));
+
+    ndiff_output.forEach((line) => {
+        if (line.startsWith("+")) {
+            output_lines.push(line);
+        } else if (line.startsWith("-")) {
+            output_lines.push(line);
+        } else if (line.startsWith("?")) {
+            changes_list = parse_questionmark_line(line);
+            output_lines[output_lines.length - 1] = apply_color(
+                output_lines[output_lines.length -1], changes_list);
+        } else {
+            output_lines.push(line);
+        }
+    });
+
+    const emphasize_codes = (string) => {
+        return "\u001b[34m" + string.slice(0,1) + "\u001b[0m" + string.slice(1);
+    };
+    output_lines = _.map(output_lines, emphasize_codes);
+
+    return output_lines.join("\n");
+}
+
+module.exports = { diff_strings };
+
+// Simple CLI for this module
+// Only run this code if called as a command-line utility
+if (require.main === module) {
+    // First two args are just "node" and "mdiff.js"
+    const argv = require('minimist')(process.argv.slice(2));
+
+    if (_.has(argv, "help")) {
+        console.log(process.argv[0] + " " + process.argv[1] +
+            " [ --help ]" +
+            " string_0" +
+            " string_1" +
+            "\n" +
+            "Where string_0 and string_1 are the strings to be diffed"
+        );
+    }
+
+    const output = diff_strings(argv._[0], argv._[1]);
+    console.log(output);
+}
diff --git a/zerver/fixtures/markdown_test_cases.json b/zerver/fixtures/markdown_test_cases.json
--- a/zerver/fixtures/markdown_test_cases.json
+++ b/zerver/fixtures/markdown_test_cases.json
@@ -19,6 +19,11 @@
       "expected_output": "<p>Hamlet once said</p>\n<div class=\"codehilite\"><pre><span></span>def func():\n    x = 1\n\n    y = 2\n\n    z = 3\n</pre></div>\n\n\n<p>And all was good.</p>",
       "text_content": "Hamlet once said\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n\n\n\nAnd all was good."
     },
+    {
+      "name": "test",
+      "input": "it's lunch time",
+      "expected_output": "<p>it's lunch time</p>"
+    },
     {
       "name": "codeblock_trailing_whitespace",
       "input": "Hamlet once said\n~~~~\ndef func():\n    x = 1\n\n    y = 2\t\t\n\n    z = 3       \n~~~~\nAnd all was good.",
@@ -93,7 +98,6 @@
       "name": "dangerous_block",
       "input": "xxxxxx xxxxx xxxxxxxx xxxx. x xxxx xxxxxxxxxx:\n\n```\"xxxx xxxx\\xxxxx\\xxxxxx\"```\n\nxxx xxxx xxxxx:```xx.xxxxxxx(x'^xxxx$', xx.xxxxxxxxx)```\n\nxxxxxxx'x xxxx xxxxxxxxxx ```'xxxx'```, xxxxx xxxxxxxxx xxxxx ^ xxx $ xxxxxx xxxxx xxxxxxxxxxxx xxx xxxx xx x xxxx xx xxxx xx xxx xxxxx xxxxxx?",
       "expected_output": "<p>xxxxxx xxxxx xxxxxxxx xxxx. x xxxx xxxxxxxxxx:</p>\n<p><code>\"xxxx xxxx\\xxxxx\\xxxxxx\"</code></p>\n<p>xxx xxxx xxxxx:<code>xx.xxxxxxx(x'^xxxx$', xx.xxxxxxxxx)</code></p>\n<p>xxxxxxx'x xxxx xxxxxxxxxx <code>'xxxx'</code>, xxxxx xxxxxxxxx xxxxx ^ xxx $ xxxxxx xxxxx xxxxxxxxxxxx xxx xxxx xx x xxxx xx xxxx xx xxx xxxxx xxxxxx?</p>",
-      "marked_expected_output": "<p>xxxxxx xxxxx xxxxxxxx xxxx. x xxxx xxxxxxxxxx:</p>\n<p><code>&quot;xxxx xxxx\\xxxxx\\xxxxxx&quot;</code></p>\n<p>xxx xxxx xxxxx:<code>xx.xxxxxxx(x&#39;^xxxx$&#39;, xx.xxxxxxxxx)</code></p>\n<p>xxxxxxx&#39;x xxxx xxxxxxxxxx <code>&#39;xxxx&#39;</code>, xxxxx xxxxxxxxx xxxxx ^ xxx $ xxxxxx xxxxx xxxxxxxxxxxx xxx xxxx xx x xxxx xx xxxx xx xxx xxxxx xxxxxx?</p>",
       "text_content": "xxxxxx xxxxx xxxxxxxx xxxx. x xxxx xxxxxxxxxx:\n\"xxxx xxxx\\xxxxx\\xxxxxx\"\nxxx xxxx xxxxx:xx.xxxxxxx(x'^xxxx$', xx.xxxxxxxxx)\nxxxxxxx'x xxxx xxxxxxxxxx 'xxxx', xxxxx xxxxxxxxx xxxxx ^ xxx $ xxxxxx xxxxx xxxxxxxxxxxx xxx xxxx xx x xxxx xx xxxx xx xxx xxxxx xxxxxx?"
     },
     {
@@ -447,8 +451,7 @@
     {
       "name": "safe_html_messed_up_complexly_nested_script_tags",
       "input": "<scr<script></script>ipt type=\"text/javascript\">alert(\"foo\");</<script></script>script<del></del>>",
-      "expected_output": "<p>&lt;scr&lt;script&gt;&lt;/script&gt;ipt type=\"text/javascript\"&gt;alert(\"foo\");&lt;/&lt;script&gt;&lt;/script&gt;script&lt;del&gt;&lt;/del&gt;&gt;</p>",
-      "marked_expected_output": "<p>&lt;scr&lt;script&gt;&lt;/script&gt;ipt type=&quot;text/javascript&quot;&gt;alert(&quot;foo&quot;);&lt;/&lt;script&gt;&lt;/script&gt;script&lt;del&gt;&lt;/del&gt;&gt;</p>"
+      "expected_output": "<p>&lt;scr&lt;script&gt;&lt;/script&gt;ipt type=\"text/javascript\"&gt;alert(\"foo\");&lt;/&lt;script&gt;&lt;/script&gt;script&lt;del&gt;&lt;/del&gt;&gt;</p>"
     },
     {
       "name": "safe_html_unclosed_tag",
@@ -527,14 +530,12 @@
     {
       "name": "tex_inline",
       "input": "$$1 \\oplus 0 = 1$$",
-      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mn>1</mn><mo>⊕</mo><mn>0</mn><mo>=</mo><mn>1</mn></mrow><annotation encoding=\"application/x-tex\">1 \\oplus 0 = 1</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.72777em;vertical-align:-0.08333em;\"></span><span class=\"base\"><span class=\"mord mathrm\">1</span><span class=\"mbin\">⊕</span><span class=\"mord mathrm\">0</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">1</span></span></span></span></p>",
-      "marked_expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mn>1</mn><mo>⊕</mo><mn>0</mn><mo>=</mo><mn>1</mn></mrow><annotation encoding=\"application/x-tex\">1 \\oplus 0 = 1</annotation></semantics></math></span><span class=\"katex-html\" aria-hidden=\"true\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.72777em;vertical-align:-0.08333em;\"></span><span class=\"base\"><span class=\"mord mathrm\">1</span><span class=\"mbin\">⊕</span><span class=\"mord mathrm\">0</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">1</span></span></span></span></p>"
+      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mn>1</mn><mo>⊕</mo><mn>0</mn><mo>=</mo><mn>1</mn></mrow><annotation encoding=\"application/x-tex\">1 \\oplus 0 = 1</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.72777em;vertical-align:-0.08333em;\"></span><span class=\"base\"><span class=\"mord mathrm\">1</span><span class=\"mbin\">⊕</span><span class=\"mord mathrm\">0</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">1</span></span></span></span></p>"
     },
     {
       "name": "tex_complex",
       "input": "$$\\Phi_E = \\oint E \\cdot dA$$",
-      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><msub><mi mathvariant=\"normal\">Φ</mi><mi>E</mi></msub><mo>=</mo><mo>∮</mo><mi>E</mi><mo>⋅</mo><mi>d</mi><mi>A</mi></mrow><annotation encoding=\"application/x-tex\">\\Phi_E = \\oint E \\cdot dA</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.805em;\"></span><span class=\"strut bottom\" style=\"height:1.11112em;vertical-align:-0.30612em;\"></span><span class=\"base\"><span class=\"mord\"><span class=\"mord mathrm\">Φ</span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.32833099999999993em;\"><span style=\"top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;\"><span class=\"pstrut\" style=\"height:2.7em;\"></span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathit mtight\" style=\"margin-right:0.05764em;\">E</span></span></span></span><span class=\"vlist-s\">​</span></span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.15em;\"></span></span></span></span></span><span class=\"mrel\">=</span><span class=\"mop op-symbol small-op\" style=\"margin-right:0.19445em;position:relative;top:-0.0005599999999999772em;\">∮</span><span class=\"mord mathit\" style=\"margin-right:0.05764em;\">E</span><span class=\"mbin\">⋅</span><span class=\"mord mathit\">d</span><span class=\"mord mathit\">A</span></span></span></span></p>",
-      "marked_expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><msub><mi mathvariant=\"normal\">Φ</mi><mi>E</mi></msub><mo>=</mo><mo>∮</mo><mi>E</mi><mo>⋅</mo><mi>d</mi><mi>A</mi></mrow><annotation encoding=\"application/x-tex\">\\Phi_E = \\oint E \\cdot dA</annotation></semantics></math></span><span class=\"katex-html\" aria-hidden=\"true\"><span class=\"strut\" style=\"height:0.805em;\"></span><span class=\"strut bottom\" style=\"height:1.11112em;vertical-align:-0.30612em;\"></span><span class=\"base\"><span class=\"mord\"><span class=\"mord mathrm\">Φ</span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.32833099999999993em;\"><span style=\"top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;\"><span class=\"pstrut\" style=\"height:2.7em;\"></span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathit mtight\" style=\"margin-right:0.05764em;\">E</span></span></span></span><span class=\"vlist-s\">​</span></span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.15em;\"></span></span></span></span></span><span class=\"mrel\">=</span><span class=\"mop op-symbol small-op\" style=\"margin-right:0.19445em;position:relative;top:-0.0005599999999999772em;\">∮</span><span class=\"mord mathit\" style=\"margin-right:0.05764em;\">E</span><span class=\"mbin\">⋅</span><span class=\"mord mathit\">d</span><span class=\"mord mathit\">A</span></span></span></span></p>"
+      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><msub><mi mathvariant=\"normal\">Φ</mi><mi>E</mi></msub><mo>=</mo><mo>∮</mo><mi>E</mi><mo>⋅</mo><mi>d</mi><mi>A</mi></mrow><annotation encoding=\"application/x-tex\">\\Phi_E = \\oint E \\cdot dA</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.805em;\"></span><span class=\"strut bottom\" style=\"height:1.11112em;vertical-align:-0.30612em;\"></span><span class=\"base\"><span class=\"mord\"><span class=\"mord mathrm\">Φ</span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.32833099999999993em;\"><span style=\"top:-2.5500000000000003em;margin-left:0em;margin-right:0.05em;\"><span class=\"pstrut\" style=\"height:2.7em;\"></span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathit mtight\" style=\"margin-right:0.05764em;\">E</span></span></span></span><span class=\"vlist-s\">​</span></span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height:0.15em;\"></span></span></span></span></span><span class=\"mrel\">=</span><span class=\"mop op-symbol small-op\" style=\"margin-right:0.19445em;position:relative;top:-0.0005599999999999772em;\">∮</span><span class=\"mord mathit\" style=\"margin-right:0.05764em;\">E</span><span class=\"mbin\">⋅</span><span class=\"mord mathit\">d</span><span class=\"mord mathit\">A</span></span></span></span></p>"
     },
     {
       "name": "tex_escaped",
@@ -562,14 +563,12 @@
     {
       "name": "tex_money",
       "input": "Tickets are $5 to $20 for youth, $10-$30 for adults, so we are hoping to bring in $500 from the event ($$x \\approx 500\\$$$)",
-      "expected_output": "<p>Tickets are $5 to $20 for youth, $10-$30 for adults, so we are hoping to bring in $500 from the event (<span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>≈</mo><mn>5</mn><mn>0</mn><mn>0</mn><mi mathvariant=\"normal\">$</mi></mrow><annotation encoding=\"application/x-tex\">x \\approx 500\\$</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.75em;\"></span><span class=\"strut bottom\" style=\"height:0.80556em;vertical-align:-0.05556em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">≈</span><span class=\"mord mathrm\">5</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">$</span></span></span></span>)</p>",
-      "marked_expected_output": "<p>Tickets are $5 to $20 for youth, $10-$30 for adults, so we are hoping to bring in $500 from the event (<span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>≈</mo><mn>5</mn><mn>0</mn><mn>0</mn><mi mathvariant=\"normal\">$</mi></mrow><annotation encoding=\"application/x-tex\">x \\approx 500\\$</annotation></semantics></math></span><span class=\"katex-html\" aria-hidden=\"true\"><span class=\"strut\" style=\"height:0.75em;\"></span><span class=\"strut bottom\" style=\"height:0.80556em;vertical-align:-0.05556em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">≈</span><span class=\"mord mathrm\">5</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">$</span></span></span></span>)</p>"
+      "expected_output": "<p>Tickets are $5 to $20 for youth, $10-$30 for adults, so we are hoping to bring in $500 from the event (<span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>≈</mo><mn>5</mn><mn>0</mn><mn>0</mn><mi mathvariant=\"normal\">$</mi></mrow><annotation encoding=\"application/x-tex\">x \\approx 500\\$</annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.75em;\"></span><span class=\"strut bottom\" style=\"height:0.80556em;vertical-align:-0.05556em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">≈</span><span class=\"mord mathrm\">5</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">0</span><span class=\"mord mathrm\">$</span></span></span></span>)</p>"
     },
     {
       "name": "tex_inline_permissive_spacing",
       "input": "$$    x =  7   $$",
-      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>=</mo><mn>7</mn></mrow><annotation encoding=\"application/x-tex\">    x =  7   </annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.64444em;vertical-align:0em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">7</span></span></span></span></p>",
-      "marked_expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>=</mo><mn>7</mn></mrow><annotation encoding=\"application/x-tex\">    x =  7   </annotation></semantics></math></span><span class=\"katex-html\" aria-hidden=\"true\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.64444em;vertical-align:0em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">7</span></span></span></span></p>"
+      "expected_output": "<p><span class=\"katex\"><span class=\"katex-mathml\"><math><semantics><mrow><mi>x</mi><mo>=</mo><mn>7</mn></mrow><annotation encoding=\"application/x-tex\">    x =  7   </annotation></semantics></math></span><span aria-hidden=\"true\" class=\"katex-html\"><span class=\"strut\" style=\"height:0.64444em;\"></span><span class=\"strut bottom\" style=\"height:0.64444em;vertical-align:0em;\"></span><span class=\"base\"><span class=\"mord mathit\">x</span><span class=\"mrel\">=</span><span class=\"mord mathrm\">7</span></span></span></span></p>"
     },
     {
       "name": "tex_inline_prohibited_newline",
diff --git a/zerver/tests/test_bugdown.py b/zerver/tests/test_bugdown.py
--- a/zerver/tests/test_bugdown.py
+++ b/zerver/tests/test_bugdown.py
@@ -22,6 +22,7 @@
     ZulipTestCase,
 )
 from zerver.lib.test_runner import slow
+from zerver.lib import mdiff
 from zerver.models import (
     realm_in_local_realm_filters_cache,
     flush_per_request_caches,
@@ -165,6 +166,12 @@ def bugdown_convert(text: Text) -> Text:
     return bugdown.convert(text, message_realm=get_realm('zulip'))
 
 class BugdownMiscTest(ZulipTestCase):
+    def test_diffs_work_as_expected(self) -> None:
+        str1 = "<p>The quick brown fox jumps over the lazy dog.  Animal stories are fun, yeah</p>"
+        str2 = "<p>The fast fox jumps over the lazy dogs and cats.  Animal stories are fun</p>"
+        expected_diff = "\u001b[34m-\u001b[0m <p>The \u001b[33mquick brown\u001b[0m fox jumps over the lazy dog.  Animal stories are fun\u001b[31m, yeah\u001b[0m</p>\n\u001b[34m+\u001b[0m <p>The \u001b[33mfast\u001b[0m fox jumps over the lazy dog\u001b[32ms and cats\u001b[0m.  Animal stories are fun</p>\n"
+        self.assertEqual(mdiff.diff_strings(str1, str2), expected_diff)
+
     def test_get_full_name_info(self) -> None:
         realm = get_realm('zulip')
 
@@ -207,6 +214,14 @@ def test_mention_data(self) -> None:
         self.assertEqual(user['email'], hamlet.email)
 
 class BugdownTest(ZulipTestCase):
+    def assertEqual(self, first: Any, second: Any, msg: Text = "") -> None:
+        if isinstance(first, Text) and isinstance(second, Text):
+            if first != second:
+                raise AssertionError("Actual and expected outputs do not match; showing diff.\n" +
+                                     mdiff.diff_strings(first, second) + msg)
+        else:
+            super().assertEqual(first, second)
+
     def load_bugdown_tests(self) -> Tuple[Dict[Text, Any], List[List[Text]]]:
         test_fixtures = {}
         data_file = open(os.path.join(os.path.dirname(__file__), '../fixtures/markdown_test_cases.json'), 'r')