Repository: zulip/zulip

Issue:
Python-Markdown extensions registered with deprecated interface
Running tests with Python warnings enabled (e.g. `python3 -Wd tools/test-backend zerver.tests.test_auth_backends.GitHubAuthBackendTest.test_config_error_development`) shows these deprecation warnings:

```
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.fenced_code.FencedCodeExtension.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/api_arguments_table_generator.py:27: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  'generate_api_arguments', APIArgumentsTablePreprocessor(md, self.getConfigs()), '_begin'
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.api_arguments_table_generator.MarkdownArgumentsTableGenerator.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/api_return_values_table_generator.py:18: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  'generate_return_values', APIReturnValuesTablePreprocessor(md, self.getConfigs()), '_begin'
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.api_return_values_table_generator.MarkdownReturnValuesTableGenerator.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/nested_code_blocks.py:13: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  '_end'
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.nested_code_blocks.NestedCodeBlocksRenderer.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/tabbed_sections.py:83: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  'tabbed_sections', TabbedSectionsPreprocessor(md, self.getConfigs()), '_end')
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.tabbed_sections.TabbedSectionsGenerator.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/help_settings_links.py:68: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  md.preprocessors.add('setting', Setting(), '_begin')
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.help_settings_links.SettingHelpExtension.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/help_relative_links.py:72: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  md.preprocessors.add('help_relative_links', RelativeLinks(), '_begin')
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.help_relative_links.RelativeLinksHelpExtension.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/help_emoticon_translations_table.py:41: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  md.preprocessors.add('emoticon_translations', EmoticonTranslation(), '_end')
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.help_emoticon_translations_table.EmoticonTranslationsHelpExtension.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
/srv/zulip/zerver/lib/bugdown/include.py:18: DeprecationWarning: Using the add method to register a processor or pattern is deprecated. Use the `register` method instead.
  '_begin'
/srv/zulip-py3-venv/lib/python3.6/site-packages/markdown/core.py:124: DeprecationWarning: The 'md_globals' parameter of 'zerver.lib.bugdown.include.MarkdownIncludeCustom.extendMarkdown' is deprecated.
  ext._extendMarkdown(self)
```

See

https://python-markdown.github.io/change_log/release-3.0/#homegrown-ordereddict-has-been-replaced-with-a-purpose-built-registry
https://python-markdown.github.io/change_log/release-3.0/#md_globals-keyword-deprecated-from-extension-api



Discussion:
Hello @zulip/server-dependencies, @zulip/server-markdown members, this issue was labeled with the "area: dependencies", "area: markdown" labels, so you may want to check it out!

<!-- areaLabelAddition -->
@aero31aero can you look into these?
@zulipbot claim
Hello @aero31aero, you have been unassigned from this issue because you have not updated this issue or any referenced pull requests for over 14 days.

You can reclaim this issue or claim any other issue by commenting `@zulipbot claim` on that issue.

Thanks for your contributions, and hope to see you again soon!

Resolving patch:
diff --git a/version.py b/version.py
--- a/version.py
+++ b/version.py
@@ -44,4 +44,4 @@
 #   historical commits sharing the same major version, in which case a
 #   minor version bump suffices.
 
-PROVISION_VERSION = '111.3'
+PROVISION_VERSION = '112.1'
diff --git a/zerver/forms.py b/zerver/forms.py
--- a/zerver/forms.py
+++ b/zerver/forms.py
@@ -50,7 +50,7 @@
                             "Please contact your organization administrator to reactivate it."
 PASSWORD_TOO_WEAK_ERROR = "The password is too weak."
 AUTHENTICATION_RATE_LIMITED_ERROR = "You're making too many attempts to sign in. " + \
-                                    "Try again in %s seconds or contact your organization administrator " + \
+                                    "Try again in {} seconds or contact your organization administrator " + \
                                     "for help."
 
 def email_is_not_mit_mailing_list(email: str) -> None:
@@ -351,7 +351,7 @@ def clean(self) -> Dict[str, Any]:
                                                realm=realm, return_data=return_data)
             except RateLimited as e:
                 secs_to_freedom = int(float(str(e)))
-                raise ValidationError(AUTHENTICATION_RATE_LIMITED_ERROR % (secs_to_freedom,))
+                raise ValidationError(AUTHENTICATION_RATE_LIMITED_ERROR.format(secs_to_freedom))
 
             if return_data.get("inactive_realm"):
                 raise AssertionError("Programming error: inactive realm in authentication form")
diff --git a/zerver/lib/markdown/fenced_code.py b/zerver/lib/markdown/fenced_code.py
--- a/zerver/lib/markdown/fenced_code.py
+++ b/zerver/lib/markdown/fenced_code.py
@@ -391,7 +391,7 @@ def format_code(self, lang: str, text: str) -> str:
                                    lang=(lang or None),
                                    noclasses=self.codehilite_conf['noclasses'][0])
 
-            code = highliter.hilite()
+            code = highliter.hilite().rstrip('\n')
         else:
             code = CODE_WRAP.format(langclass, self._escape(text))
 


Test patch:
diff --git a/frontend_tests/node_tests/markdown.js b/frontend_tests/node_tests/markdown.js
--- a/frontend_tests/node_tests/markdown.js
+++ b/frontend_tests/node_tests/markdown.js
@@ -184,7 +184,7 @@ stream_data.add_sub(amp_stream);
 run_test("fenced_block_defaults", () => {
     const input = "\n```\nfenced code\n```\n\nand then after\n";
     const expected =
-        '\n\n<div class="codehilite"><pre><span></span><code>fenced code\n</code></pre></div>\n\n\n\nand then after\n\n';
+        '\n\n<div class="codehilite"><pre><span></span><code>fenced code\n</code></pre></div>\n\n\nand then after\n\n';
     const output = fenced_code.process_fenced_code(input);
     assert.equal(output, expected);
 });
@@ -294,13 +294,13 @@ run_test("marked", () => {
         {
             input: "\n```\nfenced code\n```\n\nand then after\n",
             expected:
-                '<div class="codehilite"><pre><span></span><code>fenced code\n</code></pre></div>\n\n\n<p>and then after</p>',
+                '<div class="codehilite"><pre><span></span><code>fenced code\n</code></pre></div>\n<p>and then after</p>',
         },
         {
             input:
                 "\n```\n    fenced code trailing whitespace            \n```\n\nand then after\n",
             expected:
-                '<div class="codehilite"><pre><span></span><code>    fenced code trailing whitespace\n</code></pre></div>\n\n\n<p>and then after</p>',
+                '<div class="codehilite"><pre><span></span><code>    fenced code trailing whitespace\n</code></pre></div>\n<p>and then after</p>',
         },
         {
             input: "* a\n* list \n* here",
@@ -309,12 +309,12 @@ run_test("marked", () => {
         {
             input: "\n```c#\nfenced code special\n```\n\nand then after\n",
             expected:
-                '<div class="codehilite" data-code-language="C#"><pre><span></span><code>fenced code special\n</code></pre></div>\n\n\n<p>and then after</p>',
+                '<div class="codehilite" data-code-language="C#"><pre><span></span><code>fenced code special\n</code></pre></div>\n<p>and then after</p>',
         },
         {
             input: "\n```vb.net\nfenced code dot\n```\n\nand then after\n",
             expected:
-                '<div class="codehilite" data-code-language="VB.net"><pre><span></span><code>fenced code dot\n</code></pre></div>\n\n\n<p>and then after</p>',
+                '<div class="codehilite" data-code-language="VB.net"><pre><span></span><code>fenced code dot\n</code></pre></div>\n<p>and then after</p>',
         },
         {
             input: "Some text first\n* a\n* list \n* here\n\nand then after",
diff --git a/zerver/tests/fixtures/markdown_test_cases.json b/zerver/tests/fixtures/markdown_test_cases.json
--- a/zerver/tests/fixtures/markdown_test_cases.json
+++ b/zerver/tests/fixtures/markdown_test_cases.json
@@ -23,7 +23,7 @@
     {
       "name": "ampampamp",
       "input": "& &amp; &amp;amp;\n~~~~\n& &amp; &amp;amp;\n~~~~\n    & &amp; &amp;amp;",
-      "expected_output": "<p>&amp; &amp; &amp;amp;</p>\n<div class=\"codehilite\"><pre><span></span><code>&amp; &amp;amp; &amp;amp;amp;\n</code></pre></div>\n\n\n<div class=\"codehilite\"><pre><span></span><code>&amp; &amp;amp; &amp;amp;amp;\n</code></pre></div>"
+      "expected_output": "<p>&amp; &amp; &amp;amp;</p>\n<div class=\"codehilite\"><pre><span></span><code>&amp; &amp;amp; &amp;amp;amp;\n</code></pre></div>\n<div class=\"codehilite\"><pre><span></span><code>&amp; &amp;amp; &amp;amp;amp;\n</code></pre></div>"
     },
     {
       "name": "basic_paragraph",
@@ -34,8 +34,8 @@
     {
       "name": "codeblock_multiline",
       "input": "Hamlet once said\n~~~~\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n~~~~\nAnd all was good.",
-      "expected_output": "<p>Hamlet once said</p>\n<div class=\"codehilite\"><pre><span></span><code>def func():\n    x = 1\n\n    y = 2\n\n    z = 3\n</code></pre></div>\n\n\n<p>And all was good.</p>",
-      "text_content": "Hamlet once said\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n\n\n\nAnd all was good."
+      "expected_output": "<p>Hamlet once said</p>\n<div class=\"codehilite\"><pre><span></span><code>def func():\n    x = 1\n\n    y = 2\n\n    z = 3\n</code></pre></div>\n<p>And all was good.</p>",
+      "text_content": "Hamlet once said\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n\nAnd all was good."
     },
     {
       "name": "test",
@@ -45,8 +45,8 @@
     {
       "name": "codeblock_trailing_whitespace",
       "input": "Hamlet once said\n~~~~\ndef func():\n    x = 1\n\n    y = 2\t\t\n\n    z = 3       \n~~~~\nAnd all was good.",
-      "expected_output": "<p>Hamlet once said</p>\n<div class=\"codehilite\"><pre><span></span><code>def func():\n    x = 1\n\n    y = 2\n\n    z = 3\n</code></pre></div>\n\n\n<p>And all was good.</p>",
-      "text_content": "Hamlet once said\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n\n\n\nAnd all was good."
+      "expected_output": "<p>Hamlet once said</p>\n<div class=\"codehilite\"><pre><span></span><code>def func():\n    x = 1\n\n    y = 2\n\n    z = 3\n</code></pre></div>\n<p>And all was good.</p>",
+      "text_content": "Hamlet once said\ndef func():\n    x = 1\n\n    y = 2\n\n    z = 3\n\nAnd all was good."
     },
     {
       "name": "inline_code_spaces",
@@ -63,14 +63,14 @@
     {
       "name": "codeblock_backticks",
       "input": "\n```\nfenced code\n```\n\n```inline code```\n",
-      "expected_output": "<div class=\"codehilite\"><pre><span></span><code>fenced code\n</code></pre></div>\n\n\n<p><code>inline code</code></p>",
-      "text_content": "fenced code\n\n\n\ninline code"
+      "expected_output": "<div class=\"codehilite\"><pre><span></span><code>fenced code\n</code></pre></div>\n<p><code>inline code</code></p>",
+      "text_content": "fenced code\n\ninline code"
     },
     {
       "name": "hanging_multi_codeblock",
       "input": "Hamlet said:\n~~~~\ndef speak(self):\n    x = 1\n# Comment to make this code block longer to test Trac #1162\n~~~~\n\nThen he mentioned ````y = 4 + x**2```` and\n~~~~\ndef foobar(self):\n    return self.baz()",
-      "expected_output": "<p>Hamlet said:</p>\n<div class=\"codehilite\"><pre><span></span><code>def speak(self):\n    x = 1\n# Comment to make this code block longer to test Trac #1162\n</code></pre></div>\n\n\n<p>Then he mentioned <code>y = 4 + x**2</code> and</p>\n<div class=\"codehilite\"><pre><span></span><code>def foobar(self):\n    return self.baz()\n</code></pre></div>",
-      "text_content": "Hamlet said:\ndef speak(self):\n    x = 1\n# Comment to make this code block longer to test Trac #1162\n\n\n\nThen he mentioned y = 4 + x**2 and\ndef foobar(self):\n    return self.baz()\n"
+      "expected_output": "<p>Hamlet said:</p>\n<div class=\"codehilite\"><pre><span></span><code>def speak(self):\n    x = 1\n# Comment to make this code block longer to test Trac #1162\n</code></pre></div>\n<p>Then he mentioned <code>y = 4 + x**2</code> and</p>\n<div class=\"codehilite\"><pre><span></span><code>def foobar(self):\n    return self.baz()\n</code></pre></div>",
+      "text_content": "Hamlet said:\ndef speak(self):\n    x = 1\n# Comment to make this code block longer to test Trac #1162\n\nThen he mentioned y = 4 + x**2 and\ndef foobar(self):\n    return self.baz()\n"
     },
     {
       "name": "fenced_quote",
@@ -87,7 +87,7 @@
     {
       "name": "complexly_nested_quote",
       "input": "I heard about this second hand...\n~~~ quote\n\nHe said:\n~~~ quote\nThe customer is complaining.\n\nThey looked at this code:\n``` \ndef hello(): print 'hello\n```\nThey would prefer:\n~~~\ndef hello()\n  puts 'hello'\nend\n~~~\n\nPlease advise.\n~~~\n\nShe said:\n~~~ quote\nJust send them this:\n```\necho \"hello\n\"\n```\n~~~",
-      "expected_output": "<p>I heard about this second hand...</p>\n<blockquote>\n<p>He said:</p>\n<blockquote>\n<p>The customer is complaining.</p>\n<p>They looked at this code:</p>\n<div class=\"codehilite\"><pre><span></span><code>def hello(): print &#39;hello\n</code></pre></div>\n\n\n<p>They would prefer:</p>\n</blockquote>\n<p>def hello()<br>\n  puts 'hello'<br>\nend</p>\n</blockquote>\n<p>Please advise.</p>\n<div class=\"codehilite\"><pre><span></span><code>She said:\n~~~ quote\nJust send them this:\n```\necho &quot;hello\n&quot;\n```\n</code></pre></div>",
+      "expected_output": "<p>I heard about this second hand...</p>\n<blockquote>\n<p>He said:</p>\n<blockquote>\n<p>The customer is complaining.</p>\n<p>They looked at this code:</p>\n<div class=\"codehilite\"><pre><span></span><code>def hello(): print &#39;hello\n</code></pre></div>\n<p>They would prefer:</p>\n</blockquote>\n<p>def hello()<br>\n  puts 'hello'<br>\nend</p>\n</blockquote>\n<p>Please advise.</p>\n<div class=\"codehilite\"><pre><span></span><code>She said:\n~~~ quote\nJust send them this:\n```\necho &quot;hello\n&quot;\n```\n</code></pre></div>",
       "text_content": "I heard about this second hand...\n> He said:\n> > The customer is complaining.\n> > They looked at this code:\n> > def hello(): print 'hello\n> > They would prefer:\n> def hello()\n>   puts 'hello'\n> end\n\nPlease advise.\nShe said:\n~~~ quote\nJust send them this:\n```\necho \"hello\n\"\n```\n"
     },
     {
@@ -860,39 +860,39 @@
     {
       "name": "spoilers_fenced_spoiler",
       "input": "```spoiler header\ncontent\n```\noutside spoiler\n",
-      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>content</p>\n</div></div>\n\n<p>outside spoiler</p>",
+      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>content</p>\n</div></div>\n<p>outside spoiler</p>",
       "text_content": "header (…)\noutside spoiler"
     },
     {
       "name": "spoilers_empty_header",
       "input": "```spoiler\ncontent\n```\noutside spoiler\n",
-      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>content</p>\n</div></div>\n\n<p>outside spoiler</p>",
+      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>content</p>\n</div></div>\n<p>outside spoiler</p>",
       "text_content": "(…)\noutside spoiler"
     },
     {
       "name": "spoilers_script_tags",
       "input": "```spoiler <script>alert(1)</script>\n<script>alert(1)</script>\n```",
-      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>\n</div></div>",
-      "marked_expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>&lt;script&gt;alert(1)&lt;/script&gt;\n\n</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>&lt;script&gt;alert(1)&lt;/script&gt;\n\n</p>\n</div></div>",
+      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>&lt;script&gt;alert(1)&lt;/script&gt;</p>\n</div></div>",
+      "marked_expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>&lt;script&gt;alert(1)&lt;/script&gt;\n\n</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>&lt;script&gt;alert(1)&lt;/script&gt;\n\n</p>\n</div></div>",
       "text_content": "<script>alert(1)</script> (…)\n"
     },
     {
       "name": "spoilers_block_quote",
       "input": "~~~quote\n```spoiler header\ncontent\n```\noutside spoiler\n~~~\noutside quote",
-      "expected_output": "<blockquote>\n<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>content</p>\n</div></div>\n\n<p>outside spoiler</p>\n</blockquote>\n<p>outside quote</p>",
+      "expected_output": "<blockquote>\n<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>content</p>\n</div></div>\n<p>outside spoiler</p>\n</blockquote>\n<p>outside quote</p>",
       "text_content": "> header (…)\n> outside spoiler\n\noutside quote"
     },
     {
       "name": "spoilers_with_header_markdown",
       "input": "```spoiler [Header](https://example.com) :smile:\ncontent\n```",
-      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p><a href=\"https://example.com\">Header</a> <span aria-label=\"smile\" class=\"emoji emoji-1f642\" role=\"img\" title=\"smile\">:smile:</span></p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>content</p>\n</div></div>",
+      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p><a href=\"https://example.com\">Header</a> <span aria-label=\"smile\" class=\"emoji emoji-1f642\" role=\"img\" title=\"smile\">:smile:</span></p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>content</p>\n</div></div>",
       "text_content": "Header 🙂 (…)\n"
     },
     {
       "name": "spoiler_with_inline_image",
       "input": "```spoiler header\nContent http://example.com/image.png\n```",
-      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>Content <a href=\"http://example.com/image.png\">http://example.com/image.png</a></p>\n<div class=\"message_inline_image\"><a href=\"http://example.com/image.png\"><img data-src-fullsize=\"/thumbnail?url=http%3A%2F%2Fexample.com%2Fimage.png&amp;size=full\" src=\"/thumbnail?url=http%3A%2F%2Fexample.com%2Fimage.png&amp;size=thumbnail\"></a></div></div></div>",
-      "marked_expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>Content <a href=\"http://example.com/image.png\">http://example.com/image.png</a></p>\n</div></div>",
+      "expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>Content <a href=\"http://example.com/image.png\">http://example.com/image.png</a></p>\n<div class=\"message_inline_image\"><a href=\"http://example.com/image.png\"><img data-src-fullsize=\"/thumbnail?url=http%3A%2F%2Fexample.com%2Fimage.png&amp;size=full\" src=\"/thumbnail?url=http%3A%2F%2Fexample.com%2Fimage.png&amp;size=thumbnail\"></a></div></div></div>",
+      "marked_expected_output": "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>header</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>Content <a href=\"http://example.com/image.png\">http://example.com/image.png</a></p>\n</div></div>",
       "text_content": "header (…)\n"
     }
   ],
diff --git a/zerver/tests/test_import_export.py b/zerver/tests/test_import_export.py
--- a/zerver/tests/test_import_export.py
+++ b/zerver/tests/test_import_export.py
@@ -990,7 +990,7 @@ def get_userpresence_timestamp(r: Realm) -> Set[Any]:
         original_msg = Message.objects.get(content=special_characters_message, sender__realm=original_realm)
         self.assertEqual(
             original_msg.rendered_content,
-            '<div class="codehilite"><pre><span></span><code>&#39;\n</code></pre></div>\n\n\n'
+            '<div class="codehilite"><pre><span></span><code>&#39;\n</code></pre></div>\n'
             f'<p><span class="user-mention" data-user-id="{orig_polonius_user.id}">@Polonius</span></p>',
         )
         imported_polonius_user = UserProfile.objects.get(delivery_email=self.example_email("polonius"),
diff --git a/zerver/tests/test_markdown.py b/zerver/tests/test_markdown.py
--- a/zerver/tests/test_markdown.py
+++ b/zerver/tests/test_markdown.py
@@ -725,7 +725,7 @@ def test_inline_youtube_preview(self) -> None:
         msg = """\n```spoiler Check out this Pycon Video\nhttps://www.youtube.com/watch?v=0c46YHS3RY8\n```"""
         converted = markdown_convert_wrapper(msg)
 
-        self.assertEqual(converted, '<div class="spoiler-block"><div class="spoiler-header">\n\n<p>Check out this Pycon Video</p>\n</div><div class="spoiler-content" aria-hidden="true">\n\n<p><a href="https://www.youtube.com/watch?v=0c46YHS3RY8">https://www.youtube.com/watch?v=0c46YHS3RY8</a></p>\n<div class="youtube-video message_inline_image"><a data-id="0c46YHS3RY8" href="https://www.youtube.com/watch?v=0c46YHS3RY8"><img src="https://i.ytimg.com/vi/0c46YHS3RY8/default.jpg"></a></div></div></div>')
+        self.assertEqual(converted, '<div class="spoiler-block"><div class="spoiler-header">\n<p>Check out this Pycon Video</p>\n</div><div class="spoiler-content" aria-hidden="true">\n<p><a href="https://www.youtube.com/watch?v=0c46YHS3RY8">https://www.youtube.com/watch?v=0c46YHS3RY8</a></p>\n<div class="youtube-video message_inline_image"><a data-id="0c46YHS3RY8" href="https://www.youtube.com/watch?v=0c46YHS3RY8"><img src="https://i.ytimg.com/vi/0c46YHS3RY8/default.jpg"></a></div></div></div>')
 
         # Test youtube urls in normal messages.
         msg = '[Youtube link](https://www.youtube.com/watch?v=0c46YHS3RY8)'
@@ -896,7 +896,7 @@ def make_inline_twitter_preview(url: str, tweet_html: str, image_html: str='') -
         msg = '```spoiler secret tweet\nTweet: http://twitter.com/wdaher/status/287977969287315456\n```'
         converted = markdown_convert_wrapper(msg)
 
-        rendered_spoiler = "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n\n<p>secret tweet</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n\n<p>Tweet: {}</p>\n{}</div></div>"
+        rendered_spoiler = "<div class=\"spoiler-block\"><div class=\"spoiler-header\">\n<p>secret tweet</p>\n</div><div class=\"spoiler-content\" aria-hidden=\"true\">\n<p>Tweet: {}</p>\n{}</div></div>"
         self.assertEqual(converted, rendered_spoiler.format(
             make_link('http://twitter.com/wdaher/status/287977969287315456'),
             make_inline_twitter_preview('http://twitter.com/wdaher/status/287977969287315456', normal_tweet_html)))
@@ -2134,8 +2134,6 @@ def test_html_entity_conversion(self) -> None:
             <div class="codehilite"><pre><span></span><code>&amp;copy;
             &amp;copy;
             </code></pre></div>
-
-
             <p>Test quote:</p>
             <blockquote>
             <p>&copy;</p>