Repository: apache/airflow

Issue:
Unable to use nested lists in DAG markdown documentation
**Apache Airflow version**: 2.0.2

**What happened**:

Tried to use the following markdown as a `doc_md` string passed to a DAG

```markdown
- Example
    -  Nested List
```

It was rendered in the web UI as a single list with no nesting or indentation.

**What you expected to happen**:

I expected the list to display as a nested list with visible indentation.

**How to reproduce it**:

Try and pass a DAG a `doc_md` string of the above nested list.
I think the bug will affect any markdown that relies on meaningful indentation (tabs or spaces)

doc_md code block collapsing lines
**Apache Airflow version**: 2.0.0 - 2.1.0

**Kubernetes version**: N/A

**Environment**:

- **Cloud provider or hardware configuration**: Docker on MacOS (but also AWS ECS deployed)
- **OS** (e.g. from /etc/os-release): MacOS Big Sur 11.3.1
- **Kernel** (e.g. `uname -a`): Darwin Kernel Version 20.4.0
- **Install tools**:
- **Others**:

**What happened**:

When a code block is a part of a DAG's `doc_md`, it does not render correctly in the Web UI, but collapses all the lines into one line instead.

**What you expected to happen**:

The multi line code block be rendered with line breaks preserved.

**How to reproduce it**:
Create a DAG with `doc_md` containing a code block:

````python
from airflow import DAG

DOC_MD = """\
# Markdown code block

Inline `code` works well.

```
Code block
does not
respect
newlines
```
"""

dag = DAG(
    dag_id='test',
    doc_md=DOC_MD
)
````

The rendered documentation looks like this:
<img src="https://user-images.githubusercontent.com/11132999/119981579-19a70600-bfbe-11eb-8036-7d981ae1f232.png" width="50%"/>

**Anything else we need to know**: N/A




Discussion:
Thanks for opening your first issue here! Be sure to follow the issue template!

#14564 I suspect this pull request may be the reason since it appears to strip spaces and tabs from every line of the markdown. A minor issue, but as a lover of nested lists, would be great to get them working again 😄 
![airflow_issue2](https://user-images.githubusercontent.com/6248193/120771475-d31f5180-c551-11eb-97ea-feb73325e029.png)
![airflow_issue1](https://user-images.githubusercontent.com/6248193/120771481-d581ab80-c551-11eb-879e-c4c7c2c168be.png)

This also affects code blocks, which is probably more important than nested lists, but it shows that markdown docstrings are quite limited.

[textwrap.dedent](https://docs.python.org/3/library/textwrap.html#textwrap.dedent) could well be part of the solution here.
I will try and reproduce this.
The cause is the same as https://github.com/apache/airflow/issues/16138#issuecomment-851688673. Instead of `str.lstrip()` each line, the parser need to dedent the lines more smartly to preserve internal indentation.
`textwrap.dedent` would do the job here wouldn't it?
I think so (mentioned in the linked comment), but someone needs to actually try.
Thanks for opening your first issue here! Be sure to follow the issue template!

I will look into this @kaxil 
@sweco to create a code block, you have to leave a blank line between each line. See [PEP 257](https://www.python.org/dev/peps/pep-0257/) and [Markdown guidelines](https://www.markdownguide.org/basic-syntax/) for more details.

This is how I did it:

```python
from airflow import DAG

DOC_MD = """\
# Markdown code block

Inline `code` works well.

    Code block

    does 

    respect

    newlines

"""

dag = DAG(
    dag_id='md-doc',
    doc_md=DOC_MD
)

```
![md-doc-2](https://user-images.githubusercontent.com/9834450/120181457-63396000-c20d-11eb-9818-25e050804ab2.png)

```python
from airflow import DAG

DOC_MD = """\
# Markdown code block

Inline `code` works well.

    `Code block`

   `does`

    `respect`

    `newlines`

"""

dag = DAG(
    dag_id='md-doc',
    doc_md=DOC_MD
)

```
![md-doc](https://user-images.githubusercontent.com/9834450/120181913-fa9eb300-c20d-11eb-868f-e710501cd264.png)

@kaxil consider closing this issue.
Hey @Dr-Denzy, thank you for looking into this, but I believe that what you managed to produce are not multi-line code blocks but rather:

- In the first case just regular paragraphs
- In the second case, regular paragraphs each with inline (one-line) code

If I'm right, these are not code blocks. According to the Markdown Guide, you can create a code block in two ways: [classic code block](https://www.markdownguide.org/basic-syntax/#code-blocks) and [fenced code block](https://www.markdownguide.org/extended-syntax/#fenced-code-blocks).

### Classic code block

A classic code block is created by indenting a block by 4 spaces (Bitbucket requires this to be 8 spaces), but it does not seem to work with either 4 or 8 spaces.

```python
DOC_MC = """\
# Markdown code block

Inline `code` works well.

    Code block
    does not
    respect
    newlines

"""
```

Normally this would look as in the Markdown guide, but Airflow just creates a normal paragraph.

<img width="40%" src="https://user-images.githubusercontent.com/11132999/120184077-d7293780-c210-11eb-9505-e43a9c8a64db.png" />

### Fenced code block
Created by delimiting the code block by three backticks (that's what I used in the first example):

````python
DOC_MC = """\
# Markdown code block

Inline `code` works well.

```
Code block
does not
respect
newlines
```
"""
````

This creates a `code` element in the HTML DOM but it joins the lines:
<img width="40%" src="https://user-images.githubusercontent.com/11132999/120184471-50288f00-c211-11eb-86b2-92bc53dcf076.png" />
<img width="40%" src="https://user-images.githubusercontent.com/11132999/120184765-a39add00-c211-11eb-8c9a-e63b818842d8.png" />
The cause is [the Markdown-HTML rendering function](https://github.com/apache/airflow/blob/9c8391a13f6ba29749675cf23f2f874f96b0cc8c/airflow/www/utils.py#L343-L350) uses `lstrip()`, so all leading indentations are just gone. It should be modified to use a more sophisticated method, such as `textwrap.dedent()` (I didn’t think entirely though whether changing to the function would just work).

Resolving patch:
diff --git a/airflow/www/utils.py b/airflow/www/utils.py
--- a/airflow/www/utils.py
+++ b/airflow/www/utils.py
@@ -16,6 +16,7 @@
 # specific language governing permissions and limitations
 # under the License.
 import json
+import textwrap
 import time
 from urllib.parse import urlencode
 
@@ -344,9 +345,7 @@ def wrapped_markdown(s, css_class='rich_doc'):
     """Convert a Markdown string to HTML."""
     if s is None:
         return None
-
-    s = '\n'.join(line.lstrip() for line in s.split('\n'))
-
+    s = textwrap.dedent(s)
     return Markup(f'<div class="{css_class}" >' + markdown.markdown(s, extensions=['tables']) + "</div>")
 
 


Test patch:
diff --git a/tests/www/test_utils.py b/tests/www/test_utils.py
--- a/tests/www/test_utils.py
+++ b/tests/www/test_utils.py
@@ -218,3 +218,39 @@ def test_wrapped_markdown_with_indented_lines(self):
         )
 
         assert '<div class="rich_doc" ><h1>header</h1>\n<p>1st line\n2nd line</p></div>' == rendered
+
+    def test_wrapped_markdown_with_raw_code_block(self):
+        rendered = wrapped_markdown(
+            """\
+            # Markdown code block
+
+            Inline `code` works well.
+
+                Code block
+                does not
+                respect
+                newlines
+
+            """
+        )
+
+        assert (
+            '<div class="rich_doc" ><h1>Markdown code block</h1>\n'
+            '<p>Inline <code>code</code> works well.</p>\n'
+            '<pre><code>Code block\ndoes not\nrespect\nnewlines\n</code></pre></div>'
+        ) == rendered
+
+    def test_wrapped_markdown_with_nested_list(self):
+        rendered = wrapped_markdown(
+            """
+            ### Docstring with a code block
+
+            - And
+                - A nested list
+            """
+        )
+
+        assert (
+            '<div class="rich_doc" ><h3>Docstring with a code block</h3>\n'
+            '<ul>\n<li>And<ul>\n<li>A nested list</li>\n</ul>\n</li>\n</ul></div>'
+        ) == rendered