Repository: mondeja/mkdocs-include-markdown-plugin (python)

Issue:
Markdown links inside code blocks are interpreted
I seem to have run into a problem when using `{% markdown-include %}` to include a markdown file that contains a code block, which in turn contains something that looks like a valid markdown link (but in fact it's a C++11 lambda definition).

Here's an example. I have a markdown file examples/lambda.md that contains a C++ code snippet, with C++11 lambdas:

`examples/lambda.md`

    This is an example of a C++11 lambda:

    ```cpp
    const auto lambda = []() { .... };
    ````

If I include this markdown file into another document using `{% include-markdown %}`, the `[]()` string in the code block gets interpreted as an empty link

`docs/main.md`

    Hello! Let's learn C++!

    {% include-markdown "../examples/lambda.md" %}

The output then looks like this:

---

Hello! Let's learn C++!

This is an example of a C++11 lambda:

```cpp
const auto lambda = [](../examples) { ... };
```
---

Notice the "../examples" string rendered in the lambda codeblock. I suspect the reason is that `[]()` is a markdown syntax for links, which somehow gets interpreted during the include process, even though it is inside of a code-block. If I change the code example to contain space between `[]` and `()`, then it gets rendered correctly (i.e. no extra strings get injected)

```
const auto lambda = [] () { ... };
```


Pull request:
Don't rewrite relative links inside codeblocks
Fixes #78

Patch:
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
index dd9f16a..0047463 100644
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 3.2.1
+current_version = 3.2.2
 
 [bumpversion:file:mkdocs_include_markdown_plugin/__init__.py]
 
diff --git a/mkdocs_include_markdown_plugin/__init__.py b/mkdocs_include_markdown_plugin/__init__.py
index 9896d6e..0028b88 100644
--- a/mkdocs_include_markdown_plugin/__init__.py
+++ b/mkdocs_include_markdown_plugin/__init__.py
@@ -1,2 +1,2 @@
 __title__ = 'mkdocs_include_markdown_plugin'
-__version__ = '3.2.1'
+__version__ = '3.2.2'
diff --git a/mkdocs_include_markdown_plugin/process.py b/mkdocs_include_markdown_plugin/process.py
index 0446cd9..432f15c 100644
--- a/mkdocs_include_markdown_plugin/process.py
+++ b/mkdocs_include_markdown_plugin/process.py
@@ -88,6 +88,103 @@ MARKDOWN_LINK_DEFINITION_REGEX = re.compile(
 )
 
 
+def transform_p_by_p_skipping_codeblocks(markdown, func):
+    '''Apply a transformation paragraph by paragraph in a Markdown text using
+    a function.
+
+    Skip indented and fenced codeblock lines, where the transformation never is
+    applied.
+    '''
+    _inside_fcodeblock = False            # inside fenced codeblock
+    _current_fcodeblock_delimiter = None  # current fenced codeblock delimiter
+    _inside_icodeblock = False            # inside indented codeblock
+
+    lines, current_paragraph = ([], '')
+
+    def process_current_paragraph_lines():
+        lines.extend(func(current_paragraph).splitlines(keepends=True))
+
+    for line in markdown.splitlines(keepends=True):
+        if not _inside_fcodeblock and not _inside_icodeblock:
+            lstripped_line = line.lstrip()
+            if any([
+                lstripped_line.startswith('```'),
+                lstripped_line.startswith('~~~'),
+            ]):
+                _inside_fcodeblock = True
+                _current_fcodeblock_delimiter = line[:3]
+                lines.append(line)
+                if current_paragraph:
+                    process_current_paragraph_lines()
+                    current_paragraph = ''
+            elif (
+                # 5 and 2 including newline character
+                (line.startswith('    ') and len(line) == 5) or
+                (line.startswith('\t') and len(line) == 2)
+            ):
+                _inside_icodeblock = True
+                lines.append(line)
+                if current_paragraph:
+                    process_current_paragraph_lines()
+                    current_paragraph = ''
+            else:
+                current_paragraph += line
+        else:
+            lines.append(line)
+            if _current_fcodeblock_delimiter:
+                if line.lstrip().startswith(_current_fcodeblock_delimiter):
+                    _inside_fcodeblock = False
+                    _current_fcodeblock_delimiter = None
+            else:
+                if not line.startswith('    '):
+                    _inside_icodeblock = False
+
+    process_current_paragraph_lines()
+
+    return ''.join(lines)
+
+
+def transform_line_by_line_skipping_codeblocks(markdown, func):
+    '''Apply a transformation line by line in a Markdown text using a function.
+
+    Skip indented and fenced codeblock lines, where the transformation never is
+    applied.
+    '''
+    _inside_fcodeblock = False            # inside fenced codeblock
+    _current_fcodeblock_delimiter = None  # current fenced codeblock delimiter
+    _inside_icodeblock = False            # inside indented codeblock
+
+    lines = []
+    for line in markdown.splitlines(keepends=True):
+        if not _inside_fcodeblock and not _inside_icodeblock:
+            lstripped_line = line.lstrip()
+            if any([
+                lstripped_line.startswith('```'),
+                lstripped_line.startswith('~~~'),
+            ]):
+                _inside_fcodeblock = True
+                _current_fcodeblock_delimiter = line[:3]
+            elif (
+                # 5 and 2 including newline character
+                (line.startswith('    ') and len(line) == 4) or
+                (line.startswith('\t') and len(line) == 1)
+            ):
+                _inside_icodeblock = True
+            else:
+                line = func(line)
+        else:
+            if _current_fcodeblock_delimiter:
+                if line.lstrip().startswith(_current_fcodeblock_delimiter):
+                    _inside_fcodeblock = False
+                    _current_fcodeblock_delimiter = None
+            else:
+                if not line.startswith('    '):
+                    _inside_icodeblock = False
+        lines.append(line)
+
+    return ''.join(lines)
+
+
 def rewrite_relative_urls(
     markdown: str, source_path: Path, destination_path: Path,
 ) -> str:
@@ -131,24 +228,27 @@ def rewrite_relative_urls(
             + m.string[href_end:match_end]
         )
 
-    markdown = re.sub(
-        MARKDOWN_LINK_REGEX,
-        partial(found_href, url_group_index=3),
-        markdown,
-    )
-    markdown = re.sub(
-        MARKDOWN_IMAGE_REGEX,
-        partial(found_href, url_group_index=3),
-        markdown,
-    )
-    markdown = re.sub(
-        MARKDOWN_LINK_DEFINITION_REGEX,
-        partial(found_href, url_group_index=2),
+    def transform(paragraph):
+        paragraph = re.sub(
+            MARKDOWN_LINK_REGEX,
+            partial(found_href, url_group_index=3),
+            paragraph,
+        )
+        paragraph = re.sub(
+            MARKDOWN_IMAGE_REGEX,
+            partial(found_href, url_group_index=3),
+            paragraph,
+        )
+        return re.sub(
+            MARKDOWN_LINK_DEFINITION_REGEX,
+            partial(found_href, url_group_index=2),
+            paragraph,
+        )
+    return transform_p_by_p_skipping_codeblocks(
         markdown,
+        transform,
     )
 
-    return markdown
-
 
 def interpret_escapes(value: str) -> str:
     '''Replaces any standard escape sequences in value with their usual
@@ -187,36 +287,10 @@ def filter_inclusions(new_start, new_end, text_to_include):
 
 def increase_headings_offset(markdown, offset=0):
     '''Increases the headings depth of a snippet of Makdown content.'''
-    _inside_fcodeblock = False            # inside fenced codeblock
-    _current_fcodeblock_delimiter = None  # current fenced codeblock delimiter
-    _inside_icodeblock = False            # inside indented codeblcok
-
-    lines = []
-    for line in markdown.splitlines(keepends=True):
-        lstripped_line = line.lstrip()
-
-        if not _inside_fcodeblock and not _inside_icodeblock:
-            if any([
-                lstripped_line.startswith('```'),
-                lstripped_line.startswith('~~~'),
-            ]):
-                _inside_fcodeblock = True
-                _current_fcodeblock_delimiter = line[:3]
-            elif line.startswith('    '):
-                _inside_icodeblock = True
-            elif line.startswith('#'):
-                line = '#' * offset + line
-        else:
-            if _current_fcodeblock_delimiter:
-                if lstripped_line.startswith(_current_fcodeblock_delimiter):
-                    _inside_fcodeblock = False
-                    _current_fcodeblock_delimiter = None
-            else:
-                if not line.startswith('    '):
-                    _inside_icodeblock = False
-        lines.append(line)
-
-    return ''.join(lines)
+    return transform_line_by_line_skipping_codeblocks(
+        markdown,
+        lambda line: ('#' * offset + line) if line.startswith('#') else line,
+    )
 
 
 def filter_paths(filepaths, ignore_paths=[]):
diff --git a/setup.cfg b/setup.cfg
index 7adacee..c22fd90 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,6 +1,6 @@
 [metadata]
 name = mkdocs_include_markdown_plugin
-version = 3.2.1
+version = 3.2.2
 description = Mkdocs Markdown includer plugin.
 long_description = file: README.md
 long_description_content_type = text/markdown


Test patch:
diff --git a/tests/test_process.py b/tests/test_process.py
index 01621f3..758db61 100644
--- a/tests/test_process.py
+++ b/tests/test_process.py
@@ -22,7 +22,7 @@ from mkdocs_include_markdown_plugin.process import (
             '''
         Here's a [link](../CHANGELOG.md) to the changelog.
 ''',
-            id='relative link',
+            id='relative-link',
         ),
         pytest.param(
             '''Here's a [link whose text is really long and so is broken across
@@ -33,7 +33,7 @@ multiple lines](CHANGELOG.md) to the changelog.
             '''Here's a [link whose text is really long and so is broken across
 multiple lines](../CHANGELOG.md) to the changelog.
 ''',
-            id='multiline link',
+            id='multiline-link',
         ),
         pytest.param(
             '''
@@ -44,7 +44,7 @@ Check [this link](foobar.md) for more information
             '''
 Check [this link](includes/feature_a/foobar.md) for more information
 ''',
-            id='relative link down',
+            id='relative-link-down',
         ),
         pytest.param(
             '''Here's a [link](CHANGELOG.md#v1.2.3) to the changelog.
@@ -53,7 +53,7 @@ Check [this link](includes/feature_a/foobar.md) for more information
             'docs/nav.md',
             '''Here's a [link](../CHANGELOG.md#v1.2.3) to the changelog.
 ''',
-            id='link with hash',
+            id='link-with-hash',
         ),
         pytest.param(
             '''Here's a [link][changelog] to the changelog.
@@ -66,7 +66,7 @@ Check [this link](includes/feature_a/foobar.md) for more information
 
 [changelog]: ../CHANGELOG.md
 ''',
-            id='link reference',
+            id='link-reference',
         ),
         pytest.param(
             '''Here's a diagram: ![diagram](assets/diagram.png)''',
@@ -80,7 +80,7 @@ Check [this link](includes/feature_a/foobar.md) for more information
             'README',
             'docs/home.md',
             '''Build status: [![Build Status](../badge.png)](../build/)''',
-            id='image inside link',
+            id='image-inside-link',
         ),
         pytest.param(
             '''[Homepage](/) [Github](https://github.com/user/repo)
@@ -89,14 +89,44 @@ Check [this link](includes/feature_a/foobar.md) for more information
             'docs/nav.md',
             '''[Homepage](/) [Github](https://github.com/user/repo)
 [Privacy policy](/privacy)''',
-            id='absolute urls',
+            id='absolute-urls',
         ),
         pytest.param(
             '''[contact us](mailto:hello@example.com)''',
             'README',
             'docs/nav.md',
             '''[contact us](mailto:hello@example.com)''',
-            id='mailto urls',
+            id='mailto-urls',
+        ),
+        pytest.param(
+            '''```cpp
+// Some code in which rewrites shouldn't be proccessed.
+// https://github.com/mondeja/mkdocs-include-markdown-plugin/issues/78
+const auto lambda = []() { .... };
+```
+''',
+            'README',
+            'examples/lambda.md',
+            '''```cpp
+// Some code in which rewrites shouldn't be proccessed.
+// https://github.com/mondeja/mkdocs-include-markdown-plugin/issues/78
+const auto lambda = []() { .... };
+```
+''',
+            id='cpp-likelink-fenced-codeblock',
+        ),
+        pytest.param(
+            '''\t
+\tconst auto lambda = []() { .... };
+
+''',
+            'README',
+            'examples/lambda.md',
+            '''\t
+\tconst auto lambda = []() { .... };
+
+''',
+            id='cpp-likelink-indented-codeblock',
         ),
     ),
 )