Repository: executablebooks/markdown-it-py

Issue:
Finalise port of non-standard rules to python
These are just non-standard additional rules:

- [x] replacements (Simple typographic replacements)
- [x] smartquotes (Convert straight quotation marks to typographic ones)
- [ ] linkify (Replace link-like texts with link nodes.)

They are currently in the repo as their original `.js` files

Discussion:
codecov[bot]: # [Codecov](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=h1) Report
> Merging [#78](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=desc) (1f15019) into [master](https://codecov.io/gh/executablebooks/markdown-it-py/commit/a29c8bd14f72fbd02f6d95ef3ec4451d267053b4?el=desc) (a29c8bd) will **decrease** coverage by `0.62%`.
> The diff coverage is `73.21%`.

[![Impacted file tree graph](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/graphs/tree.svg?width=650&height=150&src=pr&token=7JsxEEV2w3)](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #78      +/-   ##
==========================================
- Coverage   95.53%   94.91%   -0.63%     
==========================================
  Files          75       78       +3     
  Lines        3740     3851     +111     
==========================================
+ Hits         3573     3655      +82     
- Misses        167      196      +29     
```

| Flag | Coverage Δ | |
|---|---|---|
| pytests | `94.91% <73.21%> (-0.63%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [markdown\_it/extra/\_\_init\_\_.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvZXh0cmEvX19pbml0X18ucHk=) | `0.00% <0.00%> (ø)` | |
| [markdown\_it/extra/linkify\_it.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvZXh0cmEvbGlua2lmeV9pdC5weQ==) | `0.00% <0.00%> (ø)` | |
| [markdown\_it/main.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvbWFpbi5weQ==) | `90.98% <60.00%> (-1.33%)` | :arrow_down: |
| [markdown\_it/rules\_core/linkify.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvcnVsZXNfY29yZS9saW5raWZ5LnB5) | `92.77% <92.77%> (ø)` | |
| [markdown\_it/parser\_core.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvcGFyc2VyX2NvcmUucHk=) | `100.00% <100.00%> (ø)` | |
| [markdown\_it/rules\_core/\_\_init\_\_.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvcnVsZXNfY29yZS9fX2luaXRfXy5weQ==) | `100.00% <100.00%> (ø)` | |
| [markdown\_it/common/utils.py](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78/diff?src=pr&el=tree#diff-bWFya2Rvd25faXQvY29tbW9uL3V0aWxzLnB5) | `68.60% <0.00%> (+1.16%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=footer). Last update [a29c8bd...1f15019](https://codecov.io/gh/executablebooks/markdown-it-py/pull/78?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

chrisjsewell: The final piece of the puzzle!
Thanks 🙏 I should be able to have a look later today

tsutsu3: Is this implementation good?

If it's OK, increase coverage
chrisjsewell: Well it passes all the markdown-it fixture tests. So that's always a good start 😀

@hukkinj1 might also want to have a look?
tsutsu3: > Nice! I would suggest a change in how the "linkify enabled but not installed" case is handled, that should result in overall a bit less code, and avoids giving `LinkifyIt.pretest` a secondary task.

Oh, thanks. :smile:
chrisjsewell: @hukkinj1 have you got any more notes on this?

Patch:
diff --git a/markdown_it/main.py b/markdown_it/main.py
index 826a815..5e8fed9 100644
--- a/markdown_it/main.py
+++ b/markdown_it/main.py
@@ -11,6 +11,11 @@ from .rules_core.state_core import StateCore
 from .renderer import RendererHTML
 from .utils import AttrDict
 
+try:
+    import linkify_it
+except ModuleNotFoundError:
+    linkify_it = None
+
 
 _PRESETS = AttrDict(
     {
@@ -41,8 +46,7 @@ class MarkdownIt:
         self.options = {}
         self.configure(config)
 
-        # var LinkifyIt    = require('linkify-it')
-        # self.linkify = LinkifyIt()  # TODO maybe see https://github.com/Suor/autolink
+        self.linkify = linkify_it.LinkifyIt() if linkify_it else None
 
     def __repr__(self):
         return f"{self.__class__.__module__}.{self.__class__.__name__}()"
diff --git a/markdown_it/parser_core.py b/markdown_it/parser_core.py
index 6422149..7e7ced2 100644
--- a/markdown_it/parser_core.py
+++ b/markdown_it/parser_core.py
@@ -8,14 +8,14 @@
 
 from .ruler import Ruler
 from .rules_core.state_core import StateCore
-from .rules_core import normalize, block, inline, replace, smartquotes
+from .rules_core import normalize, block, inline, replace, smartquotes, linkify
+
 
-# TODO linkify
 _rules = [
     ["normalize", normalize],
     ["block", block],
     ["inline", inline],
-    #   [ 'linkify',        require('./rules_core/linkify')        ],
+    ["linkify", linkify],
     ["replacements", replace],
     ["smartquotes", smartquotes],
 ]
diff --git a/markdown_it/rules_core/__init__.py b/markdown_it/rules_core/__init__.py
index 401e527..16d796c 100644
--- a/markdown_it/rules_core/__init__.py
+++ b/markdown_it/rules_core/__init__.py
@@ -4,3 +4,4 @@ from .block import block  # noqa: F401
 from .inline import inline  # noqa: F401
 from .replacements import replace  # noqa: F401
 from .smartquotes import smartquotes  # noqa: F401
+from .linkify import linkify  # noqa: F401
diff --git a/markdown_it/rules_core/linkify.js b/markdown_it/rules_core/linkify.js
deleted file mode 100644
index 7c3ffc8..0000000
--- a/markdown_it/rules_core/linkify.js
+++ /dev/null
@@ -1,133 +0,0 @@
-// Replace link-like texts with link nodes.
-//
-// Currently restricted by `md.validateLink()` to http/https/ftp
-//
-'use strict';
-
-
-var arrayReplaceAt = require('../common/utils').arrayReplaceAt;
-
-
-function isLinkOpen(str) {
-  return /^<a[>\s]/i.test(str);
-}
-function isLinkClose(str) {
-  return /^<\/a\s*>/i.test(str);
-}
-
-
-module.exports = function linkify(state) {
-  var i, j, l, tokens, token, currentToken, nodes, ln, text, pos, lastPos,
-      level, htmlLinkLevel, url, fullUrl, urlText,
-      blockTokens = state.tokens,
-      links;
-
-  if (!state.md.options.linkify) { return; }
-
-  for (j = 0, l = blockTokens.length; j < l; j++) {
-    if (blockTokens[j].type !== 'inline' ||
-        !state.md.linkify.pretest(blockTokens[j].content)) {
-      continue;
-    }
-
-    tokens = blockTokens[j].children;
-
-    htmlLinkLevel = 0;
-
-    // We scan from the end, to keep position when new tags added.
-    // Use reversed logic in links start/end match
-    for (i = tokens.length - 1; i >= 0; i--) {
-      currentToken = tokens[i];
-
-      // Skip content of markdown links
-      if (currentToken.type === 'link_close') {
-        i--;
-        while (tokens[i].level !== currentToken.level && tokens[i].type !== 'link_open') {
-          i--;
-        }
-        continue;
-      }
-
-      // Skip content of html tag links
-      if (currentToken.type === 'html_inline') {
-        if (isLinkOpen(currentToken.content) && htmlLinkLevel > 0) {
-          htmlLinkLevel--;
-        }
-        if (isLinkClose(currentToken.content)) {
-          htmlLinkLevel++;
-        }
-      }
-      if (htmlLinkLevel > 0) { continue; }
-
-      if (currentToken.type === 'text' && state.md.linkify.test(currentToken.content)) {
-
-        text = currentToken.content;
-        links = state.md.linkify.match(text);
-
-        // Now split string to nodes
-        nodes = [];
-        level = currentToken.level;
-        lastPos = 0;
-
-        for (ln = 0; ln < links.length; ln++) {
-
-          url = links[ln].url;
-          fullUrl = state.md.normalizeLink(url);
-          if (!state.md.validateLink(fullUrl)) { continue; }
-
-          urlText = links[ln].text;
-
-          // Linkifier might send raw hostnames like "example.com", where url
-          // starts with domain name. So we prepend http:// in those cases,
-          // and remove it afterwards.
-          //
-          if (!links[ln].schema) {
-            urlText = state.md.normalizeLinkText('http://' + urlText).replace(/^http:\/\//, '');
-          } else if (links[ln].schema === 'mailto:' && !/^mailto:/i.test(urlText)) {
-            urlText = state.md.normalizeLinkText('mailto:' + urlText).replace(/^mailto:/, '');
-          } else {
-            urlText = state.md.normalizeLinkText(urlText);
-          }
-
-          pos = links[ln].index;
-
-          if (pos > lastPos) {
-            token         = new state.Token('text', '', 0);
-            token.content = text.slice(lastPos, pos);
-            token.level   = level;
-            nodes.push(token);
-          }
-
-          token         = new state.Token('link_open', 'a', 1);
-          token.attrs   = [ [ 'href', fullUrl ] ];
-          token.level   = level++;
-          token.markup  = 'linkify';
-          token.info    = 'auto';
-          nodes.push(token);
-
-          token         = new state.Token('text', '', 0);
-          token.content = urlText;
-          token.level   = level;
-          nodes.push(token);
-
-          token         = new state.Token('link_close', 'a', -1);
-          token.level   = --level;
-          token.markup  = 'linkify';
-          token.info    = 'auto';
-          nodes.push(token);
-
-          lastPos = links[ln].lastIndex;
-        }
-        if (lastPos < text.length) {
-          token         = new state.Token('text', '', 0);
-          token.content = text.slice(lastPos);
-          token.level   = level;
-          nodes.push(token);
-        }
-
-        // replace current node
-        blockTokens[j].children = tokens = arrayReplaceAt(tokens, i, nodes);
-      }
-    }
-  }
-};
diff --git a/markdown_it/rules_core/linkify.py b/markdown_it/rules_core/linkify.py
new file mode 100644
index 0000000..83091d3
--- /dev/null
+++ b/markdown_it/rules_core/linkify.py
@@ -0,0 +1,141 @@
+import re
+
+from ..common.utils import arrayReplaceAt
+from ..common.normalize_url import normalizeLinkText, normalizeLink, validateLink
+from .state_core import StateCore
+from ..token import Token
+
+
+LINK_OPEN_RE = re.compile(r"^<a[>\s]", flags=re.IGNORECASE)
+LINK_CLOSE_RE = re.compile(r"^</a\s*>", flags=re.IGNORECASE)
+
+HTTP_RE = re.compile(r"^http://")
+MAILTO_RE = re.compile(r"^mailto:")
+TEST_MAILTO_RE = re.compile(r"^mailto:", flags=re.IGNORECASE)
+
+
+def isLinkOpen(string: str):
+    return LINK_OPEN_RE.search(string)
+
+
+def isLinkClose(string: str):
+    return LINK_CLOSE_RE.search(string)
+
+
+def linkify(state: StateCore):
+    blockTokens = state.tokens
+
+    if not state.md.options.linkify:
+        return
+
+    if not state.md.linkify:
+        raise ModuleNotFoundError("Linkify enabled but not installed.")
+
+    for j in range(len(blockTokens)):
+        if blockTokens[j].type != "inline" or not state.md.linkify.pretest(
+            blockTokens[j].content
+        ):
+            continue
+
+        tokens = blockTokens[j].children
+
+        htmlLinkLevel = 0
+
+        # We scan from the end, to keep position when new tags added.
+        # Use reversed logic in links start/end match
+        assert tokens is not None
+        i = len(tokens)
+        while i >= 1:
+            i -= 1
+            assert isinstance(tokens, list)
+            currentToken = tokens[i]
+
+            # Skip content of markdown links
+            if currentToken.type == "link_close":
+                i -= 1
+                while (
+                    tokens[i].level != currentToken.level
+                    and tokens[i].type != "link_open"
+                ):
+                    i -= 1
+                continue
+
+            # Skip content of html tag links
+            if currentToken.type == "html_inline":
+                if isLinkOpen(currentToken.content) and htmlLinkLevel > 0:
+                    htmlLinkLevel -= 1
+                if isLinkClose(currentToken.content):
+                    htmlLinkLevel += 1
+            if htmlLinkLevel > 0:
+                continue
+
+            if currentToken.type == "text" and state.md.linkify.test(
+                currentToken.content
+            ):
+                text = currentToken.content
+                links = state.md.linkify.match(text)
+
+                # Now split string to nodes
+                nodes = []
+                level = currentToken.level
+                lastPos = 0
+
+                for ln in range(len(links)):
+                    url = links[ln].url
+                    fullUrl = normalizeLink(url)
+                    if not validateLink(fullUrl):
+                        continue
+
+                    urlText = links[ln].text
+
+                    # Linkifier might send raw hostnames like "example.com", where url
+                    # starts with domain name. So we prepend http:// in those cases,
+                    # and remove it afterwards.
+                    if not links[ln].schema:
+                        urlText = HTTP_RE.sub(
+                            "", normalizeLinkText("http://" + urlText)
+                        )
+                    elif links[ln].schema == "mailto:" and TEST_MAILTO_RE.search(
+                        urlText
+                    ):
+                        urlText = MAILTO_RE.sub(
+                            "", normalizeLinkText("mailto:" + urlText)
+                        )
+                    else:
+                        urlText = normalizeLinkText(urlText)
+
+                    pos = links[ln].index
+
+                    if pos > lastPos:
+                        token = Token("text", "", 0)
+                        token.content = text[lastPos:pos]
+                        token.level = level
+                        nodes.append(token)
+
+                    token = Token("link_open", "a", 1)
+                    token.attrs = [["href", fullUrl]]
+                    token.level = level + 1
+                    token.markup = "linkify"
+                    token.info = "auto"
+                    nodes.append(token)
+
+                    token = Token("text", "", 0)
+                    token.content = urlText
+                    token.level = level
+                    nodes.append(token)
+
+                    token = Token("link_close", "a", -1)
+                    token.level = level - 1
+                    token.markup = "linkify"
+                    token.info = "auto"
+                    nodes.append(token)
+
+                    lastPos = links[ln].last_index
+
+                if lastPos < len(text):
+                    token = Token("text", "", 0)
+                    token.content = text[lastPos:]
+                    token.level = level
+                    nodes.append(token)
+
+                blockTokens[j].children = tokens = arrayReplaceAt(tokens, i, nodes)
diff --git a/setup.py b/setup.py
index 1cadd2b..48e5fa7 100644
--- a/setup.py
+++ b/setup.py
@@ -67,6 +67,7 @@ setup(
             "mistletoe-ebp~=0.10.0",
             "panflute~=1.12",
         ],
+        "linkify": ["linkify-it-py~=1.0"],
     },
     zip_safe=False,
 )


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index ea60c77..9ce9069 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -40,7 +40,7 @@ jobs:
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
-        pip install .[testing]
+        pip install .[testing,linkify]
     - name: Run pytest
       run: |
         pytest --cov=markdown_it --cov-report=xml --cov-report=term-missing
diff --git a/tests/test_api/test_main.py b/tests/test_api/test_main.py
index dd0891b..ad220cd 100644
--- a/tests/test_api/test_main.py
+++ b/tests/test_api/test_main.py
@@ -8,7 +8,14 @@ def test_get_rules():
     md = MarkdownIt("zero")
     # print(md.get_all_rules())
     assert md.get_all_rules() == {
-        "core": ["normalize", "block", "inline", "replacements", "smartquotes"],
+        "core": [
+            "normalize",
+            "block",
+            "inline",
+            "linkify",
+            "replacements",
+            "smartquotes",
+        ],
         "block": [
             "table",
             "code",
diff --git a/tests/test_port/fixtures/linkify.md b/tests/test_port/fixtures/linkify.md
index 1e86cac..9edb78f 100644
--- a/tests/test_port/fixtures/linkify.md
+++ b/tests/test_port/fixtures/linkify.md
@@ -55,3 +55,45 @@ http://example.com/(c)
 .
 <p><a href="http://example.com/(c)">http://example.com/(c)</a></p>
 .
+
+before line
+.
+before
+github.com
+.
+<p>before
+<a href="http://github.com">github.com</a></p>
+.
+
+after line
+.
+github.com
+after
+.
+<p><a href="http://github.com">github.com</a>
+after</p>
+.
+
+before after lines
+.
+before
+github.com
+after
+.
+<p>before
+<a href="http://github.com">github.com</a>
+after</p>
+.
+
+before after lines with blank line
+.
+before
+
+github.com
+
+after
+.
+<p>before</p>
+<p><a href="http://github.com">github.com</a></p>
+<p>after</p>
+.
\ No newline at end of file
diff --git a/tests/test_port/test_fixtures.py b/tests/test_port/test_fixtures.py
index 0475f20..85d9f8c 100644
--- a/tests/test_port/test_fixtures.py
+++ b/tests/test_port/test_fixtures.py
@@ -8,6 +8,22 @@ from markdown_it.utils import read_fixture_file
 FIXTURE_PATH = Path(__file__).parent.joinpath("fixtures")
 
 
+@pytest.mark.parametrize(
+    "line,title,input,expected",
+    read_fixture_file(FIXTURE_PATH.joinpath("linkify.md")),
+)
+def test_linkify(line, title, input, expected):
+    md = MarkdownIt().enable("linkify")
+    md.options["linkify"] = True
+    text = md.render(input)
+    assert text.rstrip() == expected.rstrip()
+
+    # if not install linkify-it-py
+    md.linkify = None
+    with pytest.raises(ModuleNotFoundError):
+        md.render(input)
+
+
 @pytest.mark.parametrize(
     "line,title,input,expected",
     read_fixture_file(FIXTURE_PATH.joinpath("smartquotes.md")),