Repository: Python-Markdown/markdown (Python)

Issue:
TOC:Anchor link written in Japanese does not work
I'm using the extension TOC with slugify_unicode for Japanese.  
And I'm using anchor links.  
In some cases, Japanese anchor link does not work.  
That is when Japanese characters contains dakuon(for example 'ba') 
or handakuon(for example 'pa').   
I think this is because the characters in the generated ID 
and the characters in the header are different. 

Sample Markdown:

```
[TOC]
[anchor link to プログラム](#プログラム)  
[anchor link to ぷろぐらむ](#ぷろぐらむ)  

##プログラム

##ぷろぐらむ
```

Generated html:
```
<div class="toc">
<ul>
<li><a href="#フロクラム">プログラム</a></li>
<li><a href="#ふろくらむ">ぷろぐらむ</a></li>
</ul>
</div>
<p><a href="#プログラム">anker link to プログラム</a><br />
<a href="#ぷろぐらむ">anker link to ぷろぐらむ</a>  </p>
<h2 id="フロクラム">プログラム</h2>
<h2 id="ふろくらむ">ぷろぐらむ</h2>

```

The result I expect is:
```
<div class="toc">
<ul>
<li><a href="#プログラム">プログラム</a></li>
<li><a href="#ぷろぐらむ">ぷろぐらむ</a></li>
</ul>
</div>
<p><a href="#プログラム">anker link to プログラム</a><br />
<a href="#ぷろぐらむ">anker link to ぷろぐらむ</a>  </p>
<h2 id="プログラム">プログラム</h2>
<h2 id="ぷろぐらむ">ぷろぐらむ</h2>
```
As far as I can tell, this depends on how the unicodedata.normalize() 
method arguments are used.   
In other words, I think we need to change the first argument 
from "NFKD" to "NFKC".  

Reference: [Difference Between NFD, NFC, NFKD, and NFKC Explained with Python Code | by Xu LIANG | Towards Data Science][reference1]  

I'm not familiar with unicode.  
And this is my first post.
Please investigate.  

[reference1]:https://towardsdatascience.com/difference-between-nfd-nfc-nfkd-and-nfkc-explained-with-python-code-e2631f96ae6c




Discussion:
First of all, I'm not familiar with Japanese to know what is correct. However, even in English there is occasionally a change in a character when generating a slug. So, yes, you sometimes need to account for that when manually creating your own links to those generated slugs.

For all I know, converting `プ` to `フ` in a slug might be the correct behavior. Or maybe is is clearly not correct. I have no way of knowing personally. But if we change to a different method, could that cause a similar error in another language? I'm not sure. 

We need to be very careful here about what changes we make. For example, if we make a change will that cause pre-existing links to slugs in existing document to now be wrong when those documents are re-rendered using the new version of Markdown?
Some historical investigation. I think the initial purpose of `unicodedata.normalize('NFKD', value).encode('ascii', 'ignore')` was converting Extended Latin characters to ASCII, e.g. `café` to `cafe`.

#970 added support for Unicode IDs, but it left the first part of that code unchanged, so the normalization happens also when `sligify_unicode` is in use.

Interesting that that PR adds a test for Japanese! It checks that `ダ` gets changed to `タ`, so probably the author of #970 @Hsn723 thought that it's the expected behavior.

If we just replace NFKD with NFKC in all cases, it will break the old behavior of ASCII-fying Extended Latin. But I *think* it should be safe to do it (or disable normalization at all) when `slugify_unicode` is in use. After all it's a relatively new option, so changing it won't hurt many users.
@mitya57 thanks for doing the research on this. I think your proposal makes sense. If I passed `café` into `slugify_unicode` I would expect to get back `café`, not `cafe` Therefore disabling normalization all together makes sense to me. 

We provide two options (1) normalize to ASCII and (2) preserve Unicode as-is (only normalizing whitespace). If users want other behavior, they can provide their own function or use some third-party provided function.
> @mitya57 thanks for doing the research on this. I think your proposal makes sense. If I passed `café` into `slugify_unicode` I would expect to get back `café`, not `cafe`

It already works like that. To replace `é` with `e`, one needs both NFKD normalization *and* the `.encode('ascii', 'ignore')` thing: the first part will decompose U+00E9 `é` into two characters (U+0065 `e` and U+0301 COMBINING ACUTE ACCENT), and the second part will remove the second character because it can't be encoded in ASCII. If you use `slugify_unicode`, the second part won't happen.

What I meant is that I see why normalization is needed with ASCII slugs: it is an essential part of making them ASCII. But I don't see a need in normalization when we use unicode slugs.
Thanks for reminding me how the normalization works (I forget those details sometimes). In any event, I agree, we don't need normalization for unicode slugs.



Patch:
diff --git a/docs/change_log/index.md b/docs/change_log/index.md
index d7487a324..aed19e3f0 100644
--- a/docs/change_log/index.md
+++ b/docs/change_log/index.md
@@ -3,6 +3,10 @@ title: Change Log
 Python-Markdown Change Log
 =========================
 
+Under development: version 3.3.5 (a bug-fix release).
+
+* Make the `slugify_unicode` function not remove diacritical marks (#1118).
+
 Feb 24, 2021: version 3.3.4 (a bug-fix release).
 
 * Properly parse unclosed tags in code spans (#1066).
diff --git a/markdown/extensions/toc.py b/markdown/extensions/toc.py
index d64ec1695..965ba4a83 100644
--- a/markdown/extensions/toc.py
+++ b/markdown/extensions/toc.py
@@ -23,16 +23,19 @@
 import xml.etree.ElementTree as etree
 
 
-def slugify(value, separator, encoding='ascii'):
+def slugify(value, separator, unicode=False):
     """ Slugify a string, to make it URL friendly. """
-    value = unicodedata.normalize('NFKD', value).encode(encoding, 'ignore')
-    value = re.sub(r'[^\w\s-]', '', value.decode(encoding)).strip().lower()
+    if not unicode:
+        # Replace Extended Latin characters with ASCII, i.e. žlutý → zluty
+        value = unicodedata.normalize('NFKD', value)
+        value = value.encode('ascii', 'ignore').decode('ascii')
+    value = re.sub(r'[^\w\s-]', '', value).strip().lower()
     return re.sub(r'[{}\s]+'.format(separator), separator, value)
 
 
 def slugify_unicode(value, separator):
     """ Slugify a string, to make it URL friendly while preserving Unicode characters. """
-    return slugify(value, separator, 'utf-8')
+    return slugify(value, separator, unicode=True)
 
 
 IDCOUNT_RE = re.compile(r'^(.*)_([0-9]+)$')


Test patch:
diff --git a/tests/test_syntax/extensions/test_toc.py b/tests/test_syntax/extensions/test_toc.py
index 04893e3e4..83c990fd0 100644
--- a/tests/test_syntax/extensions/test_toc.py
+++ b/tests/test_syntax/extensions/test_toc.py
@@ -534,9 +534,9 @@ def testPermalinkWithUnicodeInID(self):
         from markdown.extensions.toc import slugify_unicode
         self.assertMarkdownRenders(
             '# Unicode ヘッダー',
-            '<h1 id="unicode-ヘッター">'                                                            # noqa
+            '<h1 id="unicode-ヘッダー">'                                                            # noqa
                 'Unicode ヘッダー'                                                                  # noqa
-                '<a class="headerlink" href="#unicode-ヘッター" title="Permanent link">&para;</a>'  # noqa
+                '<a class="headerlink" href="#unicode-ヘッダー" title="Permanent link">&para;</a>'  # noqa
             '</h1>',                                                                               # noqa
             extensions=[TocExtension(permalink=True, slugify=slugify_unicode)]
         )
@@ -545,9 +545,19 @@ def testPermalinkWithUnicodeTitle(self):
         from markdown.extensions.toc import slugify_unicode
         self.assertMarkdownRenders(
             '# Unicode ヘッダー',
-            '<h1 id="unicode-ヘッター">'                                                        # noqa
+            '<h1 id="unicode-ヘッダー">'                                                        # noqa
                 'Unicode ヘッダー'                                                              # noqa
-                '<a class="headerlink" href="#unicode-ヘッター" title="パーマリンク">&para;</a>'  # noqa
+                '<a class="headerlink" href="#unicode-ヘッダー" title="パーマリンク">&para;</a>'  # noqa
             '</h1>',                                                                           # noqa
             extensions=[TocExtension(permalink=True, permalink_title="パーマリンク", slugify=slugify_unicode)]
         )
+
+    def testPermalinkWithExtendedLatinInID(self):
+        self.assertMarkdownRenders(
+            '# Théâtre',
+            '<h1 id="theatre">'                                                            # noqa
+                'Théâtre'                                                                  # noqa
+                '<a class="headerlink" href="#theatre" title="Permanent link">&para;</a>'  # noqa
+            '</h1>',                                                                       # noqa
+            extensions=[TocExtension(permalink=True)]
+        )