Difference Between NFD, NFC, NFKD, and NFKC Explained with Python Code | by Xu LIANG | Towards Data Science

Recently I am working on an NLP task in Japanese, one problem is to convert special characters to a normalized form. So I have done a little research and write this post for anyone who has the same need.
Japanese contains different forms of the character, for example, Latin has two forms, full-width form, and half-width.
In the above example, we can see the full-width form is very ugly and is also hard to utilizing for the following processing. So we need to convert it to a normalized form.
Use NFKC method.
>>> from unicodedata import normalize>>> s = "株式会社KADOKAWA Future Publishing">>> normalize('NFKC', s)株式会社KADOKAWA Future Publishing
There are 4 kinds of Unicode normalization forms. This article give a very detailed explanation. But I will explain the difference with a simple and easy understanding way.
First, we could see the below result for an intuitive understanding.
アイウエオ ==(NFC)==> アイウエオアイウエオ ==(NFD)==> アイウエオアイウエオ ==(NFKC)==> アイウエオアイウエオ ==(NFKD)==> アイウエオパピプペポ ==(NFC)==> パピプペポパピプペポ ==(NFD)==> パピプペポパピプペポ ==(NFKC)==> パピプペポパピプペポ ==(NFKD)==> パピプペポパピプペポ ==(NFC)==> パピプペポパピプペポ ==(NFD)==> パピプペポパピプペポ ==(NFKC)==> パピプペポパピプペポ ==(NFKD)==> パピプペポabcABC ==(NFC)==> abcABCabcABC ==(NFD)==> abcABCabcABC ==(NFKC)==> abcABCabcABC ==(NFKD)==> abcABC123 ==(NFC)==> 123123 ==(NFD)==> 123123 ==(NFKC)==> 123123 ==(NFKD)==> 123+-.~)} ==(NFC)==> +-.~)}+-.~)} ==(NFD)==> +-.~)}+-.~)} ==(NFKC)==> +-.~)}+-.~)} ==(NFKD)==> +-.~)}
There are two classification methods for these 4 forms.
# 1 original form changed or not- A(not changed): NFC & NFD- B(changed): NFKC & NFKD# 2 the length of original length changed or not- A(not changed): NFC & NFKC- B(changed): NFD & NFKD
abcABC ==(NFC)==> abcABCabcABC ==(NFD)==> abcABCabcABC ==(NFKC)==> abcABCabcABC ==(NFKD)==> abcABC# 1 original form changed or not- A(not changed): NFC & NFD- B(changed): NFKC & NFKD
The first classification method is based on whether the original form is changed or not. More specifically, A group does not contain K but B group contains K. What does K means?
D = Decomposition C = CompositionK = Compatibility
K means compatibility, which is used to distinguish with the original form. Because K changes the original form, so the length is also changed.
>>> s= '...'>>> normalize('NFKC', s)'...'>>> len(s)1>>> len(normalize('NFC', s))1>>> len(normalize('NFKC', s))3>>> len(normalize('NFD', s))1>>> len(normalize('NFKD', s))3
パピプペポ ==(NFC)==> パピプペポパピプペポ ==(NFD)==> パピプペポパピプペポ ==(NFKC)==> パピプペポパピプペポ ==(NFKD)==> パピプペポ# 2 the length of original length changed or not- A(not changed): NFC & NFKC- B(changed): NFD & NFKD
This second classification method is based on whether the length of the original form is changed or not. A group contains C(Composition), which won’t change the length. B group contains D(Decomposition), which will change the length.
You might be wondering why the length is change? Please see the test below.
>>> from unicodedata import normalize>>> s = "パピプペポ">>> len(s)5>>> len(normalize('NFC', s))5>>> len(normalize('NFKC', s))5>>> len(normalize('NFD', s))10>>> len(normalize('NFKD', s))10
We can find the “decomposition” method doubles the length.
This is because the NFD & NFKD decompose each Unicode character into two Unicode characters. For example, ポ(U+30DD) = ホ(U+30DB) + Dot(U+309A) . So the length change from 5 to 10. NFC & NFKC compose separated Unicode characters together, so the length is not changed.
You can use the unicodedata library to get different forms.
>>> from unicodedata import normalize>>> s = "パピプペポ">>> len(s)5>>> len(normalize('NFC', s))5>>> len(normalize('NFKC', s))5>>> len(normalize('NFD', s))10>>> len(normalize('NFKD', s))10
Length
Usually, we can use either of NFKC or NFKD to get the normalized form. The length won’t make trouble only if your NLP task is length sensitive. I usually use the NFKC method.
Check out my other posts on Medium with a categorized view!GitHub: BrambleXuLinkedIn: Xu LiangBlog: BrambleXu