Chinese Natural Language (Pre)processing: An Introduction | by Sidney Kung | Towards Data Science

In general, when people think of Natural Language Processing (NLP), they tend to restrict it to English. This is due to the idea that English is the only language that can be applied. Because of this linguistic bias, I decided to investigate how to preprocess Chinese text data for NLP.
First, I would like to thank my cohort mate David Bruce for pointing out this disparity. In his blog post on Learning a New Language in a Word Cloud, he shared that Professor Emily M. Bender coined “the Bender Rule” which encourages people to at least name the language(s) they’re working on for NLP.
So the question is, “how can we perform NLP on Chinese?” In this blog, I’ll be walking through the basic preprocessing steps that occur in English NLP, including data cleaning, stemming/lemmatization, tokenization and stop words. TL;DR: not all of these steps are necessary for Chinese text data!
Text normalization is a method for standardizing text to prepare it for the tokenization, vectorization and classification steps. With english, the first step would be to convert all text to lowercase. Because Chinese characters are not capitalized to begin with, there’s no need for that data cleaning step.
Next comes stemming or lemmatization. Compared to English, there is also no concept of a stem in Chinese. Therefore, there is no need to perform this step either! So far, it seems like that preprocessing Chinese text data requires less steps than English text data; making the process (surprisingly) a little easier.
It’s worth noting, however, that there’s a concept similar to stems in this language, and they’re called Radicals. Radicals are basically the building blocks of Chinese characters. All Chinese characters are made up of a finite number of components which are put together in different orders and combinations. Radicals are usually the leftmost part of the character. There are around 200 radicals in Chinese, and they are used to index and categorize characters.
For instance, the radicals for female and horse come together for the character mother. This character uses the phonetic sound for horse but the gender indicator of female.
Therefore, procedures like stemming and lemmatization are not useful for Chinese text data because seperating the radicals would change the word’s meaning entirely.
Another difference is that Chinese handles plurality much differently. A basic example is that instead of a single word for them, it’s split into two characters 他们. We wouldn’t be able to get to the root of this word without completely changing the meaning.
Just to drive this point even further, the final reason why we can’t preprocess Chinese in the same way is because it uses different grammatical tenses for past, present and future. In Chinese, characters are added onto the main verb. This could be considered adding to stem, but it’s a little more tricky because the additional character could appear at the front or end, depending on the context. Here is a quick, basic example.
With stemming and lemmatization in English, you can just remove the ing or ed to get to the root word and meaning. But with Chinese, it’s not that simple because like I mentioned before, there’s no concept of “root word” in Chinese.
I could go on all day about the differences between English and Chinese. But the bottom line is that NLP is pretty different between the two languages, and these fundamental linguistic differences are very important for any computer scientist to understand.
Before I get into stop words, let’s tokenize some text first. Tokenizing breaks up text data into shorter pre-set strings, which help build context and meaning for the machine learning model. For this example, we will be looking a short excerpt from the textbook A New China.
飞机是今天晚上七点钟准时降落在北京首都国际机场的。学校外事处派的一位张先生接我们。
Translated to English, it means:
The plane landed at Beijing Capital International Airport at 7 o’clock this evening. The foreign affairs office sent Mr. Zhang to come pick us up.
It’s worth noting that Google Translate somewhat incorrectly translated the second sentence as “A Mr. Zhang from the school’s foreign affairs office picked us up.” This could be due the Chinese Room Theory, which argues that a computer cannot truly have an “understanding” of language, no matter how perfectly things are translated, and no matter how human-like the program behaves.
For the sake of simplicity, I’ll only tokenize the first sentence. We can use jieba text segmentation module to tokenize Chinese. Fun fact, “结巴” in Chinese means “to stutter.” Here is a link to their github for more information.
With this output, the tokenization was somewhat successful. The translated words are as follows: airplane, is, today, night, seven o’clock, punctually, land, in, beijing, capital international airport, and of.
There are three things that you may immediately notice:
The output has tags next to each wordSome words were not fully separatedThere are “useless” words, aka stop words
The output has tags next to each word
Some words were not fully separated
There are “useless” words, aka stop words
These “tags” label the part of speech. There are 24 part of speech tags and 4 proper name category labels in the jieba package’s existing dictionary.
北京 and 首都国际机场 should have actually been considered a single phrase because that’s the full name of the airport. Also, 七点钟 could have been separated into individual characters; but that’s up for debate.
To fix the words that should and should not have been separated, we can manually create custom words within the package’s dictionary.
jieba.add_word('北京首都国际机场', freq=None, tag='nt')
This line of code sets the name of the airport as a custom token, with the place name tag. We can repeat this code for each of the individual custom words we want to set, and then the tokenization should be able to pick that up if we run it again. Obviously this process can get tedious with a much larger text file, so it’s suggested that you find a well established dictionary beforehand, outside of what jieba provides by default.
In NLP, stop words are “meaningless” words that make the data too noisy or ambiguous. In our example sentence, the stop words are 是, 在 and 的. We could manually filter them out, but that’s also very tedious. Just like with English, there are pre-set lists of stop words out there. There are about 119 official stop words in Chinese, and they can be viewed on this website. Instead of manually removing them, could import the stopwordsiso package for a full list of Chinese stop words. More information can be found here. And with this, we can easily create code to filter out any stop words in large text data.
After all of this work, our hypothetical data is ready to be vectorized with TF-IDF Factorization and fed into a classification model! Obviously with real text data, we would be working with more than just one sentence. And this can present many other challenges.
This was just a rough introduction to preprocessing Chinese text data for NLP, and may not totally represent the process for more complex data.
And finally, just like with English, further procedures can be done with NLP, such as sentiment analysis. Check out this blog about Chinese sentiment analysis using SnowNLP.
Blog Post on the Importance of Non-English NLP
Blog Post on Word Segmentation for Languages Without Spaces Between Words
Blog Post on Improving jieba’s Tokenization
Another Chinese Stop Words Package, stopwords-zh
TF-IDF on Chinese Text Data
Stanford NLP Group’s Study on Chinese NLP & Speech Processing