Beginner’s Guide to Sentiment Analysis for Simplified Chinese using SnowNLP | by Ng Wai Foong | Towards Data Science

By reading this article, you will be exposed to a technique for analyzing the sentiments of any text in Simplified Chinese. This tutorial will be based on Simplified Chinese but it can be used on Traditional Chinese as well due to the fact that SnowNLP is capable of converting Traditional Chinese to Simplified Chinese.
There will be 4 sections in this tutorial:
Setup and installationUsage and API callTraining modelConclusion
Setup and installation
Usage and API call
Training model
Conclusion
In this tutorial, I will be installing the modules required via pip on Windows operating system. Let’s start by getting the files from an open-source modules called SnowNLP which is a Simplified Chinese Text Processing module. Although this module has not been updated for quite some time, the results is good enough for most use cases.
Go to the following link and download the required files. Once it is completed, extract the zipped files to a directory of your preference. You should have a snownlp-master folder which contains all the necessary files.
SnowNLP supports Python 3 as stated in the official Github Link. In this tutorial, I will be using Python 3.7.1 in a virtual environment.
You can easily install this module via the following command:
pip install snownlp
Once it is completed, you will be able to the the output that indicates the version number. In case you have missed it, feel free to check it via the following command:
pip list
I am using version 0.12.3 for this tutorial. Once you have finished, proceed to the next section.
Let’s explore a little more on the basic API calls that can be used to pre-process the input text.
You have to import it and do the initialization via the SnowNLP class as follow:
You initialize it with your input text as parameter. It is recommended to prefix with a u to indicate that this is an Unicode string. This syntax has been in used since Python 2.0 but was later removed in version 3.0 to 3.2. From 3.3 onward, the u prefix is a valid syntax for Unicode string.
Chinese is a unique language in a sense that there is no spacing between words unlike majority of the languages in the world. This makes it difficult to determine the number of words in a sentences as a word can be a combination of 1 or more Chinese characters. Hence, when performing any natural language processing, you need to split the whole chunk of text into words, a process known as Tokenization. You can easily use the following command to do the tokenization:
You should get the following results as output:
['我', '喜欢', '看', '电影', '。']
Most of the time, we will be more interested in the part-of-speech tags which refer to the relationship with adjacent and related words in a sentence. If you have difficulty understand this, think of it as putting a tag to identify a word is a noun, adverb, verb, adjective or etc. Use the following code to get the tags of each word:
The function tags returns a zip object which can be unzipped using the list function. You should get the following result:
[('我', 'r'), ('喜欢', 'v'), ('看', 'v'), ('电影', 'n'), ('。', 'w')]
Each of the words will be paired with the respective tags. Kindly refer the list below to know more about the meaning of certain tags (this is not a complete list as I could not find any documentation on it in the official site):
r: refers to pronoun, word that replaces a noun in a sentence.
v: refers to verb, a word used to describe an action, state, or occurrence.
n: refers to noun, a word used to identify any of a class of people, places, or things.
w: refers to punctuation, a symbol used in writing to separate sentences and their elements and to clarify meaning
There is also a function to get the pinyin of each character. However, tones are not included in the pinyin. Example of getting the pinyin is as follow:
You should get the following output:
['wo', 'xi', 'huan', 'kan', 'dian', 'ying', '。']
As I have mentioned earlier, this module are meant for Simplified Chinese. If you would like to process Traditional Chinese, kindly use the following code to convert the text to Simplified Chinese:
You should get the following output”
'这家伙是坏人。'
Up until now, the input text consists of just one sentence. If you are using a paragraph as input, you should split it into chunk of sentences before running any API call. To do that, type the following command:
Check if you get the following output:
['在茂密的大森林里', '一只饥饿的老虎逮住了一只狐狸', '老虎张开大嘴就要把狐狸吃掉', '“慢着”', '狐狸虽然很害怕但还是装出一副很神气的样子说', '“你知道我是谁吗', '我可是玉皇大帝派来管理百兽的兽王', '你要是吃了我', '玉皇大帝是决不会放过你的”']
There is also an option to identify keywords from the sentences. You can pass an integer parameter indicating the number of keywords to get from the input. I personally tested it to extract 5 keywords and it works great for news article and short stories. Type the following command:
The extracted keywords are as follow:
['狐狸', '大', '老虎', '大帝', '皇']
If keywords aren’t what you are looking for, you can try to use the summary to extract synopsis(important sentences) from it. The module will split the text into sentences and extract what it feels are the most important sentences. Similar to keywords, it accepts an integer as parameter that determine the number of summary. If the sentences in your text is less than the input, it will output the whole text as summary. Use the following command:
You should get the following output:
['老虎张开大嘴就要把狐狸吃掉', '我可是玉皇大帝派来管理百兽的兽王', '玉皇大帝是决不会放过你的”', '一只饥饿的老虎逮住了一只狐狸', '你要是吃了我']
The focus of this tutorial is on sentiment analysis. However, most of the time, you need to do text pre-processing in order to reduce the input text for better accuracy. Hence, you can use the other API calls mentioned above. Let’s test it with some sample text:
Check if you get the following results (the comments are not part of the output, they are added for easier viewing):
0.7853504415636449 #这个产品很好用0.5098208142944668 #这个产品不好用0.13082804652201174 #这个产品是垃圾0.5 #这个也太贵了吧0.0954842128485538 #超级垃圾0.04125325276132508 #是个垃圾中的垃圾
The value output range from 0 to 1 with 0 represent negative sentiment while 1 represent positive sentiment. As you can see, the results are not that bad consider the fact that most of the results are on point except for the 0.5 value which should be a negative sentiment. Please be noted that the sentiment is trained on comments made when purchasing a product. If you are testing it on other domain, the results will be really bad. You can check the following files to find out more on the training data used:
snownlp-master/snownlp/sentiment/neg.txtsnownlp-master/snownlp/sentiment/pos.txt
snownlp-master/snownlp/sentiment/neg.txt
snownlp-master/snownlp/sentiment/pos.txt
You will notice that the word 贵 appeared about 600+ times in both neg.txt and pos.txt. This is the reason why the module output a neutral 0.5 value for the sentiment. In contrast, the word 垃圾 appeared 200+ times in neg.txt and only 35 times in pos.txt. To solve this issue, we can train our own model using custom text dataset. It will be explained in the next section.
Prepare your own dataset by collecting the positive and negative sample sentences in two text folders. I created them in the snownlp-master folder and named them as custom_pos.txt and custom_neg.txt. Each example sentences is separated in a newline as follows:
今天明明是周六,我就不想工作,你看他,好意思吗?一直在偷懒...
Once you have your dataset ready, let run the following code (change the name accordingly):
You should get an output file called custom_sentiment.marshal.3 in the snownlp-master folder. Do not be surprise by the .3 extension at the end of the file. To use the output model, you can do one of the following:
Modify the code in snownlp-master/snownlp/sentiment/__init__.py. Change the data path to the directory of the newly output marshal.3 file.Go to snownlp-master/snownlp/sentiment folder. Create a new folder called backup and place both sentiment.marshal and sentiment.marshal.3 to the backup folder. Copy custom_sentiment.marshal.3 from snownlp-master folder and place it into snownlp-master/snownlp/sentiment folder. Rename it to sentiment.marshal.3.
Modify the code in snownlp-master/snownlp/sentiment/__init__.py. Change the data path to the directory of the newly output marshal.3 file.
Go to snownlp-master/snownlp/sentiment folder. Create a new folder called backup and place both sentiment.marshal and sentiment.marshal.3 to the backup folder. Copy custom_sentiment.marshal.3 from snownlp-master folder and place it into snownlp-master/snownlp/sentiment folder. Rename it to sentiment.marshal.3.
Personally, I prefer the second method as modifying code can be risky at times. Kindly note that you only need the .3 file. You can test the result by re-initialize the SnowNLP module and run the following code (replace the text with something that similar to your training data):
There will be some differences in the output results based of your training data.
Apart from sentiment, you can train segmentation and tags as well. Refer to the following code to do the training for segmentation:
We will be using the following code for tags:
This article demonstrated a simple and effective way to use a Python module called SnowNLP for sentiment analysis for Simplified Chinese. At the time of this writing, the official documentation still lacks in quality in explaining certain functionality. Feel free to read the code and explore it to get a better understanding on the techniques used by the developer. There are a lot other modules available for natural language processing and each one of them have their own advantages and disadvantages. Kindly evaluate them according to your use cases and make use of it in your projects. Thanks and have a good day ahead!