SlowGuess/NAACL_2022_OCR huggingface dataset:
Ming Fang $^{1}$  Shi Zong $^{1}$  Jing Li $^{2}$  Xinyu Dai $^{1}$  Shujian Huang $^{1}$  Jiajun Chen $^{1}$
fangming@smail.nju.edu.cn,
{szong, daixinyu, huangsj, chenjj}@nju.edu.cn
$^{2}$ Department of Computing, The Hong Kong Polytechnic University, HKSAR, China
jing-amelia.li@polyu.edu.hk
Complaining is a speech act that expresses a negative inconsistency between reality and human expectations. While prior studies mostly focus on identifying the existence or the type of complaints, in this work, we present the first study in computational linguistics of measuring the intensity of complaints from text. Analyzing complaints from such a perspective is particularly useful, as complaints of certain degrees may cause severe consequences for companies or organizations. We create the first Chinese dataset containing 3,103 posts about complaints from Weibo, a popular Chinese social media platform. These posts are then annotated with complaints intensity scores using Best-Worst Scaling (BWS) method. We show that complaints intensity can be accurately estimated by computational models with the best mean square error achieving 0.11. Furthermore, we conduct a comprehensive linguistic analysis around complaints, including the connections between complaints and sentiment, and a cross-lingual comparison for complaints expressions used by Chinese and English speakers. We finally show that our complaints intensity scores can be incorporated for better estimating the popularity of posts on social media.<sup>1</sup>
Complaining is caused by the gap between reality and people's expectations (Olshtain and Weinbach, 1985). Brown et al. (1987) state that the purpose of complaining is not to confirm that the two parties have reached an agreement but to face-threatening acts. People use complaints to express their concerns or dissatisfaction based on the severity and urgency of situations.
Researchers from linguistics and psychology have long pointed out that people may shape their
complaints to varying degrees (Olshtain and Weinbach, 1985; Jenkins and Cangemi, 1979; Trosborg, 2011). Leech (2016) classifies complaints as conflicting speech acts. Mild complaints can reach the purpose of venting emotions to promote mental health, but serious complaints can lead to hatred and even bullying behaviors (Iyiola and Ibidunni, 2013). In computational linguistics, prior studies primarily focus on building automatic classification models for identifying the existence of complaints (Preotiuc-Pietro et al., 2019). Most recently, Jin and Aletras (2021) provided a dataset annotated with different severity levels of complaints based on the theory of pragmatics, including four distinct categories "no explicit reproach", "disapproval", "accusation" and "blame".
Among these studies, we note one missing piece is to measure the intensity of complaints. To illustrate this point, consider two sentences from the newest annotated dataset from Jin and Aletras (2021): can i complain to you about the coffee i just received ? and virgin media as usual full of lies lies lies ! !!. Although these two complaints may have the same type "accusation", it is clear that they are different regarding the degree of complaints. As another example, totally not cool. and please reply my dm asap ! !! are both classified as "disapproval". However, the latter makes a stronger complaint. Analyzing different complaints levels can also be beneficial. Companies need to regularly monitor the feedback from users, as certain complaints may significantly impact the reputation of their products. Organizations or governments need to monitor people's biggest complaints to understand their urgent needs.
In this work, we analyze the intensity of complaints on social media. To the best of our knowledge, it is the first computational linguistics study that tries to automatically capture the complaints intensity from text. We present the first Chinese complaints intensity dataset, consisting of 3,103 posts
Our annotated corpus is publicly available at https://github.com/nlpfang/complaint_intensity.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1742 - 1754 July 10-15, 2022 ©2022 Association for Computational Linguistics
from Weibo. We then show that the complaints intensity can be measured from text by building computational models. We further demonstrate the necessity and importance of understanding complaints intensity. This includes a detailed analysis that distinguishes the differences between our complaint intensity scores and original sentiment scores. As a pilot study for complaints in Chinese, we also perform a cross-lingual analysis to understand the differences in the complaint expressions used in Chinese and English. We have some interesting empirical findings. For example, we observe that English speakers tend to use more ironic expressions than Chinese speakers. Finally, we show how our annotated corpus can help predict the popularity of posts on social media.
In this section, we present the first Chinese dataset that is annotated towards the intensity of the complaints reflected from text.
We collect data from Weibo, $^{2}$  a famous social media platform in China that is similar to Twitter. As posts about complaints only account for a minority of the total posts on Weibo, in this work we consider education domain – an area that is the primary focus for most families in China, which generally raises hot debates and complaints about current education policies. We selected a set of keywords related to complaints, including抱怨 (complaint),不公平 (unfair), and举报 (report). We then randomly sampled 5 hashtags around these keywords and collected Weibo posts from these hashtags. We collected a total of 4,490 Weibo posts from August 2020 to May 2021.
Pre-processing. We notice that the hashtag on Weibo is usually a sentence (in "#...#" format), rather than a phrase like its Twitter counterparts. To ensure a certain amount of content generated by users, we filtered out posts with less than 10 words and more than 200 words (without hashtags). For each post, we removed the name of the author, location tags, and URLs. We also converted emoticon into text format. Finally, 3,103 Weibo posts remain for annotation. Table 1 shows the breakdown statistics in our corpus.
Hashtag	Num.
#代表建议让学生在校内完成家庭作业# (#The representative suggested that students should complete their homework on campus#)	762
#江苏明确教师不得用手机布置作业# (#Jiangsu Province makes it clear that teachers are not allowed to use mobile phones to assign homework#)	534
#院士不建议普通孩子学奥数# (#Academician does not recommend ordinary children to learn Mathematical Olympiad#)	627
#西安外国语大学封闭管理# (#Close management of Xi'an International Studies University#)	598
#人大法硕复试30余人成绩0分# (#More than 30 people scored 0 in the postgraduate examination of law at Renmin University#)	582
Total	3,103
2.2 Data Annotation
Complaints Levels. Our goal is to measure the intensity of complaints from text. We adopt the definition from Jenkins and Cangemi (1979), which quantifies the complaints into five levels, as shown in Table 2. Higher levels indicate stronger complaints.
Table 1: Hashtags and number of collected Weibo posts in our annotated corpus.
Level	Description
1	a little anxiety and disgust
2	deliberately expressing anxiety
3	actively looking for ways to solve anxiety
4	frustrated behavior
5	depression, fear, and despair
Table 2: Guideline used in annotation process for distinguishing different levels of complaints, adopted from Jenkins and Cangemi (1979).
In pilot studies, we test the feasibility of using these levels as the annotation guideline for the annotators, along with the potential mismatches between Chinese and English speakers. We observe that annotators are able to make comparison between complaints of different degrees. As discussed later, our annotations also achieve high agreement between annotators.
Best-Worst Scaling (BWS). In this work, we annotate the complaint intensity using Best-Worst Scaling, proposed by Louviere and Woodworth (1991). We choose this method as it can produce more stable and fined-grained scores than directly scoring (Kiritchenko and Mohammad, 2017). We note similar methods have been applied to various tasks, including measuring offensiveness (Hada et al., 2021) and intimacy (Pei and Jurgens, 2020) in the computational linguistic literature.
In BWS annotation, annotators are provided with
2www.weibo.com
4-tuples randomly generated that meet certain criteria. Annotators are then asked to select the strongest complaint item and the weakest complaint item within each 4-tuple. In practice, we randomly generated  $2n$  distinct 4-tuples, with  $n$  being the number of posts. This amount of tuples is considered to be sufficient for getting reliable scores from annotation (Kiritchenko and Mohammad, 2017). We assign the complaint intensity score for each post by using the percentage of strongest cases minus the weakest cases, ranging from -1 to 1.
Annotation Quality. To ensure the quality of our annotations, we manually annotated 100 posts and asked all annotators to annotate them beforehand. We removed annotators whose accuracy is less than  $70\%$  on these golden annotations. To get highly reliable results, we got each tuple annotated by 3 annotators. In total, we received more than 14,000 annotations from 15 annotators.
We follow the literature (Kiritchenko and Mohammad, 2017) and measure the quality of annotations by using score-to-half reliability (SHR). SHR score is calculated by randomly splitting all the tuples into two halves and then computing the correlation between these two groups. We repeat the above process 100 times. The average SHR score is 0.91, which indicates strong reliability.
2.3 Data Analysis
We first analyze the distribution for the annotated complaint intensity scores in our corpus. As shown in Figure 1 (Left), we observe a normal distribution for the number of posts across different complaint scores, with most of the posts having intensity of complaints within -0.2 and 0.2.
We also observe that the length of complaint posts (intensity  $>0$ ) is longer than that of noncompliant posts (intensity  $< 0$ ) in Figure 1 (Right). By examining our data, we observe it is because stronger complaints contain more details with more aspects. For example, in bin 5 of Table A1 (in Appendix A), the target of complaint changes from school (school) to dissatisfaction with图书馆 (library) and even accuses the behavior of the guard (security guard). This is the halo effect in psychology: if something leaves a wrong impression, everything related to it becomes terrible. On the contrary, we observe
that most non-complaining posts contain only plain expressions, and people will not describe too much after expressing their opinions on the matter.
Figure 1: Distribution for the number and length of posts over complaints intensity in our corpus.
3 Predicting the Intensity of Complaints
In Section 2, we have a dataset annotated with the intensity of complaints from -1 to 1. We now build computational models for predicting the intensity of complaints of a given post.
Support Vector Regression (SVR). We use support vector regression as our first baseline model. We experiment with two different input sentence representations: bag of  $\{2,3,4\}$ -gram features and 300-dimensional GloVe embeddings (Pennington et al., 2014). Results in Table 4 use an RBF (Radial Basis Function) kernel. We observe similar results using other kernels in practice (e.g., linear kernel).
Bidirectional LSTM. We also experiment with a bidirectional Long Short-Term-Memory (Bi LSTM) model. The LSTM and the average pooling layer concatenation are passed through a linear layer with a tanh activation, producing a score between -1 and 1. We use two sets of embedding for input layers: Glove (Pennington et al., 2014) and BERT for embedding (Li et al., 2020). The attention mechanism is also considered. Other hyperparameters for the models are a batch size of 64, a learning rate of 1e-3, 13 epochs with early stopping, and a dropout of 0.5 to avoid overfitting.
Pre-trained Models. We finally experiment with pre-trained models, including BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019), and ERNIE (Zhang et al., 2019). For all pretrained models, we add a linear layer as a regression layer to the model. We then fine-tune these models using a mean square error loss objective. We set the batch size to be 16 and learning rate to be 2e-5. The model is trained for 3 epochs. All hyperparameters are selected using a held-out dev set.
<sup>3</sup>Requirements are: (1) no two 4-tuples are the same; (2) no two posts within a 4-tuple are identical; (3) each post appears approximately in the same number of 4-tuples; (4) each pair of posts appears approximately in the same number of 4-tuples.
Bin	Weibo posts	Scores
1	以前在没有手机的年代，孩子们都是自己记作业。我觉得有助于形成自我管理能力 (In the era when there were no mobile phones, children kept their homework by themselves. I think it helps to form self-management ability.)	-1
2	最好的解决办法就是没有家庭作业，对老师对家长都好 (The best solution is to have no homework, which is good for teachers and parents.	-0.56
3	现在的学生作业为啥都得用手机做? (Why do students have to use mobile phones to do their homework now?)	+0.12
4	我是真的觉得用手机交作业很烦 (I find it really annoying to hand in homework with my mobile phone.)	+0.4
5	气死了！食堂涨价，超市关门，就没人管理嘛? (Mad! The price of the canteen increases, and the supermarket closes, no one manages it?)	+1
We evaluate the model performances for our complaint intensity prediction task in two settings: (1) mix hashtag, where we combine Weibo posts from different hashtags together, and (2) cross hashtag, where the posts for train, dev and test sets are separately from different hashtags. We use Pearson's correlation and MSE (Mean Square Error) as metrics for all our experiments.
Mix Hashtag. We combine Weibo posts from different hashtags together and then split them into  $80\%$  for training,  $10\%$  for validation, and  $10\%$  for test. Results are shown in Table 4. We observe that RoBERTa outperforms all other models and reaches Pearson up to 0.79, followed by the LSTM model. The SVR model has the worst performance.
Cross Hashtag. We choose four of total five collected hashtags as the train and development set, and hold out the rest one for test. We report the average value after five experimental runs in Table 4. We observe under cross hashtag setting, models get comparable performances to mix hashtag setting. It indicates models seem to learn common linguistic cues between different hashtags.
We perform an error analysis to shed light on the limitations of our best-performing model. A prediction is defined to be wrong when the difference between ground truth and the predicted score is greater than 0.5. We randomly sample 100 errors and manually inspect them. All errors are divided into three categories:  $43\%$  of errors are because of irony expression in complaints,  $29\%$  are due to implicit expressions, and  $28\%$  are due to the insufficient and vague expressions.
Table 3: Sample posts with complaints intensity scores. We divide our scoring scale (from -1 to 1) into 5 bins of size 0.4 (i.e., bin 1 refers to scores ranging from -1.0 to -0.6, bin 2 from -0.6 to -0.2, etc.). More examples are provided in Table A1 in Appendix A.
Models	Mix Hashtag	Cross Hashtag
r	MSE	r	MSE
SVR (\{2,3,4\}-gram)	0.36	0.46	0.35	0.46
SVR (GloVe)	0.49	0.36	0.47	0.38
LSTM (GloVe)	0.69	0.24	0.65	0.27
LSTM Attn (Glove)	0.72	0.22	0.70	0.25
LSTM (BERTembed)	0.76	0.15	0.75	0.16
ERNIE	0.76	0.14	0.76	0.14
BERT	0.77	0.20	0.75	0.23
RoBERTa	0.79	0.11	0.78	0.11
Table 4: Pearson's  $r$  and Mean Square Error (MSE) on two datasets for predicting the intensity of complaints.
Ironic Expressions. We observe that most errors happen when the posts contain ironic expressions. Users use positive words such as "perfect" or "great" to express dissatisfaction, which are misleading models to ignore the implicit complaints. A typical example is as follows: 食堂的涨价消息比封校政策来的快, 这学校真好 (The news of price increase in the cafeteria is coming faster than the school closure policy. This school's management is really good).
Implicit Expressions. The model struggles with complaints expressed in more subtle ways. These complaints do not contain any prominent negative words, but through other means like strike a chord or entrust. In the following example, the user hopes that managers can personally experience the status quo to understand the user's dissatisfaction. Therefore, predicting them correctly requires more contextual understanding: 真的非常极其的希望校领导也能来感受一下封校的生活 (I really hope that school leaders can also come and experience the life of the closed school).
4During COVID-19 pandemic, Chinese universities restrict students from going outside and limit their activities within campus. It thus causes students' complaints.
Vague Expressions. We observe that the model is likely to be confused by vague or incomplete expressions in the posts. Consider the following example:赶上 this破事，如果教育真的公平不如直接取消复试吧(Encountered this shit. If the education is really fair, it is better to cancel the exam). The hypothetical relationship using 如果 (if) is an expression of uncertainty (Wei et al., 2018) and there are no prominent emotional words in the text. Thus, our model fails to understand the speaker's intention well.
4 Complaints as an Emotion
From Table 3, we notice stronger complaints seem to be associated with negative emotion words. Prior studies also point out that complaints can be treated as an influential emotional dimension (Iyiola and Ibidunni, 2013). Then a natural question to ask is whether existing sentiment models have already been able to predict complaints intensity scores and our annotation efforts are actually not needed?
In this section, we demonstrate the necessity of building corpus annotated with complaints intensity, by showing the model trained on standard sentiment datasets fails to do well in our complaints intensity prediction task. We also show that analyzing complaints can be a useful complement for sentiment analysis.
4.1 Differences between Complaints and Sentiment
In sentiment analysis, models normally output a score between 0 and 1, indicating how likely a post is to express negative emotion. Here we make the assumption that the most negative emotion may lead to the strongest complaint. We first examine if these probability scores from sentiment models can be used as intensity scores for measuring complaints intensity.
Setup. To ensure a fair comparison, we select a newly developed dataset on COVID-19, collected also from Weibo using hashtags related to COVID-19 by Lyu et al. (2020). The dataset contains 21,174 posts with fine-grained emotion annotations.
In our experiments, we follow the same steps in Section 2.1 to pre-process this dataset. As our pre-processing steps result in a category imbalance issue, we merge categories with similar emotions.
Specifically, we merge labels "fear", "anger", "disgust", and "sadness" into "negative" category and merge labels "gratitude", "surprise", and "optimism" into the "positive" category. Finally, we have 8,783 posts in the negative category and 8,336 posts in the positive category.
We use BERT to train a sentiment model using the above COVID-19 data. We also sample  $80\%$  of our corpus for developing our BERT-based complaints model. The performances of both models are compared on the left  $20\%$  annotated posts. During evaluation, the sentiment scale from 0 to 1 is linearly mapped to our complaints intensity interval from -1 to 1.
Results. Results are shown in Table 5. We observe that using the probability scores from sentiment models shows decent performance on our complaints intensity prediction task. It indicates a clear connection between complaints and emotions. We also observe that models trained on our annotated corpus outperform sentiment model, demonstrating the necessity of building such corpus for complaints intensity estimation.
Model	Pearson's r	MSE
Complaint	0.76	0.20
Sentiment	0.71	0.24
Table 5: Performances of sentiment model and complaint model for complaints intensity prediction task.
Valence and Arousal. We also quantitatively studied the correlation between complaints and sentiment through Valence-Arousal. Valence can be positive or negative and corresponds to the standard dimension of sentiment analysis; Arousal, which can be low or high and express the degree (Vorakitphan et al., 2020). We use the VA score annotated by Xu et al. (2021), which contains 11,310 simplified Chinese words. The valence and arousal ratings include scores -3 to +3 for valence rating and scores 0 to 4 for arousal rating.
We identify sets of words in the Valence-Arousal lexicon that have high valence scores  $(>2)$ , low valence scores  $(< -2)$ , high arousal scores  $(>3)$ , and low arousal scores  $(< 2)$ . A similar approach is used in Hada et al. (2021). We average the scores of tokens from the above four dimensions in each post and calculate the correlation with our complaints intensity. Results in Table 6 show low valence and high arousal are more correlated with complaints intensity compared to the other two dimensions.
To the best of our knowledge, there does not exist a Chinese Weibo dataset annotated with continuous sentiment intensity. We note some datasets with discrete sentiment levels, like Douban movie short comments dataset (Ma et al., 2011).
Dimension	Pearson's r
High valence	0.02
Low valence	0.31
High arousal	0.18
Low arousal	0.05
4.2 Complaints Help Sentiment Analysis
We now show that analyzing complaints could be helpful for the binary sentiment analysis task.
Models. We still use the COVID-19 dataset discussed in Section 4.1 for the binary sentiment classification task. We experiment with the SVM and BiLSTM-Attention models. The complaints score is added as an additional feature input to the model.
Results. Table 7 shows the results of the models on the sentiment classification task. Overall, we observe that the models with the complaint feature perform better than the original model. It demonstrates that a simple add-on can boost the prediction accuracy of sentiment classification for non-neural and traditional neural models. We also provide the performance of BERT for reference in Table 7.
Table 6: Pearson's correlation between the complaints intensity scores and emotion dimensions.
Models	P	R	F1
SVM	0.51	0.49	0.50
+ complaint	0.53	0.50	0.51
BiLSTM-Att	0.72	0.70	0.71
+ complaint	0.74	0.71	0.72
BERT	0.79	0.76	0.77
Table 7: Results for binary sentiment prediction. F1 score of models with complaint feature is significantly better than the original model ( $p$ -value  $< 0.01$ ,  $t$ -test).
We are interested in what types of tokens sentiment model and complaint model try to capture. We thus take the BiLSTM-Attention model trained for sentiment classification task in Section 4.2 and our complaints model for comparison. We visualize the attention weights extracted from the above two models for the following example: 准备这么久的考试推迟真是绝了呵呵 (After preparing for so long, the exam is now postponed. It's absolutely speechless. Hmm. How interesting.). We observe that sentiment model assigns high attention weights for tokens 绝了 (speechless) and 呵呵 (Hmm. How interesting.), both expressing emotions. However, our complaint model puts high weights on tokens 推迟 (postponed) and 考试 (exam).
These are tokens that reflect the reasons for complaining. These differences again demonstrate the need for building a specific dataset for complaints intensity.
Figure 2: Attention weights for the sample sentences in sentiment model and our complaint model.
5 Cross-lingual Analysis
Our newly collected complaints intensity dataset is written in Chinese, while current existing datasets (Preotiuc-Pietro et al., 2019; Jin and Aletras, 2021) contain English tweets. It provides us an opportunity to understand the linguistic differences for complaints made by Chinese and English speakers on social media.