SlowGuess/NAACL_2022_OCR huggingface dataset:
Our cross-lingual analysis is performed in the following way. First, we evenly sample 200 complaint tweets released in Jin and Aletras (2021) from their defined four categories. Similarly, we divide our annotated data with intensity greater than 0 (as complaint posts) into 4 bins for sampling. We ask 5 in-house annotators to mark and then use majority voting to decide if a post makes a direct or indirect complaint, along with the strategy used to make complaints.
Direct and Indirect Complaints. According to Boxer (2010), the speech of complaint can be divided into direct and indirect complaints. In social media, direct complaints are addressed to a complainee who is held responsible for the complaint action, and the addressee is fully or partially responsible for this behavior. Here is an example for a direct complaint: the school's management of the school so inhumane?) While indirect complaints refer that the recipient is not primarily responsible for perceived complaints and is more about the evaluation of the target or event, such as:这样做让我们的压力很大,真无语 (This puts us under a lot of pressure, so speechless).
We compare the percentages of direct and indirect complaints for two languages based on our annotations. Results show that indirect expressions are more likely to be used in Chinese posts (80%
are indirect complaints). On the contrary,  $91\%$  of English tweets make complaints in a direct way. This finding seems to be consistent with the study of Deng et al. (2019), which demonstrates that Chinese people tend to use indirect expressions.
Strategy. In pragmatics, strategy is an appropriate countermeasure adopted to achieve the purpose of language communication. House and Kasper (1981) conducted a comparative study of English and German complaints from the direct degree and emotional markers. Anna (2008) discussed the choice of Chinese and English complaining strategies and proposed an explicit-implicit strategy. Implicit complaints are very subtle and even use metaphors to express complaints about the target, which requires more semantic information to capture. In contrast, explicit complaints can be further divided into types of with-redress and without-redress. With-redress is the strategy of request for repair; without-redress usually contains complaint targets or objects, which can be easily identified by recognizing an obvious complaint word or phrase.
Table 8 shows that strategy varies across languages. We find that the Chinese are more inclined to without-redress strategy, while the most frequent strategy used by Americans is with-redress strategies. It provides some empirical support for findings in Anna (2008).
Strategy	Chinese	English
Implicit	65%	12%
With-redress	13%	78%
Without-redress	22%	10%
Irony. Irony implies the opposite of the literal meaning. Dealing with non-literal means is a challenging task. On Twitter, Reyes et al. (2012) used specific hashtags as gold labels to detect irony in a supervised learning setting, such as #irony and #sarcasm. Attardo (2013) observed that native speakers are usually able to process the meaning of sarcasm automatically, but the ability of second language learners to infer meaning from context varies greatly. In Section 3.3, we observe irony counts for the majority of errors made by our model.
We analyze the number of complaints using irony. To detect ironic expressions, we separately use the Chinese irony dataset of Tang and Chen (2014) and the English dataset of Van Hee et al. (2018) to train the Bi-LSTM model. Results
showed that  $10\%$  of Chinese data contained irony, and  $26\%$  of English data contained irony. It shows that English speakers use ironic expressions more often compared to Chinese speakers. Further, we conduct part-of-speech analysis on these ironic expressions. Table 9 shows that Chinese irony has the highest proportion of nouns, followed by verbs; while in English irony, verbs are the most, followed by nouns. In addition, there are more adjectives and adverbs in English than in Chinese.
Table 8: Percentages of strategies across languages.
Part of Speech	Chinese	English
Nouns	31.2%	27.9%
Verbs	21.8%	35.2%
Adjectives	3.1%	10.7%
Adverbs	9.9%	11.9%
Table 9: Percentages for POS tags across languages.
Limitations. We note a few limitations for our cross-lingual analysis. One limitation is domain mismatch. Our Chinese posts are collected from education domain, while English posts are collected from domains including food or online service. People may exhibit different behaviors when making complaints. We also note that the sample size for making comparisons is rather small, due to budget issues for experts annotations. In future work, we will perform a large-scale comparison by using the data collected from the same domains and utilizing crowdsourcing annotations.
6 Predicting Post Popularity
Finally, we demonstrate that complaint intensity scores from our computational models can help estimate the post popularity on social media. We envision incorporating these scores into existing social media monitoring systems to improve their prediction accuracy.
Task. Predicting the popularity of content on social media has been extensively studied in literature (Szabo and Huberman, 2010; Hong et al., 2011; Bao et al., 2013; Carta et al., 2020). Our task is to predict the popularity of a Weibo post. Specifically, given the popularity prediction  $p(t_{i-1})$  at time  $t_{i-1}$ , we wish to predict the popularity  $p(t_i)$  at next time step  $t_i$ . The popularity  $p(t)$  is measured by the number of blog posts under the topic at time  $t$ .
Methods. We follow Szabo and Huberman (2010) and consider the following baseline that only uses early prevalence for prediction:
$$
\ln \widehat {p} (t _ {i}) = \alpha_ {1} \ln p (t _ {i - 1}) + \alpha_ {2},
$$
where  $\alpha_{1}$  and  $\alpha_{2}$  are learnable coefficients. It is justified as a strong baseline in Bao et al. (2013) that a strong correlation exists between logarithmically transformed popularity and early popularity.[6]
The popularity of posts on social media can be measured by multiple dimensions (Bao et al., 2013). To show the effectiveness of our complaint scores, we add in the complaint intensity as a new term to estimate the final logarithmic popularity:
$$
\ln \widehat {p} (t _ {i}) = \beta_ {1} \ln p (t _ {i - 1}) + \underbrace {\beta_ {2} d _ {c} (t _ {i - 1})} _ {\text {c o m p l a i n t s d e n s i t y}} + \beta_ {3},
$$
where  $d_{c}(t_{i - 1})$  is the complaints density at time  $t_{i - 1}$ , calculated by the ratio between the sum of the complaint intensity of blog posts per unit time to the number of all blog posts.  $\beta_{1},\beta_{2}$  and  $\beta_{3}$  are learnable new coefficients from data.
Setup. We collected a new set of 4,973 posts under 8 hashtags on Weibo from March 2021 to November 2021, which are shown in Table A2 (in Appendix B). We pre-process these posts using the same steps in Section 2.1. Within each hashtag,  $80\%$  of the posts are used for training, and the left  $20\%$  are used for testing. We set the time step to be two hours. RMSE (root mean square error) and MAE (mean absolute error) are used to evaluate predicted results.
Results. We first examine the relationship between complaints density and post popularity as a sanity check. Results show a strong positive correlation with an upper cluster slope of 0.95.
We report our post popularity prediction results in Table 10. We observe our method that combines complaint density outperforms the baseline method. In Figure 3 we also show the comparison between our predictions and real values for a specific hashtag: #巨人教育宣布倒闭# (#JuRen Education Group announces its bankruptcy#. We observe that adding complaints scores help better estimate the post popularity, especially in the early stages. It is probably because complaints are likely to draw users' attention to engage in discussions and hence boost the popularity of events.
There have been various studies for complaints in linguistics, economics, and public opinion re
Method	RMSE	MAE
Baseline	0.41 ± 0.01	0.35 ± 0.01
+ complaints density	0.35 ± 0.02	0.33 ± 0.01
Table 10: RMSE and MAE for popularity prediction.
Figure 3: Comparison between actual post popularity and our predictions for hashtag #JuRen Education Group announces its bankruptcy#. RMSE = 0.32, MAE = 0.30.
search. In Olshtain and Weinbach (1985), complaints are defined as what happened does not meet people's expectations, making people dissatisfied and blaming others. Kolodinsky (1995) explained the characteristics of consumers' complaining behavior from an economic point of view. Liu and Yen (2016) analyzed complaints about public transportation. Complaints on social media have also drawn great attention in recent years. Andreassen and Streukens (2013) focused on the study of the difference between social media complaints and traditional complaints, and argue that social media complaints are a unique way for consumers to express dissatisfaction. Balaji et al. (2015) studied the causes of complaints and found most of the complaints occur after a double deviation caused by dissatisfaction with the last solution. Motivated by these, in this paper, we collect complaints from Weibo, a widely used social media application.
In the area of linguistic studies on computational sociology, Meinl and Ebba (2010) studied the complaints act sequence in eBay reviews through 200 annotated English and German reviews. Ganesan and Zhou (2016) collected 2,500 reviews from Yelp and Walmart about commodity, then manually categorized them into 5 categories: negative only, complaint, positive only, raise, and irrelevant. Preotiuc-Pietro et al. (2019) focused on binary classification between complaints and non-complaints in various domains, such as food, car, online service, e-commerce. Jin and Aletras (2021) categorized complaints into 4 categories: no explicit reproach, disapproval, accusation, and blame. In this work,
$^{6}$ Bao et al. (2013) further proposed to use link density and diffusion depth for popularity prediction. Above methods were tested on WISE 2012 challenge dataset. We tried our best but are not able to have access to this dataset. In our own collected dataset, we do not have link or diffusion information.
we present the first study of estimating the intensity of complaints from text.
Our work is also related to prior work on emotion detection and sentiment intensity estimation (Mohammad and Bravo-Marquez, 2017; Cortis et al., 2017). Kiritchenko and Mohammad (2017) created a Twitter dataset annotated with sentiment intensity. We have discussed the connections between complaints and sentiment in detail in Section 4.
In this paper, we present the first study of measuring the intensity of complaints from text. We build a corpus of 3,103 Chinese Weibo posts about complaints, annotated with complaints intensity scores using Best-Worst Scaling method. We then demonstrate that our corpus supports the development of automatic computational models for accurate complaints intensity predictions. Furthermore, we study the connections between complaints and sentiment, and perform a cross-lingual comparison for complaints expressions between Chinese and English. We finally show that our complaints intensity scores help better estimate the posts popularity on social media.
Ethical Concerns
In this paper, we collect a complaint dataset from Weibo. The tools we use to collect posts comply with Weibo's terms of service. We will follow Weibo's policy for content redistribution to release our annotated corpus. Specifically, we will not release any user information or demographic data, including the authors' names, ages, and origins.
We recruited part-time research assistants for our annotation task. Annotators were warned that the complaint posts might contain offensive or upsetting content. Annotators were shown only anonymized posts and agreed not to make attempts to de-anonymize them. We did not collect any personal data from the annotators before, after, or during the annotation task. Moreover, we pay them 15.7 USD/hour and at most 14 hours per week.
We would like to thank the anonymous reviewers for their valuable feedback. This work was partially supported by a grant from the Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. PolyU/25200821) and CCF-Baidu Open Fund (No. 2021PP15002000).
Tor W Andreassen and Sandra Streukens. 2013. Online complaining: understanding the adoption process and the role of individual and situational characteristics. Managing Service Quality: An International Journal.
Anna. 2008. Comparative Analysis of Complaint Strategies in Direct Complaint of Chinese and American College Students. Ph.D. thesis, Dalian University of Technology.
Salvatore Attardo. 2013. Intentionality and irony. *Irony and Humor: From pragmatics to discourse*, pages 39-58.
MS Balaji, Subhash Jha, and Marla B Royne. 2015. Customer e-complaining behaviours using social media. The Service Industries Journal, 35(11-12):633-654.
Peng Bao, Hua-Wei Shen, Junming Huang, and Xue-Qi Cheng. 2013. Popularity prediction in microblogging network: a case study on sina weibo. In Proceedings of the 22nd international conference on world wide web, pages 177-178.
Diana Boxer. 2010. How to gripe and establish rapport. Speech act performance: Theoretical, empirical and methodological issues, pages 163-178.
Penelope Brown, Stephen C Levinson, and Stephen C Levinson. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press.
Salvatore Carta, Alessandro Sebastian Podda, Diego Reforgiato Recupero, Roberto Saia, and Giovanni Usai. 2020. Popularity prediction of instagram posts. Information, 11(9):453.
Keith Cortis, André Freitas, Tobias Daudert, Manuela Huerlimann, Manel Zarrouk, Siegfried Handschuh, and Brian Davis. 2017. SemEval-2017 task 5: Fine-grained sentiment analysis on financial microblogs and news. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 519-535, Vancouver, Canada. Association for Computational Linguistics.
Xinmei Deng, Sieun An, and Chen Cheng. 2019. Cultural differences in the implicit and explicit attitudes toward emotion regulation. *Personality and Individual Differences*, 149:220-222.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Kavita Ganesan and Guangyu Zhou. 2016. Linguistic understanding of complaints and praises in user reviews. In Proceedings of NAACL-HLT, pages 109-114.
Rishav Hada, Sohi Sudhir, Pushkar Mishra, Helen Yannakoudakis, Saif M Mohammad, and Ekaterina Shutova. 2021. Reddit: Norms of aggressiveness for english reddit comments. arXiv preprint arXiv:2106.05664.
Liangjie Hong, Ovidiu Dan, and Brian D Davison. 2011. Predicting popular messages in twitter. In Proceedings of the 20th international conference companion on World wide web, pages 57-58.
J. House and G. Kasper. 1981. Politeness Markers in English and German. Rasmus Rask Studies in Pragmatic Linguistics, Volume 2, Conversational Routine.
OO Iyiola and OS Ibidunni. 2013. The relationship between complaints, emotion, anger, and subsequent behavior of customers. IOSR Journal of Humanities and Social Sciences, 17(6):34-41.
William M Jenkins and Joseph P Cangemi. 1979. Levels of intensity of dissatisfaction: A model. Education, 99(4).
Mali Jin and Nikolaos Aletras. 2021. Modeling the severity of complaints in social media. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2264-2274, Online. Association for Computational Linguistics.
Svetlana Kiritchenko and Saif Mohammad. 2017. Bestworst scaling more reliable than rating scales: A case study on sentiment intensity annotation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 465-470, Vancouver, Canada. Association for Computational Linguistics.
Jane Kolodinsky. 1995. Usefulness of economics in explaining consumer complaints. Journal of Consumer Affairs, 29(1):29-54.
Geoffrey Leech. 2016. Principles of pragmatics. Routledge.
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. arXiv preprint arXiv:2011.05864.
Weng-Kun Liu and Chia-Chun Yen. 2016. Optimizing bus passenger complaint service through big data analysis: Systematized analysis for improved public sector management. Sustainability, 8(12):1319.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Jordan J Louviere and George G Woodworth. 1991. Best-worst scaling: A model for the largest difference judgments. Technical report, Working paper.
Xiaoting Lyu, Zhe Chen, Di Wu, and Wei Wang. 2020. Sentiment analysis on chinese weibo regarding covid-19. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 710–721. Springer.
Hao Ma, Dengyong Zhou, Chao Liu, Michael R Lyu, and Irwin King. 2011. Recommender systems with social regularization. In Proceedings of the fourth ACM international conference on Web search and data mining, pages 287-296.
Meinl and M. Ebba. 2010. Electronic complaints: An empirical study on British english and german complaints on ebay.
Saif Mohammad and Felipe Bravo-Marquez. 2017. WASSA-2017 shared task on emotion intensity. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 34-49, Copenhagen, Denmark. Association for Computational Linguistics.
E Olshtain and L Weinbach. 1985. Complaints: A study of speech act behavior among native and non-native speakers of hebrew. the pragmatic perspective.
Jiaxin Pei and David Jurgens. 2020. Quantifying intimacy in language. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5307-5326, Online. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Daniel Preotiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically identifying complaints in social media. arXiv preprint arXiv:1906.03890.
Antonio Reyes, Paolo Rosso, and Davide Buscaldi. 2012. From humor recognition to irony detection: The figurative language of social media. Data & Knowledge Engineering, 74:1-12.
Gabor Szabo and Bernardo A Huberman. 2010. Predicting the popularity of online content. Communications of the ACM, 53(8):80-88.
Yi-jie Tang and Hsin-Hsi Chen. 2014. Chinese irony corpus construction and ironic structure analysis. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1269-1278.
Anna Trosborg. 2011. Interlanguage pragmatics: Requests, complaints, and apologies, volume 7. Walter de Gruyter.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. 2018. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 39-50.
Vorakit Vorakitphan, Marco Guerini, Elena Cabrio, and Serena Villata. 2020. Regexit or not regexit: Aspect-based sentiment analysis in polarized contexts. In The 28th International Conference on Computational Linguistics (COLING'2020).
Zhongyu Wei, Junwen Chen, Wei Gao, Binyang Li, Lanjun Zhou, Yulan He, and Kam-Fai Wong. 2018. An empirical study on uncertainty identification in social media context. In *Social Media Content Analysis: Natural Language Processing and Beyond*, pages 79–88. World Scientific.
Xu Xu, Jiayin Li, and Huilin Chen. 2021. Valence and arousal ratings for 11,310 simplified chinese words. Behavior Research Methods, pages 1-16.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
A More Sample Posts
We provide more sample posts in Table A1 from our dataset grouped according to the 5 bins defined in Table 3.
Table A1: More sample posts for each of the 5 bins. Words in bold are some points of concern.
B Data Used for Post Popularity Prediction
We collected blog posts under 8 topics from Weibo to verify the relationship between complaint density and popularity. Table A2 shows the hashtag contents, along with the number and time of collection.
Hashtag	Number	Start From
#巨人教育宣布倒闭# (#JuRen Education announces bankruptcy#)	1,663	2021/8/31
#官方回应开学典礼学生晕倒无人扶# (#The official responded to the situation where the student fainted and no one helped at the opening ceremony#)	331	2021/9/2
#芝加哥大学24岁中国留学生被枪杀# (#A 24-year-old Chinese student at the University of Chicago was shot dead#)	179	2021/11/11
#教育部将抑郁症筛查纳入学生体检# (#Ministry of Education incorporates depression screening into student physical examination#)	1,023	2021/10/31
#江苏一建停考# (#Jiangsu Province Level One Architect Examination Suspended#)	286	2021/8/24
#计算机二级证书有必要吗# (#Is it necessary to take the second-level computer certificate#)	242	2021/11/9
#西安外国语大学封闭管理# (#Closed management of Xi'an International Studies University#)	287	2020/9/19
#大连理工大学支教# (#Supporting Teaching at Dalian University of Technology#)	962	2021/3/8
Table A2: The hashtag and its number used in the application.