SlowGuess/NAACL_2021_OCR huggingface dataset:
We explore whether or not the severity level of a complaint helps in complaint identification. We use the same data set as Preotiuc-Pietro et al. (2019), where each tweet is annotated as a complaint or not and our severity level annotations.<sup>7</sup>
8.1 Predictive models
We first adapt three multi-task learning models based on bidirectional recurrent neural networks recently proposed by Rajamanickam et al. (2020) for jointly modeling abusive language detection and emotion detection. We also adapt our M-RoBERTa $_{Emo}$  model in a multi-task setting using two variants. We use the severity complaint prediction as an auxiliary task and the binary complaint prediction as the main task to train different MTL models. All models are trained on the two tasks and updated at the same time with a joint loss:
$$
L = (1 - \alpha) L _ {c o m} + \alpha L _ {s e v}
$$
where  $L_{com}$  and  $L_{sev}$  are the losses of complaint identification and severity level classification tasks respectively.  $\alpha$  is a parameter to control the importance of each loss.
MTL-Hard Sharing We adapt the MTL-Hard Sharing model of Rajamanickam et al. (2020), where a single encoder is shared and updated by both tasks. We first pass GloVe embedding representations to a shared stacked BiGRU encoder. Then the output of the shared encoder is fed to two different BiGRU-Att models specific to each task (complaint detection and severity level identification) separately. Finally, we add an output layer with a sigmoid and a softmax activation function for binary and multi-class prediction respectively.
MTL-Double Encoder Instead of sharing a single encoder, the MTL-Double Encoder model (Rajamanickam et al., 2020) utilizes two stacked BiGRU encoders, where one is task-specific (complaint detection only) and the another one is shared by both tasks. We pass the output of the shared encoder to a BiGRU-Att model for severity level prediction. We also concatenate the output of the task-specific and shared encoder and pass it to another BiGRU-Att model for complaint prediction. The rest of the architecture is the same as the MTL-Hard Sharing model.
MTL-Gated Double Encoder The MTL-Gated Double Encoder model (Rajamanickam et al., 2020) has the same architecture as the MTL-Double Encoder. The outputs from two stacked BiBRU-Att encoders are concatenated by assigning a weight to each representation  $[(1 - \beta)$  for the output of the task-specific encoder layer and  $\beta$  for the output of the shared one)] that controls the importance of the two representations.
MTL-M-RoBERTa $_{Emo}$  We also adapt our best performing model in the severity prediction task (M-RoBERTa $_{Emo}$ ) to support multi-task learning by adding an extra output layer for binary complaint prediction (MTL-M-RoBERTa $_{Emo}$ ).
MTL-M-RoBERTa $_{Emo}$ -DE We pass the M-RoBERTa $_{Emo}$  embedding to two separate RoBERTa encoders, i.e. double encoder (DE), followed by two classifiers for binary complaint and severity level prediction (MTL-M-RoBERTa $_{Emo}$ -DE).
8.2 Experimental Setup
Baselines We compare MTL models with the following baselines on binary complaint identification: (1) Logistic Regression with bag-of-words using distant supervision $^{8}$  (LR-BOW + DS) (Preotiuc-Pietro et al., 2019); (2) A standard BiGRU-Att model (BiGRU-Att); (3) A RoBERTa base model without combining linguistic information (RoBERTa) (Jin and Aletras, 2020); (4) A BERT base model without combining linguistic information (BERT) which has been shown to achieve state-of-the-art results in binary complaint identification (Jin and Aletras, 2020); and (5) replacing M-RoBERTa $_{Emo}$  with BiGRU-Att in the MTL-M-RoBERTa $_{Emo}$  and MTL-M
$^{8}$ Distant supervision method uses larger 'noisy' data to further boost the performance of the model
For a tweet that is a non-complaint, we assign an extra class for severity (i.e. 'No Complaint Severity').
Model	Acc	P	R	F1
Single-task Learning
LR-BOW+DS (Preotiuc-Pietro et al., 2019)	81.2	-	-	79.0
BiGRU-Att	79.2 ± .05	79.2 ± .06	74.5 ± .05	74.5 ± .05
RoBERTa (Jin and Aletras, 2020)	87.6 ± .03	86.6 ± .03	86.9 ± .03	86.6 ± .03
BERT (Jin and Aletras, 2020)	88.0 ± .03	87.1 ± .03	87.3 ± .04	87.0 ± .03
Multi-task Learning
MTL-BiGRU-Att	77.2 ± .05	75.4 ± .04	75.7 ± .04	75.4 ± .04
MTL-BiGRU-Att-DE	75.7 ± .05	74.1 ± .05	74.6 ± .04	74.1 ± .05
Rajamanickam et al. (2020)
MTL-Hard Sharing	75.2 ± .04	73.5 ± .05	71.5 ± .04	72.1 ± .05
MTL-Double Encoder	74.6 ± .03	72.7 ± .04	71.7 ± .03	72.0 ± .04
MTL-Gated Double Encoder	74.7 ± .03	73.4 ± .04	70.4 ± .03	71.1 ± .03
Ours
MTL-M-RoBERTaEmo	89.0 ± .04	88.2 ± .03	88.4 ± .03	88.2† ± .03
MTL-M-RoBERTaEmo-DE	88.9 ± .04	88.3 ± .04	88.3 ± .03	88.1 ± .04
Table 4: Accuracy (Acc), Precision (P), Recall (R) and macro F1-Score (F1) for binary complaint prediction (± std. dev.). Best results are in bold.  $\dagger$  indicates statistically significant improvement over BERT (Jin and Aletras, 2020) in STL (t-test,  $p < 0.05$ ).
RoBERTa  $E_{mo}$ -DE models (MTL-BiGRU-Att and MTL-BiGRU-Att-DE).
Hyperparameters We train the MTL-BiGRU-Att and MTL-BiGRU-Att-DE model with the same hyperparameters as BiGRU-Att in complaint severity prediction. For the MTL-Hard Sharing, MTL-Double Encoder and MTL-Gated Double Encoder model, the hidden size of the stacked BiGRU encoder(s) and BiGRU-Att models is  $h = 128$ ,  $h \in \{64, 128, 256, 512\}$ . We set  $\beta$  in MTL-Gated Double Encoder and the remaining parameters in three models to be the same as Rajamanickam et al. (2020). We train MTL-M-RoBERTa  $E_{mo}$  and MTL-M-RoBERTa  $E_{mo}$ -DE with a learning rate  $l = 1e - 6$ ,  $l \in \{1e - 5, 5e - 6, 1e - 6\}$ . The rest of the parameters is the same as M-RoBERTa  $E_{mo}$  in the complaint severity prediction. The parameter  $\alpha$  which controls the importance of the two losses is set to .1,  $\alpha \in \{.001, .01, .1, .3, .5\}$ .
8.3 Results
Table 4 shows results of the single-task learning (STL) and multi-task learning (MTL) models on the complaint identification task. Overall, we observe that all MTL models using M-RoBERTa  ${}_{E{mo}}$  perform better than the majority of STL models, indicating severity detection improves binary complaint identification. MTL-M-RoBERTa  ${}_{E{mo}}$  outperforms all other models achieving 88.2 macro F1, followed by MTL-M-RoBERTa  ${}_{E{mo}}$  -DE with
88.1 F1. This confirms our hypothesis that complaint identification can be benefited by the complaint severity level information when jointly learning these two tasks simultaneously. Also, MTL-BiGRU-Att performs better than BiGRU-Att in STL achieving 75.4 F1 while the results of BiGRU-Att (74.5 F1) and MTL-BiGRU-Att-DE (74.1 F1) are comparable. We notice that the models proposed by Rajamanickam et al. (2020) (i.e. MTL-Hard sharing, MTL-Double Encoder and MTL-Gated Double Encoder) achieve low performance with only the MTL-Hard Sharing model performing slightly better than the others with 72.1 macro F1. We speculate that adding one or more extra BiGRU encoders before the BiGRU-Att model is an overly complex structure for our data set.
8.4 Analysis
We investigate the influence of recognizing severity levels of complaints on binary complaint identification in our MTL setting. We analyze predictive results by inspecting predictions from the previous best performing model BERT (STL) and MTL-M-RoBERTa $_{Emo}$  models in a random fold (out of 10 CV folds). We observe that  $9.8\%$  of predictions flip, where the number of complaints flipping to non-complaints is noticeably larger  $(88.2\%)$  than that of non-complaints flipping to complaints  $(11.8\%)$ . Similarly, we also compare predicted results between BiGRU-Att (STL) and MTL-BiGRU-Att in the same fold. The flipping percentage  $(6.9\%)$  is lower than BERT and MTL-M-RoBERTa $_{Emo}$  while the proportions of one class flipping to an
<sup>9</sup>We experiment with all MTL models using the same training and evaluation method as in the complaint severity prediction task.
Tweet	BERT	MTL-M-RoBERTaEmo	Actual Label	Severity Label
What's your secret to poaching eggs?Mine never look that good.	Complaint	Non-complaint	Non-complaint	No Complaint Severity
<URL> How bad do you really want a ps4 this year? Get a pre-owned playsta-tion 4 at a very low dis <URL>	Complaint	Non-complaint	Non-complaint	No Complaint Severity
So, I'm now having to check my <USER> forester's oil each month. Put 4 quarts in today, got about 2 out. #smh	Non-complaint	Complaint	Complaint	Disapproval
Is this how you fix the exhaust of your <USER> in #belarus? <URL>	Non-complaint	Complaint	Complaint	Blame
Table 5: Complaint classification examples by BERT (Jin and Aletras, 2020) and our MTL-M-RoBERTa  ${}_{E{mo}}$  compared to the actual labels.
other are consistent (83.4% and 16.6% respectively). These indicate that complaint severity information encapsulates complementary information for the model to predict non-complaints accurately.
Table 5 shows flipping examples from BERT (STL) and MTL-M-RoBERTa  $E_{mo}$ . From the first two rows, we see that the MTL model is not affected by negation (e.g. 'never') and negative terms (e.g. 'bad', 'very low') using the extra knowledge provided by the severity level prediction task. Also, in the last two examples, complaints are expressed in a more subtle way that rarely contains typical complaint-related terms. This indicates the MTL model is able to detect this type of complaints correctly because the severity level information encourages the model to learn to distinguish between such stylistic idiosyncrasies.
We further observe that  $11.2\%$  of wrong predictions remain the same for the two models, where complaints and non-complaints account for  $59.0\%$  and  $41.0\%$  respectively which means severity features benefit more posts that are complaints to be classified accurately. On the other hand, the model still has difficulty in predicting some non-complaint posts which might happen because of the lower performance of severity detection when used as an auxiliary task in the MTL setting.
We presented the first study on severity level of complaints in computational linguistics. We developed a publicly available data set of tweets labeled with four categories based on theory of pragmatics. We modeled complaint severity level prediction as a new multi-class classification task and conducted experiments using different transformer-based networks combined with linguistic features reaching
up to 55.7 macro F1. We further used a multi-task learning setting to jointly model binary complaint prediction and complaint severity classification as an auxiliary task achieving new state-of-the-art performance on complaint detection (88.2 macro F1). In the future, we plan to apply our methods on a multilingual setting across different platforms.
We would like to thank Areej Alokaili, Daniel Preotiuic-Pietro and Danae Sanchez Villegas for their helpful feedback. NA is supported by ESRC grant ES/T012714/1.
Our work has received approval from the Ethics Committee of our institution (Ref. No 031504) and complies with Twitter data policy for research.[11]
Md Shad Akhtar, Asif Ekbal, and Erik Cambria. 2020. How intense are you? Predicting intensities of emotions and sentiments using stacked ensemble. IEEE Computational Intelligence Magazine, 15(1):64-75.
Irean Navas Alejo, Toni Badia, and Jeremy Barnes. 2020. Cross-lingual Emotion Intensity Prediction. arXiv preprint arXiv:2004.04103.
Nikolaos Aletras and Benjamin Paul Chamberlain. 2018. Predicting twitter user socioeconomic attributes with network and language information. In Proceedings of the 29th ACM International Conference on Hypertext and Social Media, pages 20-24.
Ron Artstein and Massimo Poesio. 2008. Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555-596.
$^{10}$ Severity prediction is less accurate in MTL than in a single task setting.
<sup>11</sup>https://developer.twitter.com/en/developer-terms/agreement-and-policy
Shreesh Kumara Bhat and Aron Culotta. 2017. Identifying leading indicators of product recalls from online reviews using positive unlabeled learning and domain adaptation. In Eleventh International AAAI Conference on Web and Social Media.
Rich Caruana. 1997. Multitask Learning. Machine learning, 28(1):41-75.
Kyunghyun Cho, Bart Van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. arXiv preprint arXiv:1406.1078.
Kristof Coussement and Dirk Van den Poel. 2008. Improving Customer Complaint Management by Automatic Email Classification Using Linguistic Style Features as Predictors. Decision Support Systems, 44(4):870-882.
Toner Danisman and Adil Alpkocak. 2008. Feeler: Emotion Classification of Text Using Vector Space Model. In AISB 2008 Convention Communication, Interaction and Social Intelligence, volume 1, page 53.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), page 4171-4186.
Paul Ekman. 1992. An Argument for Basic Emotions. Cognition & Emotion, 6(3-4):169-200.
Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
J Forster and B Entrup. 2017. A Cognitive Computing Approach for Classification of Complaints in the Insurance Industry. In IOP Conference Series: Materials Science and Engineering, volume 261, page 012016. IOP Publishing.
D Gunawan, RP Siregar, RF Rahmat, and A Amalia. 2018. Building Automatic Customer Complaints Filtering Application based on Twitter in Bahasa Indonesia. In Journal of Physics: Conference Series, volume 978, page 012119. IOP Publishing.
Jiahua Jin, Xiangbin Yan, You Yu, and Yijun Li. 2013. Service Failure Complaints Identification in Social Media: A Text Classification Approach.
Mali Jin and Nikolaos Aletras. 2020. Complaint identification in social media with transformer networks. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1765-1771.
Leila Nasiri Kakolaki and Mohsen Shahrokhi. 2016. Gender Differences in Complaint Strategies among Iranian Upper Intermediate EFL Students.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
Janice Laksana and Ayu Purwarianti. 2014. Indonesian Twitter Text Authority Classification for Government in Bandung. In 2014 International Conference of Advanced Informatics: Concept, Theory and Application (ICAICTA), pages 129-134. IEEE.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv preprint arXiv:1907.11692.
Antonios Maronikolakis, Danae Sánchez Villegas, Daniel Preotciuc-Pietro, and Nikolaos Aletras. 2020. Analyzing political parody in social media. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4373-4384.
Febina Merson and Roseline Mary. 2017. A Text Mining Approach to Identify and Analyse Prominent Issues from Public Complaints. International Journal of Advanced Research in Computer and Communication Engineering, 6(3).
E Olshtain and L Weinbach. 1985. Complaints: A Study of Speech Act Behavior among Native and Nonnative Speakers of Hebrew. The Pragmatic Perspective.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Daniel Preotiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically Identifying Complaints in Social Media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5008-5019.
Daniel Preotciuc-Pietro, Vasileios Lampos, and Nikolaos Aletras. 2015. An analysis of the user occupational class through Twitter content. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1754-1764.
Wasifur Rahman, Md Kamrul Hasan, Amir Zadeh, Louis-Philippe Morency, and Mohammed Ehsan Hoque. 2019. M-BERT: Injecting Multimodal Information in the BERT Structure. arXiv preprint arXiv:1908.05787.
Santhosh Rajamanickam, Pushkar Mishra, Helen Yannakoudakis, and Ekaterina Shutova. 2020. Joint Modelling of Emotion and Abusive Language Detection. arXiv preprint arXiv:2005.14028.
H Andrew Schwartz, Salvatore Giorgi, Maarten Sap, Patrick Crutchley, Lyle Ungar, and Johannes Eichstaedt. 2017. Dlatk: Differential Language Analysis Toolkit. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 55-60.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019. How to Fine-Tune BERT for Text Classification? In China National Conference on Chinese Computational Linguistics, pages 194-206. Springer.
Donna Hurst Tatsuki. 2000. If my complaints could passions move: An interlanguage study of aggression. Journal of Pragmatics, 32(7):1003-1017.
Zhengxi Tian, Wenge Rong, Libin Shi, Jingshuang Liu, and Zhang Xiong. 2018. Attention Aware Bidirectional Gated Recurrent Unit Based Framework for Sentiment Analysis. In International Conference on Knowledge Science, Engineering and Management, pages 67-78. Springer.
Suhatati Tjandra, Amelia Alexandra Putri Warsito, and Judi Prajetno Sugiono. 2015. Determining Citizen Complaints to the Appropriate Government Departments Using KNN Algorithm. In 2015 13th International Conference on ICT and Knowledge Engineering (ICT & Knowledge Engineering 2015), pages 1-4. IEEE.
Anna Trosborg. 2011. Interlanguage pragmatics: Requests, complaints, and apologies, volume 7. Walter de Gruyter.
Adam Tsakalidis, Nikolaos Aletras, Alexandra I Cristea, and Maria Liakata. 2018. Nowcasting the stance of social media users in a sudden vote: The case of the greek referendum. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 367-376.
Guda Van Noort and Lotte M Willemsen. 2012. Online Damage Control: The Effects of Proactive Versus Reactive Webcare Interventions in Consumer-generated and Brand-generated Platforms. Journal of Interactive Marketing, 26(3):131-140.
Camilla Vásquez. 2011. Complaints Online: The Case of Tripadvisor. Journal of Pragmatics, 43(6):1707-1717.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in neural information processing systems, pages 5998-6008.
Svitlana Volkova and Yoram Bachrach. 2016. Inferring Perceived Demographics from User Emotional Tone and User-Environment Emotional Contrast. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1567-1578.
Yansen Wang, Ying Shen, Zhun Liu, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2019. Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7216-7223.
Anbang Xu, Zhe Liu, Yufan Guo, Vibha Sinha, and Rama Akkiraju. 2017. A New Chatbot for Customer Service on Social Media. In Proceedings of the Conference on Human Factors in Computing Systems (CHI), pages 3506-3510.
Wei Yang, Luchen Tan, Chunwei Lu, Anqi Cui, Han Li, Xi Chen, Kun Xiong, Muzi Wang, Ming Li, Jian Pei, et al. 2019. Detecting Customer Complaint Escalation with Recurrent Neural Networks and Manually-Engineered Features. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 56-63.
Yuxiang Zhang, Jiamei Fu, Dongyu She, Ying Zhang, Senzhang Wang, and Jufeng Yang. 2018. Text Emotion Distribution Learning via Multi-Task Convolutional Neural Network. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4595-4601.