Venue: NAACL 2021

# A Million Tweets Are Worth a Few Points:

Tuning Transformers for Customer Service Tasks

Amir Hadifar\({}^{\dagger}\) Sofie Labat\({}^{\ddagger}\) Veronique Hoste\({}^{\ddagger}\)

Chris Develder\({}^{\dagger}\) Thomas Demeester\({}^{\dagger}\)

###### Abstract

In online domain-specific customer service applications, many companies struggle to deploy advanced NLP models successfully, due to the limited availability of and noise in their datasets. While prior research demonstrated the potential of migrating large open-domain pretrained models for domain-specific tasks, the appropriate (pre)training strategies have not yet been rigorously evaluated in such social media customer service settings, especially under multilingual conditions. We address this gap by (i) collecting a multilingual social media corpus containing customer service conversations (865k tweets), (ii) comparing various pipelines of pretraining and fine-tuning approaches, (iii) applying them on 5 different end tasks. We show that pretraining a generic multilingual transformer model on our in-domain dataset, before finetuning on specific end tasks, consistently boosts performance, especially in non-English settings.1

Footnote 1: Dataset and code available at [https://github.com/hadifar/customerservicetasks](https://github.com/hadifar/customerservicetasks)

_This paper is organized as follows. In Section 2 we describe the results of our experiments. In Section 3 we discuss the results of our experiments. In Section 4 we present the results of our experiments. In Section 5 we compare the results of our experiments with those of our experiments. In Section 6 we summarize the results of our experiments. In Section 7 we provide some conclusions. Finally, in Section 8 we conclude the results of our experiments. In Section 9 we discuss the results of our experiments. In Section 10 we discuss the results of our experiments. In Section 11 we discuss the results of our experiments. In Section 12 we discuss the results of our experiments. In Section 13 we discuss the results of our experiments. In Section 14 we discuss the results of our experiments. In Section 15 we discuss the results of our experiments. In Section 16 we discuss the results of our experiments. In Section 17 we discuss the results of our experiments. In Section 18 we discuss the results of our experiments. In Section 19 we discuss the results of our experiments. In Section 19 we discuss the results of our experiments. In Section 10 we discuss the results of our experiments. In Section 11 we discuss the results of our experiments. In Section 11 we discuss the results of our experiments. In Section 12 we discuss the results of our experiments. In Section 13 we discuss the results of our experiments. In Section 14 we discuss the results of our experiments. In Section 15 we discuss the results of our experiments. In Section 16 we discuss the results of our experiments. In Section 17 we discuss the results of our experiments. In Section 18 we discuss the results of our experiments.



customer service database for decision support and machine fault diagnosis. Gupta (2011) extracted a set of sentiment and syntactic features from tweets for customer problem identification tasks. Molino et al. (2018) introduced the Customer Obsession Ticket Assistant for ticket resolution, using feature engineering techniques and encoder-decoder models. Highly popular pretrained language models, such as BERT Devlin et al. (2019), have also been explored for different customer service tasks: Hardalov et al. (2019) considered re-ranking candidate answers in chatbots, while Deng et al. (2020) proposed BERT-based topic prediction for incoming customer requests.

Although the performance gains obtained by pretraining language models are well-established, they need further exploration in terms of multilinguality. Some studies Pires et al. (2019); Karthikeyan et al. (2019); Wu et al. (2019) have investigated the transferability of multilingual models on different tasks, but they do not consider the effect of progressive pretraining on a smaller and less diverse multilingual corpus, as we will do.

## 3 Methodology

### Architecture

We selected some of the most popular publicly available pretrained language models to explore transfer learning properties for CRM classification tasks: RoBERTa Liu et al. (2019), XLM Conneau et al. (2020), and BERTweet Nguyen et al. (2020). These models are pretrained on the English Wikipedia and BookCorpus Zhu et al. (2015), CommonCrawl in 100 languages, and 850M English tweets, respectively. The XLM and BERTweet pretraining procedure is based on RoBERTa, which itself is a transformer-based Masked Language Model MLM; Devlin et al. (2019). All of these models require a different classifier 'head' for each target task to estimate the probability of a class label.

### Transfer Strategies

We adopt a straightforward approach to transfer learned representations: we continue pretraining the considered transformer models on a 4-lingual corpus of customer service Twitter conversations (see Section 4.1), i.e., the overall domain of all considered sub-tasks. After that, we apply additional adaptation for cross-lingual transfer Section 5.1), as well as cross-task transfer Section 5.2).

The following notations are used throughout the rest of this paper to describe pretraining stages:

* further **p**retraining the original MLM on our 4-lingual tweet conversation corpus.
* **f**inetuning the pretrained model extended with the MLP classifier on the target task
* freezing the pretrained model (i.e., feature extraction mode), only training the top classifier on the target task.

We thus indicate several multistage procedures: e.g., XLM\(\rightarrow\pi\to\varphi\) indicates that the XLM model is further pretrained on the in-domain tweet corpus, followed by finetuning on the end task.

## 4 Experimental Setup

We focus our experiments on text classification problems that are commonly dealt with by customer service teams. First, we describe our Twitter conversation corpus used for in-domain finetuning (Section 4.1), then we introduce the target tasks and corresponding datasets Section 4.2). For most target tasks, we hold out 10% of the data for testing, while the remaining part is used for training. We then utilize 10-fold cross-validation on the training data to select optimal hyper-parameters for each end task. When the dataset comes with a predefined train-test split, we keep that. For the pretrained transformer models RoBERTa, XLM, BERTweet), we use the publicly available 'base' versions.

### Twitter Conversation Corpus

Our corpus for in-domain prefrtaining was crawled using Twitter's API.2 The collected dataset is small compared to the original language models' data, but still larger than most corpora which SMEs have at their disposal. As such, it represents an easily collectable customer service dataset that SMEs can leverage to boost models on their own data. The tweets were gathered between May and October 2020. We started by gathering a list of 104 companies, all active on Twitter, in the sectors of telecommunication, public transportation, and airlines. We aimed for four different languages (English, French, Dutch, German).

Footnote 2: [https://developer.twitter.com/en/docs](https://developer.twitter.com/en/docs)

We preprocessed the data by removing conversations not covering at least one client/company interaction, or containing undefined languages. We further converted usernames and links into the special tokens @USER and @HTTP_URL, respectively, 

and converted emojis into corresponding strings.3 The resulting corpus contains 865k tweets over 275k conversations in the four target languages (see Table 1).

Footnote 3: We used [https://github.com/carpedm20/emoji](https://github.com/carpedm20/emoji) to convert emojs

Even though our corpus contains data from different sectors, we noticed that the dialogue flow, customer intents, and structure of conversations are fairly comparable across the target sectors (cf. Fig. 1). Examples of often recurring types of tweets are expressions of gratitude towards customers, requests for information, or typical ways to reply to complaints. Hence, we expect this corpus to be useful not only for companies that fall under one of the included sectors, but also for other companies that provide customer services over tweets.

### CRM Tasks and Datasets

**Complaint Prediction -** Timely complaint detection is of utmost importance to organizations, as it can improve their relationship with customers and prevent customer churns. Preqiuc-Pietro et al. (2019) and Greenleaf et al. (2015) proposed two datasets for identifying complaints on social media which contain 3,499 and 5,143 instances, respectively. The former (**Complain-2**) covers two types of companies (airline companies and telecommunication), while the latter (**Complain-9**) consists of data from nine domains such as food, car, software, etc. Both datasets are in English. To experiment with cross-lingual tuning for complaint prediction, we use the French complaint dataset for railway companies from (**Complain-R**; Ruytenbeek et al. 2020). Since all their 201 conversations are labeled as complaints, for training, we complemented them with negative sampling from French railway conversations in our own Twitter corpus. For testing, we annotated 200 held-out conversations.

**Churn prediction -** Customer churn implies that a customer stops using a company's service, negatively impacting its growth. Churn prediction is cast as a binary classification task (churn or non-churn) on any input text. We utilize the data provided by Amiri and Daume III (2015) with tweets from three telecommunication brands, resulting in a corpus of 4,339 labelled English tweets.

**Subjectivity Prediction -** Detecting subjectivity in conversations is a key task for companies to efficiently address negative customer feelings or reward loyal customers. It may also serve as a filtering task for more fine-grained tasks such as emotion identification. We annotated 8,174 Dutch conversations from our Twitter corpus (Section 4.1). A dialogue is judged "subjective" if at least one of the customer turns contains emotions (explicit or implicit), and otherwise "objective".

**Relevance Prediction -** The goal of this task is to determine whether an incoming text is relevant for further processing or not. We use data from GermEval 2017 (Task A) which contains over 28k short length messages from various social media and web sources on the German public train operator Deutsche Bahn Wojatzki et al. (2017). For this dataset, the evaluation is measured on two evaluation sets: one collected from the same time period as the training and development set (viz. synchronic), and another one containing data from a later time period (viz. diachronic).

**Polarity Prediction -** For this task, a system has to classify the sentiment that resides in a given text fragment according to polarity (positive, negative, or neutral). Polarity prediction has often been applied on reviews, by predicting the attitude or senti

\begin{table}
\begin{tabular}{c c|r r} \hline \hline Language & \multicolumn{1}{l}{\(|\text{convs}|\)} & \multicolumn{1}{l}{\(|\text{tweets}|\)} \\ \hline English & \(135.1\)k & \(406.3\)k \\ French & \(60.9\)k & \(212.9\)k \\ Dutch & \(45.6\)k & \(141.0\)k \\ German & \(33.0\)k & \(104.5\)k \\ \hline All & \(274.6\)k & \(864.7\)k \\ \hline \hline \end{tabular}
\end{table}
Table 1: Number of collected conversations (\(|\text{convs}|\)) and tweets (\(|\text{tweets}|\)) for each language.

Figure 1: 2D visualization of the first hidden-state representations of the XLM on randomly sampled tweets for six companies, in three different sectors using t-SNE Van der Maaten and Hinton (2008). Each color stands for a different sector, while the marker types represent the different companies, and each data point corresponds to a single operator tweet (covering multiple languages).

 

ment of a reviewer with respect to some topic. We use the GermEval 2017 (Task B) dataset (Wojatzki et al., 2017) (cf. supra) to analyze the polarity of the Deutsche Bahn customers' feedback. We also use the polarity dataset from Sanders (2011) (**Sanders**).

## 5 Results and Discussion

We now present our findings for two finetuning scenarios: transfer across languages and across tasks. Section 5.1 investigates the effect of unsupervised multilingual pretraining. Section 5.2 then explores how to further improve by finetuning the pretrained language models on similar tasks.

### Language Transferability

We compare the pretrained transformer experiments with the following baselines: majority-class (to get an idea of class imbalance), logistic regression (LR) and support vector machine (SVM) with tf-idf features. For the three transformer models, we compare the feature extraction setting \((\rightarrow{\varnothing})\) with finetuning \((\rightarrow{\varphi})\) on the target task. On the multilingual XLM, we measure the impact of first pretraining \((\rightarrow\pi)\) on our multilingual tweet corpus, after which both transfer settings are again tested on the target tasks. Table 2 reports the results (in terms of accuracy and F1 scores), including scores from literature when available ('Reference'). It should be noted that the reference scores are not state-of-the-art, but they are the scores communicated in the original dataset papers.

Only for the English tasks (Complaint-2 and Churn), results for BERTweet and RoBERTa are reported. The monolingual tweet-based model BERTweet outperforms all other models when fine-tuned on these tasks. Although a large domain-specific mono-lingual language model seems a fine choice, it may not be available for other languages. We therefore investigate the impact of a multilingual generic model (XLM was not specifically pretained on tweets), and the impact of additional finetuning on our dedicated twitter corpus.

In general, transformer models finetuned on the end task strongly outperform frozen ones. For the non-English tasks, the model XLM\(\rightarrow{\varnothing}\) with the frozen XLM encoder shows weak performance, in some cases below the baselines. The model XLM\(\rightarrow{\varphi}\) finetuned on the end task performs better. For the non-English tasks, the XLM model pretrained on our Twitter corpus and finetuned on the tasks (XLM\(\rightarrow{\pi}{\rightarrow}\)\({\varphi}\)) in all cases outperforms the finetuned XLM by a few percentage points and the baselines by an even larger margin. The performance differences between XLM\(\rightarrow{\varphi}\) and XLM\(\rightarrow{\pi}{\rightarrow}\)\({\varphi}\) clearly underscore the importance of in-domain multilingual pretraining. Furthermore, the results of XLM\(\rightarrow{\pi}{\rightarrow}\)\({\varphi}\) for the English tasks suggest that additional pretraining on a moderately small, in-domain dataset can make the performance of the multilingual XLM model comparable to the monolingual RoBERTa.

\begin{table}
\begin{tabular}{l c c c c c c c c c c c c} \hline \hline  & \multicolumn{3}{c}{**Complaint-2**} & \multicolumn{3}{c}{**Complaint-R**} & \multicolumn{2}{c}{**Churn**} & \multicolumn{2}{c}{**Subjectivity**} & \multicolumn{2}{c}{**Relevance**} & \multicolumn{2}{c}{**Polarity**} \\  & \multicolumn{3}{c}{(English)} & \multicolumn{3}{c}{(French)} & \multicolumn{3}{c}{(English)} & \multicolumn{3}{c}{(Dutch)} & \multicolumn{3}{c}{(German)} & \multicolumn{2}{c}{(German)} \\ \cline{2-14}

- & 78.3\({}^{[2]}\) &
- & 85.2\({}^{[3]}\) & 86.8\({}^{[3]}\) & 66.7\({}^{[3]}\) & 69.4\({}^{[3]}\) \\ \hline BERweet \(\rightarrow{\varnothing}\) & 80.5 & 71.6 &
- & 79.3 & 55.2 &
- \\ BERweet \(\rightarrow{\varphi}\) & **90.0** & **86.1** &
- & **93.0** & **90.0** &
- \\ RoBERTa \(\rightarrow{\varnothing}\) & 77.9 & 74.5 &
- & 78.3 & 59.7 &
- \\ RoBERTa \(\rightarrow{\varphi}\) & 87.5 & 85.1 &
- & 88.4 & 84.8 &
- \\ KLM \(\rightarrow{\varnothing}\) & 76.2 & 61.6 & 44.0 & 30.5 & 61.6 & 55.7 & 63.8 & 62.4 & 83.1 & 84.7 & 64.5 & 66.8 \\ XLM \(\rightarrow{\varphi}\) & 85.4 & 83.4 & 54.0 & 46.2 & 84.1 & 75.3 & 73.4 & 72.9 & 91.6 & 91.7 & 76.1 & 73.5 \\ XLM \(\rightarrow{\pi}{\rightarrow}{\varnothing}\) & 81.8 & 76.8 & 56.5 & 54.1 & 79.7 & 66.0 & 71.6 & 71.1 & 84.4 & 85.3 & 65.1 & 68.0 \\ XLM \(\rightarrow{\pi}{\rightarrow}{\varphi}\) & 86.9 & 82.7 & **62.0** & **61.9** & 87.8 & 83.7 & **74.6** & **74.2** & **92.7** & **92.5** & **78.7** & **76.1** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Classification results (accuracy ACC and F1-score) on CRM tasks using pretrained language models with two settings for pretraining: Feature extraction \((\rightarrow{\varnothing})\) and finetuning \((\rightarrow{\varphi})\). Missing values (‘-’) are due to unavailable reference scores, or a language mismatch between model and task.
 

Another promising observation is that the hyper-tuned classical baselines, such as SVM, are strong competitors compared to _frozen_ language models, especially on tasks that are highly sensitive to domain-specific features. For instance, for churn prediction, keywords such as 'switch to', 'quit' and 'change provider' can easily be triggered by the SVM, while _frozen_ pretrained models have not learned to identify these features. This finding might be helpful to achieve better insight into the operational aspects of _frozen_ neural models compared to simple classical approaches.

As a side result (not explicitly included in this work) we found that the multistage pretraining (XLM\(\rightarrow\pi\)) leads to better performance when incorporating multiple languages compared to a single language. The performance drops especially when training data from a single language (e.g., Dutch) is fed into the model, which is then evaluated on other languages (e.g., English).

### Task Transferability

We now investigate to what extent representations tuned on a related task can help for a given target task. In particular, Complaint-9 is the end task, and we compare the effect of finetuning on the end task only, vis-a-vis first finetuning on a related task and then on the end task. For the related task, we experiment with Complaint-2 and Sanders, as shown in Table 3. We observe that there seems to be no clear merit in the additional finetuning step on a small related end task. Pretraining on our larger Twitter corpus, however, still increases effectiveness.

## 6 Conclusion

We investigated multilingual and across-task transfer learning for customer support tasks, based on transformer-based language models. We confirmed prior insights that finetuning the models on low-resource end tasks is important. Additional pretraining on a moderately sized in-domain corpus, however, provides a complementary increase in effectiveness, especially in the non-English setting and starting from a generic multilingual language model. We provide a newly collected multilingual in-domain corpus for customer service tasks and derive the aforementioned findings from experiments using it on five different tasks.

## 7 Acknowledgments

This research received funding from the Flemish Government under the Research Program Artificial Intelligence - 174B09119. We would also like to thank the anonymous reviewers for their valuable and constructive feedback.

## References

* Amiri and Daume III (2015) Hadi Amiri and Hal Daume III. 2015. Target-dependent churn classification in microblogs. In _Proceedings of AAAI_.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhry, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of ACL_.
* Cui et al. (2017) Lei Cui, Shaohan Huang, Furu Wei, Chuanqi Tan, Chaoqun Duan, and Ming Zhou. 2017. SuperAgent: A customer service chatbot for e-commerce websites. In _Proceedings of ACL_.
* Deng et al. (2020) Xin Deng, Ross Smith, and Genevieve Quintin. 2020. Semi-supervised learning approach to discover enterprise user insights from feedback and support. _arXiv preprint arXiv:2007.09303_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT_.
* Greenleaf et al. (2015) Dylan Greenleaf, Jianze Liu, Yuanbo Wang, and Antoine Rigaut. 2015. Final project report: Identifying customer complaints on social media.
* Gupta et al. (2010) Narendra Gupta, Mazin Gilbert, and Giuseppe Di Fabbrizio. 2010. Emotion detection in email customer care. In _Proceedings of WS_.
* Gupta (2011) Narendra K Gupta. 2011. Extracting descriptions of problems with product and services from twitter data. In _Proceedings of SWSM_.
* Gulrajani et al. (2018)

\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Test Dataset** & \multicolumn{3}{c}{**Complaint-9**} \\ \hline

- \\ SVM (tf-idf) & 78.6 &
- \\ Preotiuc-Pietro et al. & 79.0 &
- \\ \hline XLM \(\rightarrow\varphi\) & 78.6 & 79.3 & 80.1 \\ XLM \(\rightarrow\pi\)\(\rightarrow\varphi\) & 82.4 & 80.0 & 82.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: F1 test scores on Complaint-9 for finetuning on the end task alone (indicated as **C-9**) vs. on Complaint-2 or Sanders and again on Complaint-9 (columns **C-2 \& C-9**, respectively, **S \& C-9**.).
 

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: adapt language models to domains and tasks. In _Proceedings of ACL_.
* Hardalov et al. (2019) Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2019. Machine reading comprehension for answer re-ranking in customer support chatbots. _Information_.
* Hui and Jha (2000) Siu Cheung Hui and G Jha. 2000. Data mining for customer service support. _Information & Management_.
* Karthikeyan et al. (2019) Kaliyaperumal Karthikeyan, Zihan Wang, Stephen Mayhew, and Dan Roth. 2019. Cross-lingual ability of multilingual bert: An empirical study. In _Proceedings of ICLR_.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danoj Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Molino et al. (2018) Piero Molino, Huaixiu Zheng, and Yi-Chia Wang. 2018. Cota: improving the speed and accuracy of customer support through ranking and deep networks. In _Proceedings of ACM-SIGKDD_.
* Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020. BERTweet: a pre-trained language model for English Tweets. In _Proceedings of EMNLP_.
* Pan and Yang (2009) Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. _IEEE Transactions on knowledge and data engineering_.
* Pires et al. (2019) Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual bert? In _Proceedings of ACL_.
* Preotiuc-Pietro et al. (2019) Daniel Preotiuc-Pietro, Mihaela Gaman, and Nikolaos Aletras. 2019. Automatically identifying complaints in social media. In _Proceedings of ACL_.
* Ruder (2019) Sebastian Ruder. 2019. _Neural Transfer Learning for Natural Language Processing_. Ph.D. thesis, National University of Ireland, Galway.
* Ruytenbeek et al. (2020) Nicolas Ruytenbeek, Ilse Depraetere, and Sofie Decock. 2020. Linguistic (in)directness in twitter complaints: a contrastive analysis of railway complaint interactions. _Journal of pragmatics_.
* Sanders (2011) Niek J Sanders. 2011. Sanders-twitter sentiment corpus. _Sanders Analytics LLC_.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_.
* Wojatzki et al. (2017) Michael Wojatzki, Eugen Ruppert, Sarah Holschneider, Torsten Zesch, and Chris Biemann. 2017. GermEval 2017: shared task on aspect-based sentiment in social media customer feedback. In _Proceedings of GermEval_.
* Wu et al. (2019) Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Emerging cross-lingual structure in pretrained language models. In _Proceedings of ACL_.
* Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In _Proceedings of ICCV_.