RoBERTaIQ: An Efficient Framework for Automatic Interaction
Quality Estimation of Dialogue Systems
Saurabh Gupta

Xing Fan

Derek Liu

gsaur@amazon.com
Alexa AI, Amazon
Seattle, USA

fanxing@amazon.com
Alexa AI, Amazon
Seattle, USA

derecliu@amazon.com
Alexa AI, Amazon
Seattle, USA

Benjamin Yao

Yuan Ling

Kun Zhoubenjamy@amazon.com
Alexa AI, Amazon
Seattle, USA

yualing@amazon.com
Alexa AI, Amazon
Seattle, USA

zhouku@amazon.com
Alexa AI, Amazon
Seattle, USA

Tuan-Hung Pham

Chenlei Guo

hupha@amazon.com
Alexa AI, Amazon
Seattle, USA

guochenl@amazon.com
Alexa AI, Amazon
Seattle, USA

ABSTRACT
Automatically evaluating large scale dialogue systems’ response
quality is a challenging task in dialogue research. Existing automated turn-level approaches train supervised models on Interaction
Quality (IQ) labels or annotations provided by experts, which iscostly and time-sensitive. Moreover, the small quantity of annotated data limits the trained model’s ability to generalize to the longtail and out of domain cases. In this paper, we propose a learning
framework that improves the model’s generalizability by leveraging
various unsupervised data sources available in large-scale conversational AI systems. We mainly rely on the following three techniques
to improve the performance of dialogue evaluation models: First,
we propose extending the RoBERTa model to encode multi-turn dialogues to capture the temporal differences between different turns.
Second, we add two additional pretraining processes on top of enhanced multi-turn RoBERTa to take advantage of large quantity ofexisting historical dialogue data through self-supervised training.
Third, we perform fine-tuning on IQ labels in a multi-task learning
setup, leveraging domain-specific information from other tasks. We
show that the above techniques significantly reduce annotated datarequirements. We achieve the same F1 score on IQ prediction task
as our baseline with only 5% of IQ training data and further beat
the baseline by 5.4% absolute F1 score if we use all of the training
data.
ACM Reference Format:
Saurabh Gupta, Xing Fan, Derek Liu, Benjamin Yao, Yuan Ling, Kun Zhou,
Tuan-Hung Pham, and Chenlei Guo. 2021. RoBERTaIQ: An Efficient Framework for Automatic Interaction Quality Estimation of Dialogue Systems.
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
KDD ’21, Aug 14–18, 2021, Virtual Event© 2021 Association for Computing Machinery.
ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00
https://doi.org/10.1145/nnnnnnn.nnnnnnn

In Proceedings of DeMaL, 2𝑛𝑑 International Workshop on Data-Efficient
Machine Learning (KDD ’21). ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

INTRODUCTION

Large scale conversational agents like Amazon Alexa, Apple Siri,and Google Assistant have set a standard for conversational AI
with the ability to integrate seamlessly across a wide range offunctionalities. Such systems are complex in nature with many
sequential components, such as Automatic Speech Recognition(ASR), Natural Language Understanding (NLU), Dialogue Manager,
and Natural Language Generation. As the scope of these systems
is increasing to cover more scenarios and applications, it becomesvital to automatically evaluate the response quality of these agentsto estimate user satisfaction. In particular, identifying problematic
responses where the user was left dissatisfied can be useful inimproving dialogue agents over time with data driven learning[3, 16, 26].
Previous approaches for automated dialogue evaluation can beclassified into Dialogue-level or Turn-level, based on whether weare evaluating multiple exchanges at once or each exchange (user’sutterance and agent’s response) individually. The PARADISE (PARAdigm for DIalogue System Evaluation) framework is the most wellknown evaluation framework proposed for evaluating dialoguelevel user satisfaction [23]. In PARADISE, a linear regression model
is fitted to predict the dialogue-level user satisfaction for a given set
of manually extracted input features and user ratings. In contrast
to rating the dialogue as a whole, approaches such as Interaction
Quality (IQ) [19] were proposed to capture user satisfaction atturn level. Here, an SVM model [4] is learned based on the ratings
provided by human annotators, while the input features were automatically extracted based on interaction parameters and emotions.
More recent approaches [1, 2, 11] extend the IQ framework anduse models like Gradient Boosting Decision Tree (GBDT) [6], Recurrent Neural Networks (RNN) [17], and Long Short-Term Memory
Networks (LSTM) [18] to encode the dialogue session. However,in addition to using textual data, these approaches also rely on

KDD ’21, Aug 14–18, 2021, Virtual Event

Gupta et al.

Table 1: Examples of Dialogue Sessions with different turnlevel IQ labels (Unsatisfied = 1, Satisfied = 0)#

Timestamp(in seconds)

Dialogue Session

𝜏 =0

𝜏 =4

[USER] Stop
[AGENT] <null>𝜏 =0[USER] Play maj and dragons.
[AGENT] Sorry, I can’t find that.

1

𝜏 =0

[USER] Play hello[AGENT] Here’s Hello, by Pop Smoke.

1

𝜏 =6

[USER] Play hello by Adele[AGENT] Here’s hello by Adele

0

𝜏 =0

[USER] Play hello[AGENT] Here’s Hello, by Pop Smoke.

0𝜏 = 60[USER] Play hello by Adele[AGENT] Here’s hello by Adele

0

𝜏 =0

[USER] show me shark videos[AGENT] Here’s what I found (playing video)

1

𝜏 =8

[USER] play baby shark on amazon prime[AGENT] Here’s Baby Shark , by Pinkfong , on Amazon Music.

0

1

1

2

IQ label[USER] Play hello[AGENT] Here’s Hello, by Pop Smoke.

3

4

5

input features generated by internal components, such as NLU/ASR
confidence scores and Dialogue Status. These signals introduce
dependencies on internal components and force the model to besystem specific. As a result, our work does not leverage these signals; instead, we focus on more powerful model architectures thatcan capture user satisfaction using the textual and temporal information alone.
We hypothesize that user satisfaction can be inferred using explicit and implicit user/agent behaviors that exist in the dialogue
session. Dialogues 1 and 2 in Table 1 are examples of explicit user
and agent behavior, respectively. In dialogue 1, the user terminatedthe request as the agent did not play the right song. In dialogue
2, the agent failed to handle the request due to an error in entity
resolution, caused by ASR error. Dialogues 3 and 4 capture user’s
intention implicitly and highlight the importance of temporal information. In dialogue 3, the user did not intend to listen to Pop
Smoke and thus immediately interrupted the agent by rephrasing
the original request. However, in dialogue 4, the user listened to“Hello, by Pop Smoke” for 𝜏 = 60 seconds before issuing the next request. This arguably leads to the conclusion that the user intended
to listen to Pop Smoke and Adele thereafter. Dialogue 5 shows why
it is important to capture context from other turns. The agent’s
action in the first turn did not satisfy user’s requirement as the user
was looking for Baby Shark specifically.
These examples emphasize the significance of capturing dialogue context as precisely as possible, to correctly estimate user
satisfaction. From the perspective of offline dialogue evaluation,
the dialogue context should not only include the previous and thefollowing turns, but also the temporal differences between different
turns. To this end, we design a novel transformer based dialogue
encoder, so as to utilize its self-attention mechanism [22] acrosstokens of different turns of the dialogue, while making the modelaware of the temporal differences between turns. We build on topof RoBERTa encoder [13] and refer to our model as RoBERTaIQ.

Another major challenge imposed by automatic dialogue evaluation is collecting large amount of human annotations or IQ labels,
which can be costly and time consuming. Recent advances in pretraining using self-attention encoder architectures like BERT [5]and RoBERTa [13] have been commonly used in many NLP applications. Such models are usually trained on massive general text
corpora like English Wikipedia. However, the underlying differenceof linguistic patterns between general text and dialogues makes
existing pretrained language models less useful in practice. Wu et al.
[24] have successfully shown that pretraining for task-oriented dialogues can be more useful than using general pretrained language
models. However, there are only a few related works that leverage
pretraining for automated dialogue evaluation. Liang et al. [10]learn dialogue feature representation with a self-supervised dialogue flow anomaly detection task, while Sinha et al. [20] train text
encoders via noise contrastive estimation (NCE) [8]. Inspired by the
success of domain-adaptive (DAPT) and task-adaptive pretraining(TAPT) [7, 9], we adopt the multi-stage pretraining process on large
scale historical dialogue data and IQ task training data. Furthermore,
we make our training process more data efficient by following theMulti-Task DNN learning framework for NLU [12]. We cast ourlearning process in a multi-task setting leveraging large amountsof cross-task data and regularization benefits. When fine-tuning,
we learn to jointly predict turn-wise IQ label, domain and intent.
The domain and intent signals are obtained from a separate NLU
classification system and do not introduce additional annotation
costs.
In summary, we make the following contributions:• We design a novel transformer based dialogue encoder: RoBERTaIQ for inferring turn-level user satisfaction in multi-turn
dialogues.
• We show the effectiveness of RoBERTaIQ in capturing dialogue context and temporal information across turns by comparing it with previous state of the art discourse-structureaware text encoders.
• We propose a data efficient learning framework to significantly reduce the amount of annotated data required forlearning RoBERTaIQ. We leverage unlabelled historical dialogue data for pretraining. We perform fine-tuning in a
multi-task learning setup to further utilize readily available
signals like Domain and Intent. Unlike other works that use
these signals as input features [1, 11], our model uses themas supervision signals to reduce training data (IQ labels)requirement.
The rest of the paper is organized as follows. Section 2 reviews
existing work. Section 3 presents baselines and our approach forautomatic dialogue evaluation. Section 4 presents our experimental
results. Section 5 shows different ablation studies. We conclude ourpaper in Section 6. Appendices A and B contain hyperparameter
information and case studies, respectively.

2

RELATED WORK

Recent works on evaluation of response quality in dialogue systems[1, 2, 11] are closely related to our work. While [2] use human engineered NLP features, [1, 11] propose IQ prediction models that use
input features directly from raw dialogue turn contents and system

RoBERTaIQ: An Efficient Framework for Automatic Interaction Quality Estimation of Dialogue Systems

KDD ’21, Aug 14–18, 2021, Virtual Event

metadata (e.g. ASR/NLU scores). However, we see the reliance on
system metadata as a limitation, and design our approach suchthat no system metadata is required as input features to the model.
While the above approaches focus on dialogue evaluation in Spoken
Langugage Understanding (SLU) systems, there is another line ofwork that focuses more on evaluation of open-domain chit-chatstyle dialogues. Lowe et al. [14] proposed a supervised approachcalled ADEM to mimic human annotator’s assessment of response
appropriateness, while Tao et al. [21] proposed an unsupervised
method called RUBER. Both of these approaches use RNN based
encoders. However, both ADEM and RUBER metrics result in poorcorrelation with human judgements [27]. Zhao et al. [27] propose
RoBERTa-eval, which uses a powerful RoBERTa based text encoder
to represent the dialogue context. Recently, Sinha et al. [20] propose MaUdE, which uses a BERT based text encoder to encode theutterances, followed by an RNN to model dialogue transitions. We
adapt MaUdE and RoBERTa-eval to our use-case. We use them asbaselines to analyze their shortcomings and design our dialogue
encoder with enhanced contextual and temporal representations.

3

METHODOLOGY

In this section, we first define the notations and provide the problem definition. Next, we present the baseline model architectures
adapted to our use-case: MaUdE and RoBERTa-eval. We then sharethe details of our proposed architecture: RoBERTaIQ that encodes aflattened dialogue text sequence and explain how each dialogue session is processed before inputting to this model. Next, we introduce
how we obtain the datasets used for experiments, followed with
explanation of the training procedure that involves pretraining andmulti-task fine-tuning.

3.1

Notations and Problem Definition

We consider a dataset 𝐷 of 𝑀 multi-turn dialogue sessions, suchthat 𝐷 = {𝑆 𝑗 }𝑀
𝑗=1 , and every session 𝑆 is an ordered set of 𝑁 turns:
𝑁 . Here 𝑖 indicates the index of turn, and each turn 𝑡
𝑆 = {𝑡𝑖 }𝑖=1𝑖consists of a pair (𝑄𝑖 , 𝑅𝑖 ), where 𝑄𝑖 is the user’s query and 𝑅𝑖 is
the agent’s response to query 𝑄𝑖 . Each turn 𝑡𝑖 also has a timestamp𝜏𝑖 associated with it, which is the time at which 𝑄𝑖 was received by
the agent. Any two successive turns have a time gap of less than aminute. Given a dialogue session S and a reference turn 𝑡𝑟𝑒 𝑓 = 𝑡𝑖
for some 𝑖 ∈ {1, . . . , 𝑁 }, the goal of our model is to predict 𝐼𝑄𝑠𝑐𝑜𝑟𝑒of turn 𝑡𝑟𝑒 𝑓 . 𝐼𝑄𝑠𝑐𝑜𝑟𝑒 = 0 if the agent’s response 𝑅𝑟𝑒 𝑓 to query 𝑄𝑟𝑒 𝑓
is satisfactory from user’s perspective, and 1 otherwise. We focus
on offline turn-level dialogue evaluation, which means that we haveboth previous turns and following turns available at the time ofevaluating 𝑡𝑟𝑒 𝑓 .

3.2

Baseline models

3.2.1 MaUdE++. MaUdE (Metric for automatic Unreferenced dialogue Evaluation) was proposed by Sinha et al. [20] for online
dialogue evaluation. Here, we adapt MaUdE’s Dialogue-structureaware encoder for offline evaluation, and slightly modify the architecture such that it can encode other meta information about eachturn, such as domain, intent, timestamp, active screen availability
etc. We use similar metadata features as used by Ling et al. [11].
We refer to this modification as MaUdE++.

Figure 1: MaUdE++ (baseline model) architecture

As shown in Figure 1, MaUdE++ first computes the encoding
for each turn and then passes the turn encodings through a bidirectional GRU to compute the dialogue session embedding. Thissession embedding is concatenated with other features and fed tothe classifier for IQ prediction. Considering a dialogue session 𝑆
with 𝑛 turns: {(𝑄 1, 𝑅1 ), ..., (𝑄𝑛 , 𝑅𝑛 )}, we compute the IQ score as:fi = 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎𝐶𝐿𝑆 ([𝐶𝐿𝑆]; 𝑄𝑖 ; [𝑆𝐸𝑃]; 𝑅𝑖 )ei = (fi ; metai )−→←−hn, h1 = 𝐵𝑖𝐺𝑅𝑈 (e1, e2, ..., en )−→ ←−𝐼𝑄𝑠𝑐𝑜𝑟𝑒 = 𝜎 (𝑊 .(eref ; hn ; h1 ))

(1)
(2)
(3)
(4)

Here, [CLS] refers to a special token prefixed to user query, [SEP]refers to a special token inserted between the query and response,“;” denotes the concatenation operator, ei refers to the encoding ofturn 𝑡𝑖 , metai refers to the concatenated encodings of categorical−→←−and real-valued features of turn 𝑡𝑖 . hn and h1 are the final hidden
states of the bidirectional GRU in either direction and eref refers
to the encoding of the reference turn for which we want to makethe IQ score prediction. To compute the text representation fi , weuse the [CLS] token encoding from the RoBERTa encoder. We startwith a pretrained RoBERTa model and finetune it end-to-end withgradients coming from IQ classification loss.
3.2.2 RoBERTa-eval. RoBERTa-eval was proposed by Zhao et al.
[27] as a robust dialogue response evaluator. It produces a vector dgiven a context c and a response 𝑅𝑟𝑒 𝑓 and then finally calculates

KDD ’21, Aug 14–18, 2021, Virtual Event

Gupta et al.

Figure 2: RoBERTaIQ model architecture
its score via a Multilayer Perceptron with a sigmoid function. Considering a dialogue session 𝑆 with 𝑛 turns: {(𝑄 1, 𝑅1 ), ..., (𝑄𝑛 , 𝑅𝑛 )}:𝑐 = [[𝐶𝐿𝑆]; 𝑄 1 ; [𝑆𝐸𝑃]; 𝑅1 ; [𝑆𝐸𝑃]; 𝑄 2 ; [𝑆𝐸𝑃]; . . . 𝑄𝑟𝑒 𝑓 ](5)

d = 𝑅𝑜𝐵𝐸𝑅𝑇 𝑎𝐶𝐿𝑆 (𝑐; [𝑆𝐸𝑃]; 𝑅𝑟𝑒 𝑓 )(6)
𝐼𝑄𝑠𝑐𝑜𝑟𝑒 = 𝜎 (𝑀𝐿𝑃 (d))(7)
Here, c, the dialogue context, is a flattened sequence of user
queries and agent responses from previous turns, including thequery of reference turn, for which we want to predict the IQ score.
Note that a limitation of this model is that it can only encode theleft dialogue context, i.e. turns that happened before 𝑄𝑟𝑒 𝑓 .

3.3

Our approach: RoBERTaIQ

Figure 2 shows the RoBERTaIQ model architecture diagram. Unlike previous works that rely on textual features and many othersystem specific signals [11, 15] which require feature engineering
efforts, RoBERTaIQ relies solely on the textual features, i.e. user’sutterances and agent’s responses. RoBERTaIQ model is built on topof RoBERTa-base model with modifications at the input layer. To
differentiate between the utterances and responses, we add two
special tokens to the vocabulary, [USER] and [AGENT] and prefix
them to each utterance and response respectively. By doing this,we create a single flat sequence for the whole dialogue. We limit
the dialogue length to 512 tokens. Table 2 shows how a dialogue
session is pre-processed.
Temporal difference encoding: In addition to capturing thetext contextual information as shown above, we also capture thetime difference between multiple turns in case of a multi-turn dialogue. Capturing the time difference is an important factor for IQ
prediction as users are likely to immediately interrupt the agent if
they do not get the right response. They might even rephrase their

Table 2: Pre-processing of a dialogue session

Timestamp(in seconds)

Dialogue Session

𝜏 =0[USER] Play fearless.
[AGENT] Playing fearless by Pink Floyd.

𝜏 =2[USER] Stop
[AGENT] <null>[USER] Play fearless by Taylor Swift.
[AGENT] Here’s fearless by Taylor Swift.
Pre-Processed form
[USER] Play fearless [AGENT] Playing fearless by Pink Floyd[USER] Stop [AGENT] [USER] Play fearless by Taylor Swift[AGENT] Here’s fearless by Taylor Swift𝜏 =7request or add more information and hope the agent can take theaction they expected in the follow-up turn.
To make the model aware of these temporal differences between
the turns, we first select a reference turn in the dialogue session
and refer to its timestamp as 𝜏𝑟𝑒 𝑓 . This turn is selected at random
when pretraining. During fine-tuning, this is the turn for which wehave the IQ label. We then calculate the time difference Δ𝑖 for all
turns with respect to 𝜏𝑟𝑒 𝑓 : Δ𝑖 = 𝜏𝑖 − 𝜏𝑟𝑒 𝑓 , where 𝜏𝑖 is the timestampof turn 𝑖. Δ𝑖 for all turns are then discretized using equal widthbinning. We create 16 bins to represent equal sized intervals in Δ𝑖 ’s
range of [−60, 60] seconds and map Δ𝑖 to its respective time bin:
[𝐵𝐼 𝑁𝑖 ]. The corresponding time-bin embeddings are added to eachtoken of the turn at the input layer of the model, depending on
the turn’s bin, as shown in Figure 2. These embeddings are learned

RoBERTaIQ: An Efficient Framework for Automatic Interaction Quality Estimation of Dialogue Systems

from scratch. The number of bins is decided in a way to ensure a
uniform distribution of turns across the bins. We reserve a special
bin: [𝐵𝐼 𝑁 0 ] for reference turn’s tokens. [𝐵𝐼 𝑁 0 ] is the key indicatorusing which the model recognizes the reference turn.
Task specific heads: As shown in Figure 2, the model has various heads: MLM (Masked Language Modeling) and classifier heads
for IQ (Interaction Quality), Domain and Intent classification. During multi-stage pretraining, we only use the MLM head for calculating the loss and updating the weights. The MLM head operates
on the output representations of tokens. For multi-task fine-tuning,
we use the classification heads for calculating the loss. Each of theclassification heads takes the encoded representation of the [CLS]
token as input. Each classifier head has a dense layer, followed bya projection layer. All heads are initialized randomly. The output
size of the projection layer is equal to the number of labels of therespective task.
Note that we do not design the architecture with real-time/online
IQ prediction in mind. We focus on offline evaluation where wehave previous and next turns available, when evaluating the qualityof the reference turn. However, our design is easily extensible toonline evaluation, in which case, the reference turn will always be
the last turn in the dialogue session.

3.4

Datasets

Historical Dialogue Sessions: We randomly sample around 2
million English dialogue sessions between users and Alexa fromanonymized logged historical data. We do not use any task specifichuman annotations for these dialogue sessions. These sessions span
many NLU domains and intents, and contain turns where the users
had both good and bad experiences. As described later, we use these
dialogue sessions for the first stage of pretraining.
Interaction Quality (IQ) dataset: This dataset is sampled from
Alexa Live Traffic and is annotated with IQ labels provided by experts: 0 (Non-defect or satisfactory experience) and 1 (Defect). Only
one turn per dialogue session has a defect/non-defect label, which
we refer to as the reference turn. The reference turns are labelledfrom the end user’s perspective. For example, considering turn 1 as
the reference turn in Table 2, the annotators would give it an IQ
label of 1 (defective), as the agent did not play the song intended
by the user in that turn. To get the IQ labels, we use a similar Response Quality annotation workflow as described in [2]. We
have around 500K dialogues for training, 100K for development set,and 100K for testing. The training and test sets are sampled fromdifferent time periods. This leads to a test set that has a different domain distribution than the training set, and also some new domainsthat are not present in the training set. All the turns have domain
and intent strings that were produced by a separate NLU system.
The IQ prediction task is the primary task at which we want to do
better with the least amount of human annotated data possible. For
evaluation, we focus on F1-score for the defective class as the binaryclassification metric. We do so because our dataset is imbalanced(25% defect and 75% non-defect) and identifying dissatisfactoryturns is of more importance.

KDD ’21, Aug 14–18, 2021, Virtual Event

Out-of-domain (OOD) testset: This dataset is sampled fromannotated IQ test data. It has 30K instances in total and is only used
for evaluation in particular to see the benefits and limitations ofpretraining and multi-task learning on out-of-domain instances.
To ensure that there is no overlap between the domains/intentsof the turns in this testset with the training set, we sample threedifferent fractions (5%, 10% and 25%) from the IQ training dataset(500K instances), and use these subsets for training the models.

3.5

Pretraining: Domain adaptive and Task
adaptive

Masked Language Modeling is a common pretraining strategy fortransformer based architectures in which a random sample of thetokens in the input sequence is selected and replaced with thespecial [MASK] token. The MLM loss function is the cross-entropy
loss on predicting the masked tokens. Following Liu et al. [13], we
conduct token masking dynamically with each batch by masking
15% of the tokens. RoBERTaIQ is initialized from RoBERTa-base andis further pretrained as described below. The MLM loss function isdefined as:
𝐿𝑚𝑙𝑚 = −𝑀Õlog 𝑃 (𝑥𝑚 )

(8)

𝑚=1

where 𝑀 is the total number of masked tokens and 𝑃 (𝑥𝑚 ) is thepredicted probability of token 𝑥𝑚 .
Following [7], we perform the first stage of pretraining on unlabelled historical dialogue sessions data, which we refer to as
Domain Adaptive Pretraining (DAPT). Similar to [9], we then further pretrain this model using the MLM loss on the IQ training
dataset in the second stage, which we refer to as Task Adaptive
Pretraining (TAPT). Note that both DAPT and TAPT do not require
any task specific labels.

3.6

Multi-task (MT) fine-tuning

After the multi-stage pretraining process, we finetune the model on
the main downstream task of IQ prediction, with additional heads
for Domain and Intent prediction. Our hypothesis is that we canbenefit from both cross-task data and the regularization effects ofMT, especially when the IQ data is small. The multi-task loss isdefined in Equation 9:Õ𝐿(𝜃 ) =
𝜆𝜓 𝑙 (𝑦𝜓𝑖 , 𝑓𝜓 (𝐸𝑛𝑐𝐶𝐿𝑆 (𝑥𝜓𝑖 )))(9)𝑥𝜓𝑖 ,𝑦𝜓𝑖 ∈𝐷𝜓

where 𝜓 refers to one of the tasks (IQ, Domain, Intent), 𝑥 𝑖 , 𝑦𝑖 referto raw dialogue features and task labels respectively, 𝐸𝑛𝑐𝐶𝐿𝑆 (𝑥𝜓𝑖 )refers to the encoding of [𝐶𝐿𝑆] token after passing 𝑥𝜓𝑖 through theshared RoBERTaIQ encoder, 𝑓𝜓 is the respective task classifier, 𝑙 isthe cross-entropy loss and 𝜆𝜓 is the task weight. We empiricallyset 𝜆𝐼𝑄 = 1, and 𝜆𝑑𝑜𝑚𝑎𝑖𝑛 = 𝜆𝑖𝑛𝑡𝑒𝑛𝑡 = 0.5.

4

EXPERIMENTS

In this section, we first compare RoBERTaIQ with other baselines
to see the effects of different model architectures. We then show
the results of RoBERTaIQ with multi-stage pretraining and multitask fine-tuning with varying amounts of IQ training data. All theKDD ’21, Aug 14–18, 2021, Virtual Event

Gupta et al.

experiments were conducted on an AWS p3.16xlarge instance with
8 GPUs. All the numbers reported with “±” prefixes denote absolutedifferences in the metric w.r.t the corresponding baseline. Other
training details and hyperparameters can be found in the Appendix
A.

4.1

Comparison with baselines

Table 3: RoBERTaIQ vs baselines on IQ test set (100K examples)
Perf(%)

Accuracy

F1

Precision

RecallMaUdE++ (Text features only) [20]MaUdE++ (+ system metadata)

86.5
+ 2.1

77.4
+ 1.9

78.0
+ 2.7

76.9
+ 1.1

RoBERTa-eval [27]

- 3.3

- 5.4

- 5.9

-5

RoBERTaIQ (This work)

+ 4.2

+ 6.1

+6

+6

Using 100% IQ training data

Table 3 shows the performance comparison between RoBERTaIQ
and other baselines on IQ test set. We use the full IQ training data
(500K instances) to train all the models. The RoBERTa encoder
weights are initialized with a pre-trained model 1 and are finetuned
end-to-end with gradients coming from IQ classification loss. We
do not apply any multi-tasking or pretraining strategies for thiscomparison. Using system metadata features helps increase theperformance of MaUdE++ by 1.9% F1 score, but RoBERTaIQ, whichuses only the text features, still outperforms it by 4.2% absolute F1score. Please refer to Appendix B for a case study between MaUdE++
and RoBERTaIQ.
RoBERTa-eval performs worse than the MaUdE++ baseline by
5.4% F1 score. This is mainly due to the fact that RoBERTa-eval sees
only the left context (previous turns).

4.2

RoBERTaIQ Full Results on IQ testset

Table 4: Model performance comparison with pretraining
and finetuning on varying amounts of IQ training data. All
the rows show evaluation metrics on IQ testset (100K instances)
Perf (%)
IQ (5% Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task
IQ (10% Training Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task
IQ (25% Training Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task
IQ (50% Training Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task
IQ (100% Training Data)
Scratch (Baseline)+ Multi-task+ DAPT+ TAPT+ DAPT + Multi-task+ TAPT + Multi-task

Accuracy

F1

Precision

Recall

84.7
+2.7
+0.5
+3.8

72.2
+5.2
+1.8
+7.1

75.1
+4.5
-1.1
+6.3

69.5
+5.7
+4.5
+7.5

88.2
+0.5
+1.1
+1.4

78.6
+1.9
+2.7
+2.9

81.6
-2.3
-0.2
+0.8

75.9
+5.8
+5.3
+4.8

89.7
0.0
+0.8
+0.5

82.1
+0.2
+1.5
+1.0

81.7
-0.6
+0.2
-0.3

82.4
+1.0
+3.1
+2.5

90.1
+0.2
+0.6
+0.6

82.2
+0.6
+1.9
+1.4

84.2
-0.3
-1.8
-0.1

80.4
+1.5
+5.6
+2.9

90.7
-0.3
+0.4
+0.5
+0.1
+0.1

83.5
-0.3
+0.7
+1.2
+0.3
+0.44

84.0
-0.4
+0.2
+0.6
+0.6
+0.7

82.9
-0.1
+1.4
+1.8
+0.1
+0.3

see TAPT providing a further boost over DAPT.

Table 4 shows the performance of RoBERTaIQ model on IQ testset
with different training settings. To show the respective benefits ofpretraining and multi-task learning, we train models with varying
amount of IQ training data. “Scratch” refers to training on IQ only,
without pretraining or multi-task learning. In DAPT runs, we startwith a RoBERTaIQ model pretrained on historical dialogue sessions
and finetune on IQ data. In TAPT runs, we start with the DAPT
pretrained model, and further pretrain on the IQ training data with
MLM loss (without using the IQ labels) and then fine-tune on IQ
training data with the classification loss. We perform TAPT experiments only for the cases where we use 100% of available IQ training
data. For all the Multi-task runs, we include other classification
tasks (Domain and Intent) in addition to IQ for fine-tuning.

Effects of Multi-task learning: The performance improvements that come with multi-task learning vary with the IQ training
dataset size. We see maximum benefits, i.e. an increase in F1 scoreby an absolute 5.2% when we use only 5% IQ training dataset. The
improvements diminish with increasing IQ training dataset size,
even leading to a slight decrease in F1 score when 100% for IQ
dataset is used for training. This leads to the conclusion that theadditional tasks help the model learn better through extra supervision when IQ training data is small. But after a certain point as
the training dataset size increases, cross-task knowledge transfer
becomes less useful and instead the regularization effects of othertasks start hurting the performance on the primary task of IQ prediction.

Effects of DAPT and TAPT: As can be seen from Table 4, DAPT
provides consistent benefits in terms of boosting the performanceon IQ prediction task, improving the F1 score by absolute 2.7% in
the best case. This successfully demonstrates knowledge transferfrom unlabelled historical data to the downstream task of IQ prediction. For the scenario where we use 100% of IQ training data, we

Overall, we find that combining pretraining and multi-task learning provides gains as big as 7.1% F1 score improvement with smaller
training dataset sizes. In other words, using these techniques, we
can significantly reduce the amount of training dataset (IQ annotations), which is both costly and time-consuming to collect. Thiscan be especially useful for the new or the long-tail domains, forwhich we do not have IQ annotations available.

1 https://huggingface.co/roberta-base

RoBERTaIQ: An Efficient Framework for Automatic Interaction Quality Estimation of Dialogue Systems

4.3

Table 6: Contributions of different tasks in Multi-task learning

RoBERTaIQ Results on Out-of-Domain(OOD) Dataset

Table 5: Model performance comparison with pretraining
and finetuning on varying amounts of IQ training data. All
the rows show evaluation metrics on Out-of-domain dataset(30K instances)
Perf (%)

Accuracy

F1

KDD ’21, Aug 14–18, 2021, Virtual Event

Precision

RecallIQ (5% Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task

83.71
+1.92
+1.28
+3.54

73.75
+2.12
+3.33
+4.84

73.42
+5.38
-0.48
+8.21

74.09
-0.92
+7.65
+1.69

IQ (10% Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task

88.08
-0.30
+1.97
+0.59

79.28
+0.86
+4.32
+1.78

85.61
-5.2
-0.42
-1.75

73.82
+6.05
+8.25
+4.63

IQ (25% Data)
Scratch (Baseline)+ Multi-task+ DAPT+ DAPT + Multi-task

90.77
+.10
+1.66
+0.99

84.73
+0.49
+3.18
+1.93

86.64
-1.35
+0.17
+.02

82.9
+2.25
+6.13
+3.76

Table 5 shows the evaluation results on the OOD dataset. DAPT
consistently provides improvements in F1 scores, irrespective of
the size of IQ training dataset used. However, multi-task learning
provides improvements only with small IQ training set (5%). The
improvements are almost negligible when training dataset size increases, and are much lesser than that of DAPT. This is expected as
the knowledge that comes from domain and intent prediction is nolonger useful, as we are evaluating on a dataset that has no domains
and intents in common with the training dataset. This observationshows the limitations of our multi-task learning setup and emphasizes the importance of choosing domain agnostic auxiliary tasks
whose knowledge is transferable to the target task of interest.

5 ABLATION STUDIES
5.1 Contributions of different tasks in
Multi-task learning
Here, we gauge the contributions of different tasks that aid in
improving the performance on IQ prediction. We conduct the experiments with 5% and 10% IQ training set and other tasks, andremove the tasks one by one to see their impact. As shown in Table
6, the task of Intent prediction is more helpful for improving theperformance of IQ prediction. This is expected as there are specificintents that capture user sentiments like pleasantries or insults.
Other intents capture instances where the users try to terminate
the current turn or exit a skill if the agent is not doing the rightthing. These behaviors (as captured by NLU intents) are directly
related to user experience and hence get reflected in improved IQ
prediction accuracy.

Perf (%)
IQ (5% Data)
All tasks (Baseline)
No Domain
No Intent
IQ (10% Data)
All tasks (Baseline)
No Domain
No Intent

5.2

Accuracy

F1

Precision

Recall

87.4
-0.3
-1.5

77.4
-1.1
-3

79.6
-1.2
-3.9

75.2
-0.9
-2.1

88.7
-0.2
-1.3

80.5
-0.8
-1.7

79.3
-0.8
-2.1

81.7
-0.8
-1.2

Effect of time encoding in RoBERTaIQ

To see the importance of capturing the time difference between
the reference turn and other turns, we do a study where we do
not provide any temporal information to the model through thetime bin embeddings described in Section 3.3. As shown in Table
7, we see a drop of almost 9% in F1 score if we do not capture thetemporal differences between the turns.
Table 7: Effect of time difference encoding in RoBERTaIQ

Perf (%)
IQ (100% Data)
RoBERTaIQ
No Time encodings

6

Accuracy

F1

Precision

Recall

90.7
-5.6

83.5
-8.8

84
-9.1

82.9
-8.5

CONCLUSION AND FUTURE WORKIn this work, we presented a framework for automatic turn-level
dialogue evaluation in large scale Conversational AI systems. We
introduce a new modeling architecture on top of RoBERTa that ismore suitable for encoding dialogue sessions while capturing the
temporal differences across different turns. We showcase that thisarchitecture alone boosts the F1 score by 4.2% over other modelarchitectures used in dialogue evaluation literature, while making
use of only the textual features. To make the training more data
efficient, we propose multi-stage pretraining and multi-task learning approaches. This helps us leverage large amount of historical
dialogue sessions and other system signals, like Domain and Intent,
which are readily available. These approaches significantly reduce
the requirement of obtaining annotated Interaction Quality (IQ)
data. With only 5% of annotated training data, we achieve the same
F1 performance as our baseline on IQ testset.
One of the limitations of our multi-task learning approach isthat the tasks used are domain specific, which makes it challenging to transfer the knowledge to out-of-domain cases. For future
work, we plan to incorporate more auxiliary tasks for multi-task
learning that are agnostic to dialogue domains, like user utterance
rephrase detection, predicting cohesion between request and response and next session prediction [25]. Pretraining proved to bevery promising in improving the IQ prediction performance forboth in-domain and out-of-domain cases. In the future, we want toconsider much larger scale pretraining by increasing the numberof unlabeled historical dialogue sessions by an order of magnitude.

KDD ’21, Aug 14–18, 2021, Virtual Event

REFERENCES[1] Praveen Kumar Bodigutla, Aditya Tiwari, Spyros Matsoukas, Josep Valls-Vargas,
and Lazaros Polymenakos. 2020. Joint Turn and Dialogue level User Satisfaction Estimation on Multi-Domain Conversations. In Findings of the Association
for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 3897–3909. https://www.aclweb.org/anthology/2020.findingsemnlp.347[2] Praveen Kumar Bodigutla, Longshaokan Wang, Kate Ridgeway, Joshua Levy,
Swanand Joshi, Alborz Geramifard, and Spyros Matsoukas. 2019. DomainIndependent turn-level Dialogue Quality Evaluation via User Satisfaction Estimation. arXiv preprint arXiv:1908.07064 (2019).
[3] Z. Chen, X. Fan, Y. Ling, and C. Guo. 2020. Pre-Training for Query Rewriting in a
Spoken Language Understanding System. In ICASSP 2020 - 2020 IEEE International
Conference on Acoustics, Speech and Signal Processing (ICASSP). 7969–7973. https://doi.org/10.1109/ICASSP40776.2020.9053531[4] Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machinelearning 20, 3 (1995), 273–297.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:
Pre-training of deep bidirectional transformers for language understanding. arXiv
preprint arXiv:1810.04805 (2018).
[6] Jerome H. Friedman. 2001. Greedy function approximation: A gradient boostingmachine. The Annals of Statistics 29, 5 (2001), 1189 – 1232. https://doi.org/10.
1214/aos/1013203451[7] Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy,
Doug Downey, and Noah A Smith. 2020. Don’t Stop Pretraining: Adapt Language
Models to Domains and Tasks. arXiv preprint arXiv:2004.10964 (2020).
[8] Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation:
A new estimation principle for unnormalized statistical models. In Proceedingsof the Thirteenth International Conference on Artificial Intelligence and Statistics.
297–304.
[9] Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning
for text classification. arXiv preprint arXiv:1801.06146 (2018).
[10] Weixin Liang, James Zou, and Zhou Yu. 2020. Beyond User Self-Reported Likert
Scale Ratings: A Comparison Model for Automatic Dialog Evaluation. arXiv
preprint arXiv:2005.10716 (2020).
[11] Yuan Ling, Benjamin Yao, Guneet Kohli, Tuan-Hung Pham, and Chenlei Guo. 2020.
IQ-Net: A DNN Model for Estimating Interaction-level Dialogue Quality with Conversational Agents. In Proceedings of the KDD 2020 Workshop on Conversational
Systems Towards Mainstream Adoption co-located with the 26TH ACM SIGKDD
Conference on Knowledge Discovery and Data Mining (SIGKDD 2020), VirtualWorkshop, August 24, 2020 (CEUR Workshop Proceedings, Vol. 2666), Giuseppe Di
Fabbrizio, Surya Kallumadi, Utkarsh Porwal, and Thrivikrama Taula (Eds.). CEURWS.org. http://ceur-ws.org/Vol-2666/KDD_Converse20_paper_12.pdf[12] Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multitask deep neural networks for natural language understanding. arXiv preprint
arXiv:1901.11504 (2019).
[13] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A
robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692(2019).
[14] Ryan Lowe, Michael Noseworthy, Iulian V. Serban, Nicolas Angelard-Gontier,
Yoshua Bengio, and Joelle Pineau. 2018. Towards an Automatic Turing Test:
Learning to Evaluate Dialogue Responses. arXiv:1708.07149 [cs.CL][15] Dookun Park, Hao Yuan, Dongmin Kim, Yinglei Zhang, Matsoukas Spyros, YoungBum Kim, Ruhi Sarikaya, Edward Guo, Yuan Ling, Kevin Quinn, Pham Hung,
Benjamin Yao, and Sungjin Lee. 2020. Large-scale Hybrid Approach for Predicting
User Satisfaction with Conversational Agents. arXiv:2006.07113 [cs.HC][16] Pragaash Ponnusamy, Alireza Roshan Ghias, Chenlei Guo, and Ruhi Sarikaya.
2020. Feedback-Based Self-Learning in Large-Scale Conversational AI Agents.
Proceedings of the AAAI Conference on Artificial Intelligence 34, 08 (Apr. 2020),
13180–13187. https://doi.org/10.1609/aaai.v34i08.7022[17] Louisa Pragst, Stefan Ultes, and Wolfgang Minker. 2017. Recurrent neural network
interaction quality estimation. In Dialogues with Social Robots. Springer, 381–393.
[18] Niklas Rach, Wolfgang Minker, and Stefan Ultes. 2017. Interaction quality estimation using long short-term memories. In Proceedings of the 18th Annual SIGdial
Meeting on Discourse and Dialogue. 164–169.
[19] Alexander Schmitt and Stefan Ultes. 2015. Interaction quality: assessing thequality of ongoing spoken dialog interaction by experts—and how it relates touser satisfaction. Speech Communication 74 (2015), 12–36.
[20] Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William L
Hamilton, and Joelle Pineau. 2020. Learning an Unreferenced Metric for Online
Dialogue Evaluation. arXiv preprint arXiv:2005.00583 (2020).
[21] Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. 2017. RUBER: An
Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems.
arXiv:1701.03079 [cs.CL][22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all

Gupta et al.

you need. arXiv preprint arXiv:1706.03762 (2017).
[23] Marilyn Walker, Candace Kamm, and Diane Litman. 2000. Towards developing
general models of usability with PARADISE. Natural Language Engineering 6, 3
& 4 (2000), 363–377.
[24] Chien-Sheng Wu, Steven C.H. Hoi, Richard Socher, and Caiming Xiong. 2020.
TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented
Dialogue. In Proceedings of the 2020 Conference on Empirical Methods in Natural
Language Processing (EMNLP). Association for Computational Linguistics, Online,
917–929. https://www.aclweb.org/anthology/2020.emnlp-main.66[25] Ruijian Xu, Chongyang Tao, Daxin Jiang, Xueliang Zhao, Dongyan Zhao, andRui Yan. 2020. Learning an Effective Context-Response Matching Model with
Self-Supervised Tasks for Retrieval-based Dialogues. arXiv:2009.06265 [cs.CL][26] Siyang Yuan, Saurabh Gupta, Xing Fan, Derek Liu, Yang Liu, and Chenlei Guo.
2021. Graph Enhanced Query Rewriting for Spoken Language Understanding
System. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP).
[27] Tianyu Zhao, Divesh Lala, and Tatsuya Kawahara. 2020. Designing Precise andRobust Dialogue Response Evaluators. arXiv:2004.04908 [cs.CL]

RoBERTaIQ: An Efficient Framework for Automatic Interaction Quality Estimation of Dialogue Systems

KDD ’21, Aug 14–18, 2021, Virtual Event

Table 8: Comparison of RoBERTaIQ model predictions with MaUdE++. Reference turn in each session is boldfaced.
#

Timestamp(in seconds)

Dialogue Session

𝜏 =0[USER] how much do amungus plushies cost
[AGENT] One moment, for amungus plushies this is the closest I could find. The first one is Winsterch Kids Cats Stuffed Animal Toys

𝜏 =9[USER] how much do among us plushies cost
[AGENT] One moment. Showing results for among us plushies. The first one is Bedtime Originals Among Us Plush Toy.

𝜏 =0

[USER] Start joke responder[AGENT] Ok, Here’s Joke Responder Welcome to joke responder. Please tell me a joke.

𝜏 =6[USER] why did the skeleton not cross the road
[AGENT] I don’t know, why did the skeleton not cross the road

𝜏 = 11[USER] because he didn’t have the guts!
[AGENT] Ha ha ha. So much talent! So many laughs!

𝜏 =0[USER] show me shark videos[AGENT] Here’s what I found (playing video)

𝜏 =8[USER] play baby shark on amazon prime[AGENT] Here’s Baby Shark , by Pinkfong , on Amazon Music.

1

2

3

A

TRAINING SETUP

MaUdE++
prediction

RoBERTaIQ
prediction

1

0

1

0

1

0

1

0

1

IQ label

B

CASE STUDY

We use the HuggingFace Transformers 2 library for all our training
and evaluation code. Table 9 and 10 show the hyperparameters for
pretraining and fine-tuning, respectively.

Table 7 shows a comparison of model predictions of RoBERTaIQ
with MaUdE++ . We use the better performing version of MaUdE++,i.e., the one that also uses system metadata as input features.

Table 9: Hyperparameters for Pretraining (DAPT and TAPT)

Dialogue 1: By looking at turn 1 (reference turn) in isolation, it
appears that the agent gave the right response. However, turn 2
makes it clear that turn 1 had an ASR error; “Among Us” was incorrectly recognized as “amungus”. RoBERTaIQ’s encoder was able
to recognize this, as it has the ability to apply self-attention mechanism across tokens of different turns, and hence correctly predicted
IQ score = 1, for turn 1.

Hyperparameter

Assignment

maximum epochs
MLM masking probability
max learning rate
optimizer
batch size per GPU
gradient accumulation steps
learning rate decay
effective batch size
Adam epsilon
Adam betas

20
0.15
5e-4
AdamW
4
32
Linear
4*32*8 = 1024
1e-6
0.9, 0.999

Table 10: Hyperparameters for Fine-tuning
Hyperparameter

Assignment

maximum epochs
patience (for early stopping)max learning rate
dropout
optimizer
batch size per GPU
gradient accumulation steps
learning rate decay
effective batch size
Adam epsilon
Adam betas

20
4
1e-5
0.1
AdamW
4
8
Linear
4*8*8 = 256
1e-6
0.9, 0.999

2 https://github.com/huggingface/transformers

Dialogue 2: In turn 2, the user is interacting with a “Joke Insider”
3P skill. MaUdE++ predicts that the response is unsatisfactory due
to presence of “I don’t know”. However, RoBERTaIQ made a better
sense of the response using the previous and next turns, and predicted the response to be satisfactory (IQ score = 0).
Dialogue 3: MaUdE++ failed to recognize from the context thatthe user wanted to listen to Baby Shark, while RoBERTaIQ figuredthat out and correctly predicted IQ score = 1 as the agent showed
the user irrelevant shark videos in turn 1.