NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo JapanRSLDE at NTCIR-16 DialEval-2 Task
Fan Li

Tetsuya Sakai

Waseda UniversityJapan
lif@akane.waseda.jp

Waseda University
Japantetsuyasakai@acm.org

ABSTRACT2

In this paper, we report the work of the RSLDE team at the dialogueevaluation (DialEval-2) task of NTCIR-16, including the Chineseand English dialogue quality (DQ) and nugget detection (ND) subtasks. We implemented two sentence-level baselines that fine-tune
BERT and XLNet along with a linear layer for the ND subtask. In
addition, we propose a model based on BERT to capture the structure and context information of a customer-helpdesk dialogue. Thisdialogue-level model modifies the input and embeddings of BERT.
We add a Transformer encoder layer over this model as our thirdmodel for the ND subtask and first model for the DQ subtask. The
second model for the DQ subtask is the same dialogue-level model
but without the Transformer encoder layer. Our XLNet model generated the best run for both English and Chinese ND subtasks. Our
dialogue-level model outperformed the baselines for the Chinese
DQ subtask but not for the English DQ subtask.

The developments in the field of natural language processing (NLP)
and machine learning (ML) have recently led to an increasing interest in task-oriented dialogue (TOD) systems. Such system assists
users in accomplishing their goals by providing meaningful system
responses. Compared to a traditional help desk, an automatic dialogue agent enhanced through the TOD system is able to handle
customer inquires at any time of the day, which is more economicand efficient. However, evaluating such systems is usually dependent on human annotators, and is thus time-intensive and expensive. It is therefore of significant interest to propose an automatic
dialogue evaluation system to alleviate this problem.
For the traditional closed-domain spoken dialogue system (suchas a slot-filling task), most automatic evaluation studies follow
the PARADISE framework [13]. However, evaluating customerhelpdesk dialogues is related to but extremely different from evaluating slot-filling dialogues. Customer-helpdesk dialogues discuss
diverse problems about products and services, as opposed to slotfilling dialogues, which list up all the required slot in advance [17].
Therefore, one of the most significant discussions occurring incustomer-helpdesk dialogue systems is the creation of an appropriate automatic method, which is the aim of the DialEval-2 task.
Several methods were proposed in the NTCIR-15 DialEval-1
task [16]. Most of the participants utilized a pretrained language
model (e.g., BERT [4]) in their networks. These Transformer-based
pretrained language models were pretrained on a large corpus of unlabeled data and are able to transfer knowledge efficiently to downstream tasks. For example, the IMTKU team used XLM-RoBERTa, avariant of BERT, by fine-tuning various transfer learning recipes forboth DQ and ND subtasks [5]. Most of their runs outperformed thebaselines. Similarly, the TUA1 team employed a pretrained BERT
network for extracting features of dialogues before feeding them toa Bi-LSTM network [6]. Furthermore, they applied a self-attention
network and several feed-forward neural network layers over this
Bi-LSTM network. As a result, the proposed network can generatepredictions that are very close to human annotations.
In addition to BERT and its variants, XLNet is another successful
and representative Transformer-based pretrained language model.
In contrast to BERT, which uses autoencoder (AE) language modeling for its training, XLNet applies a generalized autoregressive (AR)language modeling, and outperformed BERT on 20 tasks, often bya large margin, including question answering, natural language inference, sentiment analysis, and document ranking [15]. Therefore,
one objective of this study is to investigate BERT and XLNet, thetwo most successful Transformer-based language models, in termsof their performance on DQ and ND subtasks.
Another important part of the DialEval-1 task is an approach to
representing the structure of a dialogue. At NTCIR-15, the IMTKUTEAM NAMERSLDESUBTASKS
Nugget Detection (Chinese, English)
Dialogue Quality (Chinese, English)

KEYWORDS
Pre-trained language model, Transformer, Dialogue representation

1

INTRODUCTION

The RSLDE team participated in the English and Chinese dialogue
quality (DQ) and nugget detection (ND) subtasks of DialEval-2 [11].
The task details of the two subtasks can be found in the overview
paper [11]. We propose three models for the ND subtask and twomodels for the DQ subtask.
In the dialogue quality subtask, our proposed models include a
modified pre-trained BERT model. By modifying the input sequenceand position embeddings, this BERT model can capture the dialogue structure and extract dialogue context information. We also
adopt this dialogue-level model in the nugget detection subtask,
and implement a vanilla BERT model for comparison. Meanwhile,
we use another pre-trained Transformer-based language model,
XLNet [15], to generate customer and helpdesk nuggets.
The rest of this paper is organized as follows. Section 2 presents
previous related studies, whereas section 3 describes our proposed
approaches. Our submission and an analysis of results are reported
in section 4. Finally, section 5 provides some concluding remarks.

66

RELATED WORKNTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japanteam utilized some tokenization tricks to better represent the structure [5]. the RSLNV team used two different encoders for the sentence and its context and applied one attention layer over the RNNs
to understand the dialogue context [1]. The TUA1 team embeded
not only each token but also its speaker identity and added thisspeaker embedding into the BERT output as a compound input tothe Bi-LSTM network [6]. Inspired by Liu [8], who obtained therepresentation of each sentence of a document by modifying the
input sequence and the embeddings of BERT, we propose a similar
approach to better obtain the representation of each utterance of acustomer-helpdesk dialogue.

AR： [

]
Unidirectional prediction

BERT：

[

]
Bidirectional prediction[

PermutationAR]

Permutation and unidirectional prediction

Figure 1: Comparison of AR, AE, and permutation AR

3 APPROACHES
3.1 Sentence-level Baseline
Considering the fact that some nuggets can involve several typical
patterns of words, one of our attempts is to tackle an ND subtask
as a sentence-level classification problem, which is similar to thesentiment analysis task. Based on this attempt, we apply transfer learning by fine-tuning a pretrained BERT and XLNet, as ourbaselines for the ND subtask.
3.1.1 BERT. BERT [4] is a widely used Transformer-based language model that uses autoencoder (AE) language modeling for its
pre-training. Rather than applying a left-to-right language model,
the authors adopted masked language model (MLM) for the BERT
pretraining. MLM masks 15% of all WordPiece tokens in each sequence at random and then requires the model to predict them,which thus enabled BERT to fuse the left and right contexts of asentence and provide a bidirectional representation. In addition,
Devlin et al. [4] also use a next sentence prediction (NSP) task thatjointly pretrains text-pair representations. This enables BERT totransfer the pre-trained knowlodge on a wider range of downstreamtasks. These novel procedures allow BERT to outperform the othersimilar Transformer-based model, such as GPT and GPT-2 [9] [10].
The BERT model we used for our sentence-level baseline is basedon the pre-trained uncased base model of a HuggingFace’s Transformers [14]. There are 12 layers, 12 attention heads, 768 neurons,
and 110M parameters in total. First, the dialogues are split by turn,and each turn contains one or more utterances from either a customer or a helpdesk. Second, as the input of the BERT model, werebuild two datasets that contain only utterances of the customer
and helpdesk respectively. In this way, we transferred the ND subtask to a classification task. As the only difference between them,for classification task, the target label is a one-hot vector, whereas
for the ND subtask, it has a probability distribution.

Factorization order: 3

2

4

1

Factorization order: 2

4

3

1

Factorization order: 1

4

2

3

Factorization order: 4

3

1

2

Figure 2: Illustration of the permutation language modelingobjective for predicting 𝑥 3 given the same input sequence x
but with different factorization orders (duplicated from [15])motivated Yang et al. [15] to propose a generalized autoregressive
method called permutation language modeling.
Permutation language modeling not only retains the benefitsof AR models but also allows models to capture bidirectional contexts [15]. We compare AR, BERT, and permutation language modeling in Figure 1. During the training process, the permutation
method, as shown in Figure 2, first shuffles the input tokens, andthen predicts words for training according to a traditional autoregressive method. Therefore, for a sequence x of length 𝑇 , there
are 𝑇 ! different orders for conducting a valid autoregressive factorization, which makes XLNet able to gather information from all
positions on both sides. In addition, as another major contributionof XLNet, it follows the key ideas of Transformer-XL [3], i.e., thesegment recurrence mechanism and relative encoding scheme for
effectively learning long sentences. For further details, we refer thereader to the original study [15]. We use XLNet base models bothfor English and Chinese tasks. Similar to BERT, the base model has
12 layers, 12 attention heads, 768 neurons, and 110M parameters in

3.1.2 XLNet. Like BERT, XLNet [15] is also based on Transformer,
but is pre-trained using a generalized autoregressive (AR) language
model. Traditional AR language modeling seeks to estimate theprobability distribution of a text corpus using an autoregressivemodel. Specifically, given a text sequence x, AR language modelingfactorizes the likelihood into either a forward or a backward product, and thus has the drawback of being a unidirectional network,i.e., it can only reach either the left or right context of the evaluated token. However, as demonstrated with BERT, bidirectional
pre-training is very important for language representations, which

67

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 1: Official Runs Details

total. We use the XLNet-base-cased model of HuggingFace’s Transformers [14] for the English subtask and HFL’XLNet-base chinesemodel [2] for the Chinese subtask. The training details are the sameas those of BERT.

Task

Language
EnglishDQ

3.2

Chinese

Dialog-level Model

For the baseline ND models above, we assumed that all utterances
in a dialogue are independent to each other, i.e., are context-free.
However, this assumption may lead to a loss of context informationand thus limit the performance of our automatic evaluating systems.
To address this limitation, inspired by Liu [8], who proposed amethod to represent the sentence in a document for an extractive
summarization, we present a BERT-based dialogue-level model
to better represent the dialogue structure and better capture thecontext information in a dialogue. Because we need all informationof a dialogue to estimate the quality, this BERT-based dialogue-level
model is also apparently suitable for a DQ subtask.

EnglishND
Chinese

Run
0
1
0
1
0
1
2
0
1
2

Model
Dialogue-BERT+Transformer
Dialogue-BERT
Dialogue-BERT+Transformer
Dialogue-BERT
XLNet Baseline
BERT Baseline
Dialogue-BERT+Transformer
XLNet Baseline
BERT Baseline
Dialogue-BERT+Transformer

Batch size
12

8Transformer encoder layer to extract dialogue-level features focusing on the ND or DQ subtask from the BERT outputs:

3.2.1 Encoding Dialogue. Extracting context information to obtain
the representation of each utterance is one of the most important
procedures for both ND and DQ tasks. However, as described above,
the pre-training objective of BERT is to obtain the representationof a masked token rather than that of a sentence. In addition, although BERT has segmentation embeddings for indicating different
sentences, it only has two labels (sentences A and B), which meansthat it is inappropriate for dealing with a dialogue that has morethan two turns. Therefore, following the key idea of Liu [8], we
modify the input sequence and embeddings of BERT such that it
can obtain the representation of each utterance in a dialogue.
As illustrated in Figure 3, we insert [CLS] and [SEP] tokens
at the beginning and end of each utterance, respectively, and theembeddings of [CLS] are used as the representation of this utterance.
Similar to using BERT in a classification task, the [CLS] token isconsidered to be able to aggregate features from the corresponding
sentence. Hence, this modification makes BERT able to intuitively
obtain the respective representation of each utterance in a dialogue.
Meanwhile, we use interval segment embeddings to distinguish
multiple utterances within a dialogue. Specifically, for 𝑠𝑒𝑛𝑡𝑖 , weassign a segment embedding 𝐸𝐴 or 𝐸𝐵 conditioned on 𝑖 as odd oreven. Moreover, as described in the task details, a dialogue is formedby merging all consecutive posts by either a customer or helpdesk,
which means that customer and helpdesk turns alternately appear.
For example, using 𝑏𝐶 and 𝑏𝐻 to indicate customer and helpdesk
turns respectively, a dialogue [𝑏𝐶 , 𝑏𝐻 , 𝑏𝐶 , 𝑏𝐻 , 𝑏𝐶 ] will be assigned
as [𝐸𝐴 , 𝐸𝐵 , 𝐸𝐴 , 𝐸𝐵 , 𝐸𝐴 ]. In other words, for the segment embeddingsof BERT, all customer turns are assigned as 𝐸𝐴 , and all helpdesk
turns are assigned as 𝐸𝐵 , which informs the model of dialogue
structure.
In the following, 𝑇𝑖 denotes the vector of the 𝑖-th [CLS] symbolfrom the top BERT layer and will be used as the representation forutterances 𝑠𝑒𝑛𝑡𝑖 .

ℎ˜𝑙 = LN(hl−1 + MHAtt(hl−1 )),

(1)

l

(2)

𝑙

l

ℎ = LN( h̃ + FFN( h̃ )),where ℎ 0 = PosEmb(T ) and 𝑇 are the utterance vectors output
by BERT. PosEmb represents positional encoding, a function thatindicates the information about the relative or absolute positionof the utterance in the dialogue. LN is the layer normalization
operation, and MHAtt is the multi-head attention operation. The
superscript 𝑙 indicates the depth of the stacked layer.
For the ND subtask, we applied two densely connected NN layers over this Transformer encoder for generating customer andhelpdesk nuggets. Meanwhile, for the DQ subtask, we first employed a global average pooling layer [7] to obtain the representation of the entire dialogue, and then applied three densely connected
NN layers over it to generate A, E, and S score, respectively.
In addition, in order to verify the effect of this Transformer layer,
we also implemented a baseline model for the DQ subtask, i.e., a
dialogue embedding BERT model with a simple dense layer placed
over it. In other words, the Transformer encoder layer was excluded.

4 EXPERIMENTS
4.1 Run description
The details of our official runs are shown in Table 1. For the dialoguelevel model, BERT and the other layers (Transformer encoder anddense layer) are fine-tuned jointly. We utilize Adam with 𝛽 1 =0.9, 𝛽 2 = 0.98 for the optimization and cross-entropy as the loss
function. All optimizers of our official run models follow the learning rate schedule of Vaswani et al. [12], with warming-up on first
10,000 steps:
𝑙𝑟 = 2𝑒 −3 · min(step−0.5, step · warmup−1.5 )All models are trained for 100 epochs. Model checkpoints are
saved once a better dev score is obtained. We chose the JSD score
for the ND subtask and the average NMD score for the DQ subtask
as our dev score.

3.2.2 Fine-tuning. After obtaining the utterance vectors of a dialogue, it is necessary to implement a layer to capture the relationship between these utterances. In our model, we employ a68

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Input Dialogue

[CLS]

cust

utte

[SEP]

[CLS]

help

utte

[SEP]

[CLS]

cust

utte

[SEP]

Token EmbeddingsInterval Segment
Enbeddings
Position Embeddings
BERT

……

……

……

Fine-tuning Layers

Figure 3: Overview architecture of the dialogue-level model. Here “cust" and “helpdesk" are abbreviations for customer andhelpdesk, respectively, and “utte" indicates an utterance.
Table 2: Chinese Nugget Detection Results
Run
RSLDE-run0
BL-LSTMRSLDE-run2
RSLDE-run1
BL-popularity
BL-uniform

Mean JSD
0.0560 (1)0.05850.06070.06340.18640.2042Run
RSLDE-run0
BL-LSTMRSLDE-run1
RSLDE-run2
BL-popularity
BL-uniform

Table 5: Chinese Dialogue Quality (E-score) Results

Mean RNSS
0.1604 (1)0.16510.17120.17200.29010.3371Run
RSLDE-run0
RSLDE-run1
BL-LSTM
BL-uniform
BL-popularity

Mean RSNOD
0.16600.17250.18540.24960.2569Run
RSLDE-run0
RSLDE-run1
BL-LSTM
BL-popularity
BL-uniform

Mean NMD
0.1222 (2)0.12860.15790.17100.2106

Table 6: English Nugget Detection Results
Table 3: Chinese Dialogue Quality (A-score) Results
Run
BL-LSTM
BL-popularity
RSLDE-run0
RSLDE-run1
BL-uniform

Mean RSNOD
0.23010.23200.24380.2446
0.2767Run
RSLDE-run0
RSLDE-run1
BL-popularity
BL-LSTM
BL-uniformRun
RSLDE-run0
BL-LSTMRSLDE-run2
RSLDE-run1
BL-popularity
BL-uniform

Mean NMD
0.15370.15510.15770.17720.2500Mean JSD
0.0557 (1)0.06250.0676
0.06910.18640.2042Run
RSLDE-run0
BL-LSTMRSLDE-run2
RSLDE-run1
BL-popularity
BL-uniform

Mean RNSS
0.1615 (2)0.17220.17780.18530.29010.3371

Table 7: English Dialogue Quality (A-score) Results
Table 4: Chinese Dialogue Quality (S-score) Results
Run
RSLDE-run0
RSLDE-run1
BL-LSTM
BL-popularity
BL-uniform

Mean RSNOD
0.19380.19640.19980.20620.2959Run
RSLDE-run1
RSLDE-run0
BL-popularity
BL-LSTM
BL-uniform

Run
BL-popularity
BL-LSTMRSLDE-run0
RSLDE-run1
BL-uniform

Mean NMD
0.12290.12430.12880.15230.2565

4.2

Mean RSNOD
0.23200.23210.26150.27250.2767

Run
BL-popularity
BL-LSTMRSLDE-run1
RSLDE-run0
BL-uniform

Mean NMD
0.15770.17800.1896
0.19570.2500Results and Discussion

We list our experiment results of our official runs for the DialEval-2
task in Tables 2 to 9. The numbers in parentheses indicate the ranking of our runs according to the official results [11]. The organisers

69

NTCIR 16 Conference: Proceedings of the 16th NTCIR Conference on Evaluation of Information Access Technologies, June 14-17, 2022 Tokyo Japan

Table 8: English Dialogue Quality (S-score) Results
Run
BL-LSTM
BL-popularity
RSLDE-run0
RSLDE-run1
BL-uniform

Mean RSNOD
0.19860.20620.20780.21540.2959Run
BL-popularity
RSLDE-run0
RSLDE-run1
BL-LSTM
BL-uniform• The XLNet model has an outstanding language understanding capability for customer-helpdesk dialogues.
• Considering the structure and context information of a dialogue is important for the dialogue nugget detection.
In future studies, it will be worth searching for technical aspects
that make the hyperparameters suitable for the transfer learning ofa Transformer-based model’s.

Mean NMD
0.12880.13810.14380.14670.2565ACKNOWLEDGEMENT
We thank the DialEval-2 organisers and the NTCIR chairs for giving
us the opportunity to participate in this task.

Table 9: English Dialogue Quality (E-score) Results
Run
BL-LSTMRSLDE-run0
RSLDE-run1
BL-uniform
BL-popularity

Mean RSNOD
0.17450.18320.18890.24960.2569Run
RSLDE-run0
BL-LSTMRSLDE-run1
BL-popularity
BL-uniform

Mean NMD
0.14290.14310.14440.17100.2106REFERENCES[1] Ting Cao, Fan Zhang, Haoxiang Shi, Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai,
Injae Lee, Kyungduk Kim, and Inho Kang. 2020. RSLNV at the NTCIR-15 Dialogue
Evaluation (DialEval-1) Task. In Proceedings of the 15th NTCIR Conference on
Evaluation of Information Access Technologies. 57–61.
[2] Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu.
2020. Revisiting Pre-Trained Models for Chinese Natural Language Processing.
In Proceedings of the 2020 Conference on Empirical Methods in Natural Language
Processing: Findings. Association for Computational Linguistics, Online, 657–668.
https://www.aclweb.org/anthology/2020.findings-emnlp.58[3] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan
Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixedlength context. arXiv preprint arXiv:1901.02860 (2019).
[4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert:
Pre-training of deep bidirectional transformers for language understanding. arXiv
preprint arXiv:1810.04805 (2018).
[5] Mike Tian-Jian Jiang, Yueh-Chia Wu, Sheng-Ru Shaw, Zhao-Xian Gu, Yu-Chen
Huang, Min-Yuh Day, Cheng-Jhe Chiang, and Cheng-Han Chiu. 2020. IMTKU
Multi-Turn Dialogue System Evaluation at the NTCIR-15 DialEval-1 Dialogue
Quality and Nugget Detection. In Proceedings of the 15th NTCIR Conference on
Evaluation of Information Access Technologies. 68–74.
[6] Xin Kang, Yunong Wu, and Fuji Ren. 2020. TUA1 at the NTCIR-15 DialEval Task.
In Proceedings of the 15th NTCIR Conference on Evaluation of Information AccessTechnologies. 53–56.
[7] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard,
Wayne Hubbard, and Lawrence Jackel. 1989. Handwritten digit recognition witha back-propagation network. Advances in neural information processing systems
2 (1989).
[8] Yang Liu. 2019. Fine-tune BERT for extractive summarization. arXiv preprint
arXiv:1903.10318 (2019).
[9] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. (2018).
[10] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever,
et al. 2019. Language models are unsupervised multitask learners. OpenAI blog
1, 8 (2019), 9.
[11] Sijie Tao and Tetsuya Sakai. 2022. Overview of the NTCIR-16 Dialogue Evaluation(DialEval-2) Task. In Proceedings of NTCIR-16. to appear.
[12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all
you need. In Advances in neural information processing systems. 5998–6008.
[13] Marilyn A. Walker, Diane J. Litman, Candace A. Kamm, and Alicia Abella. 1997.
PARADISE: A Framework for Evaluating Spoken Dialogue Agents. In 35th Annual
Meeting of the Association for Computational Linguistics and 8th Conference of theEuropean Chapter of the Association for Computational Linguistics. Association
for Computational Linguistics, Madrid, Spain, 271–280. https://doi.org/10.3115/
976909.979652[14] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue,
Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.
2019. Huggingface’s transformers: State-of-the-art natural language processing.
arXiv preprint arXiv:1910.03771 (2019).
[15] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov,and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language
understanding. Advances in neural information processing systems 32 (2019).
[16] Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang. 2020. Overview of
the NTCIR-15 Dialogue Evaluation (DialEval-1) Task. In Proceedings of the 15th
NTCIR Conference on Evaluation of Information Access Technologies. 13–34.
[17] Zhaohao Zeng, Cheng Luo, Lifeng Shang, Hang Li, and Tetsuya Sakai. 2018.
Towards Automatic Evaluation of Customer-Helpdesk Dialogues. Journal of
Information Processing 26 (2018), 768–778. https://doi.org/10.2197/ipsjjip.26.768

of DialEval-2 provided three baseline models: a LSTM baseline, auniform baseline, and a popularity baseline. The details of thesethree baselines can be found in the overview paper [11].
As shown in Table 2 and Table 6, although our XLNet baselinemodel is simply a sentence-level model, it still outperformed allother participant runs for both English and Chinese ND tasks. In
other words, our XLNet baseline model outperformed our BERT
baseline model for the sequence labeling task, which is in agreementwith the results found by the author of XLNet [15]. In addition, bycomparing the results of RSLDE-run2 and RSLDE-run1, as shown in
Table 2 and Table 6, it can be observed that the dialogue-level model
outperformed the BERT baseline model although they both used
BERT, which means that it is meaningful to consider the structure
and context information of a customer-helpdesk dialogue to detect
its nugget.
According to Tables 3 to 5, except for the mean RSNOD of the Ascore, both RSLDE-run0 and RSLDE-run1 outperformed the official
baselines for the Chinese DQ subtask. Moreover, for most situations,the score of RSLDE-run0 is higher than that of RSLDE-run1, whichmeans that our Transformer encoder layer was able to extract thefeatures of the whole dialogue and evaluate its quality. However,as shown in Tables 7 to 9, neither model outperformed the official
baselines for the English DQ subtask. This could be because thehyperparameters we used for the transfer learning are unsuitable
for the English task.

5

CONCLUSION

In this paper, we proposed several models for automatically evaluating customer-helpdesk dialogues, i.e., two sentence-level models
that respectively fine-tune BERT and XLNet as our baseline models,
and two dialogue-level models. The two dialogue-level models modify the input sequence and embeddings of BERT to better capture
the structure and context of a dialogue. One also uses a Transformer
encoder to extract the dialogue-level feature, and the other, whichlacks this Transformer encoder, is our baseline model for the DQ
subtask.
Based on the official results, the following can be concluded:70