Learning Locality and Isotropy in Dialogue Modeling
Authors: Han Wu
Haochen Tan
Mingjie Zhan
Gangming Zhao
Shaoqing Lu
Ding Liang
Linqi Song
Venue: ICLR 2023 poster (2023)
Keywords: dialogue system
representation learning
feature space calibration

Abstract: Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context representations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.e., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, to build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks. More in-depth analyses further confirm the effectiveness of our proposed approach. We release the code at https://github.com/hahahawu/SimDRC.

Published as a conference paper at ICLR 2023
LEARNING LOCALITY AND ISOTROPY IN DIALOGUE MODELING
Han Wu Haochen Tan 12, Mingjie Zhan?, Gangming Zhao?, Shaoqing Lu", Ding Liang', Linqi Songl.2*
DDeaartment of Computer Science, City University of Hong Kong
City University of Hong Kong Shenzhen Research Institute
"Sensetime Research
(hanwu32-c, haochetan-2 @my cityu. edu.hk 1 linqi. song@city.. edu. edu.hk
ABSTRACT
Existing dialogue modeling methods have achieved promising performance on various dialogue tasks with the aid of Transformer and the large-scale pre-trained language models. However, some recent studies revealed that the context represen- tations produced by these methods suffer the problem of anisotropy. In this paper, we find that the generated representations are also not conversational, losing the conversation structure information during the context modeling stage. To this end, we identify two properties in dialogue modeling, i.c., locality and isotropy, and present a simple method for dialogue representation calibration, namely SimDRC, build isotropic and conversational feature spaces. Experimental results show that our approach significantly outperforms current state-of-the-art models on three open-domain dialogue tasks with eight benchmarks. More in-depth analyses fur- ther confirm the effectiveness of our proposed approach. We release the code at attps://githut hww.om/whhwawwwu/aooooooooaaaaaa Simnn/iimmm
INTRODUCTION
Dialogue modeling (Serban et al., 2016 Mehri et al. 2019 Liu et al. 2021 is to encode the raw text of the input dialogue to the contextual representations. Although the Transformer-basee dialogue modeling methods (Hosseini-Asi) et al. 2020; Liu et al., 2021) have achieved great success on various dialogue tasks, there are still some impediments these methods that are not well explored nowadays. Specifically, recent studies (Ethayarajh) 2019, Su et al. 2022) have revealed that on dialogue generation tasks, the representations produced by existing dialogue modeling methods are anisotropic, i.e. features occupy narrow cone in the vector space, thus leading to the problem of degeneration. To alleviate this problem, previous solutions (e.g. SimCTG) (Su et al. 202T 2022) encourage the model to learn isotropic token embeddings by pushing away the representations of distinct tokens. While building the more discriminative and isotropic feature space, these methods still ignore learning dialogue-speciff features, such as inter-speaker correlations and conversational structure information, in the dialogue modeling stage. Therefore, a question is naturally raised- are the representations produced by existing dialogue modeling methods really conversational?
To answer this question, in Figure aa), we showcase the cosine similarity matrix of token repre- sentations produced by BART (Lewis al. 2020) that is well trained response generation task. First, we can easily observe the phenomenon of anisotropy from the heatmap where the similarities of distinct tokens are relatively high, over 0.5 for most token pairs. Then, Figure T[b) illustrates the similarity heatmap of token representations produced by SimCTG where the color is faded on the whole, suggesting the problem of anisotropy is relaxed. However, another critical problem still remains, is that the representations of tokens in different utterances are nearby to each other, making the utterance indistinguishable on the token representations. features can be captured from the token similarity matrix while the matrix is produced by "dialogue modeling" method trained on the dialogue task using dialogue data. Ideally, we expect that the
undesirable that no conversational
Corresponding author.

---

Published as a conference paper at ICLR 2023
Hour yourenaage
[[Ck]
Let Son
121 11 To
Figure 1: Illustaatoons of the token cosine similarity matrix produced by BART (a), SimCTG (Su et al. 2022) (b) and our proposed SimDRC (c).
representations of tokens within an utterance are close to voice a concentrated idea of the utterance, and the representations of different utterances are discriminative and isotropic to convey the maximal information of the dialogue. Accordingly, the ideal similarity matrix of token representations should be similar to Figure (c)) where tokens within utterance intensive and different utterances easily distinguishable on representations. Our motivation is that humans pay more attention to the central idea of the utterance rather than how the utterance is organized by words when humans utter, and humans also prefer to express more information with fewer utterances (Woolnough et al., (021).
Based on the above observation and motivation, we identify two properties, i.e., locality and isotropy in dialogue modeling, and then present SimDRC, a simple dialogue representation calibration method, which encourages the model to aggregate the representations of tokens within an utterance and push away the representations of distinct utterances. We evaluate our approach on three open-domain dialogue tasks, including multi-turn dialogue response generation (Li et al. 2017). conversational response retrieval (Lowe et al. 2015) and conversational semantic role labeling (Xu et al., [02TT) The experimental results show that our approach achieves comparable or better performance against the current state-of-the the-art methods across the three dialogue tasks on both automatic and human evaluations. In-depth analyses towards the effects of the locality and isotropy further verify the effectiveness of our proposed approach.
and the measurements of
RELATED WORK
2.1 DIALOGUE MODELING
Dialogue modeling is to transform the raw text of the dialogue to readabbe representations. which is an indispensable step to most dialogue tasks (Li al. 2017] Liu et al. 2021). To achieve
this goal, conventional approaches (Serban al. 2016 2017 Xing et al. (018)) with recurrent neural networks (RNN) (Hochreiter Schmidhuber 1997| Mikolov et al.) 2010) prefer to hierarchically
learn the representations due to the long distance dependency problems in RŇNs. With the remarkable success of Transformer (Vaswani et al.) 2017) and pre-trained language models (Devlin et al. [2019] Raffel et al. (0200) on various NLP tasks, Transformer-a based dialogue modeling methods (Hosseini Asi et al. 2020 Gu et al. 2020 Liu et af. 2021] Wa et al. 202T) are widely used, and significantly outperform the traditional methods many dialogue tasks, such response generation (Li et al.) 2017) and response retrieval (Lowe et al., (015).. In this work, we concentrate on the standard Transformer-based dialogue modeling method, which directly encodes the flattened dialogue context with the pre-trained language models. By studying the geometry of the representation space of the model, we find that the contextual representations produced by existing dialogue modeling methods are not isotropic and conversational.
2.2 REPRESENTATION CALIBRATION
Outside dialogue modeling, many other representation learning approaches also attempt to normalize their feature distributions from different perspectives. A bunch of studies theoretically verify that
2

---

Published as a conference paper at ICLR 2023
uniformly distributing points unit hypersphere provide good representations (Bojanowski & Joulin 2017; Wang & Isola 2020). Specifically, Wang & Isola (2020) directly uses the alignment and uniforitt properties to guide the representation learning. finally achieving comparable even better performance on downstream tasks than the original contrastive learning framework. With the rapid spread of Transformer, set of works (ongg et al. [02T]] Su al. (022) also empirically explore the intrinsic properties of representations produced by the models, and find that the contextual words representations from most models anisotropy, occupying narrow cone in the feature space (Ethayarajh 2019). This observation strongly agrees with our intuition that refining dialogue feature space a desirable step. To this problem. SimCTG (Su et al.) (022) presents token-level contrastive learning framework to calibrate the model's representation space regarding the anisotropy problem. However, the essential difference with our work that we normalize the feature space on multi-granularity levels instead of just the token level, which matches better with the structural characteristics of the dialogue and produces more isotropic and conversational representations.
3 METHODOLOOGY
3.1 DIALOGUE MODELING
Given dialogue D {ui U2 ... uNl consisting of N utterances, where 1 W4,2, ...: W6, ] is sequence of words, the goal of dialogue modeling is to learn the con- textual representation
{h1,1.... ha.f.... by.lug1 over the dialogue context. Typically, the Transformer-bart based dialogue modeling method obtain the dialogue contextual representations by taking the utterances as a consecutive sequence with special separate tokens, which presented as:
H Prl.M(u1 [EOU] [EOU] Un [FOU] [CONTEXT]).
where [EOU] and [CONTEXT] are special tokens which would further be used to represent the whole utterance or dialogue. [EOU] and [CONTEXT] are respectively inserted as the last token of each utterance and the last token of the whole dialogue.
3.2 LOCALITY AND ISOTROPY DISTANCE
Conventional wisdom says that good dialogue contextual representation should be informative, discriminative and structurally sensitive (Gu et al. 2020 Wu et al. [222] Suu et al. 2022).. Therefore, we identify two properties for good dialogue representations as tollowing:
Locality: tokens within an utterance are mapped to nearby features, thus concentrating on expressing core idea of the utterance.
Isotropy: utterance representations are roughly uniformly distributed in the feature space, thus preserving as much information of the whole dialogue as possible and enabling the utterance discriminative to each other.
For locality, encourages the learned feature representations of the tokens within an utterance be similar. In practice, the special token (i.c.. [EOU]) is often served as the representative token of the utterance to speak for the utterance (Zhu by bridging the gap between each token and the representative token of the utterance it belongs. The benefits of this design are 1) the representations of the tokens within an utterance are pushed close to the representation of the utterance representative token for better expressing the utterance's s concentrated idea; 2) the representative token are encouraged to combine the information from all tokens within the utterance, thus becoming more informative and representative. Formally, given dialogue D = [w1,1. Wi ..., WN, N.Jaxll }. for each token [ in the context, we obtain the locality value by calculating the cosine similarity between the token and the corresponding utterance representative token W/.
2020 Liu al. 202T). Therefore. we model locality
h he,jall
locality_value,.u
hhsll |l [
3

---

Published as a conference paper at ICLR 2023
where b is the representation of token way- The value will be larger when the two vectors closer. Then, we compute the overall locality distance for the whole dialogue context as:
N bull
locality cestane_disttnceppo u1
Σ
locality_valuu _aaitty_aluuee_pp (i.))*
+ +
UN
N
For isotropy, it encourages the model push away the features of distinct utterances, thus preserving the maximal information of the dialogue and enabling the utterance discriminative and isotropic. Similar to locality, model isotropy by enlarging the gap between distinct representative tokens. Formally, given two distinct utterances u, and 4j we obtain the isotropy value by computing the cosine similarity between two utterance representative tokens,
T 5.jo:l Alull
isotropy_valuee (i.))
(4)
Similarly, the overall isotropy distance of the whole dialogue context is formulated as:
N
isotropy ddistance
isotropy isotropy_value
(5)
(N
11
3.3 ADJUSTING FEATURE SPACE
Based on the locality distance and isotropy distance, we present locality loss, isotropy loss and SimDRC loss to adjust the feature space for modeling better representations.
Locality loss tations the corresponding utterance representative token. Therefore, we introduce locality loss, aiming at maximizing the locality distance defined in Equation
attempts to close the representations of the tokens within an utterance to the represen-
N 1ul-1
Clocality
max 81 locality locality_valucc..
(6)
11 ... + uN N
where 61 E [0, 1] is the margin value, When 81 equals to 0, the loss degrades to 0.
Isotropy loss tries push away the representations of distinct utterances. To this end, we introduce isotropy loss to minimize the isotropy distance defined in Equation 5
Lisntropy
max(0, 82 _satropy
N
N
1=
where 82 E [-1, is the margin value. When dg equals to -1, the loss degrades to O.
SimDRC loss combines the locality loss and isotropy loss to calibrate the representation space. Formally, present the SimDRC loss as,
LSimDRC acCiocality (0) (1 0i) Lisotropy (8)
where 5 E 1] is the margin value, shared between the locality loss and isotropy loss. is the syper-parameter of loss weight. The SimDRC loss degenerates to locality loss when a = 1, and to isotropy loss when -1 < 5 < 0 or 0, and to 0 when 8 -1. It is worth noting that, the isotropy loss is more sparse because it only works on a few utterance representative tokens while the locality loss computes all tokens within the utterances. Therefore, the model would be biased modeling
4

---

Published as a conference paper at ICLR 2023
DuilyDislgg
LCCC
Model
Method
BERTSSooe
BERTSoee
Pt
X1
F7
P7
RT
F7
Dis2/4
groady
12.31
10.51
0.044
0.206
0.133/0.511
heam
12.29
12.85
12.05
0.501/0.676
-6.21
0.214
0.17840.34
BAKT
sucicus
12,50
12:55
12.15
-3.58
0,410
0,503/0.666
7.55
9.34
8.14
-6.19
0,215
0.172/0.221
consmattive
6.36
5.22
6.50
-.22
0.270
0,273/0.611
4.64
5.62
4,81
6.31
0.175
0,155
greedy
11.39
0.408
0.216
0.17840.33
SimCTG
besn
0.270
0.237A0.603
0.375
0.15500.03
(p-0.5)
sucleus
10.85
12.56
10.96
-3.62
0.395
0,28900,672
6.82
9.31
7.75
6.18
0,206
0.133/0.481
contmadtive
11.35
12:77
0.488
0.283/0.650
0.218
0.173/05504
SinDRC
groody
12.91
0.238
0:275/0:644
0.92
10:755
0.15410,444
(5-0.7
beam
12.67
14.12
12.89
-3.9
0.408
0,285/0.651
7.65
9.33
0.216
0.172/0.524
sucleus
12.85
14.14
12.97
-3.55
0.419
0,318/07111
7.77
9.48
8.32
-6.21
0.219
0.181M.543
(w-1.3)
controotive
12.73
0.408
0.32800.447
6.92
0.207
0.13410.457
Table 1: Results of automatic evaluation on the DailyDialog and LCCC datasets. BAS means the BARTScore (Yuan al. 202T). BT means the BLEURT score (Sellam et al. 2020).
more local features we give moderate values of 5 and a (e.g. ő=0, a=0.5). To this problem, we empirically scale up the isotropy loss by applying the large value of å and the small value of a.
4
EXPERIMENTS
We evaluate our method on three kinds of dialogue tasks, i.e., multi-turn dialogue response generation (Li et al. 2017), conversational response retrieval (Lowe et al. (2013) and conversational semantic role labeling (uu et al. 202T). Note that we just focus open-domain dialogue tasks in this work since previous studies (Jakobovits et al.) 2022) revealed that most task-oriented datasets, like MultiWOZ (Budzianowski et al. 2018) are essentially not conversational and contextualized while method aims to learn these features dialogue modeling stage.
4.1 MULTI-TURN DIALOGUE RESPONSE GENERATION
Task & Data Multi-turn dialogue response generation is to automatically produce the human-like response given the dialogue context. Hopefully, we expect the generated responses are diverse, fluent, coherent and informative. We evaluate our approach DailyDialog (Li et al. (017) and LCCC (Wang et al. 2020), wherein DailyDialog is a multi-turn open-domain English dialogue dataset and LCCCC is open-domain Chinese short-text conversation dataset.
Models & Decoding We compare the vanilla BART and DialoGPT models to their fine-tuned versions with SimCTG and SimDRC. For each model, we conduct the inference using four decoding strategies, including 1) greedy search; 2) beam search (beam size = 5); 3) nucleus sampling (p 0.95)
(Holtzman et al. 2019); 4) contrastive decoding (k = 5, 0.6) (Su et al., 2022).
Evaluation It is long-standing problem accurately and comprehensively evaluate the quality of the generated texts (Celikyilmaz et al.) 2020). The traditional n-gram overlap and text matching metrics, such as BLEU (Papineni et al. 2002) and ROUGE (Lin) 2004), are not good choices to open-domain dialogue systems which permit significant diversity and allow multiple plausible outputs for given input. Therefore, we choose measure the generation quality over following automatic evaluation metrics, including BERTScore (Zhang et al., 2019), BARTScore (Yuan et al. 2021
BLEURT (Sellam al. 2020), Distinct2/4 (Li [d.][221))
We also conduct the human evaluation with the help of proficient English/Chinese speakers recruited from the crowdsourcing platform. We randomly sample 200 dialogue contexts from DailyDialog and LCCC test sets, respectively. For each dialogue context, we generate responses using different models (vanilla BART, BART+SimCTG and BART+SimDRC) with different decoding strategies (greedy, beam search, nucleus sampling and contrastive decoding). Five annotators are employed to independently evaluate these 2,400 samples by giving a score ranging from to over following aspects, including fluency, informativeness. coherence, and semantic coverage"
Mooe details of human evaluation Appendix
5

---

Published as a conference paper at ICLR 2023
DiilyDilog
LOCC
Model
Plueney
Infremativeness
Coherence
Coverage
Fluency
Informatieness
Coherence
Coverage
Agreement
0.65
0.56
0.71
0.72
0.66
0.78
0.70
0.68
groedy
4,62
3
3.72
2.50
4.65
2.84
3.58
2.09
3.5]
BART
nucleus
1.66
3,5
3.53
2.44
4.64
2.96
3.57
2,06
contrastive
4.63
2,00
2.44
1.27
4.61
3.00
3.18
1.79
greedy
4.53
3.50
3.69
2.51
1.62
2.87
3.63
2,05
SimCTO (p-0.5)
4.65
2.54
nucleus
4.69
3.52
3.52
2.45
4.42
2.6
3.31
2.13
contrastive
4,70
3,57
3,70
2.66
4.68
2.94
3,66
2.13
SimDRC (6-0.7.
groety
4.58
2.87
3.22
1.83
3.03
3.39
1,86
4.68
3.53
3,67
2.63
4.64
2.92
3.67
nuclous
4.76
3,66
3,90
2.71
4.68
3
3.70
2.23
o=0.3)
contrastive
4.62
3.69
3.77
2.61
4.50
2,95
3.27
1.99
Table 2: Results of human evaluation on the DailyDialog and LCCC datasets.
Results & Discussion Table lists the automatic evaluation results of the different methods (i.e. BART, BART+SimCTG and BART+SimDRC? with the different decoding strategies. On both datasets, we can see that SimDRC, especially when equipped with nucleus sampling, achieves the best performance across the board (sign test p < 0.01 compared that SimDRC can produce diverse (high distinct scores), semantically consistent (high BERTScores, BLEURT scores and BARTScores) and informative (high distinct and BART scores) responses. Inconsistent with the findings of SimCTG where their human evaluation showed that both SimCTG and contrastive decoding could improve the quality of the generated texts on the two datasets in English and Chinese, our automatic evaluation results find the difference that SimCTG+contrrasivee performs comparably with our method on the LCCC dataset, but performs poorly on the DailyDialog dataset. In contrast, SimDRC constantly performs well regardless of the decoding methods and datasets used, suggesting SimDRC is a more robust solution. We also conduct ablation studies regarding the locality and isotropy training objectives. We find that after removing either locality loss or isotropy loss, the quality of generated texts degrades to a certain extent (details Section (.2).
indicating
Table 2 summaries the results of human evaluation on DailyDialog and LCCC datasets. The first row indicates the inter-rater agreements calculated by Fleiss' kappa coefficient (Fleiss) 1971). In the rest part, we find that the results of human evaluation are highly consistent with automatic evaluation's showing that SimDRC+nucleus significantly outperforms baselines across the board. The exception is that SimDRC+comtrastive obtains better informativeness score than SinDRC+nucleus on DailyDialog. We think this is reasonable since contrastive decoding encourages the model to generate the response with fewer repetitions and more novel terms. It is also in line with the results of automatic evaluation that SimDRC+contrastive achieves better distinct scores on DailyDialog.
4.2 CONVERSATIONAL RESPONSE RETRIEVAL
Task & Data Conversational response retrieval (Lowe et al. 2015) is to automatically select the appropriate response from the corpus given prefix context. Unlike response generation which produces the target sequence word by word, response retrieval finds the potential responses by matching the context representations with the candidate utterance representations. We evaluate our approach on three commonly used response retrieval datasets, including Douban (Wu et al. (017)). E_ commerce (Zhang et al. 2018) and Ubuntu (Lowe al. 2015), wherein Douban a daily chit-chat dataset while E commerce and Ubuntu are domain-specific.
Model & Baselines As our approach is aechttectrr-aggnstii and can be applied to most existing models, choose the current performinn response retrieval model, i.e. BERT-FP (Han et al. 2021), as our backbone. BERT-FP is a simple BERT-basee retrieval model but benefits from the fine- tme-ggained post-training that reflects the characteristics of the multi-turn dialogue. We directly apply SimDRC or SimCTG to the hidden states from the last layer of the original BERT-FP model.
Results & Discussion Table summarizes the evaluation results on three datasets. We can see that our approach (BERT-FP + SimDRC) achieves comparable or better performance against the
We put the results of DialoGPT models in Appendix C The findings and conclusions the DialoGPT models similar the BART models".

---

Published as a conference paper at ICLR 2023
Ubeuta
Douhan
E-commeerca
Rio 01
Rgg 685
MAP
MRR
Pg8
R1081
R10
Rig
R1081
R1o82
R1o 655
Prev. SoTA
0.914
0.964
0.995
0.633
0.669
0,489
0.311
0,520
0.867
0.865
0.948
0.993
SimCTO(Ge0055
0.934
0.324
SimDRC(5=00.8
0.913
0,963
0.995
0.643
0.81
0.510
0.336
0.536
0.872
0.869
0,959
0.991
only w/ locality
0.113
0,963
0.994
0.637
0.080
0.514
0.325
01.32
0.856
0.808
0.969
0.992
only w/ inetropy
0.912
0,963
0.993
0.638
0.680
0.510
0.325
0,530
0.867
0.866
0.958
0.991
Table 3: Evaluation results the Ubuntu, Douban and E-comercce corpus. reproducing results based on the public code released by (Han et al. (021).
indicates
DuConv
NewsDialog
PersonaDDalog
Method
Fly
crous
Flu!
Flaros
Flittr
Flun
Flaross
Prev. SoTA
89.47
84.57
90.15
80.86
55.54
84.24
71.82
36.89
75.46
SinCTG (p 0.2)
89.28
83.17
90.27
80.86
57.28
84.18
71.50
43.45
73.83
SinDRCI 0.5.
89.83
84.75
90.51
81.22
57.77
84,50
74.03
48.64
76.84
0= 0.2)
only w/ locality
89.61
87.33
90.28
81.63
33..5
84.82
72.74"
43.66
75.39
only w/ isotropy
89.30
80.35
71.43
Table 4: Evaluation results on the DuConv, PersonalDialog and NewsDialog datasets.
baselines over all metrics, indicating that SimDRC is also an effective solution to retrieval- based tasks. Specifically, our approach reaches the new state-of-the-ard performance on the Douban corpus (sign test p 0.01), but we find few performance drops on the Ubuntu and E-commerce corpus. We think this is because there is an extremely small room for performance improvements since the previous best-performing model has achieved over 99% accuracy Rio @5. For ablation studies, we find separately employing locality or isotropy also outperforms the baselines.
4.3 CONVERSATIONAL SEMANTIC ROLE LABELING
Task & Data Conversational semantic role labeling (CSRL) (Xu al.) 2021 challenging dialogue understanding task, which extends the standard SRL task (Palmer et al. (010)) to dialogue scenario and aims to find the potential arguments over the entire conversation with the given predicate. Following previous work (Xu et al. (021), we conduct experiments on three CSRL datasets, namely DuConv, NewsDialog and PersonalDialog DuConv annotations are split into 80%/10%/10% as train/dev/test sets, while NewsDialog and PersonalDialog used as out-of-domain test sets.
Model & Baselines We adopt CSAGN (Wu et al. 2021) model as our baseline method. CSAGN firstly obtains the dialogue representations from pre-trained BERT, and then employs predicate- aware module and conversation structure-awar graph network to capture high-level features. We apply SimDRC or SimCTG on the generated representations from the last layer of BERT.
Results & Discussion Table 4 shows the evaluation results on three datasets. We can that our approach (CSAGN+SimDRO) significantly outperforms all baseline models across the board (sign test with p-value < 0.01), suggesting that SimDRC helps the model to capture both local and conversational features. SimCTG helps the model find more cross-arguments on two out-of-domain test sets while also missing more intra-arguments. This is in line with the characteristies of SimCTG because pushing away all tokens makes the representations more isotropic but less concentrated on local information. Superior SimCTG, SimDRC improves both locality and isotropy. For ablation studies, we can find 1) individually adopting locality loss also outperforms the baselines; 2) locality consistently enhances the capacity of local feature modeling while isotropy tends enhance the capacity of cross-turn feature modeling.
5
IN-DEPTH ANALYSES
5.1 CONVERSATIONAL COHERENCE
main concern our approach that SimDRC may weaken the intrinsic correlations among utterances since it tries to push away the utterance representations and make them discrimina-
7

---

Published as a conference paper at ICLR 2023
care oe DuCame
CHAOO
2.4
Lockltt
241
EA
100h 9-0
(b)
Figure 2: (a) The effects of and 6. SimDRC-a/6 is the average score of all models trained with the candidate 8la. (b) The effect of margin value 5 when a = 0.5 on the DuConv test set.
tive. Intuitively, the most straightforward idea improving the quality of dialogue modeling should be strengthening the connections among utterances, thus making the dialogue more coher- ent. To this end, we study the conversational coherence in SimDRC-basee models. As there are available methods automatically measure the conversational coherence based on the contex- tual representations, we take the average score of the similarities between the dialogue embed- ding and the utterance embeddings the coherence score, which is formulated as: coherence
N E Vi, III reoeeul] where Vu; the utterance representation of utterance uy and Veontext is the dialogue representation. The motivation behind is that the utterances of a coherent dialogue should consistently follow main topic, which is expressed as the dialogue embedding in our settings. We calculate the coherence score the DailyDialop dataset. As shown in Table 5. SimDRC obtains the highest coher- ence score over other methods while the score of SimCTG is even lower than the vanilla BART model. We think this is because SimCTG constantly focuses on pushing away all token representations instead of modeling the conversational structure-aware repre- sentations, like SimDRC. Moreover, on the CSRL task. SimDRC greatly improves CSAGN's capacity of identifying cross-arguments, consistently showing its ability to enhance conversational coherence.
T Veuntess
Model
BART
SimCTG
SimDRC 0.413
Coherence
0.407
0.405
Table 5: Conversational coherence scores on DailyDialog.
5.2 EFFECTS OF LOSS WEIGHT AND MARGIN VALUE
Here, we explore how the loss weight a and margin value affects the performance of the response generation task. On the DailyDialog dataset, fine-tune the BART+SimDRC models with different loss weights
[0.1, 1,.10.. 0.9] and margin values E 10.1, 0.9) and measure the perplexity on the test set. We experimentally find that the large value of a (e.g. 0.6 0.9) and the marginal values of (e.g. 0.1 ~ 0.2, 0.8 ~ 0.9) would hurt the performance, which is consistent with the discussion in the last paragraph of Section 3.3 To this end, we narrow the candidate sets of a and 5 to a E [0.1. 0.5] and 8 E j0.3, 0.7]. Figure a plots the results of the models trained with different hyper-parameters can see that SimDRC consistently achieves comparable or better performance than the baselines. And it obtains the best performance when 0.3 and 8 0.5.
For the CSRL task, we find the variations a do not lead big weave of the performance. Based on this observation, we fix the value of 0.5 to study the effects of margin value 5 E [0.1, 0.9]. Figure (b) illustrates the results of the models trained with different margin values. We see that SimDRC obtains better scores at the most time. Still, the marginal values of 8 would affect the performance. Furthermore, individually introducing isotropy loss to CSAGN is not an effective solution while only using locality loss performs comparably. The optimal values of a and 8 generally differ from task to task. It would take much effort to search the best hyper-parameters for different tasks. With the extensive experiments in this work, we find that the settings of a [0.2, 0.4) and 8 E [0.4, 0.6] work well for most tasks.
5.3 MEASUREMENTS OF LOCALITY AND ISOTROPY DISTANCE
To ensure that SimDRC exactly optimizes the model regarding the properties of locality and isotropy, measure the locality distance and isotropy distance along with the training process. Specifically,
8

---

Published as a conference paper at ICLR 2023
Locaity Locdity of
x
Truining Steps
Figure 3: The values of locality distance (a), isotropy distance (b) and development loss calculated by checkpoints in different training steps.
sdlt hustt, haf.
hamb full.
pid
27 3" 1TP"N yIa
74 TTP'
2IS F (c) SimDRC Diiiydilogg
7 71""
(4) BART on Dallydisog
(b) SiCCTG on Dallddalog
(40 BART LCOC
(4) SinCTG un LOOC
30 SimDRC oz LCOC
Figure Token similarity matrices of different models trained on DailyDialog and LCCC.
we fine-tune the BART-based models on the DailyDialog training set and measure the locality and isotropy distances on the development set, following the 3quationnaann and Figure3]plots the values of locality, isotropy distances and the development loss at different training stages. We that SimDRC (red line) obtains much better locality and isotropy distances than the vanilla BART (blue line) and SimCTG (green line), suggesting that SimDRC is essentially working these two properties. Additionally, larger fluctuations on these two distance values are spotted when training with BART and SimCTG, while the values are steady with SimDRC, especially in terms of isotropy distance. We think this is also the reason that SimDRC has a more smooth development loss curve than baselines.
5.4 VISUALIZATION OF TOKEN SIMILARITY MATRIX
In this part, we visualize the token similarity matrices of different models, i.e. vanilla BART, BART+SimCTG and BART+SimDRC. We first fine-tune the models on DailyDialog and LCCC datasets, and then employ the well-trained models to encode the dialogue context and generate the token similarity matrices. see that I) The token similarities of vanilla BART are relatively high and uniform, i.e. the token close to most of the other tokens representations; 2) the token similarities overall decreased after introducing SimCTG, indicating it alleviates the problem anisotropy, especially on the LCCC corpus. However, the utterances still indistinguishable SimCTG also ignores modeling the conversational features; 3) the models with SimDRC generates isotropic and conversational representations, making all utterance discriminative on representations. The observations are in agreement with our motivations and discussions above.
showcases the resulted heatmaps on these two datasets. We can
representations, suggesting that
6 CONCLUSION AND FUTURE WORK
In this work, we present a simple dialogue representation calibration method to encourage the model to leam isotropic and conversational features in dialogue modeling stage. Experimental results show that our method achieves impressive performance on three dialogue tasks with eight benchmarks. For future work, applying SimDRC into dialogue pre-training would be promising research direction.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We sincerely appreciate the valuable and constructive comments from the reviewers. This work was supported part by the Changsha Science and Technology Program International and Regional Science and Technology Cooperation Project under Grants kh2201026 the Hong Kong RGC grant ECS 21212419, the Technological Breakthrough Project of Science, Technology and Innovation Commission of Shenzhen Municipality under Grants JSGG20201 102 102162000001, InnoHK initiative, the Government of the HKSAR, Laboratory for Al-Powered Financial Technologies, and the Tencent AI Lab Rhino-Bing Bird Gift Fund with Hong Kong RGC RMGS.
ETHICS STATEMENT
In this work, we use the publicly released datasets to train/valid/test our models. Generally, these previous works have considered the ethical issues when creating the datasets. For the datasets we used in this work, we have manually checked some samples, and do not find any obvious ethical concerns, such as violent or offensive content. We will also release the source code and the well-trained models along with friendly instructions to support its correct use. However, we still need to emphasize that text generation is not as controllable as we think. It still would generate some novel or unexpected words occasionally. We may take the actions to decrease the generation diversity to alleviate this problem.
REPRODUCIBILITY STATEMENT
In this work, present simple method for dialogue representation calibration. We have provided detailed descriptions of our proposed algorithm in Section For different downstream evaluated
tasks, we elaborate the experimental settings and hyper-parameters in Section 4 and Appendix B All datasets used in this work have been publicly released.
REFERENCES
Piotr Bojanowski and Armand Joulin. Unsupervised learning by predicting noise. In International Conference Machine Learning, pp. 517-526. PMLR, 2017.
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gasic. Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task- oriented dialogue modelling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 5016-5026, 2018.
Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. Evaluation of text generation: A survey. arXiv preprint arXiv:2006. 14799, 2020.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, 2019.
Yihe Dong, Jean-Bao Caan-Baptiste Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793-2803. PMLR, 2021.
Kawin Ethayarajh. How contextual contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 55 55-65, 2019.
Joseph L Fleiss. Measuring nominal scale agreement among many raters. Psychological bulletin, 76 (5):378, 1971.
10

---

Published as a conference paper at ICLR 2023
Jia-Chen Gu, Tianda Li, Quan Liu, Zhen-Hua Ling, Zhiming Su, Si Wei, and Xiaodan Zhu. Speaker- aware bert for multi-turn response selection in retrieval- based chatbots. In Proceedings of the 29th ACM International Conference Information & Knowledge Management, pp.
2020.
Janghoon Han, Taesuk Hong. Byoungjae Kim, Youngjoong Ko, and Jungyun Seo. Fine- tine-griied post- training for improving retrieval-basee dialogue systems. In Proceedings of the 2021 Conference of the North American Chupter of the Association for Computational Linguisties: Human Language Technologies, pp. 1549-1558, 2021.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural compulation, 9(8): 1735-1780, 1997.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Ehsan Hosseini-Ass Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for lrik-triented dialogue. Advances in Neural Information Processing Systems, 13:20179-20155 2020.
Alice Shoshana Jakobovits, Francesco Piccinno, and Yasemin Altun. What did you say? task-oriented dialog datasets are not conversationall? arXiv preprint arXiv:2203.0343) 2022.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising eequence-to-sqquencc pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. A
objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, pp. 110-119, 2016.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: manually labelled multi-turn dialogue dataset. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 986-995, 2017.
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp. 74-81, 2004.
Longxiang Liu, Zhuosheng Zhang, Hai Zhao, Xi Zhou, and Xiang Zhou. Filling the gap of utterance- aware and speaker-aware representation for multi-turn dialogue. In The Thirty-Fifth AAAI Confer- ence on Artificial Intelligence (AAAI-21), 2021.
Ryan Lowe, Nissan Pow, Tulian Vlad Serban, and Joelle Pineau. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 285-2944 294, 2015.
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. Pretraining methods for dialog context representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3836-3845, 2019.
Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjcev Khudanpur. Recurrent neural network based language model. In Interspeech, volume 2. pp. 1045-1048. Makuhari, 2010.
Martha Palmer, Daniel Gildea, and Nianwen Xue. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 3(1):1-103, 2010.
Kishore Papineni, Salim Roukos, Todd Ward. and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.

---

Published as a conference paper at ICLR 2023
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. Bleurt: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, 2020.
Julian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Tulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Yixuan Su, Fangyu Liu, Zaiqiao Meng, Tian Lan, Lei Shu, Ehsan Shareghi, and Nigel Col- lier. Tacl: Improving bert pre-training with token-aware contrastive learning. arXiv preprint arXiv:2111.04198, 2021.
Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. contrastive framework for neural text generation. arXiv preprint arXiv:2202.0644 06417, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International Conference on Machine Learning, pp. 9929-9939, PMLR, 2020.
Yida Wang, Pei Ke, Yinhe Zheng, Kaili Huang, Yong Jiang, Xiaoyan Zhu, and Minlie Huang. A large-scale chinese short-text conversation dataset. In CCF International Conference on Natural Language Processing and Chinese Computing, pp. 91-103. Springer, 2020.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi,
Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, et al. Huggingface's transformers: State-of-the-ar natural language processing. arXiv preprint arXiv: 1x1.v:1910.03771 2019a.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint arXiv:1901.08149 2019b.
Oscar Woolnough, Cristian Donos, Patrick S Rollo, Kiefer J Forseth, Yair Lakretz, Nathan E Crone, Simon Mischer-Bauma Stanislas Dehaene, and Nitin Tandon. Spatiotemporal dynamics of orthographic and lexical processing in the ventral visual pathway. Nature Human Behaviour, 5(3): 389-398, 2021.
Han Wu, Kun Xu, and Linqi Song. Csagn: Conversational structure aware graph network for conversational semantic role labeling. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 2312-2317, 2021.
Yu Wu, Wei WU, Chen Xing, Ming Zhou, and Zhoujun L.. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 496449668 505,505,22 2017.
Chen Xing. Yu Wu. Wei Wu, Yalou Huang, and Ming Zhou. Hierarchical recurrent attention network for response generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018
Kun Xu, Han Wu, Linfeng Song, Haisong Zhang, Lingi Song, and Dong Yu. Conversational semantic role labeling. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:2465-222 24775, 2021.
12

---

Published as a conference paper at ICLR 2023
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34, 2021.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference Learning Representations, 2019,
Zhuosheng Zhang, Jiangtong Li, Pengfei Zhu, Hai Zhao, and Gongshen Liu. Modeling multi-turn conversation with deep utterance aggregation. In Proceedings of the 27th International Conference on Computational Linguisties, pp. 3740-3752 2018.
Henghui Zhu, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. Who did they respond to? conversation structure modeling using masked hierarchical transformer. In Proceedings the AAAI Conference on Artificial Intelligence, volume 34, pp. 1144-9948, 2020.
13

---

Published as a conference paper at ICLR 2023
A HUMAN EVALUATION INSTRUCTIONS
Please rate the quality of the generated response based the given dialogue context and the target response over following aspects: (1) Fluency; (2) Informativeness; (3) Coherence; (4)Semantic Coverage. We provide some instructions for your rating.
A.1
FLUENCY
This measures whether the generated text has formatting problems, capitalization obviously ungrammatical sentences (e.g., fragments, missing components) that make the text difficult read. The definitions of different scores are:
5: The text is fluent, grammatically correct, and has no errors. It is easy to read.
4: The text is grammatically correct, but has a few spelling or capitalization errors, which does not affect your understanding.
3: The text has minor errors both grammar and spelling. The errors slightly affect your understanding.
2: The text has major errors in both grammar and spelling. The errors make the text hard to read.
1: The text does not make sense and is unreadable.
A.2 INFORMATIVENESS
This measures whether the generated text has diverse, informative, novel or logically related contents. The definitions of different scores are:
5: The text contains very diverse, informative and novel contents. It is enjoyable to read the text.
4: The text contains many informative and novel contents. (Choose this score when you hesitate between 3 and 5.)
3: The text contains some new information but also contains few repetitions of the context.
2: The text only contains few informative and new terms. (Choose this score when you hesitate between 1 and 3.)
1: The text is dull, repetitive and has new information. All contents are from the dialogue context.
A.3 COHERENCE
This measures whether the generated text is semantically and factually consistent with dialogue context. The definitions of different scores are:
5: The text is semantically, factually and topically consistent with the dialogue context. All contents of the text related to the source text be inferred from the source.
4: The text is very related to the context but has minor inconsistency or contradictions that do not affect its overall relevance.
3: The text is related to the context but has some obvious inconsistency and contradictions.
2: The text is slightly consistent with the context. Many inconsistency and contradictions to the context can be found.
1: The text totally inconsistent with the context. is semantically or factually contradicted to the context.
14

---

Published as a conference paper at ICLR 2023
A.4 SEMANTIC COVERAGE
This measures how many semantic content units from the target response covered by the generated text. The definitions of different scores are:
5: All semantic content units of the target text be found from the generated text. They are semantically consistent.
4: Most of the content units of the target text can be found from the generated text while few missing units do not affect the overall coverage.
3: Some semantic content units can be found from the generated text but also miss some important units.
2: Most of semantic content units are not covered. Only a few insignificant units can be found the generated text.
1: The text does not have any overlapping semantic content units with the target text.
We recruit five human workers to annotate 24,000 samples. To make sure the workers are fairly paid, pay 0.1 dollars for each sample. Therefore, the total amount spent on participant compensation is 2,400 dollars. The annotators take 48 hours to finish the task, suggesting the hourly wage for each worker is 10 dollars.
B MORE DETAILS OF THE TASKS
B. MULTI-TURN DIALOGUE RESPONSE GENERATION
Training We fine-tune the models on the DailyDialog and LCCC datasets for 8k and 40k steps, respectively. We use batch size of 128 and truncate the training samples maximum length of 256. The parameters of the models are initialized from Huggingface Libraries (Wolf et al. (0199) and updated by Adam optimizer (Kingma & Ba 2013) with a learning rate of 3e-5. The margin values of SimCTG and SimDRC are set to 0.5 and 0.7, respectively. The loss weight is 0.3. All hyper-parameters are selected from the development set. The training process on the DailyDialog and LCCC datasets takes 0.7 hours and hours on four A100 GPUs, respectively.
Evaluation We conduct automatic evaluations for the response generation task on following metries: 1) BERTScore (Zhang al. 2019) which calculates the similarities of token representations between the generated response and the target response using the pre- pre-trained BERT model; 2) BARTS Yuuan et al.) 2021) which estimates the difficulties of converting the generated text the reference output by the text generation method; 3) BLEURT (Sellam et al.) (0200) which is reference- based text generation metries that robust both domain and quality drifts; and 4) Distinct2/4 (Li et al. 2016) which computes the generation repetition at different n n-gram levels.
B.2 CONVERSATIONAL RESPONSE RETRIEVAL
Training For each dataset, we fine-tune the model on the training set for epochs, and save the best model according to the performance on the development set. The model parameters are initialized from the post-training checkpoint released by Han et al. (202T) and updated by Adam optimizer with a learning rate 15 The margin values in SimCTG and SimDRC are set to 0.5 and 0.8, respectively. The value of in SimDRC is set to 0.6. All hyper- paper--rrameterss are selected on the development set. The training process on each dataset takes around 10 hours on a single A100 GPU.
Evaluation Following previous work (Zhang al. 2018 Han et al.) 2021 we use Recall (i.e. Rlo = (1, 5)) as our evaluation metric, which indicates the probabilities whether the correct answer stands in the top candidates given 10 samples. For the Douban benchmark, also compute the values of mean average precision (MAP) and mean reciprocal rank (MRR), and precision one (P@1) since the context in this dataset may contain multiple positive responses.
15

---

Published as a conference paper at ICLR 2023
B.3 CONVERSATIONAL SEMANTIC ROLE LABELING
Training We follow the previous work [Wu et al. 2021) and solve the CSRL task as the sequence labeling problem. We keep the training settings same CSAGN's. The parameters of the model are initialized from the pre-trained BERT, and updated by Adam optimizer with a linear learning rate schedule. The margin values of SimCTG and SimDRC are set 0.2 and 0.5, respectively. The loss weight a in SimDRC is set to 0.2. All hyper-parameters parameters are selected on the development set. The training process the DuCony training takes around 2 hours
A A100 GPUs.
Evaluation Following (Xu et al. 2021 Wu et al., (021). we report the Fl ut, Fl Fintre and F1 Finter scores over the (predicate, argument, label) tuples. The arguments are categorized into two types, i.e. intra-arguments and aogsmmrggsenns, according to whether the argument appears in the same turn with the predicate or not. Therefore, we calculate the intra and inter scores on intra- and cross-arguments, respectively.
B.4 RESULTS OF DIALOGPT ON RESPONSE GENERATION TASK
C
MORE EVALUATION RESULTS
BERTScore
Model
Method
BARTScorrT
BLEURTT
Dis2/41
Pt
Rt
Ff
greedy
12.13
10.22
10.96
-3.82
0.382
0.303/0.695
beam
12.18
12.63
11.71
-3.90
0.383
0.300/0.671
DialoGPT
nucleus
12.22
12.65
12.05
-3.70
0.386
0.306/0.692
contrastive
10.14
11.92
10.56
-4.19
0.247
0.288/0.653
greedy
11.31
9.69
10.00
-3.98
0.371
0.271/0.622
SimCTG (p=0.6)
beam
12.01
12.46
12.15
-3.73
0.375
0.273/0.632
nucleus
10.54
11.63
10.65
-3.79
0.366
0.269/0.627
contrastive
12.12
12.62
12.22
-3.71
0.375
0.274/0.63
greedy
12.05
12.10
12.01
-3.90
0.322
0.271/0.633
SimDRC (6=0.5,
beam
12.63
13.75
12.65
-3.69
0.385
0.277/0.634
nucleus
12.74
13.88
12.78
-3.65
0.399
0.317/0.713
a=0.2)
contrastive
12.62
13.15
12.45
-3.64
0.392
0.322/0.744
Table 6: Results of automatic evaluation on the DailyDialog dataset.
Table shows the results of DialoGPT models DailyDialog. Since there suitable DialoGPT models for Chinese, we only evaluate on the DailyDialog dataset here.
D MORE IN-DEPTH ANALYSES
D.1 VISUALIZATION OF SELF-ATTENTION WEIGHTS
In this part, we visualize the self-attention weights of vanilla BART and BART+SimDRC trained on DailyDialog. As shown in Figure we can see that 1) in vanilla BART, all tokens are primarily attended to the dialogue representative token, i.e. 
 in our cases. This would lead to the problem that all tokens receive much the same information and be nearby each other representations. This is exactly the problem of anisotropy; 2) with the aid of SimDRC, the tokens are encouraged to be concentrated the tokens within the same utterance and discriminative to tokens in different utterances. Therefore, we believe that SimDRC also essentially learns locality and isotropy on attention weights, while it explicitly calibrates token embeddings.
16

---

Published as a conference paper at ICLR 2023
(8) Vanilla BART on Dailydialog
163
PPP
ypu y
(b) SimDRC on Dailydialog
Figure 5: Attention weights visualization of different models trained on DailyDialog.
-03
[m)
treu
Presken
righ
-01
(a) Vanilla BART
(b) BART + DiaSE
Figure 6: Comparison of vanilla BART and BART with dialogue state embeddings.
D.2 DIALOGUE STATE EMBEDDING VS. SIMDRC
A straightforward idea distinguish different utterance might be simply adding dialogue state embedding (DiaSE) (Wolf et al., 2019b Xu et al.) (021)) to token embeddings. To this end, we introduce additional randomly initialized dialogue state embedding layer into vanilla BART to differentiate different turns. During encoding, these additional embeddings are added with word embeddings and positional embeddings. As shown in Figure (bb), even with the help of the turn indicator, the problem of anisotropy still remains. We think the reason is that DiaSE is essentially another kind of "segment embedding" which capable of capturing deep semantic or conversational features.
D.3 EXAMPLES OF GENERATED TEXTS
17

---

Published as a conference paper at ICLR 2023
Context
SimDRC
Label
A: #| 38
T
!!
#A. 1
f/h
1
#H X8
N -.UL.BBAR
A: M T
M - R
B: #TT..
B
-
A: D
$ S
#EXT B1
Table 7: More generated examples on LCCC.
18

---

Published as a conference paper at ICLR 2023
Have you got any experience in advertising?
B: Yes, have working in the Public Relations Section company in the past two years. plan
the advertising campaign and couperate the work of artists and lypographers.
Context
A: Do you have any practical experience a secretary?
B: Yes. sir. acted as a secretary in an insurance company.
A: Have you had any experience with computer?
BART
Yes, have used a computer several times in the past two years, I've learned how to use very well.
SimCTG
Yes, have worked in computer company for two years and I have computer of my own.
SimDRC
Yes, have worked in computer company for two years and I have used Microsoft Word and Excel.
A: Listen, Karen, need your help. don't know anyone here yel.
I'm glad help you. What's wrong?
My mother-in-law just into the hospital in L.A. Hank and will be flying there tonight.
B: I'm sorry hear it. What's wrong with her?
Context
A: Doctors aren't sure yer. But the real problem is Suzy. She has a bad cold, and we don't 't want to take
her with But we don't know who can stay with her.
B: You need a babysitter.
A: Yes, I'd ask Jill, the girl I've had before, but need someone Baa.riiggh..aahaee even for two nights.
BART
That's a good idea. I'll ask Jill. She's very good.
SimCTG
I'm sonry to hear that, but don't 't think Jill would be able to do that.
SimDRC
don't think Jill would be able to do it. She's not good at babysitting because she's only 17.
911 emergency. What the problem?
B: would like report break-in.
Where was the break-in?
Context
B: It happened my house.
A: Was anything stolen??
B: haven't It checked yct.
A: Why do you think there was break-in?
BART
don't 't know, but think was a robbery.
SimCTG
There was a brcak-in at my house last night.
SimDRC
helieve it was a break-in because the door was left open.
Excuse me. have an appointment with at nine. May come in?
Yes, come please. am Mr. Li. must be My Liu, right?
Context
Yes, My Liu. Thanks.
B: I'd like start this interview with questions. Why do you think
you are qualified for this position?
BART
am very interested in this position and I very competent this
think I am qualified for this position because have worked this company for more than five years and have experience in this field.
SimCTG
Well, first of all, I think am good leam-player and have lot of experience in this line of work. Second, enjoy working with people and have a good working relationship with my colleagues. Finally, I am very interested this position.
SimDRC
A: How do you get road news?
Context
B: I've driver mate in my car!
A: So you have all the news on broudcast?
BART
Yes, I've got all the news my car!
SimCTG
Yes. I've on the TV, too.
SimDRC
Yes. I've all the news on the radio in my car.
Table 8: More generated examples on DailyDialog.
19


Reviews:
[{"review_id": "Z7l0mYcCin", "reviewer": "Reviewer_uyyF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces SimDRC for dialogue representation learning. SimDRC capture the dialogue structure by locality loss and isotropy loss. The locality loss maximizes the cosine similarity of the representations of the tokens within an utterance, while the isotropy minimizes the cosine similarity of the representations of the tokens from different utterances. The authors applied SimDRC to fine-tune BART on 3 different dialogue tasks, i.e., multi-turn dialogue response generation, conversational response retrieval, and conversational semantic role\nlabeling. The experimental results showed that SimDRC improved the naive fine-tuning strategy in terms of automatic and human evaluation metrics.", "summary_of_the_review": "This paper proposed a novel method to improve dialog representation learning. But the experiments are only limited to open domain dialogue datasets. ", "strength_and_weaknesses": "Strengths:\n\n- The paper is well organized and easy to follow.\n- The authors presented extensive experimental results on 3 different dialogue tasks with 2 different language model (BART and DialoGPT).\n\nWeakness:\n\n- I am not fully convinced that \"most task-oriented datasets, like MultiWOZ are essentially not conversational and contextualized\" as stated in the paper. There are plenty of multi-turn task-oriented datasets (e.g., SGD, STAR, RiSAWOZ, BiTOD, CrossWoz ). It would make the paper stronger if authors could show the advantage of the proposed method in some task-oriented datasets.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KaDBfQZb5JH", "reviewer": "Reviewer_uSDJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes techniques for generation dialog representation that is both local and isotropic in order to address the problem of anisotropy in conversation modeling. The paper attempted to apply locality and isotropic constraints at both the token and utterance levels. The locality constraint ensures that each token representations is close to the utterance representation while the isotropic constraint ensures that the utterance representations are pushed away from each other. The proposed representation was then applied to 3 dialog related tasks including multi-turn response generation, response retrieval and dialog semantic role labeling. The paper is well written and well motivated. The paper is also easy to follow.", "summary_of_the_review": "The main idea of the paper seems novel and reasonable. However, there's concern with some of the conclusions drawn by the paper especially the broad applicability to different dialog modeling related tasks. \n\nIt is not clear to me that the isotropic constraint did not impact the conversational coherence despite the results shown in Table 5. Would be great to understand how the dialog representation is calculated and if explicitly adding to the problem formulation to ensure conversational coherence would be more beneficial. \n\nAlso, the sensitivity for training hyper-parameter raises concerns for reproducibility. I wonder if using contrastive learning loss would have been better than using max-margin losses, which would minimize the number of hyperparameters to be searched for.", "strength_and_weaknesses": "Pros:\nThe use of locality and isotropic constraints is adequately demonstrated as show in the embedding similarity heat map in Figure 1&4. When applied to response generation, the technique was able to improve the vanilla BART model on both DailyDialog and LCCC datasets. \n\nCon:\nMost of the cons are already mentioned in section 5 under in-depth analysis. \nConversational Coherence - One would expect the isotropy between utterance representations to reduce the conversational coherence. However, the analysis showed that SimDRC is actually more conversationally coherent than the vanilla BART and SimCTG. While this is encouraging, it is not clear how the dialog representation v_{context} is obtained in the article since dialog representation is not explicitly stated in the problem formulation. Can the author(s) explain how this is calculated? Would be great if the coherent score can be evaluated on Ubuntu dialog also.\n\nSensitivity to loss weight and margin value -  The optimal values of these hyper-parameters seem to be dataset dependent which make it unreliable. However, the authors providing ranges of value for each task alleviates this problem to a considerable extent. Nevertheless, it would still be desirable to have more robustness over these hyper-parameter values.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "F1ZlYGGUDtq", "reviewer": "Reviewer_YMbd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem of how to make it more conversational for the generated dialogue representation from Transformers or large-scale pretrained language models. The authors studied related literatures on the issues that dialogue modeling is not isotropic and conversational, and analyzed the recent representation calibration approaches. Based on these, the authors proposed a new dialogue representation calibration method, namely SimDRC focusing on locality and isotropy to justify the challenges. \n\nThe paper first gave the definitions of locality and isotropy in dialogue modeling, then defined the locality loss, isotropy loss and SimDRC loss to adjust feature space to model the dialogue representation. \n\nThen the paper applied the new approach on the state-of-the-art models for 3 dialogue tasks: multi-turn dialogue response generation, conversational response retrieval, conversational semantic role labeling. The results of the experiments demonstrated that by combining SimDRC and state-of-the-art models in these tasks, better results have been achieved across all the 3 tasks than other related methods. \nThe analysis of conversational coherence, effects of loss weight and margin value, the measurements of locality and isotropy as well as the visualization of token similarity metrics were presented. ", "summary_of_the_review": "This paper created a new approach for generated dialogue representation calibration based on locality and isotropy, demonstrated a consistent performance gain in 3 conversational tasks. It was organized very well to articulate the new methods, experiments and results. It is a good extension of the state-of-the-arts.  ", "strength_and_weaknesses": "Strength: \n\nThis paper defines a new approach for generated dialogue representation calibration based on locality and isotropy. The experiments showed a consistent performance gain in 3 conversational tasks. \nThe literature study in the paper was complete and the analysis of the experiments was also in-depth. \nThe paper was organized very well with good articulation and the analysis of the technical problems, the methodology and the corresponding results. \n\nWeakness: \n\nSimCTG seems to be the major focus that the paper is comparing SimDRC with. The paper articulated the difference between the two methods in section 2 (token-level vs. multi-granularity levels), and the results of the experiments in section 4, as well as the conversation coherence analysis in section 5.1. It is better if the authors could include the following analysis: \n\na) why is SimDRC a better methods that SimCTG in dialogue related tasks? what is missing in SimCTG that make it is not suitable for conversational tasks? \n\nb) what other tasks will SimDRC be good for other than the 3 tasks included in the paper? Is it good for document generation tasks? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-Ncyd99_L5", "reviewer": "Reviewer_bCi9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the problem that the existing large language models fail to learn the dialogue-specific features. The paper tries to solve the locality and isotropy problem for dialogue generation modes by encouraging the model to aggregate the representation of tokens within an utterance and push away the representation of distinct utterances. ", "summary_of_the_review": "The motivation of this paper is not convincing. There is a gap between the motivation and the proposed model.", "strength_and_weaknesses": "The motivation of this paper is not convincing. I think the inter-speaker correlation can not be well-modeled by the generation model. It would be better to use the social network to enhance the generation model rather than use the model itself. \n \nIt is not novel to define a distance and add it to the model via a loss function. \n \nBy using the locality loss, and isotropy loss, the token-level similarity will be revised to the pattern shown in Figure 1 (c). But I do not think the inter-speaker correlations and conversational structure information are well-captured by the generation model. There is a gap between the motivation and the proposed model.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster