Logic Unveils Truth, While Disguise Obscures It: Transition Logic Augmented Response Selection for Multi-Turn Dialogue
Authors: Tingchen Fu
Xueliang Zhao
Lemao Liu
Rui Yan
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-turn dialogue
dialogue retrieval

Abstract: Multi-turn response selection aims to retrieve a response for a dialogue context from a candidate pool and negative sampling is the key to its retrieval performance. However, previous methods of negative samples tend to yield false negatives due to the one-to-many property in open-domain dialogue, which is detrimental to the optimization process. To deal with the problem, we propose a sequential variational ladder auto-encoder to capture the diverse one-to-many transition pattern of multiple characteristics in open-domain dialogue. The learned transition logic thus assists in identifying potential positives in disguise. Meanwhile, we propose a TRIGGER framework to adjust negative sampling in the training process such that the scope of false negatives dynamically updates according to the model capacity. Extensive experiments on two benchmarks verify the effectiveness of our approach.

Logic Unveils Truth, While Disguise Obscures It: Transition Logic Augmented Response Selection for Multi-Turn Dialogue
Tingchen Fu o Xueliang Zhao o Lemao Liu Rui Yan +xr=
Gaoling School of Artificial Intelligence, Renmin University of China
Peking University Tencent AI Lab
Beijing Key Laboratory of Big Data Management and Analysis Methods
Engineering Research Center of Gexe-Genn Intelligent Search and Recommendation, Ministry of Education
furas.fuutingghee@@maaa. com
zhaoxIpku@gmail. com
Abstract
For improving the discriminating power of re- trieval dialogue system, the key is the construc- tion of the candidate pool, or the selection the negative examples. Since the trivial solution, i.e., randomly sampling utterances from the entire train- ing set, results in too simple and informative neg- atives (Li et al., 2019), a large body of previous works (Lin et al., 2020; Su et al., 2020; Penha and Hauff, 2020) discuss how to excavate hard nega- tives that are lexically or semantically similar the golden response and therefore hard to differen- tiate. However, due to the inherent one-to-ma many property of open-domain dialogue, sometimes hard negatives are in fact positive in disguise, or false negatives (Gupta et al., 2021; Lee et al., 2022) and are detrimental to the convergence of retrieval model (Xiong et al., 2020; Zhou al., 2022).
Multi-turn response selection aims to retrieve a response for dialogue context from a can- didate pool and negative sampling is the key its retrieval performance. However, previ- ous methods of negative samples tend to yield false negatives due the mn--tommany property in open-domain dialogue, which is detrimental to the optimization process. To deal with the problem, we propose a sequential variational ladder auto-encoder to capture the diverse one- to-many transition pattern multiple charac- teristics in open-domain dialogue. The learned transition logic thus assists in identifying po- tential positives in disguise. Meanwhile, we propose TRIGGER framework adjust neg- ative sampling in the training process such that the scope of false negatives dynamically up- dates according to the model capacity. Exten- sive experiments on two benchmarks verify the effectiveness of our approach.
To verify this point, we perform pilot study (Section to find that previous negative sampling methods do yield portion of false negatives and the ratio higher than random sampling. easy to understand since previous methods tend to overlook and thus have little control over the false negative issue, except that Gupta et al. (2021) try to filter out the false negatives in a heuristic manner. Nevertheless, heuristically filtering out negatives that similar to the golden response is far from enough, and mitigating the false negative issue non-trivial problem:
is
1 Introduction
Recently, retrieval-based dialogue draws rising in- terest from the NLP community (Liu et al., 2022;
Lee et al., 2022; Tao et al., 2023; Feng et al., 2023), since it is promising way towards intelligent human machine dialogue. Moreover, the technolo- gies in building it show great potential in various applications such as tasked-oriented dialogue assis- tant (Shu et al., 2022), conversational recommenda- tion (Li et al., 2018) or the recently released interac- tive large language model (OpenAI, 2022). In this study, we focus on the core of retrieval dialogue system, i.e., the multi-turn response selection task, which aims to retrieve the best response (golden response) from pre-defined candidate pool given a dialogue context.
human-machine
A major challenge is that a randomly sampled utterance could also be more or less appropriate. although may differ from the golden response in many aspects. Owing to the one-to-many na- ture (Zhao and Kawahara, 2021; Towle and Zhou, 2022), there usually exists more than one possi- ble dialogue flow, with each flow reflecting differ- transitions in dialogue topic, user emotion and many other characteristics. To recognize and miti- gate false negatives, is required that we capture the diverse potential transition logic of multiple characteristics in open-domain dialogue, which is
TTingchen Fu and Xueliang Zhao contribute equally to this work. This work was done during their internship at Tencent Lab. "Corresponding author: Rui Yan (ruiyan@ruc. edu.cn).

---

difficult acquire (Xu al., 2021).
Model
False Negative Ratio
Another challenge lies in the balance between excavating hard negatives and removing false nega- tives (Cai et al., 2022; Yang et al., 2022). Specif- ically, we may largely avoid false negatives by al- ways selecting naive negative examples that obviously inappropriate, which is however uninfor- mative and useless. In other words, the dividing line between the false negatives and the hard neg- atives is bounded to change dynamically in the training process according to the retrieval model capacity. Though there exist some works employ- ing curriculum learning (Su et al., 2020; Penha and Hauff, 2020), their adjustment of negative sampling is performed in empirical way independent of the model capacity.
Random
3.29
Semi (Li et al., 2019)
7.16
CIR (Penha and Hauff, 2020)
7.03
Grey (Lin al., 2020)
7.42
HCL (Su el al., 2020)
6.95
Mask-and-fill (Gupta al., 2021)
5.63
Table I: The false negative ratio (%) of selected negative candidates on Douban dataset.
False Negative
Negative Updating
Model
Preventing
Semi (Li et al., 2019)
CIR(Penha and Hauff, 2020)
HCL(Su al., 2020)
Grey(Lin al., 2020)
Mask and-illi(Guptaa al., 2021)
In this rescarch, to cope with the first challenge, we propose to decompose the characteristics in multi-turn conversation into multiple dimensions and represent each with a latent label respectively. To achieve this, we design sequential variational ladder auto-encoder (SVLAE) to model the tran- sition logic of multiple characteristics and dis- entangle each other. For the second challenge, we update the negative sampling dynamically in the training process pace with the optimiza- tion of the retrieval model. Specifically, we pro- pose a TRIGGER (TRansItion loGic auGmentEd Retrieval) framework, which consists of T-step and R-step and optimizes the negative sampling process and retrieval model in two steps iteratively.
Ours
Table 2: Comparison with other negative sampling meth- ods
four previous methods. Specifically, we ask five in-house annotators to independently scan through negative responses excavated from previous neg- ative sampling methods and mark out the appro- priate and acceptable ones. Negatives marked by more than half of the annotators are identified to be false negatives. The experiment is performed on 200 dialogue context sampled from Douban (Wu et al., 2017) and for every context sample 5 negatives from each sampling method.
To summarize, our contributions are three-fold: (1) We devise a sequential variational ladder auto-encoder to model the multiple orthogonal char- acteristics in compositional and disentangled way. (2) We propose TRIGGER framework that combines the updating of negative sampling to- gether with the optimization of a retrieval model such that the criterion for negative sampling dy- namically changes to pace with the capacity of the retrieval model.
The experiment results are shown in Table 1. We can see that compared with random sampling, HCL (Su et al., 2020), CIR (Penha and Hauff, 2020), Grey (Lin al., 2020) and Semi (Li et al., 2019) have substantially more false negatives. Al- though mask-and and-fill (Gupta al., 2021) partially solves the problem thanks to its semantic limitation mechanism, it still has a higher mislabel ratio than random sampling.
benchmarks verify that when combined, our method signifi- cantly improves the existing retrieval models by a large margin and achieves a new state-of-the-ard
(3) Extensive experiments
Related Work
The task of multi-turn response selection aims at selecting response to match the human input from large candidate pool (Lowe et al., 2015; Yan et al., 2016; Zhou et al., 2016; Wu et al., 2017; Zhou al., 2018: Tao et al., 2019a; Jia al., 2020). Close lated conversational recommendation (Li et al., 2018) and interactive large language model (Ope- nAI, 2022), it has extensive application in the com- mercial area. With the recent huge success of
2 Pilot Study On False Negatives
In this pilot study, we conduct human evaluation investigate false negatives hidden in the candi- date pool, where the negative samples in the candi- date pool are constructed by random sampling or

---

PLMs (Devlin al., 2018: Liu al., 2019), post- training PLMs with diverse self-supervised tasks become popular trend and achieve impressive per- formance (Xu et al., 2020; Gu et al., 2020; Whang et al., 2021: Han et al., 2021; Fu et al., 2023).
yf
31
P
Z
y
Apart from designing new architecture or new self-supervision task, another branch of work put emphasis negative sampling. Namely, since the quality of negative candidates has great influ- ence on the retrieval model, a large body of work curate hard negative candidate by searching within the corpus (Su ct al., 2020), synthesizing from lan- guage model (Gupta et al., 2021) or combination of both (Lin et al.. 2020). However, most previous methods pay little attention to preventing false neg- atives or updating the negative sampling to adapt with the optimization of the retrieval model, albeit the semantic limitation in Gupta al. (2021) and the curriculum learning in Penha and Hauff (2020) and et al. (2020). The comparison of our meth- ods against previous ones is shown in Table 2.
Figure 1: The generation (left) and inference (right) of SVLAE architecture with 2.
4.1 Transition Logic Estimation
To represent and learn the transition logic, de- scribe each utterance in dialogue as generated by both a discrete latent label y and continuous latent feature z following Kingma and Welling (2013). Because the transition logic is influenced by multi- ple (orthogonal) factors including but not limited to the dialogue topic, dialogue acts, or speaker's emotion, it would be unwieldy to enumerate each combination of these factors with single y, which would lead to the hypothesis space of scaling ponentially with the number of factors considered. Instead, inspired from Zhao et al. (2022) and Falck et al. (2021), we propose a Sequential Vari- ational Ladder Enctoer (SVLAE) to model the multiple characteristies in a disentangled and compositional way. The generation and inference in our SVLAE architecture are shown in Figure and elaborated as below.
4 Methodology
Problem Formulation. Given dialogue con- text (u,, u22 "., uN) with u, denoting the i-th utterance and N is number of turns, the ob- jective of a retrieval model Dc, R) is to find the golden response r from a candidate pool R = fr* r1 ri....,,r. } where (TT)...,7)) are n negative samples. To enable retrieval model to differentiable multiple candidates and pick out the r*, the core is the meticulous construction of the candidate pool R.
Overview. The proposed approach has two stages: (1) transition logic estimation and (2) dy- namic negative updating. In the first stage, we train transition model to capture the transition logic of multi-level characteristics in conversa- tion. The transition model is used for detecting characteristics in multiple facets decide whether a candidate utterance potential false negative. However, with the growing model capacity, the criteria for potential false negatives should change accordingly. So in the second stage, we introduce a policy network to determine the negative sam- pling criteria regarding multi-facet characteristics according to the feedback from the retrieval model. In this way, the negative sampling process paces with the evolution of the retrieval model. At the test stuge, the transition model and the policy network are discarded so it causes no extra latency.
Generation Specifically, 1:L and latent features z
L latent labels 1:L to describe the char-
acteristics of an utterance in L different facets. In our probabilistic framework, to generate multi- turn dialogue with utterances, we first sample latent labels for every facet / E (1,2,... L}, and then sample corresponding z' from mixture of
'For example, if we consider L facets with K' the number of possible values for the 1-th facet. "single partition" model needs to encode K3 x k² x x... x k possibilities with a single f).

---

Gaussian
token after each layers to discover multiple diverse features about
N plvin) = p(vi) [ i=2
palvé I v'os)
z' NNooov t 'w'), D,, n w'')) 14 /w'J)
(4)
|+1 =oo(lb[css]
p(ziN)) [ i=1 [ N(z' mo(u'), Eo('))
p(zz v
(1)
where nO' E R is the distribution of y' calculated by goly' | u) and bicis) is the representation of [CLS] token after layers.
Optimization For optimizing the SVLAE, we exploit the evidence lower bound objective (ELBO) to jointly optimize the generation parameter and inference parameter Φb
With all L latent features 1:L i-th utterance uj generated by:
the corresponding
o fo([2]; - 1), 1,2, .....,,...
l
p(u, ) p(u; )
z²); do(w:) weer, (eerzz))
E 2 th1 log pa( u4 z )
2-1
1=1
(2)
In implementation, p(vi) is uniform distri- bution over K' possible values. pø. po(y') and Eg(y') are multi-layer perceptrons. The sequence modeling (o yea) and the utterance genera- tion dg Wet; ă) are both implemented with liight transformer.
1:1 B. (T,(g(Z))
1:I 1
am)ll M pl
vit)
KL(a(sh (
(5)
where KI. denotes the Kullback-Leibh divergence. After the training process, the transition model is fixed and used for inferring the latent label of all utterances in the training corpus.
More details about the neural parameterization could be found in appendix ??.
Inference For the recognition of latent labels 1 and latent continuous variables z I:L for ut- terance' propose to factorize the variational posterior q(z" .9 [u) as
4.2 Dynamical Negative Updating
1:1 1:L
We assume that utterances sharing more latent la- bels with r- are more likely to be false negatives and thus should be excluded from R at the begin- ning. But with the training process proceeding, the retrieval model gradually acquires the ability to discern the subtle differences in multiple facets and the exclusion criterion should also update. To achieve this, we develop a TRIGGER framework that updates the negative sampling in pace with the retrieval model in two iterative steps:
l:t u) ao(z"" g(z 1: "1
l:b
l:L w) yy"
[
w)
(3)
In this way, the inference for each facet con- ducted independently, encouraging the capturing multi-facet disentangled characteristics.
In implementation, (v' u) is parameter- ized by a 1-layer GRU and a multi- percep- tion. Regarding the inference of 2 1:L. we draw inspiration from Tenney et al. (2019): Niu al. (2022). These studies find that the representation after lower BERT layers usually encodes basic syn- tactic information while higher layers usually en- code high-level semantic information. In light of this, we prepend a special [CLS] token to each ut- terance u and employ the encoding of the [CLS]
T-step At T-step, we introduce policy network for predicting the characteristics of the utterance that is most suitable to be negative samples given the current model capacity. Then the policy net- work receives a reward from the retrieval model to update its parameter.
Specifically, the policy network P takes the pre- dicted latent label of the golden response po po(N++) as input and predicts the latent label distribu- tion of the suitably difficult negatives, denoted as T T 1:I. Then we sample the latent labels Ø T( ).92 ~ T ........ Gn TT=") and
"ee use superscript denote the facet and the subscript denote the utterance "we omit the suhscript in this paragraph to avoid the clutter of notation
the of the paper.

---

Ubuntu (Lowe et al., 2015)
Douban (Wu et al., 2016)
Dataset
Train
Validation
Test
Train
Validation
Test
# context-responss pairs
IM
500K
500K
1M
50K
6670
candidates per context
10
Avg #turns context
10.13
10.11
9.80
6.69
6.75
6.45
Avg #words per turn
13.86
17.14
17.18
15.58
14.71
16.56
Table 3: Statistics of two datasets used in our experiments.
construct a set (11, 12.1,12....0 ................................ , WL) consisting of all utterances that are predicted to have exactly the
.i....,
Algorithm 1 The proposed TRIGGER framework.
1: Input: A retrieval model D. training corpus, maximum training step for the transition model and retrieval model Mj and M2.
same latent label y ........ "." We sample neg- ative from the S(ýn, .2.....,91.. U1.), and op- timize the policy network P with policy gradi- ent (Sutton et al., 2000):
2: for + to M, do
3: Sample a mini-bated (c,r) from the training corpus. Recognize the latent labels and latent features with the inference model é.
4:
5: Generate the conversation with generation model 0.
E, -( B. (n)) D(F lcr") (6)
6: Optimize the SVLAE with the objective in Eq. 5.
7: end for
8:
for + 1 to M3 do
Intuitively, the reward for the policy network
[An new episode begins.) 10: Sample mini-batch of (c,r) training corpus.
9:
D(F cerr) is the probability that the retrieval model regards the sampled r- as a better candi- date than r". In implementation, the policy net- work is lightweight bi-dircctional transformer. More parameterization details could be found in Appendix ??.
Compute the latent label distribution of the suit- ably difficult negutives (z=" (0) and construct the set 8(T, U2.....zz. Wū.)
12:
Sample (r from the set to compose R. if m is odd then
13:
14:
(T-step)
15:
Optimize the policy network P with Eq. 6 else
16:
R-step R- the retrieval model is trained
17:
[R-step)
to discriminate the golden response r* apart from the negatives excavated by the policy network. In
18:
Optimize the retrieval model D with Eq. 7
19:
end if
20: [An episode ends.
detail, after the T-step is finished, the parameter
21: end for
of the policy network is fixed and reconstruct the set SW 12................ ........ UL) with the updated policy network. Note that possible that the safe is an empty set. If this is the case, we ra-sample y 1:L from the sampling policy TTy" 1:1)). The objective the retrieval model thus:
22: Return: the retrieval model 1).
Ubuntu Corpus V1 (Lowe et al., 2015) is a multi-turn response selection dataset in English collected from chatting logs, mainly about seeking technical support for problems in using the Ubuntu system. We use the copy shared by (Xu et al., 2016), which replaces all the numbers, URLs and paths with special placeholders.
max E ec- D(r (c,6) 1)
Compared with the original training objective, the new one in Eq. 7 restricts the scope of negatives to the set S(0,, ....... ... .z). thus mitigating the false negatives and rendering the selection of negatives pace with the optimization of the retrieval model. A high-level algorithm for our proposed frame- work shown in Algorithm
Douban Corpus (Wu et al., 2016) an open- domain Chinese dialogue dataset from the Douban website, which a popular social networking ser- vice. Note that for the test set of Douban corpus, one context could have more than one correct re- sponse the golden response is manually labeled.
5.2 Evaluation Metrics
5 Experiment
Following previous works (Tao al., 2019b; Xu et al., 2020), use recall our evaluation met- rics. The recall metric Rigkk means the correct response is within the top-k candidates scored by the retrieval model out of 10 candidates in total.
5.1 Datasets
We conduct experiments on two benchmarks: Ubuntu Corpus VI, and Douban Corpus. The statis- ties of these three datasets are shown in Table

---

Ubuntu
Doutsn
Model
Ry
Ry
Ruo e5
MAP
MRR
Pel
R
R
BERT (Devlin al., 2018)
80.8
89.7
97.5
63.3
45.4
28.0
+Seni (Li al., 2019)
81.0
89.8
97.5
60.3
63.8
46.0
28.2
47.9
+CIR (Penha
Hauff, 2020)
81.2
97.6
28.5
49.1
+Gray (Lin al., 2020)
81.5
90.1
97.5
61.5
64,8
47.1
29.1
50.7
+1CCL (Su et al., 2020)
81.5
90.2
97.6
61.7
65.9
48.0
30.4
$1.0
+MF (Crupsa al.,
81.6
30.3
.RIGGER (Ouns)
81.9
90.4
97.5
62.9
67.8
50.2'
32.5'
$2.1
SA-BERT (Gu et al., 2020)
85.5
92.8
98.3
61.9
65.9
49.6
31.3
48.1
+Semi (Li ot al., 2019)
85.8
93.1
98.9
62.3
66.4
50.0
31.7
49.0
+CR (Penha and Hauff, 2020)
86.0
93.5
99.0
62.4
66.6
50.3
31,8
49.7
+Gray (Lin et al., 2020)
86.1
93.4
99.1
62.8
67.0
50.3
32.0
50.3
(Su
86.7
99.2
33.0
MMF (Ciupta sl., 2021)
86,9
93.7
98.9
63.6
68.4
$2.0
33.5
52.9
-TRIGGER (Ouns)
87.1
93.8
98.7
64.1
69.5
$2.9
34,3
53.7
BERT TP (IIa et al., 2021)
91.1
96.2
99.4
64.4
68.0
51.2
32.4
54.2
SSemi (1. ct al,
91.2
34.3
+CI (Penha and [luff, 2020)
91.2
96.4
99.3
66.5
70.6
$4.8
35.5
$5.0
GGray (I.in al., 2020)
91.3
99.4
36.3
+ICL (Su 2020)
91.2
96.3
99.4
67.1
70.9
55.3
36.4
55.7
MMF (Ciupta al., 2021)
91.3
96-4
99.3
67.2
71.4
55.4
36.2
56.7
+TRIGGER (Ouns)
91.7"
96.6
99.4
67.9
72.4
56.5
36.7
59.5
Table 4: Evaluation results on the test sets of the Ubuntu and Douban. Numbers in bold are best results. that the improvement over the most competitive baseline statistically significant (t-test, p-value <0.05)
denotes
We use R1, &2 and R1o @5 in our experi- ment. As mentioned above, Douban Corpus con- tains more than one positive response from the candidates: also measure MAP (mean aver- age precision), MRR (mean reciprocal rank) and Pâl precision at one.
by looking up randomly initialized and learnable embedding matrix, and then the embeddings are modeled by uni-directional 3-layer transformer.
Similarly, dg (w,, Wer, ž is implemented as a 6-layer uni-directional transformer. The a is
mapped to the same dimension as the hidden repre- sentation of the transformer with a learnable matrix before prepending as a special token to the word embedding of u.
5.3 Implementation Details
Our method is implemented by Pytorch and per- formed on 2x24 GiB GeForce RTX 3090. The code is implemented with Hugging Face and the code is available https: thub. com/ Lingchenfu/EEMMLPP3--oooooo-eee
For inference the latent labels given an utter- ance, (x | u) is composed of 1-layer GRU and a multi-layer perceptron. The former encodes an utterance into dense vector u while the lat- ter maps the dense vector to K -wwy categorical distribution.
For hyper-parameter selection in the transition model, retrieval model, and policy network, we sweep the learning rate among [5e-6s 1e 5, 2e 5, 4e - 5, 5e - 5] and sweep the batch size among [1,8,16, 12] for each dataset. We only keep the last 15 turns of a dialogue context and the max- imum length of a context-candidate pair is 256. The gradient is clipped to 2.0 to avoid the gra- dient explosion. All models are learned with Adam (Kingma and Ba, 2015) optimizer with 31 0.9 and 3g 0.999. An early stop on the validation set is adopted as a regularization strategy. We re- port the averaged performance over three repetitive experiments for our method.
The policy network a bi-directional trans- former together with L dense matrices W Wi-L- The matrices map p 1:L, into L vectors in the same dimension before concatenating them together as an embedding sequence and input into the trans- former. The hidden representation after the last layer of transformer is then mapped back to the dimension K² K² k²,...,K².
5.4 Retrieval Models
As negative sampling approach, our proposed TRIGGER framework is orthogonal the retrieval models and theoretically our approach can be com- bined with any dialogue retrieval model seamlessly. To validate the universality of our approach, we per- form experiments on the following base retrieval
For transition model, po(vl y'e)) is imple- mented as transformer. Similar to word embed- ding, we obtain the embedding of yi .,....,
h/ctps://www: www.ggggggnnncccomm

---

model:
BERIFFP BERTFRRTRR
(2) The BERT (Devlin et al., 2018) is the vanilla BERT
model fine-tuned on the response selection task with no post-training. SA-BERT (Gu et al., 2020) enables the BERT model to be aware of differ- ent speakers in the concatenated context sequence by applying speaker position embedding. BERT- FP (Han et al., 2021) uses fine-grained post- training help the Bert model distinguish the golden response and negatives from the same dia- logue session.
1.03
2.00
3000
044.0
04504
6.1411
03500
03422
025
Lexical similarity
Figure 2: Retrieval Performance vs. lexical similarity on Ubuntu.
5.5 Baselines
To verify the effectiveness our approach, we draw a comparison with the following baseline method on negative sampling methods: Semi (Li et al., 2019) re-scores negative samples at differ- ent epochs to construct new negatives for training. CIR (Penha and Hauff, 2020) exploits curriculum learning and transits from easy instances difficult ones gradually. Gray (Lin et al., 2020) trains an- other generation model to synthesize new negatives. HCL (Su et al., 2020) proposes an instance- curriculum and corpus-level curriculum with a pacing function to progressively strengthen the ca- pacity of model. Mask-and-fill (Gupta et al., 2021) propose mask-and-fi fill approach that simul- taneously considers the original dialogue context well a randomly picked one to synthesize negative examples.
Ro01
3.5
423
BERTHP
082-0.0
8.4-0.86
0.00-0 ca
00866.10
0.50c
Lexical similarity
Figure 3: Retrieval Performance lexical similarity on Douban.
or noisy input? Q4: How can we understand and interpret the learned transition logic?
6.1 The Impact of Each Component
Answer to Q1 To study how each component works, conduct ablation study mainly con- sidering the following variants: -transition: The transition-aware hard negatives are replaced with randomly sampled negatives; -update: The T-step removed the transition-aware hard negatives are fixed during the training process; -feature: The la- tent features z I:LL are removed from the SVLAE architecture. -label: The latent labels y are re- moved from the SVLAE architecture
5.6 Experiment Results
The quantitative results are shown in Table 4, We can observe that our method significantly improves the original base retrieval models on most met- rics, showing the universality and robustness of our method. When combined with BERT-FP, our method achieves the new state-of-the-ard for most metrics, with an absolute improvement of 0.6% and 4.3% @ for Ubuntu benchmark and Douban benchmark respectively.
The experiment results are shown in Table 5. From the table, we could observe that: (1) There is an evident degradation when the meticulously excavated negatives are replaced with randomly sampled ones This suggests that our negative sam- pling methods are crucial to the retrieval perfor- mance, and that simply increasing the number of negative examples may not be sufficient.
6 Analysis
Apart from the overall performance, we are partic- ularly curious about and make further analysis to understand the following questions: Q1: How does each component and mechanism contribute to the overall performance? Q2: How does the lexical similarity between the context and the response in- fluence the performance of our method? Q3: How is the robustness of our method under adversarial
updating of the sampling policy in T-step plays important role its removal of it causes an ob- vious drop. This result justifies the necessity of updating hard negatives dynamically, (3) Both the latent labels y 1:L and latent features z to the model performance. This is likely because
1:L contribute

---

Ubuntu
Douban
Models
R1o @P
Ryo 08
MAP
MRR
P60
Rio 61
R1o
BERT-FP
91.1
96.2
64.4
68.0
51.2
32.4
54.2
+TIIGER (Ours)
91.7
96.6
99.4
67.9
72.4
56.5
36.7
59.5
transition
90.9
95.9
99.2
64.5
68.2
51.5
32.6
54.8
update
91.2
96.3
99.3
65.9
70.4
53.8
34,9
56.3
label
91.5
96.4
99.4
67.3
72.0
55.9
36.1
58.3
feaaure
91.4
96.3
99.4
67.0
71.6
54.2
35.5
57.4
Table 5: Results of ablation study on two benchmarks.
Rank
Transition
Probability
et al., 2022). In addition, it might render the model more robust to adversarial input or noisy input in real-world applications. To verify this point, in- spired by Jia and Liang (2017); Yuan et al. (2019); Whang et al. (2021), we alter the candidate pool of each case by substituting all the negative candi- dates with the utterances in the dialogue context. which is more challenging setting than previous scenarios (Whang et al., 2021). The experiment sults on Ubuntu and Douban are shown in Figure 2 and Figure respectively.
233-115
14.25%
2
265-211
12.90%
3
211-2265
10.45%
4
274-2100
2.20%
5
110-274
2.22%
Table 6: The top-5 2-gram transition pairs in Ubuntu and their transition probability.
Rank
Latent Label
Implication
233
Short instructions about paste operation. Reference URI..
115
According to the experiment results, all three base models deteriorate severely under attack. also reveals that relying on superficial cues is in fact a common phenomenon in PLM-based re- trieval models, not limited to BERT-FP (Han et al.. 2021). In comparison, we could see that the three models are much more robust when combined with our proposed TRIGGER strategy.
211
Audio and sound configuration in Ubuntu.
265
Sound system breakdown.
110
Brief yes/no answer.
Table 7: The implications of top-5 latent label at facet
L in Ubuntu.
they provide complementary information that can be used learn the transition logic.
6.4 Case Study
6.2 The Impact of Lexical Similarity
Answer to Q4: In this section, we aim at interpret- ing what is learned by the transition model in an intuitive way. As simplification, sider the "2-gram" transition of latent labels at the L-th facet (The most abstract one). Specifically, we first recognize the latent label facet I, for all utterances in the corpus with our transition model. Next, we investigate the statistics of latent labels I. pairs (4s 3i+1) appeared in the transition sequence [v] 1/2 ........ .NN] across the entire corpus.
Answer to Q2. To have a better understanding of the impact of lexical similarity, we bin the test set of Ubuntu and Douban into different bins according the similarity between the context and the golden response". The improvement on BERT-FP (Han et al., 2021) shown in Figure 2 and Figure 3. We could see that our method is helpful and substan- tially improve the retrieval performance on most bins. The improvement is obvious, especially in the harder scenario where the context and the golden response are less similar. We attribute the gains the transition logic, which exempts the retrieval model from the interference false negatives.
only con-
The most frequent "2-gram" transitions Ubuntu, as well as their transition probability (The percentage that the first state is succeeded by the second one, other than the frequency of the "2- gram"), is shown in Table 6. Besides, we review the 5 most frequent latent labels in facet L. By ob- serving randomly sampled 100 utterances with the corresponding latent label from the Ubuntu corpus, we manually induce their implications as shown Table 7.
6.3 Performance under Adversarial Attack
Answer to Q3: Reducing false negatives can ef- fectively mitigate the noise in supervision signal and therefore stabilize the training process (Zhou
"measured in uni-gram F1

---

Ubuntu
Douban
Models
R
R10 @2
R
MAP
MRR
P@
R1 1
R
BERT<Devlin ul., 2018)
17.0
25.0
3.2
6.5
BERT+TRIGGER
13.4
20.2
48.1"
37.6'
42.0'
23,5
13.7
21.1
BERT-FP (Han al., 2021)
52.5
57.4
69.2
32.5
36.8
21.9
12.9
BERT-FP-TRIGGER
76.2"
81.9
90.1
38.8"
43.0'
25.4"
16.2
23.5
Table 8: Evaluation results on Ubuntu and Douban under attack. model statistically significant (t-test, p-value <0.05).
denotes that the improvement over the original
7 Conclusion
In this study, target at the false negative issue in dialogue retrieval. We recognize that previous negative sampling methods lead more false neg- atives than random sampling, which detrimental to model optimization. So we propose a TRIGGER framework in which we model the inherent tran- sition logic in open-domain dialogue in multiple characteristics with our SVLAE architecture and combine the negative sampling process with the optimization of the retrieval model. In this way, the dividing line between the hard negatives and false negatives updated dynamically. Extensive experiments verify the efficacy of our approach.

---

Limitations
Qian Chen and Wen Wang. 2019. Sequential attention- based network for noetic wud-to-end/nn response selec- tion. arXiv preprint arXiv: 1901.02609.
All technologies built upon the large-scale PLM more or less inherit their potential harms (Ben- der et al., 2021). Besides, we acknowledge some specific limitations within our methods: We only verify the effectiveness of our method several recent PLM-based methods, but not on early meth- ods without PLM, like SMN (Wu et al., 2017) or ESIM (Chen and Wang, 2019). But since our ap- proach is orthogonal to the base retrieval model, we are promising that our proposal could be easily adapted to these methods.
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and
Kristina Toutanova. 2018. 8. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint arXiv:1810.04805
Fabian Falck, Haoting Zhang, Matthew Willetts, George
Nicholson, Christopher Yau, and Christopher C Holmes. 2021. Multi- falle- clustering variational au- toencoders.
Jiazhan Feng, Qingfeng Sun, Can Xu, Pu Zhao, Yaming Yang. Chongyang Tao. Dongyan Zhao, and Qing- wei Lin. 2023. Mmdialog: large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. In Proceedings of the 61th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Pupers), Toronto, Canada. Association for Computational Linguisties.
Ethical Considerations
This paper will not pose any ethical problems. First, multi-turn response selection is an old task in natu- ral language processing, and several papers about this task are published at EMNLP conferences. Sec- ond, all the datasets used in this paper have been used in previous papers. Our method should only be used to boost the performance of the retrieval dialogue system or other research use but not for any malicious purpose.
Tingchen Fu, Xueliang Zhao, and Rui Yan. 2023. Delv- ing into global dialogue structures: Structure plan- ning augmented respouse selection for multi- -trr con- versations. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Min- ing, KDD '23, page 495-505, New York, NY, USA. Association for Computing Machinery.
Acknowledgement
Jia-Chen Gu, Tianda Li, Quan Liu, Zhen Zhen-Hua Ling. Zhiming Su, Si Wei, and Xiaodan Zhu. 2020. Speaker-aware bert for multi-turn response selection in retrieval-bases chatbots. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2041-2044
This work was supported by the National Key Research and Development Program of China (No. 2020YFB 1406702), National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program and Intelli- gent Social Governance Platform, Major Innova- tion & Plamning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.
Prakhar Gupta, Yulia Tsvetkov, and Jeffrey Bigham. 2021. Synthesizing adversarial negative responses for robust response ranking and evaluation. In Find- ings of the Association for Computational Linguis- fics: ACL-IICNLP 2021, pages 3867-3883, Online. Association for Computational Linguisties.
NO.
Janghoon Han, Taesuk Hong. Byoungjae Kim. Youngjoong Ko, and Jungyun Seo. 2021. Fine- grained post-training for improving retrieval-bact dialogue systems. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1549-1559
References
Emily M Bender, Timnit Gebru, Angelina McMillan- Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Confer- ence on Fairness, Accountability, and Transparency. pages 610-623.
Qi Jia, Yizhu Liu, Siyu Ren, Kenny Q Zhu, and Haifeng Tang, 2020. Multi-tum response selection using dialogue dependency relations. arXiv preprint arXiv:2010.01.55
Yinqiong Cai, J. Guo, Yixing Fan, Qingyao Ai, Ruqing Zhang, and Xueqi Cheng. 2022. Hard negatives or false negatives: Correcting pooling bias in training neural ranking models. Proceedings of the 31 ACM International Conference on Information & Knowl- edge Management.
Robin Jia and Percy Liang. 2017. Adversarial exam- ples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 2021-2031. Copenhagen, Denmark. Association for Computational Linguistics.

---

Diederik P Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In ICLR.
Raphael Shu, Elman Mansimov, Tamer Alkhouli, Niko- laos Pappas, Salvatore Romeo, Arshit Gupta. Saab Mansour, Yi YiZhang, and Dan Roth. 2022. Dialog2api: Task-oriented dialogue with api description and ex- ample programs. arXiv preprint arXiv:2212.00900 09996..
Diederik P Kingma and Max Welling. 2013. Auto- encoding variational bayes. arXiv preprint arXiv:1312.611.
Yixuan Su, Deng Cai, Qingyu Zhou, Zibo Lin, Si- mon Baker, Yunbo Cao, Shuming Shi, Nigel Collier, and Yan Wang. 2020. Dialogue response selection with hierarchical curriculum learning. arXiv preprint
Nyoungwoo Lee, ChaeHun Park, Ho-Jin Choi, and Jaegul Choo. 2022. Pneg: Prompt-based negative response generation for dialogue response selection task. urXiv preprint arXiv:2210.17...
Jia Li, Chongyang Tao, Wei Wu, Yansong Feng, Dongyan Zhao, and Rui Yan. 2019. Sampling mat- ters! an empirical study of negative sampling strate- gies for learning of matching models in retrieval- based dialogue systems. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing pages 1291-1296, Hong Kong, China. Association for Computational Linguistics.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pages 1057-1063.
Chongyang Tao, Jiazhan Feng, Tao Shen, Chang Liu, Juntao Li, Xiubo Geng, and Daxin Jiang. 2023. Core: Cooperative training of retrievet-reranker for effec- tive dialogue response selection. In Proceedings of the 6lth Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Lin- guistics.
Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards deep conversational recommenda- tions. In Advances in Neural Information Processing Systems, volume 31, Curran Associates, Inc.
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019a. Multi- representation fusion network for multi-turn response selection in retrieval-based chatbots. In WSDM, pages 267-275.
Zibo Lin, Deng Cai, Yan Wang, Xiaojiang Liu, Hai- Tao Zheng, and Shuming Shi. 2020. The world is not binary: Learning to rank with grayscale data for dialogue response selection. arXiv preprint arXiv:2004.02421.
Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019b. One time of interaction may not be enough: Go deep with an network for response lection in dialogues. In Proceedings of the 57th an- nual meeting of the association for computational linguistics, pages 1-11.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 7
Yongkang Liu, Shi Feng, Wei Gao, Daling Wang, and Yifci Zhang. 2022. Dialogcony: A lightweight fully convolutional network for multi-view response selec- tion. arXiv preprint 2ixiv:2210. 138455
Ian Tenney, Dipanjan Das, and Ellic Pavlick. 2019. BERT rediscovers the classical NLP pipeline. Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4593- 4601, Florence, Italy, Association for Computational Linguistics.
Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dia- logue systems. In SIGDIAL, pages 285-294.
Benjamin Towle and Ke Zhou. 2022. Learn what is possible, then choose what is best: Disentangling one-to-many relations language through text-based games. In Findings of the Association for Computa- tional Linguistics: EMNLP 2022, pages 4955-4965, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Jingcheng NIu, Wenjie Lu, and Gerald Penn. 2022. Does BERT rediscover classical NLP pipeline? In Proceedings of the 29th International Conference on Computational Linguisties, pages 3143-3153. Gyeongju, Republic of Korea. International Com- mittee on Computational Linguistics.
OpenAl. 2022. Chatgpt: Optimizing language models for dialogue.
Tacsun Whang, Dongyub Lec, Dongsuk Oh, Chanhee Lec, Kijong Han, Dong-hun Lee, and Sacbyeok Lec. 2021. Do response selection models really know what's next? utterance manipulation strategies for multi-turn response selection. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 35, pages 4041--4049,
Gustavo Penha and Claudia Hauff. 2020. Curriculum learning strategies for ir: An empirical study on con- versation response ranking. Advances in Information Retrieval, 12035:699.

---

Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2016. Sequential matching network: A new archi- tecture for multi-turn response selection in retrieval- based chatbots. arXiv preprint arviv 1612.01627.
Xueliang Zhao, Lemao Liu, Tingchen Fu. Shuming Shi, Dongyan Zhao, and Rui Yan. 2022. Towards effi- cient dialogue pre-training with transferable and in- terpretable latent structure. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 10051-10063, Abu Dhabi, United Arah Emirates. Association for Com- putational Linguistics.
Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new archi- tecture for multi-turn response selection in retrieval- based chatbots. In ACL, pages 196-505.
Kun Zhou, Yeyun Gong, Xiao Liu, Wayne Xin Zhao, Yelong Shen, Anlei Dong, Jingwen Lu, Rangan Ma- jumder, Ji-rong Wen, and Nan Duan. 2022. SimANS: Simple ambiguous negatives sampling for dense text retrieval. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing: Industry Track, pages 548-559, Abu Dhabi, UAE. Association for Computational Linguisties.
Lee Xiong. Chenyan Xiong, Li, Kwok-Fung Tang. Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00800
Jun Xu, Zeyang Lei, Haifeng Wang, Zheng-Yu Yu Niu,
Hua Wu, and Wanxiang Che. 2021. Discovering di- alog structure graph for coherent dialog generation. Proceedings of the 59th Annual Meeting of Association for Computational Linguistics and the 11th International Joint Conference Natural Lan- guage Processing (Volume Long Papers), pages 1726-1739, Online. Association for Computational Linguistics.
Xiangyang Zhou, Daxiang Dong. Hua Wu, Shiqi Zhao, Dianhai Yu, Hao Tian, Xuan Liu, and Rui Yan. 2016. Multi-view response selection for human-computer conversation. In EMNLP, pages 372-381.
Xiangyang Zhou, Lu Li, Daxiang Dong, Yi Liu, Ying Chen, Wayne Xin Zhao, Dianhai Yu, and Hua Wu. 2018. Multi-turn response selection for chathots with deep attention matching network. In ACL, pages 1118 118--127
Ruijian Xu, Chongyang Tao, Daxin Jiang, Xueliang Zhao, Dongyan Zhao, and Rui Yan. 2020. Learning an effective context-ree renexx-respoose matching model with self-supervised tasks for retrieval-based dialogues. arXiv preprint arXiv:200900666.
Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2016. Incorporating loose-structured knowledge into Istm with recall gate for conversation modeling. arXiv preprint arXiv: 1rX5::1605.05100
Rui Yan, Yiping Song, and Hua Wu. 2016. Learning to respond with deep neural networks for retrieval- based human-computer conversation system. In SI- GIR, pages 55-64.
Chenxiao Yang, Qitian Wu, Jipeng Jin, Xiaofeng Gao, Junwei Pan, and Guihai Chen. 2022. Trading hard negatives and true negatives: A debiased contrastive collaborative filtering approach. In International Joint Conference on Artificial Intelligence.
Chunyuan Yuan, Wei Zhou, Mingming Li. Shangwen Lx, Fuqing Zhu, Jizhong Han, and Songlin Hu. 2019. Multi-hop selector network for multi-turn response selection in retrieval-based chatbots. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th interna- tional joint conference natural language process- ing CCMNIP-IJCNIPP pages 1-20
Tianyu Zhao and Tatsuya Kawahara. 2021. Multi- referenced training for dialogue response generation. In Proceedings of the 22nd Annual Meeting of the Special Interest Group Discourse and Dialogue, pages 190-201. Singapore and Online. Association for Computational Linguistics.


Reviews:
[{"review_id": "7nX8SM7fhQ", "reviewer": "Reviewer_wMUd", "summary": "", "questions": "1. Can the proposed method further improve performance with better PLMs?\n2. How expensive the proposed method in the perspective of computational costs in comparison with the competitors?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and expertise invested in reviewing our submission! We are grateful to receive your appraisal that our work provides a well-designed method to tackle the problem and would inspire a bunch of future research. This is indeed our original aspiration. Below, we will address each point individually.\n\n**Q1: The distinction from CVAE.**\n\n**A1**: Good question! Our proposed Sequential Variational Ladder Auto-Encoder (SVLAE) shares similarities with the CVAE in that they both involve one or more latent labels. However, SVLAE differs from CVAE in the following ways:\n\n+ Previous works using CVAE for dialogue generation [1][2] tend to view the entire dialogue context as a single observed data point $x$, and golden response as the label $y$, seldom considering the internal transition logic between each turn in $x$. In contrast, we treat the dialogue session $u_{1}, u_2, \\ldots, u_N, u_{N+1}$ as a sequence of observed data and use both latent labels $y_{1:N}$ and latent feature $z_{1:N}$ to model the sequential transition logic within dialogue.\n\n+ Multiple orthogonal characteristics influence dialogue flow, such as dialogue topic, discourse coherence, and speaker emotion. It is challenging to encode these abundant pieces of information with a single discrete or continuous latent variable. To address this, we propose a ladder-like architecture (Figure 1) to encode multiple levels of semantic abstraction with $y^{1:L}$ and $z^{1:L}$ in a compositional and disentangled manner, which is a major novelty of our SVLAE.\n\nWe are sorry for the ambiguity and will explain it more clearly in our final version.\n\n\n[1]Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders, ACL 2017.\n\n[2]A Discrete CVAE for Response Generation on Short-Text Conversation, EMNLP 2019.\n\n\n\n**Q2: Knowledge distillation-based approaches.**\n\n**A2**: Thanks for your suggestions! Following your advice, we compare with RocketQA[1] and CRA[2] using BERT and SA-BERT as base retrieval models and the results are shown below:\n\n| | R$_{10}$@1 | R$_{10}$@2 | R$_{10}$@5 |\n|---|---|---|---|\n|BERT| 80.8 | 89.7 | 97.5 |\n|BERT+RocketQA| 81.5 | 90.0 | 97.5 | \n|BERT+CRA| 81.3 | 89.9 | 97.5|\n|BERT+TRIGGER| 81.9 | 90.4 | 97.5 |\n\n\n\n| | R$_{10}$@1 | R$_{10}$@2 | R$_{10}$@5 |\n|---|---|---|---|\n|SA-BERT| 85.5 | 92.8 | 98.3 |\n|SA-BERT+RocketQA| 85.9 | 93.3 | 98.5 | \n|SA-BERT+CRA| 86.0 | 93.2 | 98.3 |\n|SA-BERT+TRIGGER| 87.1 | 93.8 | 98.7 |\n\nThe table demonstrates that our method surpasses both CRA and RocketQA, potentially due to our explicit consideration of multiple characteristics within the conversational flow, rendering our approach more adept at uncovering false negatives in open-domain dialogue. In our final version, we will integrate these experiments and reference knowledge distillation-based techniques [1][2].\n\n\n\n[1]RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering, NAACL 2021.\n\n[2] Conditional Response Augmentation for Dialogue Using Knowledge Distillation. INTERSPEECH 2020.\n\n**Q3: Marginal improvements over baselines.**\n\n**A3**:  We recognize that our approach may not yield a \"sharp\" enhancement in the retrieval performance of the Douban or Ubuntu benchmark, given their inherent dataset complexity and the limitations of the base retrieval model. Nonetheless, it is crucial to note that our proposed TRIGGER framework significantly bolsters the robustness of retrieval models, as evidenced by Table 6, which is vital for deployment and service. Additionally, our method is compatible with any base retrieval model and can be integrated with more advanced PLM encoders to further augment performance (kindly refer to our response to Q6).\n\n**Q4: Which aspects of the proposed method and experiments are meaningful in the era of LLMs.**\n\n**A4**: Good question! Despite the remarkable performance of LLM, retrieval-based dialogue systems and our proposed negative sampling techniques maintain their distinct advantages:\n\n+ Enhanced safety and control. As responses are chosen from a predefined candidate pool, retrieval-based dialogue systems offer greater content control, thereby mitigating the presence of toxic words or hallucinations in responses.\n\n+ Reduced computational cost. While LLMs generate high-quality output, the substantial computational expense of APIs (e.g., chatGPT and GPT-4) should not be disregarded. Prior research [1] examines the economic trade-off in deploying dialogue systems for customer service, revealing that LLM (GPT-3) may not be the optimal choice when considering both generation quality and API cost compared to smaller models.\n\n+ Retrieval-augmented LLM. Many researchers contend that retrieval should play a vital role in next-generation language models to enhance their factuality, interpretability, and trustworthiness. For example, LLMs are less likely to hallucinate when provided with retrieved knowledge from a given knowledge base [2][3][4]. Conversely, LLM output can be unstable due to sensitivity to prompts and decoding strategies. A potential solution is to first generate multiple candidates by using LLM to simulate various personas [5] or engage in multi-agent debates [6]. A retrieval module may then select the most suitable candidate as a response.\n\n+ Tool retrieval for LLM. External tools or APIs remain necessary for addressing higher-level tasks with LLMs. In this context, a retrieval system is required to select an appropriate tool (API) from thousands of API documents based on specific human instructions [7]. Given the functional similarity of APIs and the existence of multiple solution paths (chains of API calls), randomly sampled negative APIs may suffer from false negative issues, and our approach offers a potential remedy for this problem.\n\n\n[1]The economic trade-offs of large language models: A case study, ACL 2023.\n\n[2]Check Your Facts and Try Again: Improving Large Language Models\nwith External Knowledge and Automated Feedback, Arxiv 2023.\n\n[3]Enabling large language models to generate text with citations, Arxiv 2023\n\n[4]Rarr: Researching and revising what language models say, using language models.\n\n[5]Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration, Arxiv 2023\n\n[6]Improving factuality and reasoning in language models through multiagent debate, Arxiv 2023.\n\n[7]ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, Arxiv 2023.\n\n\n\n**Q5: I think further improvements would be achievable with other PLMs better than BERT, such as RoBERTa and T5 (or, more recently vicuna or llama-2). Can the proposed method further improve performance with better PLMs?**\n\n**A5**: Absolutely. Although our experiments primarily employ BERT-based models to maintain consistency with prior literature [1][2], our proposed TRIGGER framework remains agnostic and orthogonal to PLM architecture. This framework can be effortlessly integrated with any existing PLM to bolster its performance in dialogue retrieval. For instance, when combined with DeBERTa [3], a more recent and potent backbone, dialogue retrieval performance could be further enhanced, as illustrated below:\n\n| | R$_{10}$@1 | R$_{10}$@2 | R$_{10}$@5 |\n|---|----|----|----|\n|DeBERTa |  85.4 |  92.6 |  98.2 |\n|DeBERTa+TRIGGER |  86.5 | 93.2 | 98.3 |\n|DeBERTa-FP | 92.0 | 96.6 | 99.5 |\n|DeBERTa-FP+TRIGGER | 92.8 | 97.2 | 99.6 |\n\n\nWe believe that these results provide valuable insights into the effectiveness and versatility of our approach across different model backbones.\n\n\n[1]Dialogue Response Selection with Hierarchical Curriculum Learning, ACL 2021.\n\n[2]Synthesizing Adversarial Negative Responses for Robust Response Ranking and Evaluation, ACL 2021.\n\n[3]DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION, ICLR 2021\n\n\n\n\n**Q6: The computational costs of the proposed method?**\n\n**A6**: We appreciate the reviewer's apprehension. Our proposed method concentrates on negative sampling, offering more beneficial responses and co-evolving with the retrieval model. During inference, only the retrieval model D requires deployment, ensuring that our approach imposes no supplementary computational costs or latency during inference.\n\n\n\n**Q7: Further definition of \"model capacity\" would improve the paper reading.**\n\n**A7**: Thanks for your suggestion! Since we mainly focus on multi-turn response selection task in this study, the \"model capacity\" in our manuscript refers to the ability of a retrieval model $D(\\cdot \\mid c,\\mathcal{R})$ to discern multiple candidates in $\\mathcal{R}$ and find the appropriate response $r^+$ without fooled by the negatives. We will definitely follow your advice and add the detailed explanation in Section 4 in the final version to enhance the paper's readability.\n\n\n**Q8: It is hard to understand how can I interpret the results in Section 6.4. Do the results contribute to the performance improvements in the above sections?**\n\n**A8**: Good question! The impetus for Section 6.4 is to offer a proof-of-existence and an intuitive comprehension of the transition logic for our readers. Specifically, we can observe certain transition patterns in the Ubuntu Corpus from Table 7 and Table 8. For instance, the most prevalent pattern is $233\\to155$, signifying that a concise instruction on paste operations in the pastebin.ubuntu website (latent label $233$) typically precedes a reference to a URL (latent label $155$) within a conversation session.\nIn summary, this subsection affirms the efficacy of the proposed Sequential Variational Ladder Auto-Encoder (SVLAE) in capturing transition patterns within a conversation corpus in a more intuitive and qualitative manner, thereby facilitating an understanding of our method's effectiveness.\n\n**Q9: Grammar typos and presentation improvement**\n\n**A9**: Thanks for your suggestions! we will definitely revise the grammar typos and incorporate your advice on presentation into our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is generally well written and easy to follow.\n2. The tackled problem (false negatives in negative sampling) is important and the proposed method is reasonably designed to address the problem.\n3. The proposed method achieves the new state-of-the-art performances in Ubuntu and Douban datasets.\n4. The pilot study (Section 2) that manually inspects the ratio of false negatives in various methods is meaningful and would inspire a bunch of future researches.", "reasons_to_reject": "There is no serious reason to reject this paper. But, I have a few concerns regarding the novelty of the proposed method and experiments. It would be helpful to strengthen the following points:\n1. The distinction from CVAE (conditional variational auto-encoder): Though this paper tackles its own problem in response selection tasks and proposes a novel method, the proposed method seems sharing a lot of aspects with CVAE architecture in my perspective. Highlighting the novelty of leveraging ''conversational'' aspects would be a good initial point for this concern. \n2. Weak baselines: Among baselines, I realized that recent knowledge distillation-based approaches methods are not considered, such as [1] in passage retrieval and [2] in response selection tasks. Though they are not seriously relevant and important to cite, comparing the proposed method with them would make the contribution of this paper much clearer. \n3. Marginal improvements over baselines: Though the proposed method achieves the largest improvements among the competitors, I wonder if we should pursue such a sophisticated framework to achieve this level of performance. I think further improvements would be achievable with other PLMs better than BERT, such as RoBERTa and T5 (or, more recently vicuna or llama-2). It would be greatly helpful to discuss which aspects of the proposed method and experiments are meaningful in the era of LLMs, such as computational costs or real-world scenarios.", "questions_to_authors": "1. Can the proposed method further improve performance with better PLMs?\n2. How expensive the proposed method in the perspective of computational costs in comparison with the competitors?", "paper_topic_and_main_contributions": "This paper studies the false negative problems of conversational response selection tasks. For the purpose of better negative sampling, this paper proposes a variational approach that can consider various characteristics (or, facets) and a novel framework TRIGGER to dynamically update negatives according to the current model capacity. The proposed approach is applied in multiple retrieval models and evaluated on two famous response selection benchmarks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1]: RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering\n[2]: Conditional Response Augmentation for Dialogue using Knowledge Distillation"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Further definition of \"model capacity\" would improve the paper reading.\n2. Figure of the overall framework would make the readers better understand the entire flow of the proposed method at a glance.\n3. Line 199-201: \"the criteria for potential false negatives should change accordingly\" is a bit problematic. In response selection task, the criteria for true responses cannot be modified. It would be better to discuss it in terms of \"helpfulness\" of the negative sampels during training.\n4. Line 330-332: the set $S$ is not used in Equation 7. \n5. Section 6.2: For the arguments in Line 481-484, a preceding observation is required, such as the higher false negative ratios for the responses of lower lexical similarity. (though it is still interesting result).\n6. Line 458: missing '.'\n7. Section 6.3: Figure 2 and Figure 3 are not the results for adversarial attacks. Table 6 presents partial results for adversarial attacks (only 2 models). \n8. Section 6.4: It is hard to understand how can I interpret the results. Do the results contribute to the performance improvements in above sections?"}}, {"review_id": "9LSv0qfeH7", "reviewer": "Reviewer_oe9s", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments and suggestions, and they are exceedingly helpful for us to improve our paper. We carefully read through your comments and we would like to address your concerns point-by-point in below:\n\n\n**Q1: The false negative problem is already extensively studied**\n\n**A1**: We appreciate your guidance and recognize the extensive body of literature addressing negative sampling in the field of multi-turn response selection, as discussed in Section 3. Nonetheless, our comparison with prior negative sampling methods reveals that they either inadequately address the false negative issue or fail to implement negative updating (Table 2), thus inspiring our research. Furthermore, the human evaluation presented in Table 1 indicates that this problem remains unresolved.\n\n\n**Q2: The idea of modeling the inherent transition logic is not new.**\n\n**A2**: Thanks for your comment. The novelty of our proposed Sequential Variational Ladder Auto-Encoder (SVLAE) lies in encoding the transition of multiple characteristics in a disentangled way with a ladder-like architecture, as described in Section 4.1 and illustrated in Figure 1. The motivation behind the proposed approach is to model the dynamics of multiple orthogonal characteristics, thereby identifying possible false negatives that conform to the transition logic in one or more facets. To the best of our knowledge, we are the first to devise this variational and ladder-like architecture for negative sampling in dialogue retrieval. We are sorry for the caused ambiguity and we will definitely highlight the novelty more prominently in the final version.   \n\n\n\n**Q3: The proposed method is complex, and the improvement over baselines is marginal.**\n\n**A3**: Thank you for your remarks. We recognize that our approach may not yield a sharp enhancement in the retrieval performance of the Douban or Ubuntu benchmark, given their inherent dataset complexity and the limitations of the base retrieval model. However, as demonstrated in Table 4, the significant test (student test) reveals that TRIGGER notably surpasses the most competitive baseline method. Additionally, it is crucial to note that our proposed TRIGGER framework significantly bolsters the robustness of retrieval models, as evidenced by Table 6, which is vital for deployment. Besides, our method is compatible with any base retrieval model and can be integrated with more advanced PLM encoders to further augment performance\n\n\n**Q4: The ablation study shows little difference between the method and its several variants.**\n\n**A4**: Your observation is correct. In the development of our method, we also note that there is little difference between ours and the *-label* variant. In fact, the latent label $y^{1:L}$ is mainly used for categorizing the utterances in the corpus into different groups and constructing the negative set $S(\\tilde{y}_1,\\tilde{y}_2,\\tilde{y}_3,\\ldots,\\tilde{y}_L)$. Furthermore, the discrete $y^{1:L}$ facilitates the interpretation of the discovered transition logic, making it more explainable.  \n\nWe perform a significant test on our ablation experiments and the results are shown as below:\n|Variant | R$_{10}$@1 | R$_{10}$@2 | R$_{10}$@5 | MAP | MRR| P@1 | R$_{10}$@1 | R$_{10}$@2 |\n|------|------|------|------|------|------|------|------|------|\n|-transition | 90.9* | 95.9* | 99.2 | 64.5* | 68.2*  | 51.5* |32.6* | 54.8* |\n|-update |91.2* |96.3| 99.3| 65.9*| 70.4* |53.8* |34.9*| 56.3*|\n|-label |91.5| 96.4| 99.4| 67.3| 72.0| 55.9*| 36.1*| 58.3*|\n|-feature| 91.4*| 96.3| 99.4| 67.0*| 71.6*| 54.2*| 35.5*| 57.4*|\n\nThe numbers marked with an asterisk are significantly (t-test, $p<0.05$) inferior to the TRIGGER. \n\n\n**Q5: Large language models such as Chatgpt should perform this task well. What are their results?**\n\n**A5**: Thanks for your advice. We perform an experiment with the ChatGPT with the following prompt: \n\n\"Given the following conversation between two speakers: \n\nspeaker A: {}\n\nspeaker B: {}\n\nspeaker A: {}\n\nspeaker B: {}\n\n......\n\nand the 10 candidate replies numbered from A to J: \n\nreply A: {}\n\nreply B: {}\n\nreply C: {}\n\n......\n\nreplay J: {}\n\nwhich candidate reply is a good response to the above dialogue context to form into a fluent, coherent and consistent conversation?\n\nPlease output a rank of above 10 candidates to indicate whether the candidate reply is appropriate.\n\nThe more highly ranked the response, the more appropriate it is. For example, 'BDCAIJGHFE' means B is the best response, followed by D,C,A ... while E is the worst one.\n\nOnly output the rank in a single line. Do not explain. \" \n\nWe randomly sample 100 cases (1000 context-response pairs) from the Ubuntu benchmark The experiment results are shown below.\n|    |R$_{10}$@1 | R$_{10}$@2 | R$_{10}$@5 |\n|----|----|----|----|\n|GPT-3.5-turbo | 0.31 | 0.46 | 0.82 | \n\nFrom the table, we can observe that ChatGPT is not the final solution for the multi-turn response selection task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is easy to follow.\n2. The method is evaluated on multiple datasets and shows better performance than baselines.", "reasons_to_reject": "1. The false negative problem is already extensively studied, and the idea of modeling the inherent transition logic is not new.\n2. The proposed method is complex, and the improvement over baselines is marginal. The ablation study shows little difference between the method and its several variants. \n3. Large language models such as Chatgpt should perform this task well. What are their results?", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the false positive problem in multi-turn response selection. The paper proposes a sequential variational auto-encoder to capture the diverse one-to-many transition pattern of multiple characteristics in open-domain dialogue and a TRIGGER framework that combines the updating of negative sampling together with the optimization of a retrieval model.  Experimental results show the effectiveness of the proposed method.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "xNH9euudq3", "reviewer": "Reviewer_kX1o", "summary": "", "questions": "Is current transition modelling using a transformer structure with no pre-trained weights or load from any related pre-training models? Why is that?\n\nDuring the session, is the creation of different difficulty negatives or sets related to different facets? Can you elaborate on the process of sampling negatives from the set?\n\nNegative sampling is a very general technique in information retrieval, is there any discussion for the future work of this proposed method in other areas or domains?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and expertise invested in the reviewing process. We are grateful for your appraisal that our manuscript is of high quality and interesting. We carefully read through your insightful comments and we would like to address your concerns one by one. \n\n\n**Q1: Why dialogue retrieval methods are still meaningful and attractive for the research?**\n\n**A1**: Good point! The emergence of large language models in 2022 and 2023, alongside the impressive modeling capabilities of generative dialogue systems (e.g., ChatGPT), has sparked hopes for AGI and raised questions regarding the relevance of retrieval dialogue systems. Nevertheless, we maintain that retrieval dialogue systems warrant further study, with their values and advantages primarily residing in:\n\n+ Enhanced safety and control. As responses are chosen from a predefined candidate pool, retrieval dialogue systems offer greater content control, thereby mitigating the presence of toxic words or potentially unsafe content.\n\n+ Retrieval-augmented LLM. Many researchers contend that retrieval should play a vital role in next-generation language models to enhance their factuality, interpretability, and trustworthiness. For example, LLMs are less likely to hallucinate when provided with retrieved knowledge from a given knowledge base [1][2][3].\n\n+ Retrieval-after-LLM. LLM output can be unstable due to sensitivity to prompts and decoding strategies. A potential solution is to first generate multiple candidates using LLM to simulate various personas [4] or engage in multi-agent debates [5]. A retrieval model may then select the most suitable candidate as a response.\n\n+ Reduced computational cost. While LLMs generate high-quality output, the substantial computational expense of APIs (e.g., chatGPT and GPT-4) should not be disregarded, and prior research [6] indicates that LLM (GPT-3) may not be the optimal choice when considering both generation quality and API cost.\n\n+ Tool retrieval for LLM. External tools or APIs remain necessary for addressing higher-level tasks with LLMs, and a retrieval system is required to select an appropriate tool (API) from thousands of API documents based on specific human instructions [7]. Given the functional similarity of APIs and the existence of multiple solution paths (chains of API calls), randomly sampled negative APIs may suffer from false negative issues, and our approach offers a potential remedy for this problem.\n\nTherefore, there are still discussions on the techniques of dialogue retrieval.[8][9][10]\n\n\n[1]Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback, Arxiv 2023.  \n\n[2]Enabling large language models to generate text with citations, Arxiv 2023\n\n[3]Rarr: Researching and revising what language models say, using language models.\n\n[4]Unleashing cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration, Arxiv 2023\n\n[5]Improving factuality and reasoning in language models through multiagent debate, Arxiv 2023.\n\n[6]The economic trade-offs of large language models: A case study, ACL 2023.\n\n[7]ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs, Arxiv 2023.\n\n[8]CORE: Cooperative Training of Retriever-Reranker for Effective Dialogue Response Selection, ACL 2023\n\n[9]A Textual Dataset for Situated Proactive Response Selection, ACL 2023\n\n[10]Towards Efficient Coarse-grained Dialogue Response Selection, ACM Transactions on Information Systems 2023.\n\n\n\n\n\n\n**Q2: Hard to understand why this specific encoder or framework is proposed.**\n\n**A2**: The proposed framework aims at identifying and eliminating the potential false negatives in the negative sampling process. However, because of the one-to-many property in open-domain dialogue, the most appropriate response may vary if we consider different (and orthogonal) factors such as dialogue topic, speaker emotion, and so on, posing a great challenge to the identification of false negatives. \n\nTo deal with this, we attempt to model the transition logic of these multiple orthogonal characteristics and predict the characteristic distributions of the golden response. Given the predicted characteristic distribution, we dynamically construct and adjust a negative set $S(\\tilde{y}_1, \\tilde{y}_2, \\ldots, \\tilde{y}_L)$. Therefore, the false negatives are diminished and the difficulty of the negatives is updated, constituting our TRIGGER framework.\n\n\n\n**Q3: Is current transition modeling using a transformer structure with no pre-trained weights or load from any related pre-training models? Why is that?**\n\n**A3**: Regarding the inference module, we load weights from a (tiny-)BERT model, as different layers of the BERT model often encode varying levels of semantic abstraction [1], which assists in  acquiring the transition patterns of multiple orthogonal characteristics.\n\nFor the generation module, we employ no pre-trained weights but instead use a randomly initialized unidirectional transformer, as transition modeling inherently involves sequence modeling of latent labels, and the latent label sequence does not correspond to any existing language.\n\n\n[1]BERT Rediscovers the Classical NLP Pipeline, ACL 2019.\n\n\n**Q4: During the session, is the creation of different difficulty negatives or sets related to different facets? Can you elaborate on the process of sampling negatives from the set?**\n\n**A4**: Good question! The difficulty of a negative sample/set depends on its/their latent labels in $L$ facets, or $y^{1:L}$. In essence, the more a negative sample resembles the golden response in terms of latent labels, the more probable it is to be a false negative or hard negative.\n\nIn detail, given the predicted latent label of the golden response $p_\\theta(y^{1:L}_{N+1})$, the policy network $\\mathcal{P}$ outputs distribution $\\pi(y^1), \\pi(y^2), \\ldots, \\pi(y^L)$, from which we sample $\\tilde{y}_1 \\sim \\pi(y^1), \\tilde{y}_2 \\sim \\pi(y^2), \\ldots, \\pi(y^L) \\sim \\tilde{y}_L$. The negative set consists of all the utterances whose latent labels in $L$ facets are exactly $\\tilde{y}_1, \\tilde{y}_2, \\ldots, \\tilde{y}_L$.\n\nIn summary, the policy network establishes the latent label distribution of the negative set $S$ across $L$ facets, with the similarity between this distribution and the latent label distribution of the golden response determining the negative difficulty. We apologize for any ambiguity in lines 299-310 and will ensure that the details are clarified in the final version.\n\n\n**Q5: Is there any discussion for the future work of this proposed method in other areas or domains?**\n\n**A5**: Thanks for your suggestion! The proposed approach can be easily tranferred to and adapted other subfields:\n\n+ conversational search. This approach to information retrieval involves users interacting with an AI assistant in a conversation session to iteratively refine their queries and seek additional information. As it shares the intrinsic features of open-domain dialogue and requires negative sampling during training, our method can be integrated with existing conversational retrieval systems to enhance their performance.\n+ conversational recommendation. This system aims to provide accurate product recommendations by directly extracting user preferences from dialogue history. However, like traditional recommendation systems, it necessitates negative sampling for optimization. Given that randomly sampled negatives pose risks of uninformative sampling or false negatives, our approach can be employed to strike a balance between unearthing hard negatives and eliminating false negatives.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The problem of a false negative in dialogue retrieval is well discussed and the methods are proved through well-designed experiments in a very scientific manner. \n\nOverall, the paper is of high quality with good English, mathematical expression, tables and figures. \n\nSection 2 is really interesting which use human evaluation to identify the situation of false negative, it would be better if we slightly increase the sampling size. \n\nThe discussion and results are clear and sound. ", "reasons_to_reject": "\nCompared with more powerful generative PLM-based methods (especially LLMs), the author fails to justify why dialogue retrieval methods which can only select from existing dialogues instead of generating more diverse or new responses are still meaningful and attractive for the research. \nAll the related works about dialogue retrieval are from before 2020 and there are only limited words on its application, which makes the reviewer doubts the potential applications of dialogue retrieval in future NLP.\n\n\nThe transition between the problem to your prosed methods in the introduction is not smooth, which makes it hard to understand why this specific encoder or framework is proposed.", "questions_to_authors": "Is current transition modelling using a transformer structure with no pre-trained weights or load from any related pre-training models? Why is that?\n\nDuring the session, is the creation of different difficulty negatives or sets related to different facets? Can you elaborate on the process of sampling negatives from the set?\n\nNegative sampling is a very general technique in information retrieval, is there any discussion for the future work of this proposed method in other areas or domains?", "paper_topic_and_main_contributions": "This paper is addressing the issue of dialogue response retrieval and negative sampling, targeting to filter false negative samples from negative samples. \nThe author identifies the false negative issue by conducting human annotation and analysis. \nThe author proposes a sequential variational ladder auto-encoder to capture the transition in dialogue, and further facilitate the identification of false negative responses, which helps the PLM better use negative sampling for training.\nExperiment results prove the efficiency of the proposed methods.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Formula 2, what is f(theta)? Formula 4, what is metric W^l?"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper delves into the false negative issues within conversational response selection tasks. To enhance negative sampling, the authors put forth a variational method accounting for diverse characteristics, along with the TRIGGER framework to dynamically refresh negatives based on the model's evolving capacity. This approach is tested on various retrieval models and validated using two renowned benchmarks: the Ubuntu and Douban datasets.

Soundness scores stand at (4, 3, 4), reflecting reviewers' consensus on the paper's solidity. The presented claims are well supported by the studies and experimental data.

On the other hand, excitement scores read (3, 2, 3). A primary critique highlights the proposed solution as overly complex, with limited generalizability and practical application. The performance improvement is marginal. However, there is still a notable highlight is section 2, which presents an insightful human-centric pilot study on false negatives.
Decision: Accept-Findings