An Equal-Size Hard EM Algorithm for Diverse Dialogue Generation
Authors: Yuqiao Wen
Yongchang Hao
Yanshuai Cao
Lili Mou
Venue: ICLR 2023 poster (2023)
Keywords: dialogue systems
diverse text generation
EM algorithm

Abstract: Open-domain dialogue systems aim to interact with humans through natural language texts in an open-ended fashion. Despite the recent success of super large dialogue systems such as ChatGPT, using medium-to-small-sized dialogue systems remains the common practice as they are more lightweight and accessible; however, generating diverse dialogue responses is challenging, especially with smaller models. In this work, we propose an Equal-size Hard Expectation--Maximization (EqHard-EM) algorithm to train a multi-decoder model for diverse dialogue generation. Our algorithm assigns a sample to a decoder in a hard manner and additionally imposes an equal-assignment constraint to ensure that all decoders are well-trained. We provide detailed theoretical analysis to justify our approach. Further, experiments on two large-scale open-domain dialogue datasets verify that our EqHard-EM algorithm generates high-quality diverse responses.

Published as a conference paper at ICLR 2023
AN EQUAL-SIZE HARD EM ALGORITHM FOR DIVERSE DIALOGUE GENERATION
Yuqiao Wen' Yongchang Hao'
Yanshuai Cao
Lili Mou
1 Dept. Computing Science, Alberta Machine Intelligence Institute (Amii), University of Alberta 2 Borealis AI Canada CIFAR AI Chair, Amii
yq when@gmail, com yongchal @ualberta.cm ca
yanshuai cao@borealisa.. com doublepower wwwu@ma com
ABSTRACT
Open-domain dialogue systems aim to interact with humans through natural lan- guage texts in an open-ended fashion. Despite the recent success of super large di- alogue systems such ChatGPT, using medium-to-sine to-smoll-sszze dialogue systems remains the common practice as they are more lightweight and accessible; how- ever, generating diverse dialogue responses is challenging. especially with smaller models. In this work, we propose an Equal-size Hard Expeetation-Maxxiizaaaioo (EqHard-EM) algorithm train a multi-decoder model for diverse dialogue gen- eration. Our algorithm assigns a sample to decoder in a hard manner and ad- ditionally imposes an
constraint to ensure that all decoders are well-trained. We provide detailed theoretical analysis to justify our approach. Fur- ther, experiments on two large-scale open-domain dialogue datasets verify that our EqHard-EM algorithm generates high-quality diverse response
INTRODUCTION
Open-domain dialogue systems aim generate natural language text utterances to hold open-ended conversations with humans (Li et al. 2017a; Wang et al. 2021b). These systems have shown great success, and are scamlessly integrated into society through chatbots. The recently launched ChatGP1P model, for example, has shown remarkable conversational skills. However, ChatGPT is prohibitively large in size and requires significant human feedback during its training process, and language models without human feedback still remains the common practice (Wu et al. 202I Wang al., 2021aa Chen al., 2022), and these smaller models tend to generate generic responses such as / don't know (Li et al. 201 Wang et al. 2021b). One of the possible causes is the one-to-many mapping phenomenon in the dialogue task, where dialogue context may correspond to multiple plausible responses (Wei et al. 2019: Bao et al.) 2020 Khan et al. 2020). Learning to generate dialogue utterance is thus analogous to learning multimodal distribution of a continuous variable, where a mode refers to a peak in the distribution; in the dialogue task, a mode can be thought of as a set of similar responses. The widely used cross- entropy training is not good at capturing different modes, as it encourages the prediction to cover all plausible responses, forcing the model to learn an overly smooth distribution. Consequently, these neural dialogue models resort to generating generic responses (Wei et al. 2019).
therefore, training codiiu--t--smallsiizeee
Previous studies address generic responses mainly at training or inference time. Training tamiing-timeeae ap- proaches explore different training objectives: [i al. (2016b) apply reinforcement learning to optimize a customized reward function that discourages generic responses. Khan er al.) (2020) train generative adversarial network in the latent space to discourage generic responses because they can be easily identified by the discriminator. Among inference approaches. Vijayakumar et al. (2016) encourage diverse beam results by penalizing similar beam entities. Holtzman al. (2020) apply nucleus sampling to allow plausible but less probable words to be decoded. Wang et al. (2021b) apply label smoothing to prevent the model from being overly confident with generic responses.
code is available at http:///aniai.ooooooo/o////aaaaa

---

Published as a conference paper at ICLR 2023
Decoders
xN
xN
xN
Adapter
Adapter
Adapter
Transformer Layer
Transformer Layer
Transformer Layer
Decoder
Decoder 2
Decoder K
Transformer Layer
xN
Shared Encoder
Samples
(a) Adltt--dapper Model
(b) Equal-Size Hard Assignment
Figure 1: (a) Our aaatie neural architecture. (b) The equal-size hard assignment scheme.
Dashed circles: decoders are conceptually duplicated when we solve the assignment problem.
Another way to address the generic responses is explicitly capture different dialogue patterns with mixture models. This has been explored for diverse machine translation, where [Shen et al. (2019) train multi-decoder model with the Expectation-Maxiiizzaoon (EM) algorithm (Dempster et al. 1977) to address the one-to-many mapping phenomenon. However, their direct application of the standard EM algorithms suffers from the decoder--collapsing problem, where the multi-decoder model degenerates to single-decoder model. The authors attempt to alleviate this problem by disabling the dropout layers during the E-step of the EM algorithm, but their results show that the model is still susceptible to collapses.
To this end, we propose novel EM variant to multi-decoder diverse dialogue generation. A stan- dard EM algorithm assigns sample to all decoders by the posterior probability, thus known as Soft-EM; suffers from synchronous-trainigg collapse, where the posterior probabilities tend be similar and all decoders are also trained similarly. The Hard-EM variant trains the decoder that has the highest posterior probability; it suffers from non-training collapse due to the rich-gets-richer phenomenon. Our proposed EM algorithm avoids both types of collapses by adopting hard assign- ments and imposing an cqual-assignment constraint. Hence, we call our approach Equal-size Hard EM (aqHard-MM).
We conducted experiments on the Weibo (Gao et al. 2019) and OpenSubtitles (Lison et al.) 2018) datasets. Results show that our EqHard-EM algorithm alleviates the decoder-collapsing problem in both Soft-EM and Hard-EM, allowing the multi-decoder model to specialize and generate high- quality and diverse responses. In addition, we provide in-depth theoretical and empirical analyses to better understand our EqHard-EM algorithm.
To sum up. our contributions are three-fold: the collapse issues in soft and hard EM variants; 2) We provide detailed theoretical analysis to justify our equal assignment; and 3) We conduct extensive empirical experiments show the effectiveness of our approach.
1) We propose the EqHard-EM, algorithm alleviate
APPROACH
In this section, we first present our mixture model architecture (Subsection 2.1 Then, we propose EqHard-EM, novel EM variant, for training such mixture models (Subsection 2.2
2.1 NEURAL ARCHITECTURE
We propose to address the one-to-many mapping phenomenon in the dialogue task with mixture model. Given a dialogue context, we use a shared encoder to build an input utterance's hidden representations, based which multiple decoders generate set of output responses.
For multiple decoders, 2019), but this leads to a large number of parameters that may not fit into the memory of an ordinary GPU. To this end, we propose multi-adapter architecture (Figure Th), where different decoders share most Transformer parameters, but differ by a few inserted, thin adapter layers (Houlsby al.) 2019). In fact, the adapter model is widely used for aarameter-efficien domain adaptation (Artetxe)
possible to instantiate each with a full Transformer model (Shen et al.,
al. 2020: Wang al. 202 Ngo Trung al. 2021 We propose apply this architecture
2

---

Published as a conference paper at ICLR 2023
mixture models. Our preliminary experiments show that the multi-adapter model results in degra- dation of one BLEU point, but reduces 76% parameters compared with Tull-Tranformmr decoders. Therefore, use the multi-adapter model in main experiments.
Specifically, an adapter layer applics two projections of linear transformation and non-linear ac- tivation. Further, residual connection is included to ensure that the Transformer layers are not significantly disturbed. This process can be formulated as
AdapterLayer(x) = x + W1(
(1)
where x E _6 is the input to the adapter layer; W1 1 E Rdxd'' and W2 E Rdd xd are the projection matrices with d' < d; and is non-linear activation function such the rectified linear unit (ReLU, Agarap 2018
2.2 TRAINING METHOSSS
We propose to train our multi-adapter architecture in two stages: 1) Pretraining of the shared Transformer-layer parameters, and 2) EM training adapter-layer parameters as the mixture model. In the first stage, we pretrain T5-smal eecoddr-decoded model (Raffel et al. (2220) with stan- dard cross-entropy loss since the Transformer parameters are shared among different mixture com- ponents; in the second stage, adopt the standard treatment of adapter modules (Houlsby al., 2019) and freeze the shared Transformer parameters to minimize the number of trainable parameters for effective adaptation.
In the rest of this subsection, we focus on the second stage, i.e., training the adapter parameters of the mixture model. We first review classic EM algorithms and analyze their drawbacks. Then, propose a novel EM variant that addresses the drawbacks.
Classic EM Algorithms. Consider multi-decoder model with K decoders. Given dia- logue context e and response r, our training objective is to maximize the marginal likeliodd log Σx P(r, z klc; 0), where 0 is the model parameters and the variable 2 E {I, _K) the choice of the decoder. Since is unobserved in the training set, it can be viewed as a latent variable, and we apply the expectation expectation-maxiiiaatte maximization (EM) framework (Dempster et al. 1977) as a principled approach to training latent-variable models. Specifically, the EM algorithm alternates between the E-step and M-step:
E-step: Estimate the posterior distribution of the latent variable by applying Bayes* rule:
p = 0)P k|c; 0)
Qs(c.r) = P(= 0)
(2)
ΣK_ -(r]==kk P(z -/ c)
M-step: Maximize the expected logllikel with respect to O:
Ez-Qu(es) log 2(r,2/c;0). Σ k= Qu(c,r) P(r 2(;
(3)
We additionally assume a uniform prior for every context c as P(= klc; 0) 1/K, K, following the one-to-many assumption that each context has multiple plausible responses. This assumption simplifies Eqn. (2) as
p = k;
P(= k(c,
(4)
ΣK_ f
suggesting that the posterior is proportional to the likelihood under the uniform prior assumption.
Soff-EM. The classic EM algorithm, as explained above, assigns sample to a mixture component by the posterior distribution (2) in a soft manner, and thus is also known as Soft-EM. However, we
IIn principle, other pretrained language models may also be used. In our development, we have tried finetuning GPT-2 on the OpenSubtitles dataset. It gave a similar, but slightly lower, result: 3.26 BLEU2-F vs T5-small's s 3.46 BLEEU--F. Therefore, we chose T5-small as the base model, because it achieves higher performance with fewer parameters.
3

---

Published as a conference paper at ICLR 2023
observe that Soft-EM performs poorly in experiments due to synchronous-training collapse where all decoders are trained almost synchronously and generate similar responses. In particular, the Transformer-layer parameters are pretrained. Even with the inserted adapter layers, all decoders still behave similarly, yielding a similar posterior distribution (2). This in turn yields a similar gradient for each decoder, and result, multiple decoders virtually collapsed to decoder.
Hard-EM. Alternatively, the assignment in the E-step can be accomplished in a hard manner by choosing the best-fit mixture component
k" argmaxx P(2 k;
(5)
Then, the M-step is approximated as follows
Ex--Qu((.c P(r, 2/c; 0)] = log 2(x;==k'|c;00
(6)
Unfortunately, the Hard-EM algorithm suffers from the non-training collapse, where only one decoder is trained due to the rich-gets-richer phenomenon. That is to say, particular decoder will be selected by Eqn. (5) due to the random initialization of adapter layers. Then, that decoder will be trained with more samples and perform better than the other decoders, making it more likely to be selected by Eqn. (5). In practice, we observe that only one decoder is properly trained by the Hard-EM algorithm, making the mixture model ineffective.
Our proposed EqHard-EM. To this end, we propose an Equal-size Hard Expectation- Maximization (E) algorithm to address the collapse issues. Specifically, we adopt the hard assignments from Hard-EM to avoid synchronous-traa traaiinng collapse, whereas we impose an equal-assignment constraint ensure that all decoders are adequately trained avoid non-training collapse.
We formulate the E-step as constrained optimization problem over an assignment matrix A {0, 1 where N is the number of samples and K is the number of decoders. The element Ank indicates whether the nth sample is assigned to the kth decoder. We define the cost matrix C E I N by the posterior probability, i.e., C.k - (r(m), c(r)) as in Eqn. (2) Our equal-size hard E-step aims to
minimizea
K k= DA' Ankk
subject to
Σ == Ank = 1, for n = 1,,,,,,,
(8)
ΣN Ank = N/K, for l: 1, K
(9)
Ank E [0,
(10)
Compared with Hard-EM, we impose the constraint to ensure that every decoder is assigned an equal number of samples.
In practice, this optimization problem can be transformed into a balanced assignment problem, as shown by Figure [p. This is accomplished by duplicating each decoder and its associated costs by factor of N K to match the number of samples. Here, we enforce the batch size to be divisible by only conceptually duplicated, and it does not incur additional compu- tational cost. We solve the balanced assignment problem by the classic Hungarian algorithm (Kuhn) 1955), which was later improved by Joonker & Volgenant (1987) to achieve computational com- plexity of O(N3). This is efficient in our batch implementation and contributes to less than 1% of the total training time.
K. Note that the decoders
For inference, we follow existing literature (Zhao et al. 2017 Gu et al.) 2019 Shen et al. 2019)
and consider the multi-response generation setting, where the goal to generate a set of diverse responses given a dialogue context. We collect the outputs of different decoders by greedy decoding.
Theoretical analysis. We provide theoretical justifications for our EqHard-EM algorithm. Our equal-assignmen constraint makes sense intuitively, because the aggregated posterior should be the same as the prior distribution, suppose the posterior distribution is correctly estimated and we have
Shen et al. 2019) refer to this phenomenon as the "D2 Degeneracy" and interpret it as the bypassing of latent variables, similar to the posterior collapse of variational auto-encoders (VAE, Bowman et al., 2016 Bahuleyan al. 2018). We find the connection with VAF weak. Instead, we present our own interpretation of synchronous traiing, which is justified by empirical evidence.
4

---

Published as a conference paper at ICLR 2023
enough samples. This is given by Ee P(ein =e -
1/K for 2 = ...., , K. In this part, we develop a theorem on how equal the assignments would be in the superfect-posterion and maite cases.
....
Theorem 1. With probability at least - 8, we have
Σ (r,c)ED
c(z
$ 1 2
log
2K (
Eer
[Q(=]c.r) - p(z)e,r)
-
[D]
K
for all E [I,...,K" KJ. Here, Qz)ce,r) is estimated posterior and 'D is a dataset with finite samples.
The proof sketch is to consider finite samples (Lemma I and the estimated posterior (Lemma 2 separately, which can then be combined in a straightforward manner. See Appendix A for detailed proof.
Theorem I justifies the equal-assignment constraint, because with a high probability, the assign- ments will converge to uniform if we have large dataset and a near-correct posterior. Note that the term ((. Q=(r,, c) in the theorem mainly works for soft assignments. However, we could not adopt soft assignment in the algorithm because it leads to synchronous-traine collapse. The assignenttn approximation
theorem is nevertheless illuminating to our algorithm, which similar to Hard-EM.
3 EXPERIMENTS
3.1 SETUP
Datasets. We evaluated our approach on two large-scale open-domain dialogue datasets: Weibo (Gao et al. 2019) and OpenSubtitles (Lison et al. 2018). The Weibo dataset is compi- lation of posts and replies, crawled from Chinese micro- plocoonlooggng website. The dataset features explicit one-to-many mapping because in most cases, post corresponds to multiple replies, On the other hand, the OpenSubtitles dataset a collection movie dialogues, which are parsed from on- line subtitle files. The dataset is known to be noisy (Vinyals & Le 2013 Feng et al. 2020 Nakazawa al. 202T), and therefore demonstrates the implicit one-to-many phenomenon: movic utterance though the dataset only contains reference response. Admittedly, there exist other manually constructed, high-quality datasets such as DailyDialog (Li
may lead to many plausible responses
al. 2017b), but they usually much smaller and the one-to-many phenomenon is less severe.
Recently, Wen et al. (2022) identify significant overlapping between training, validation, and/or test splits in various open-domain dialogue datasets (including OpenSubtitles) due the oversights of data collection. We use their cleaned OpenSubtitles dataset, which contains 1M, 12K, and 12K samples for training, validation, and test, respectively. We investigated the Weibo dataset and found significant overlapping between the validation and test sets as well, so we followed their cleaning strategy and obtained 4M, 10K, and 10K training, validation, and test samples. Overall, Weibo and OpenSubtitles allow to conduct comprehensive analyses in both explicit and implicit one-to-many mapping scenarios and in different languages.
Metrics. Open-domain dialogue evaluation is currently an active area of research due to its one-to- many nature (Tao et al. 2018: Sinha et al., 2020). In particular, the standard BLEU metric, which measures the n-gram similarity between generated and reference responses, has a low correlation with human evaluation because a plausible response is not necessarily similar to the reference re- followed previous work on open-domain dialogue (Zhao et al. 2017 7 Gu et al. 2019 Li et al. 2020) and generated 10 responses for each dialogue context to compute BLEU-P, BLEU-R, and BLEU-F. Roughly, BLEU-P measures the average quality of the generated responses, whereas BLEU-R measures how well the references are covered. BLEU-F is the harmonic mean We adopted uni-gram and bi-gram BLEU scores in our experiments, because Liu et al. (2016) show that the n--gram overlap is often zero for dialogue evaluation with n > 2.
sponse (Liu et al. 2016 Therefore,
'The terminologies of BLEU-P, BLEU-R, and BLEU-F are inspired by, but different from, the precision and recall in information retrieval (Manning et al. 2008 Interested readers are referred to Zhao et al. (207 for the details of BLEU-P, BLEU-R, and BLEU-F
5

---

Published as a conference paper at ICLR 2023
BLEUI
BLEU2
Distinct n- n-gram
Pairwise
Datuset
Model
F
P
R
F
P
R
Dist-1
Dist-2
BLEU+
TS-small beam search
18.59
25.59
14.60
8.51
17.57
5.62
10.41
14.17
65.46
TT-small nucleus sampling?
20.76
26.13
17.23
10.03
17.44
7.04
16,90
23.79
44.55
Weibo
AdaLabel WWag al. 1b
14.80
21.63
11.25
5.86
13.72
3.72
10.41
14.71
62.46
DERT (Gu al. 2021
15.99
15.88
16.10
3.55
3.95
3.23
68.95
96.76
0.01
Our approach
22.65
26.17
19.96
10.13
14.55
7.77
27.03
42.02
22.32
T5-small
14.39
11.23
20.01
3,46
2.40
6.22
13.38
20.71
42.76
Open- Subtitles
T5-small nucleus samplingt
16.03
11.21
28.13
2.51
1.47
8.48
60.50
91.26
0.29
AdaLabel WWang et al., 1b
15.80
11.94
23.38
3.20
2.19
5.94
26.47
43.57
17.21
DialogBERT (Gu et al. 2021
12,86
8.71
24.50
1.20
0.68
4.94
69.70
96.49
0.02
Our approach
17.86
12.98
28.60
3.76
2.36
9.27
34.09
50.41
13.15
Table 1: Results on Weibo and OpenSubtitles. We use cleaned versions of these datasets since
the original datasets contain overlapping samples (Wen al. 2022). indicates sampling- -based approaches: DialogBERT performs poorly with beam search or greedy decoding, also reported by Wen et al. (2022). T/eThe higher/lower, the better. Underlined numbers are the best diversity scores (Dist and Pairwise-BB BLEUU) among non-sampling methods. Note that comparing diversity is only meaningful when BLEU scores are controlled.
Consequently, high-order BLEU scores mostly rely on smoothing techniques and correlate poorly with human satisfaction.
For diversity, we adopt the Dist (Li al. 201 and Pairwise-BLEU (Shen et al., 2019 scores.
Dist-n is a widely used metric that measures the percentage distinct r-grams among utterances. We follow the standard practice and report Dist-1 and Dist-2 between the generated responses for each dialogue context et al.] 2016a| Wang et al. 2021b). In addition, we follow follow[Shen et al. (2019) and report Pairwise-BLEss which measures the pairwise similarity among the responses, i.e., a low Pairwise-BL.Fi indicates high diversity.
Implementation details. We considered 10 decoders due to the setup of diverse generation evalua- tion (Zhao et al. 2017 Gu et al. 2019 Li et al. 2020). Each E-step involved 640 samples, assigning 64 samples to each decoder. More implementation details are presented in Appendix B
3.2 RESULTS AND ANALYSES
Main results. Table 1 shows our main results on Weibo and OpenSubtitles. The T5-small model is a pretrained mncodr-ddccode Transformer (Raffel et al. 2020), where we experimented with both beam search and sampling. For beam search, we set the beam size to be 10, i.e., the number of samples needed for evaluation. As known, beam search does not generate diverse responses (Vi- jayakumar et al., 2016; Holtzman et al., 2020), shown by low BLEU-R and Dist scores, but high Birtiise-BEEL The performance of nucleus sampling is inconsistent in the two experiments: on the Weibo dataset, it slightly improves both generation quality and diversity; on OpenSubtitles, how- ever, the generated responses are much more diverse, but the generation quality is not convincingly improved, as shown by the low BLEU2-F score. We examined the samples and believe this is be- cause nucleus sampling depends on the distribution temperature and the nucleus size for sampling: if the sampling scope is too small, then the diversity would not be improved much; if otherwise the sampling scope is too large, the quality may deteriorate.
By contrast, our EqHard-EM approach is able perform consistently well on both datasets by sim- ple greedy decoding. Our BLEU-R is much higher than most competing methods. This shows that our EqHard has high coverage of potential responses, largely alleviating the one-to-many issue in dialogue generation. Overall, we achieve the highest BLEU-F scores, while largely improving di- versity. In fact, we outperform all non-sampling methods in terms of both Dist and Pairwise-Bl.Hr scores.
We also experimented with recently proposed models: AdaLabel (Wang et al. (021b) is an encoder- decoder Transformer, trained with adaptive label smoothing improve diversity. DialogBERT (Gu et al. 2021) features the use of a BERT encoder (Devlin et al. 2019) to capture contextual infor- mation. They were alleged to be state-of-the--
models when the dataset-overlappinng problem was not realized. However, they perform poorly on the deduplicated datasets; our results in the diverse generation setting consistent with single-response generation (Wen et al. 2022).

---

Published as a conference paper at ICLR 2023
BLEUI
BLEU2
Distinct r-gram
Pairwise
Dataset
Model
Dist-1
Dist-2
BLEU
Bearn search
18.59
25.59
14.60
8.51
17.57
5.62
10.41
14.17
65.46
Soft-EM
17.02
27.55
12.32
6.27
16.03
3.89
10.69
12.45
86.25
Weiho
Hard-EM
4.81
3.09
10.84
2.15
1.61
3.22
4.01
4.11
Sakandom-Dynamiin
18.72
27.34
14.23
7.21
15.87
4.66
12.79
16.30
71.19
EgRandomm-Five -Fxed
20.01
25.51
16.46
8.12
14.41
5.65
19,40
25.77
46.60
EgHard-EM
22.65
26.17
19,96
10.13
14.55
7.77
27.03
42.02
22.32
Beam search
14.39
11.23
20.01
3.46
2.40
6.22
13.38
20.71
42.76
Soft-EM
15.48
14.55
16.54
3.03
2.76
3.35
12.89
15.35
72.95
Hard-EM
6.84
4.13
19.81
1.25
0.73
4.30
4.83
4.81
1.50
OpenSubrites
15.80
14.56
17.27
2.92
2.56
3.40
14.52
18.57
70.12
EqRandom-Fixen Fixx
15.89
14.59
17.46
2.93
2.56
3.43
14.73
18.96
68.89
FqHard-EM
17.86
12.98
28.60
3.76
2.36
9.27
34.09
50.41
13.15
Table 2: Comparing EM-like algorithms based on multi-adapter T5-small. The Soft-EM and Hard-
EM variants follow Shen al. (2019). who address diverse machine translation.
(a) Soft-EM
(b) Hard-EM
(c) EqHard-EN EM
100
50
000
000
000
000
10.00g 000
.00000
10000
0000
001
000
10.000
000
000
0,99
1
3
6
8
10
2
4
6
7
99
10
2
4
7
9
10
Decader Index
Decader Index
Decoder Index
Figure 2: Decoder assignment statistics for (a) Soft-EM, (b) Hard-EM, and (c) EqHard-EM. Means and standard deviations are computed on the Weibo training set.
Comparing EM variants. We conducted controlled experiments compare EM-like algorithms (Table 2 We find Soft-EM suffers from aeccroonous-tainiigg collapse, as it achieves very high Pairwise-BLEU scores. achieves even lower diversity than beam search because of the puachronous-trainigg collapse. This can be best seen by decoder assignment statistics (Figure 2 where every decoder is assigned an equal fraction samples with near-zero standard deviation.
On the other hand, the Hard-EM algorithm also performs poorly. It suffers from the non-training collapse, due to the rich-gets-richer phenomenon. As seen in Figure only Decoder 6 is trained in the experiment. Hard-EM achieves near-zero Pairwise- BLEU and the lowest Dist scores (Table 2 because the untrained decoders tend to repeat random words, which can be verified in our case stud- ies (Appendix E). As result, the generated sentences not meaningful, shown by extremely low BLEU scores.
As shown in Figure 2 EqHard-EN EM guarantees that all decoders trained with the same number of samples, preventing non-training collapse: since our algorithm adopts hard assignments (either 0 or 1), it also has high variance among different samples, preventing from synchronous- training collapse. Overall, Table 2 shows that our EqHard-EL achieves the highest BLEU-F score, as well as the best diversity scores (excluding the improperly trained Hard-EM), Appendix presents sample responses generated by Soft-EM, Hard-EM, and EqHard-EM.
Shen et al. 2019) have experimented with other EM variants including disabling the dropout and learning the prior. We analyze them in Appendix F due to the limit of space, especially as these variants are neither effective nor standard.
We further investigate whether EM-style, posterior-based assignment actually helps. We com- pare our Eq with two other equal-size hard assignment methods: EqRandom-Fis and SegRandom-Dyramte before training; this is equivalent to training each decoder on a partition of the training set. EqRandom Diraaddom-DDnami... assigns samples to decoders randomly and dynamically during training; this is essentially training a model with multiple initializations, if the number epochs is large. In both cases, ensure each decoder is assigned the same number of samples, so they equal-size hard assignments. As seen, these two methods perform similarly to each other. They are slightly better than beam search, but much worse than our HaM in terms of both quality and diversity.
The EqRandom-Fi i xed variant pre-assigns samples to decoder

---

Published as a conference paper at ICLR 2023
12.5
(a) Beam Search, length
(b) Beam Search, "?"
(c] Beam Search, "like/love
17.5
10.0
1.5
length
15.0
7.5
1.0
5.0
12.5
2.5
10.0
0.0
0.0
4
2
9
10
7
8
9 10
Beam Index
Beam Index
Bear Index
12.5
(d) EqHard-EM, length
(e) EqHard-EM, "?"
EqHard-EM, "like/love"
17.5
10.0
1.5
length
15.0
7.5
2 1.0
12.5
0.5
2.5
10.0
0.0
0.0
2
3
4 5 6
10
1 2
3 5
8 9
10
2
5
6
7 8
9 10
Decoder Index
Decoder Index
Decoder Index
Figure 3: Comparing outputs from beam search and EqHard-EM on the Weibo dataset. Subplots (a) and (d) compare the length of generated sentences; Subplots (b) and (e) compare the percentage of the question mark; and Subplots (c) and (f) compare the percentage of the word "like/love" (#X)) generated.
These controlled experiments convincingly show that our EqHard-EM is able to take advantage of the principled EM algorithm to learn a mixture of decoders, and that our model does not suffer from the collapse issue.
Analyzing different decoders. We analyze how decoders are specialized in certain aspects of dia- logue responses. Admittedly, the responses generated by different decoders would not exhibit clear clustering phenomena, because our inputs are open-domain dialogue utterances covering a wide range of topics. The generated responses must fit the input in certain way, the between-sample variance would be larger than the decoder variance. Despite this, we do find obvious patterns of certain decoders, including the length" sentence structures (e.g.. whether it is question), and the preference of certain words (Figure 33. For example, Decoder tends to generate short question responses, whereas Decoders 9 and 10 tend generate positive-sentiment utterances with the word "like/love". For comparison, we also show beam search results, which are nearly uniform among quan-
different samples in the beam. Our decoder analysis, despite its simplicity, is conducted titative fashion, as opposed to previous studies that are purely based on examples (Gu et al.) 2019) Shen et al. 2019). This provides rigorous evidence that our EqHard-EM able train multiple decoders to capture different "modes" of open-domain dialogue utterances.
Additional results. We present additional results in appendices. C Effect of the number of de- coders, D Human evaluation, E. Case study, F Analysis of additional EM variants, and G Effi- ciency analysis.
4 RELATED WORK
Open-domain dialogue generation is a challenging task as neural dialogue systems may generate generic responses such as "I don't know", which especially apparent for moderately sized models due to their limited modeling capacity (Li et al., 2016b; Wang et al. 2021b). In addition to the training-time time and inference-time time approaches mentioned Section I another line of work addresses the problem using cue words to guide the response generation (Mou et al. 2016). Yao et al. (2017) enhance the gated recurrent unit to fuse cue words that are heuristically selected based on point- wise mutual information. Gao et al. (2019) jointly train a cue word inference model along with the decoder network to further improve performance. Zou et al. (202I) apply non-autoregresssvv
We observe the Fq generates slightly shorter responses than heam search (Figures 3 and 3 By examining the utterances, find heam search tends generate repetitive words such "ha ha...
8

---

Published as a conference paper at ICLR 2023
generation incorporate cue words in the generated response. Generic responses may also be alleviated by grounding on commonsense knowledge. Zhou et (018)) use a large-scale knowledge graph to ground the generation process. |Wu et al. (020) make further improvements by retrieving and fusing relevant knowledge facts. These approaches do not directly address the one-to-many phenomenon in open-domain dialogue generation, but rely external information to make the responses more meaningful.
Large language models (LLMs) suffer less from generic responses because they have more model capacity to learn the desired distribution required for the one-to-many dialogue task. Recently, LLMs have shown success in text generation with reinforcement learning using intensive human feedback. Ouyang et al. (2022) propose InstructGPT to improve LLM's ability to follow human instructions. They warm-start the LLM by supervised learning, and then finetune with proxi- mal policy optimization (PPO, (SchulmSn et al. 2017), where the feedhack is provided by a reward model. Later, OpenAl releases ChatGPT, which follows a similar training pipeline and achieves remarkable performance for dialogue generation. These models require significant human annota- tion for both expert demonstration and reward modeling, making them inaccessible to most users and researchers. Medium-to soatlluuttt---maai coo-small-sizze dialogue systems trained by supervised learning remain the common practice for dialogue generation.
Expectation maximization (EM, Dempster et al. 1977) is principled approach training mixture models. The algorithm has been successfully applied in various fields (Dayan & Hinton 1997] Siig
& Buhhmann 2008 Li et al. 2019). Specifically in natural language processing, EM is applied to unsupervised or semi-supervised training of hidden Markov models for sequential labeling (Rabiner
& Juang 1986). known the classic Baum-Welch algorithm (Baum et al. 1970). Hofmann (1999) apply the EM algorithm for probabilistic latent semantic analysis (pLSA), where the topic is an unobserved variable. In the neural regime, the most related work to ours is Shen et al. (2019), as they apply Soft-EM and Hard-EM to diverse machine translation. We find it controversial whether diversity needed in translation; instead, we focus on dialogue generation, where diversity desired to avoid generic responses (Li et al., 2016a b). Moreover, we observe neither their Soft-EM nor Hard-EM works well, and we propose a theoretically justified EqHHard-EM variant.
Another related (but different) research topic is the mixture of experts (MoE, Jacobs et al. 1991, Hampshire & Waibel| 1992, Shazeer et al., 2017). where an expert k predicts real-valued feature vector Cg. mixed by predicted probabilities p Σx PACK. The main difference is that mix decoders' probabilities rather than features: c( = Σx P(z = & P(rlz = = k, c). However, uniform prior of P(=|c) in Sec-
do not directly optimize the marginal probability because tion makes the problem degrade to the optimization of P(riz = ki c) individually for different ks. Admittedyy might train model to learn the mixing probability P(= k/c), predicting which decoder is responsible for a context c. This, however, violates our one-to-many assumption and in- deed results in performance degradation as shown in Appendix F By contrast, our EM formulation does not predict the decoder by merely the context c, but determines which decoder fits given training sample comprising both the context e and response r.
CONCLUSION
In this work, we propose an Equal-size Hard Expectaionn-Mxximzaatoi (EqHard-EM) algorithm for multi-decoder diverse dialogue generation. Our EqHard-EM adopts the hard assignment as in Hard- EM, but additionally imposes an equal- assignment constraint ensure all decoders are well-trained. We provide detailed theoretical analysis to justify our algorithm. Extensive experiments on two large-scale datasets further confirm that our EqHard-FNN algorithm enables decoders to specialize and generate high-quality and diverse responses.
Limitation. Our paper addresses the generic responses for dialogue systems by training a mixture of decoders. This inevitably slows down the training process similar to standard EM algorithms (Shen)
et al. 2019), because we need to perform inference for all decoders even though our EqHard-EN propagates gradient to one. Despite this, find that our Hungarian algorithm has little overhead for the training process, and our approach is more efficient at inference compared with beam search, because multiple decoders be parallelized across different GPUs.

---

Published as a conference paper at ICLR 2023
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units (Rel.U).
arXiv preprint
1803.088330 2018. |tttts:::///axx www.aawwwwww...//aaaaaaaa..aaeee ht////bs/11/00 08375
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of mono-
lingual representations.
In ACL, pp. 4623-4637, 2020. URL
org/2020. ac1.020.cc1-a0 mmain.mmin.... 421
Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. for sequence-to-ssquenee models. In COLING, pp. 1672-1682, 2018. URL https://
Variational attention
2alamtaaloory.oog/011-1111
Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. PLATO: Pre-trained dialogue generation model with discrete latent variable. In ACL, pp. 85-96, 2020. URL https:
/acllancoolooy..crr2222000.0
Leonard Baum, Ted Petrie, George Soules, and Norman Weiss. A maximization technique oc- curring in the statistical analysis of probabilistic functions of Markov chains. The Annals of Mathematical Statistics, 11(1)1144-171, 1970. URL http::/www..... http::/wwww..torrrosssstttaaaaae
22399727
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. In CoNLL, pp. 10-21, 2016. URL https:
//aclanthology.orr//////// 1002
Wei Chen, Yeyun Gong, Song Wang, Bolun Yao, Weizhen Qi, Zhongyu Wei, Xiaowu Hu, Bartuer Zhou, Yi Mao, Weizhu Chen, Biao Cheng. and Nan Duan. Dialog pre-trained latent variable encoder-decoder model for dialog response generation. In ACL. pp. 4852-48644 2022. URL URRLitttps::/aaaaathhoo.co
Peter Dayan and Geoffrey E. Hinton. Using Dnpectaation-Muximizziinn for Reinforcement Learn ing. Neural Computation, 9(2):271-278, 1997. URL httpp:s///oi...r//10.1111111
1997.9.2.2 9.2.771
Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from in- complete data via the EM algorithm. Journal of the Royal Statistical Society, 39(1):1- 22, 1977. URL https::/p///. rss oolonniieenee. www.onnnneeaarllnllleeeeaaaaaaaa wi ley. com/ dotl//oo////aa /bb// 10. 1111/j. 2517- 1 197.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186, 2019.
Shaoxiong Feng, Hongshen Chen, Kan Li, and Dawei Yin. Posterior-GAN, Towards informa- tive and coherent response generation with posterior generative adversarial network. In AAAI, pp. 7778-7715, 2020. URL htttps://ojs.. aaai.aaaiia........ org/ wnw/x/ndxxxxcm/// pttppppt//attic.......a
view/ 6273
Jun Gao, Wei Bi, Xiaojiang Liu, Junhui Li, and Shuming Shi. Generating multiple diverse responses for short-text conversation. In AAAI, pp. 6383-6390, 2019. URL httttt/ss////s.... aaai /
index.
Xiaodong Gu, Kyunghyun Cho. Jung-Woo Ha, and Sunghun Kim. DialogWAE: Multimodal re-
sponse generation with conditional Wasserstein auto-encoder. In ICLR, 2019. URL https:
net. http://wwwwweeeeettrrreeeaaaaaaa
Xiaodong Gu, Kang Min Yoo, and Jung-Woo Woo Ha. DiaTogBERT: Discourse-aware response gen- cration via learning to recover and rank utterances. In AAAI, pp. 12911-12919, 2021. ÜRL https://ojs aaai. org/ Indx.. wwtpppppp//pecppppccccpppppppppp AAAI wwt.a..eeeeeew/eeeeeeeee
John B. Hampshire and Alex Waibel. The Meta-Pi network: Building distributed knowledge rep- resentations for robust multisource pattern recognition. TPAMI, 14(7):751-76, 1992. URL
https::/ieeexplore ieee. caaeeeorr//occcc orgg//ccumet//12211
10

---

Published as a conference paper at ICLR 2023
Thomas Hofmann. Learning the similarity of documents: An Croomatiin-geommeee approach to document retrieval and categorization. In NeurIPS, pp. 925- 1999.
URL
cC wttps/eppppppee.e/..ee...
(11662367c3935dee ccbl wwt.c:/9e2277770000ppepeeee
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. ICLR, 2020. URL/nttps://ppeeevveen http://rwweeeeeeeee/ddd/o///oo/o
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-cfficient transfer learning for NLP. In ICML, pp. 2790-2799, 2019. URL http:://poocediinnnss.....eee//2
houlsby19as hal.sby19ahhtm..
Robert A Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991. URL https::/doo. /ttp:////oo...//.1111111.1
neco 1991. 1..1..1..71
R. Jonker and A. Volgenant. A shortest augmenting path algorithm for dense and sparse linear assignment problems. Computing, pp. 325-340, 1987. URL /RLntttss///0d..o//111.0000
BFC2278710
Kashif Khan, Gaurav Sahu, Vikash Balasubramanian, Lili Mou, and Olga Vechtomova. Adversarial learning the latent space for diverse dialog generation. In COLING, pp. 5026-5034, 2020.
URL https: /tc//aaaaaaa www.cnnnholggy...oo org wttp:/2220...0.oog main. 441
Harold W Kuhn. The Hungarian method for the assignment problem. Naval Research Logistics
Quarterly, 2(1-2):83 97, 1955. URL /ouL:tttoss//nnnneelllrrrry /o/ com/do1/1C.
1002/nav: nav. 3800020109,
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng. Xiujun Li, Yizhe Zhang, and Jianfeng Gao. Opti- mus: Organizing sentences via pre-trained modeling of a latent space. EMNLP, pp. pp. 4678-4699, 2020.
http://22222..emm...mmmmmmm.emm emnlpmmmmmm 378
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promottin objective function for neural conversation models. In NAACL-HLT, pp. 110-119, 2016a. URL
https://aclantho.l org. Maig/N16-10100 1001111101000
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. In EMNLP, pp. 1192-1202, 2016b. URL.
https::/aclantholoog www.o//-rg/01111100101
Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, and Dan Jurafsky. Adversarial learning for neural dialogue generation. In EMNLP, pp. 2157-2169, 2017a. URL https:
paclanthology
Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, and Hong Liu. Expectation-
Maximization attention networks for semantic segmentation. In ICCV, pp. 9167-9176, thevvf com content TCCV_ 2atp/_209hhmmmm Att.://mmtttooottttttti.o. Attention Secoon_eeetoss_eeeeeeeoeeo_e for Sates__foo__Sseeeeeee Secmentation ICCV 2019 paper..... .hmll
2019. URL htttps:://ppeacceeett
Expectation www.-Maaaiiiiiaiii..mm..o
Yanran Li. Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A man-
ually labelled multi-turn dialogue dataset. In IJCNPP, pp. 986-995, 2017b. URL https:
7/acl wtt.llannoooooy.oo//ooooo
ogy. wtt.
Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In LREC, pp. 1742-1748, 2018. URL
https::/eclaani /nthh logy. owg.1orrg/113 112.11111117..011
1275.
Chia- Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.
How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation
metrics for dialogue response generation. In EMNLP, pp. 2122-2132, 2016. URL hetps:

---

Published as a conference paper at ICLR 2023
Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. Introduction Informa- tion Retrieval. Cambridge University Press, 2008. URL https:://h....... http:::///aaac.o..////////o www../ac.../..///....//
5555/1394399
Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. Sequence backward and forward sequences: A content-ing inttouuiintoouuing approach to generative short-text conversation. In COLING, pp. 3349-3358, 2016. URL
Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, Yusuke Oda, and Sadao Kurohashi. Overview the 8th Work- shop Asian Translation. In Proceedings of the Workshop on Asian Translation, pp. 1-45, 2021. URL https: http:///aaaaatthoooooo
org/ 2021 wat-
Nghia Ngo Trung, Duy Phung, and Thien Huu Nguyen. Unsupervised domain adaptation for event detection using domain--seeeiin adapters. In ACL-IJCNLP Findings, pp. 4015 4025, 2021. URL
httpss//acclanthooooyyyooo wttp:/thhwooooooog///eoeo/ee wttp wttp:////..........oooooooo//// wwwdinggsscc..n3333
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. OpenAl Blog, 2022. URL https::/ htt.alog.cc//o///////oo///// instruction following/)
Lawrence Rabiner and Bing-HWang Juang. An introduction to hidden Markov models. IEEE ASSP Magazine, 3(1):4-16, 1986. URL htttss:/ ieee. oee...rgg/bouumnntt
1165342
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter Liu, et al. Exploring the limits transfer learning with a unified text-to- text Transformer. JMLR, 21(140);1-67, 2020. URL
20-074 htm1
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal pol- icy optimization algorithms. OpenAl Blog, 2017. URL hnttps://openai cwm/blog////ooo
openai praseliness ppoo
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely gated mixture-of-experts layer. In ICLR, 2017. URL /RR.httpss///peeevvieee net/ www.mntttfutaatoooooooo wtt.:ffrrrmmiddlllll0a. ckcDDlgg
Tianxiao Shen, Myle Out, Michael Auli, and Marc'Aurehe Ranzato. Mixture models for diverse machine translation: Tricks of the trade. ICML, pp. 5719-5728, 2019. URL https:
proceedings.co www...r.....m.m............ press http////////////////// 9c. htmml
Christian D. Sigg and Joachim M. Buhmann. Expectatoon-maxiiizaiioo for sparse and nen-negative PCA. In ICML, pp. 960-967, 2008. 1390277
URL
Koustuv Sinha, Prasanna Parthasarathi, Jasmine Wang, Ryan Lowe, William I.. Hamilton, and Joelle Pincau. Learning unreferenced metric for online dialogue evaluation. ACL, pp. 2430-2441, 2020. //555/aaa..ccoo...o.oo/////////r
main. 220
Chongyang Tao, Lili Mou, Dongyan Zhao, and Rui Yan. RUBER: An unsupervised method for automatic evaluation of open-domain dialog systems. In AMAI, pp. 722-729, 2018. URL https:
wwojs.aaii.o.. www.aaaa.....n.n....o....a http:e/ppppppaaaaaaiiaic.c.ce
Ashwin K Vijayakumar, Michael Cogswell, Ramprasath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. Diverse beam search: Decoding diverse solutions from neural se- quence models. arXiv preprint arXiv: 1610.0244 2016. URL.
Oriol Vinyals and Quoc Le, A neural conversational model. arXiv preprint wwwpxiiv.orgaaaaooo abs/ URL pttttps://ariiin http:p::///aaat..oog////////.../ 05869
1660. 02424
2015.
12

---

Published as a conference paper at ICLR 2023
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. In ACL-IJCNLP Findings, pp. 1405-1418, 2021a. URL URLattpsss/aclaathhoolo org/2021. findi agg.aal................o......c. 121
Yida Wang, Yinhe Zheng, Yong Jiang, and Minlie Huang. Diversifying dialog generation via adaptive label smoothing. In ACI-IJCNLP, pp. 3507-3520, 2021b. URL https://
aclanthology. wttp///gyy..gg/222111o
Bolin Wei, Shuai Lu, Lili Mou, Hao Zhou, Pascal Poupart, Ge Li, and Zhi Jin. Why do neural dialog systems generate short and meaningless replies? comparison between dialog and translation. In ICASSP, pp. 7290-7294, 2019. URL httttps://leexpporeee leee. wtt.e/eeee....eo.eee.eeeeee documenct
8682634
Yuqiao Wen, Guoqing Luo, and Lili Mou. An empirical study on the overlapping problem of open- domain dialogue datasets. LREC, pp. 146-153, 2022. URL http: /www//wwww..... 1rec conf.
http/p/rceednnn//rre//////////te 1rec
16. 16.pff
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's Transformers: Sfate-of-the-att natural language processing. arXiv preprint arXiv:1910.03771, 2019. URL
https://aze /ttp///aaivvvv//.a///i/ wttp//ab///gi/o/// 0370.
Sixing Wu, Ying Li, Dawei Zhang, Yang Zhou, and Zhonghai Wu. Diverse and informative dialogue generation with context-specific commonsense knowledge awareness. In ACL, pp. 581 1-5220, 2020.
aww.://///22.....maaaa....... 55.55
Sixing Wu, Ying Li, Minghui Wang, Dawei Zhang, Yang Zhou, and Zhonghai Wu. More is better: Enhancing open-domain dialogue generation via multi-source heterogencous knowl- edge. In EMNLP, pp. 2286-2300, 2021. URL httppsss/acc.attooooooooooo/////
emr 1 ma in. 175
Lili Yao, Yaoyuan Zhang, Yansong Feng, Dongyan Zhao, and Rui Yan. Towards implicit content- introducing for generative short-text conversation systems. In EMNLP, pp. 2190-2199, 2017. URL
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. ACL, pp. 654-664, 2017. URL
Hao Zhou, Tom Young. Minlie Huang. Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Common- sense knowledge aware conversation generation with graph attention. In IJCAI, pp. 4623 4629, 2018. httpg/////////////aaaaaaaaooooor
Yicheng Zou, Zhihua Liu, Xingwu Hu, and Qi Zhang. Thinking clearly, talking fast: Concept-guided non-autoregressive generation for open-domain dialogue systems. In EMNLP, pp. 2215-2226, 2021. URL URL.UURLLttaaaa httphhss//aclaattllllooooo wwg.:yyyrrg2221111 emnlp-main.
A PROOF OF THEOREM
We introduce two lemmas for proving Theorem The first lemma controls the error between the empirical average and the true expectation. It is direct application of Hoeffding's inequality. The second lemma bounds the difference between the learned probability and the true underlying probability. Following the notation in Eqn. (2). we use Q(2lc, r) as the estimated posterior.
Lemma 1. Given a dataset 'D consisting of context-response pairs (c, r), for any z, and > 0, we have
Pr
Σ
Q(=lc, Ber (()lc, 2() 2xxp (-262(D)
D
(c,r)D
13

---

Published as a conference paper at ICLR 2023
Proof. Observe that Q(2(c, r) is real-valued random variable. The proof is then clear by using Hoeffding'ss s inequality.
Lemma is a quantitative description of the law of large numbers.
Lemma 2. Given any and estimation Q=2|c,r) of the true P(zle,r), we have
Ee, 1 K - ((alc..)-P(ale..r)))
Ee.rQQ (2][,,))
Proof.
1
(((c.r 1 K
Be.r Be.r(((=(e,r))
(1I)
K
E ((x.))))(or)) - -E P(el.)
(12)
E (((,)))(()c.)))-
(13)
Σ P(c,r) -(()c,r))
(14)
-Fc. [Q(2lc,r) - P((lc,)))
(15)
Here, Eqn. (12 replaces 1/K with the marginalization of P(2, over and r, and (14) follows the triangle inquualtty
Lemma bounds the absolute difference between the estimation Q==(c,r) and the true P()
Theorem 1, With probability at least
8, we have
Σ
c))
S
log
2K 8
+ Eer [[(()c,)) - p(=|c,))
D
K
2
(r.c)ED
for all E (1,..., KK. Here, 122c,,r) is an estimated posterior and D is a dataset with finite samples.
Proof. Lemma shows an inequality that upper bounds the deviation from uniform assignment
for particular z. The probability for the original inequality to hold for all is then:
Pr Vz E [K]:
Q(=le, (c.(((()(c.r)) 2 26) < 2K exp (-2e? (c.r)ED
Σ
D
Choose such that 5 = 2K exp (-2e21D)). We have
2K
Pr
Vzz
|
Σ
Q(=lc,
Ee, -E, (((e,,))))))) 2
log
< 8
D
2|D
(c.r)ED
14

---

Published as a conference paper at ICLR 2023
Then with probability least 1 8, we have:
1
Σ
Q(zle,r)
D
(e,r)ED
Q(Ele,r) (e.r(Q(([(c,r]]] + [er,((
D
(r.o)ED
2K
+ [ex(0(2/c,r)] 1
log
(w.p. 1 8, by Lemma
2
[K
+ Ee,r [[(=|c,)) P(z
log
(by Lemma
2|D]
8
concluding the proof.
B IMPLEMENTATION DETAILS
Our multi- adati- model used T5-small (Raffel et al., 2020) from HuggingFace (Wolf et 2019) as the backbone: 6 layers, 8 attention heads, an input/output dimension of 512, and an inner di- mension of 2048 for both the encoder and decoder. We implemented our multi-decoder model by inserting an adapter layer with an inner dimension of 256 after each Transformer decoder layer. We adopted 10 decoders based on the common evaluation metric of diverse text generation (Zhao et al.) 2017 Gu al. 2019). Table 3 shows comparison between our multi-adapter architecture and the full Transformer architecture. As seen, our multi-adapter model retains a competitive BLEU per- formance while only using 24% of the parameters. We adopted the multi-adapter architecture in the main experiments, because it helped our development process much.
For training, had stages: Pretraining of the shared Transformer model, and 2) EqHard- EM training of the adapter layers. In both stages, we used a learning rate 0.001 with the Adam optimizer with 3 (0.9, 0.999). We used batch size of 64 for pretraining. For our 10-decoder EqHard-EM training, we considered 640 samples in an E-step, assigning 64 samples for each de- coder.
C EFFECT OF THE NUMBER OF DECODERS
We investigate how well our approach scales with the number decoders. Figure a compar- ison between beam search and multi-decoder model under 5-, 10-, and 15-generation settings. Due to the limited resources, we conducted the analysis only on the Weibo dataset. As seen, our multi-decoder model performs similarly beam search when there only 5 decoders. However, the performance gap increases when we have more decoders, which verifies the need for our multi- decoder model.
D HUMAN EVALUATION
We conducted human evaluation to compare both the quality and diversity of the responses from T5-small (beam search) and our multi-decoder model. We again chose the Weibo dataset, because is less noisy. Five native Chinese speakers were provided with a dialogue context and generated responses of two systems, and were asked to select the better system indicate tie. For quality
Decoder Architecture
#Parameters
BLEU2-
Adapter
108M
10.13
Full Transformer
451M
11.08
Table Comparison between adapter and full- Transformor architectures for our 10-decoder model.
15

---

Published as a conference paper at ICLR 2023
GZ2
Beam Search
EqHard-EN EM
10.13
11.02
10
BLEU2-F
8.51
7.4
7.42
0
5
10
15
Beam Size Number of Decoders
Figure 4: Comparison between beam search and our multi-decoder model in 5-, 10-, and 15- generation settings on the Weibo dataset.
Model
Wins
Ties
Losses
p-value
T5-small search
32.8%
19.0%
48.2%
Overall Quality
<000001
Multi- Multi-adapter EqHard-EM
48.2%
19.0%
32.8%
T5-small beam search
4.8%
8.0%
87.2%
Diversity
<000001
Multi-adapter + EqHard EM
87.2%
8.0%
4.8%
Table 4: Human evaluation for the overall quality and diversity of the generated responses. Five annotators were invited to evaluate 100 samples on the Weibo dataset. The p-value is given by the one-sided binomial test (ignoring ties).
evaluation, annotators were presented with only one response from each model, preventing them from identifying the systems based diversity. For diversity, all 10 responses were presented. In both cases, the presentation order of the two systems was randomly shuffled, so the annotators could not tell which system generated which response.
Table shows the results of human evaluation. Our multi-decoder model shows statistically sig- nificant improvement in terms of overall quality and diversity. Especially, annotators believe approach generates more diverse utterances on 18x more samples. The results are highly consistent with the automatic metries in Table confirming the effectiveness of our approach.
E
CASE STUDY
Table 5 shows example responses from Weibo and OpenSubtitles. As seen, Soft-EM exhibits the synchronous-training collapse, where all decoders generate highly similar responses. Hard-EM, on the other hand, suffers from the non non-training collapse, where large number of decoders are not properly trained and generate low-quality text (e.g., certain words being repeated). By contrast, our EqHard-EM algorithm enables properly train all decoders to generate high-quality, diverse responses.
F ANALYSIS OF ADDITIONAL EM VARIANTS
Shen et al. 2019) also realize the collapse issues of classic EM algorithms; however, they explain the poor performance Soft-EM by drawing an analogy to the bypassing of latent variables in variational auto-encoders (VAE, Bowmann et al. 2016). Based on such interpretation, they propose no-dropout trick that disables the dropout layers for the E-step while retaining them for the M-step.
We analyze the no-dropout trick in Table 6 As seen, disabling dropout has minimum effect on the EM algorithms, and does not prevent collapses for Hard-EM and Soft-EM. This is not surprising because even (hen al, (2019) acknowledge that their no-dropout trick does not work consistently, as shown Table of Appendix C in their paper (red and blue numbers). The one-to-many phe- nomenon in dialogue setting is more severe than their machine translation setting, and it is understandable that such an ad hoc trick, without much theoretical justification, may fail in the more difficult dialogue task.
We also analyze EM variants with learned priors, where a neural network used to predict the de- coder given the dialogue context. This is another variant in Shen al. 2019). As shown, learned
16

---

Published as a conference paper at ICLR 2023
Weibo Input: 86 1 IIMM, 2E 2 Finally got iPhone, found useless, than ipad
Soft-EM
Hard-EM
EgHard-EM
3 T. N H
[repcrreepaa]
1 TEt
have one tou, but not big
river river river [repeat]
What brand is this?
P2tt -r 11
[cos] [eas]
RE - TE
have one too, but not so big
want to buy one to0, too bad I don't 't have money
Bi -1. -. A2 Fi X
T
have too, but not big
treasure treasure treasure [repeat]
Me
worse than iPad indeed
[cos]
$2 T. -
have one too, but not so big
[eas]
want to buy one too, too bad I don't 't have money
P AA, Ff 26
[cos]
1 A.A. A- BRt
have too, but not big
[eax]
This stuff good,
buy one too
2. have one too, but not so big
Htt i -
RE -r UiidiFFiipa
I have one too, but not so big
have one too, but not the ipad
f T, 12 2
13 (repeat)
(A1 IA? 12.11A3.7 T. -
have one too, but not big
think think think [repeat]
Where are you want buy one too
i8 BE/EE 22 SKA What is this? Please expllain)
# th (repeat]
Fith --1 T.
[1
reed reed reed [repeat]
want to buy one too, but don't have money
Bu -, have one too, but not so big
Fi
1 2 (repeat]
REE T T-A, Aisstid-- HR fit 1- bought one too, but it actually pretty good
snow snow snow [repeat]
12 1 1e--t 2.7 SRARS
[repeat]
What is this? Please explain
rrepeut]
This is not bad, but like
OpenSubsitles Input: and you know what happens if we object?
Soft-EM i'm not gonna object.
Hard-EM
EyHard-EM
what what what [repeat]]
don't know.
do.
victim.
what do you mean?
don't know.
and and and [repeat]
if we object, 'Il be able
don't know.
really really really [repeat] no pe pe pe [repeat]
don't know, don't know.
do.
don't know.
don't know.
gonna have to do this.
i'm not gonna be ahle anything. we're gonna
don't know.
okay okay okay [repeat] hurry hurry hurry [repeat]
don't know.
what?
do.
what?
don't
don't know.
then then then [repeat]
i'm not gonna able to this, mr. santos.
Table 5: Case studies. Translations are in italic. [repeat] indicates that further repetitions are omitted, and [eos] indicates an end-of-sentend token generated as the first token.
priors perform worse than uniform priors due to the more severe collapse problem, which is demon- strated by the extremely low BLEU scores. This is because learned prior gives the model more opportunities to select the best decoder, further worsening the siih-gets--icherr effect.
Therefore, do not consider the recurrent dropout trick and learned priors in our main experiments due to their ineffectiveness and poor performance.
G EFFICIENCY ANALYSIS
We evaluated training and inference efficiency on an 19-9900X CPU and TITAN RTX GPU, Com- pared with classic EM variants, our EqHard-EM involves an additional assignment step using the Hungarian algorithm. As shown in Table 7 the E-step, Hungarian algorithm, and M-step take 75.69%, 0.97%, and 24.31% % the total training time, respectively. This suggests that our EqHard- EM incurs negligible overhead compared with soft and hard EM variants. At inference, our multi- adapter model processes 79.1 samples per second, whereas beam search processes 16.9 samples per second. Our approach achieves 4.7x speedup because inference for each decoder can be computed in parallel across multiple GPUs, whereas beam search cannot leverage the same level of parallelism.
17

---

Published as a conference paper at ICLR 2023
BLEUT
BLEU2
Distinct n-gram
Pairwise
Dataset
Model
F
Dist-1
Dist-2
BLEU
unform prior
17.02
27.55
12.32
6.27
16.03
3.89
10.69
12.45
86.25
Soft-EM
uniforn prior, no dropout
17.13
27.53
12.43
6,30
16.02
3.92
10.88
12.77
84.77
leamndd price
10.33
8.63
12.86
3.18
2.76
3.74
27.89
54.53
1.09
learned prior. no dropout
13.58
12.95
14.27
4,50
4.66
4.35
20.56
38.31
4.02
Weibo
prior
3.09
3.22
0.00
Hard-EM
uniform prior, dropout
5.17
3.39
10,86
2.25
1.71
3.27
3.51
3.60
2.22
learned prior
5.07
3,30
10.92
2.14
1.61
3.20
5.38
learned prior, dropout
4.83
3.11
1.78
3.31
9.65
10.02
0.02
EqHard-EM
22,65
26.17
19.96
10.13
14.55
7.77
27.03
42.02
22.32
uniform prior
15:48
14.55
16.34
3.03
2.76
3.35
12.89
15.35
72,95
SoftEEM
uniform prior, no dropout
15.52
14.51
16.67
3.00
2.71
3.37
13.23
15.93
70.29
learned prior
15.45
16.50
2.76
13.01
15.45
71.86
learned prior, no dropout
15.47
14.48
16,60
2.95
2.71
3.25
13,40
16.17
67.46
Open- Subtitles
uniform prior
6.84
4.13
19.81
0.73
4.30
4.81
1,50
Hard-EM
uniform prior, dropout
9.20
5.66
24.54
1.36
0.88
6.82
9.87
10.60
0.61
learned prior
4.71
2.73
17.11
0.81
0.46
3.31
6.82
5.57
0.23
learned price, dropout
2.67
EqHard-EM
17.86
12.98
28.60
3.76
2.36
9.27
34.09
50.41
13.15
Table Comparison with additional EM variants on Weibo and OpenSubtitles.
Training Time
E-Step
M-Step
Hungarian
75.69%
24.31%
0.97%
Inference Time
Beam search
16.9 samples per second
Ours
79.1 samples per second
Table 7: Training and inference efficiency analysis. The top half shows the percentage of training time spent in the E-step. M-step, and the Hungarian algorithm (part the E-step). The bottom half compares the inference efficiency of our EqHard-EM algorithm against beam search.
ACKNOWLEDGMENTS
We would like to thank all reviewers and chairs for their comments. We would also like to thank Nidhi Hegde and Herbert Yang for their valuable suggestions. This research was supported in part by Natural Sciences and Engineering Research Council of Canada (NSERC) under Grant No. RGPIN2020-04465, the Amii Fellow Program, the Canada CIFAR AI Chair Program, the Dig- ital Research Alliance of Canada (alliancecan.ca), and a Mitacs Accelerate (Cluster) grant.
18


Reviews:
[{"review_id": "d8Ro5UXtWK", "reviewer": "Reviewer_3nYS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the issue of generic response generation by first using adapters for efficient multi-decoder frameworks, and next using a balanced EM algorithm to train the EM of the multi-decoders. The main contribution is to train the balanced EM by the classic Hungarian algorithm. Experiments compare the proposed model with other methods that address the same problem.\n", "summary_of_the_review": "Overall, I think the idea is reasonable but the experiments are a bit weak.", "strength_and_weaknesses": "Strengths: The idea is clear and the implementation of the idea should be easy.\n\nWeakness:\nThe authors argue that the E-step in the proposed model contributes to less than 1% of the total training time. I want to know if the full experimental results can be provided, including the total training time, the number of EM steps, and the exact time of the E-step and M-step in the training time. Also, the time efficiency should be compared for all methods in your experiments.\n\nThe experimental results are not convincing.  First, only results on BLEU-2 of BLEU-n are given. How about BLEU-1? Also, human evaluation results of Table1 should be given.  Also, I think a pretrained T5 should not be proper to compare here, as it is not for dialog generation. Perhaps DialogGPT is better if the authors want to compare a pre-trained model. However, I found most of the compared methods except the proposed one perform even worse than T5, which is hard to believe.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VGABuRC_C7", "reviewer": "Reviewer_wvrp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In order to solve the issue of generic responses generated by the neural dialogue systems because of one-to-many mapping phenomenon, the authors of the paper studied the state-of-the-art methods including the methods for training time and reference time, the application of the standard EM, Soft-EM, Hard EM and etc. In the paper, the analysis of the disadvantages of these methods was given and a new method EqHard-EM algorithm was proposed. \nIn EqHard-EM, this paper proposed a multi-adaptor architecture with multi-decoder and a shared encoder, adopted hard assignments from Hard-EM but impose equal-assignment constraints to ensure all decoders are adequately trained. To evaluate the new method, the paper included the experiments on Weibo and Open Subtitles. The results and the comparison with the state-of-the-are methods and EM variants have been given in the paper. In the results, the EqHard-EM demonstrated its advantages again EM variants and a few state-of-the-art methods. ", "summary_of_the_review": "The method in the paper is new and developed based on a good study of related method and innovations/extensions beyond them. It justifies the one-to-many mapping phenomenon and the results of the two experiments demonstrates advantages comparing to others. \nThis paper will benefit the readers with novel approaches. ", "strength_and_weaknesses": "Strength: \nThe study of the literature for the solutions to justify the  one-to-many mapping phenomenon was complete. The paper also analyzed the issues coming from standard EM, Soft-EM and Hard-EM. \nIn the proposed new method, new neural architecture based on multi-adaptor architecture was created on top of and beyond a few state-of-the-art methods. Training methods and the theoretical analysis was given. \nWeakness or questions: \nIt was not that clear how the Theorem 1 came up. Can the author elaborate more details of it? \nThe organization of the paper can be improved. It maybe better to have related work section (Section 4) come earlier because now there are duplications among Section 1 (introduction), Section 2.2 and Section 4. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Oad8QIGg_1F", "reviewer": "Reviewer_ov7h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores mixture models for diverse dialogue generation, following the same pipeline proposed by Shen et al., (2019). This method adopts a multi-decoder model to address the one-to-many mapping phenomenon, where a multi-adapter architecture is used to form the multi-decoder model. Based on that, the EM algorithm (soft or hard) is leveraged to learn the latent variable (i.e.,  the choice of the decoder). The authors further handle the non-training collapse of the Hard-EM algorithm by introducing equal-assignment constraints, which ensure that all decoders are adequately trained. Automatic evaluations on Weibo and OpenSubtitles datasets prove the effectiveness of introduced equal-assignment constraints, compared with the original EM algorithm.", "summary_of_the_review": "This paper investigates introducing mixture models for diverse dialogue generation and proposes EqHard-EM to handle the non-training collapse of the Hard-EM algorithm. But the experiments are not convincing enough to me, including lacking more details of all model variants in Shen et al., (2019) and full human evaluation. The E-step of Hard-EM in this paper is also different from Shen et al., (2019), and it is uncertain whether this method will be further improved when combined with the version of Shen et al., (2019).", "strength_and_weaknesses": "Strengths:\n- This paper explores mixture models for dialogue generation tasks. \n- This paper proposes EqHard-EM to handle the non-training collapse of the Hard-EM algorithm.\n\nWeaknesses:\n- It is an incremental work of Shen et al., (2019) and the novelty of the proposed method is limited. \n- It is fine to apply and extend the method of Shen et al., (2019) in dialogue generation tasks, but the experiments are not good enough, including implementing all model variants in Shen et al., (2019) and full human evaluation. \n- The main contribution of this paper is to introduce equal-assignment constraints to handle the non-training collapse of the Hard-EM algorithm. However, the E-step of Hard-EM in this paper is different from Shen et al., (2019), which leverages learned prior (lp) or uniform prior (up) to ease this issue. It is difficult to judge that this method could lead to further improvements compared with Shen et al., (2019)'s version of Hard-EM.\n\n\nDetailed Comments:\n\nOverall, this paper is easy to follow and this idea is straightforward. I like this idea of introducing mixture models for diverse dialogue generation. But the experiments are not convincing enough to me. The E-step of Hard-EM in this paper is different from Shen et al., (2019), which leverages learned prior (lp) or uniform prior (up) to ease the non-training collapse of the Hard-EM algorithm. It is not sure that this method could lead to further improvements when combined with the version of the Hard-EM algorithm of Shen et al., (2019). It is necessary to reproduce the version of Shen et al., (2019) in the dialogue generation task.\n\nWhile Shen et al., (2019) have explored four model variants and some crucial tricks (e.g., the effect of dropout) for diverse machine translation, it is better to revisit these models and details in dialogue generation. In addition, this paper lacks full human evaluations across two datasets and different methods, and automatic evaluations are not convincing enough in dialogue generation tasks. The authors only perform human evaluations on beam search and the proposed method, which should be improved.\n\n\nQuestions for the Author(s):\n- As shown in Table 5, the example of the Hard-EM algorithm tends to repeat. Since we directly optimize the conditional language model in M-step, these examples are weird. What is your thought about it?\n- Why is the E-step of the Hard-EM algorithm in this paper different from Shen et al., (2019)? ( i.e.,  argmax_{z} p (z| x,y; \\theta) v.s.  argmax_{z} p (y| z,y; \\theta)  )\n- Have you evaluated the effect of the dropout during the EM algorithm? \n- How about training different decoders instead of introducing adapter layers?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LH84s3zANF", "reviewer": "Reviewer_EYAz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed the use of Equal size hard EM algorithm to address the problem of diverse response generation in open domain dialog modeling. The paper modified the EM algorithm to overcome mode collapse issues with hard EM and synchronous training collapse issues with soft EM. Experimental results show improvement in diverse response generation on Weibo and Open subtitle datasets. Ablation studies also show that Equal size hard EM performs better than beam search, soft EM and pure hard EM. Qualitative analysis also shows that some of the modes learned unique attributes of the dialog data such as response length, question mark, and like/love. The paper also shows that the performance improves with increasing number of decoders, although the incremental improvement reduces. Overall, the paper is clearly written and well motivated.", "summary_of_the_review": "The proposed method of Equal-size hard EM is novel and well motivated and supported by experimental results. However, the paper needs to add other baselines to really evaluate the diverse response generation capability of the proposed approach.", "strength_and_weaknesses": "Strength:\nThe paper is clearly written and easy to follow. There are couple of minor typos to be fixed. The proposed method is novel and experimental results is thorough. The ablation study is also informative. \n\nWeakness:\nThe use of mixture of experts for dialog modeling is not new, but the implementation using adaptor is new. The evaluation did not include existing baseline that have used GAN or reinforcement learning to accomplish diverse response generation. Also, it is not clear if encode-decoder architecture is suitable for dialog response generation as the conditional probability modeling P(y/x) exacerbate the one-to-many problem. Would be useful if authors could compare the proposed method with finetuning vanilla GPT-2 modeling P(x,y) with the evaluated dataset as explored in [1].\n\n[1] https://arxiv.org/pdf/1908.01841.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster