Learning Distinct and Representative Modes for Image Captioning
Authors: Qi Chen
Chaorui Deng
Qi Wu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image Captioning
Discrete Mode Learning

Abstract: Over the years, state-of-the-art (SoTA) image captioning methods have achieved promising results on some evaluation metrics (e.g., CIDEr). However, recent findings show that the captions generated by these methods tend to be biased toward the "average" caption that only captures the most general mode (a.k.a, language pattern) in the training corpus, i.e., the so-called mode collapse problem. Affected by it, the generated captions are limited in diversity and usually less informative than natural image descriptions made by humans. In this paper, we seek to avoid this problem by proposing a Discrete Mode Learning (DML) paradigm for image captioning. Our innovative idea is to explore the rich modes in the training caption corpus to learn a set of "mode embeddings", and further use them to control the mode of the generated captions for existing image captioning models. Specifically, the proposed DML optimizes a dual architecture that consists of an image-conditioned discrete variational autoencoder (CdVAE) branch and a mode-conditioned image captioning (MIC) branch. The CdVAE branch maps each image caption to one of the mode embeddings stored in a learned codebook, and is trained with a pure non-autoregressive generation objective to make the modes distinct and representative. The MIC branch can be simply modified from an existing image captioning model, where the mode embedding is added to the original word embeddings as the control signal. In the experiments, we apply the proposed DML to two widely used image captioning models, Transformer and AoANet. The results show that the learned mode embedding successfully facilitates these models to generate high-quality image captions with different modes, further leading to better performance for both diversity and quality on the MS COCO dataset.

Learning Distinct and Representative Modes for Image Captioning
Qi Chen" Chaorui Deng" Qi Wu
Australian Institute for Machine Learning, University of Adelaide (qi. chen04, chaorui, deng, qi.wu01 /edeelidde eduu.e.dduc.
Abstract
Over the years, state-of-the-art (SoTA) image captioning methods have achieved promising results some evaluation metrics (e.g., CIDEr). However, recent findings show that the captions generated by these methods tend to be biased toward the "average" caption that only captures the most general mode (a.k.a, language pattern) in the training corpus, i.e,, the so-called mode collapse problem. Affected by it, the generated captions limited in diversity and usually less informative than natural image descriptions made by humans. In this paper, we seek avoid this problem by proposing Discrete Mode Learning (DML) paradigm for image captioning. Our innovative idea is to explore the rich modes in the training caption corpus learn set of "mode embeddings", and further them to control the mode of the generated captions for existing image captioning models. Specifically, the proposed DML optimizes a dual architecture that consists of an image-come congge--oditionedd discrete variational autoencoder (CdVAE) branch and mode- conditioned image captioning (MIC) branch. The CdVAE branch maps each image caption to one of the mode embeddings stored in learned codebook, and is trained with pure non-autoregressive generation objective make the modes distinct and representative. The MIC branch can be simply modified from an existing image captioning model, where the mode embedding is added to the original word embeddings the control signal. In the experiments, apply the proposed DML to two widely used image captioning models, Transformer and AoANet. The results show that the learned mode embedding successfully facilitates these models to generate high-quality image captions with different modes, further leading to better performance for both diversity and quality on the MSCOCO datase
1 Introduction
Image captioning aims to generate natural descriptions for a given image. It is widely used in many
real-world applications such as human-computer interaction, multi-modal recommendation, and hence
has attracted lots of research attention. Recently, many state-of-the-ar (SoTA) methods (20 26 57
have achieved promising results w.r.f. evaluation metries like CIDEr [47], BLEU [38], and SPICE M.
However, as discussed in [52], focusing on achieving higher scores on these metrics usually biases the image captioning models towards using only the common words, phrases and language patterns in the training corpus when describing the images (see Figure I for an example). In other words, the model automatically finds the most general mode perform captioning. As result, the generated captions are limited in diversity both semantically and syntactically. This is far away from the ability of human beings as humans are able to describe the image in various ways.
Cooee available wttpthtts////....t. / thub. cww/ttoooo//oooooooo hoombbl adewa 1 /ModeCap Authors contributed cqually. (Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Captions generated by humans:
A man holding white frishee while standing field.
man standing outside holding frishee in his hands.
man is outside holding onto frisbee disc.
smiling man in the park with frishee.
man is having good time playing frisbee.
Captions generated by existing models:
Transformer: man holding frisbee while standing in field.
Net: man holding frisbee field.
Captions generated by DML (Ours):
Mode-7: 7: There is man playing with a frishee.
Mode-32: Man in blue shirt throwing white frishee.
Mode-43: A close up of person with a frishee.
Mode-3: 3: A young man bolding white frisbee in his right hand.
Mode- 58: A man about throw frishee.
Figure 1: Captions written by humans vs. captions generated by existing models (Transformer [46 and AoANet (20)). and by our DML. The model-generated captions prefer to use common words or phrases while captions derived from humans are more informative and diverse. The proposed DML generates diverse image captions based on different mode embeddings, and some of the modes tend to yield certain language patterns. E.g., mode-7 is likely to generate captions with the pattern of "There is ..... mode-3 tends to produce complex sentences while mode-58 is prone to brief sentences.
The cause of this phenomenon is widely known as the mode collapse problem and has been discussed in many prior works generative modeling [17] Formally, given input and generative model g. mode collapse appears when the estimated output distribution Pa(x) assigns most of its probability mass to small region of the output space, despite that the real data distribution Paam (y) has a much larger variance. As a consequence, the randomly sampled outputs {y: ~ Pp(x) l E N} tend be very similar. While mode collapse is typically side effect for generative modeling, it is somewhat "welcomed" in SoTA image captioning models as it usually facilitates a higher evaluation performance on reference-based metrics like CIDEr, BLEU and SPICE. For example, CIDEr optimization [41] based methods [TT] 20 have significantly pushed the performance of image captioning new level on mainstream reference-basse evaluation metries. However, shown in B1 52 the success of CIDEr optimization could be largely attributed to its ability of reducing the modes in the generated captions.
Some recent researches in image captioning have attempted to tackle the mode collapse problem so to improve the diversity of the generated captions. Specifically, 3 33 50) adopt conditional variational autoencoders (CVAE) to encode the conditional distribution of the image captions into a now-dimensional continuous latent space Z. When performing inference, these methods randomly sample latent variables from Z and input them into the caption decoder to drive it towards using different language patterns to describe the image. In this sense, the sampled latent variables can be considered as the continuous representations of modes interpolated from all occurred modes during training, thus alleviating the mode collapse phenomena to some extent. Nevertheless, is still difficult to interpret the underlying conditional distribution, i.e., which part of Z corresponds to which kind of language patterns in the real world. Thus, for the captions derived from two sampled mode representations, it is uncertain how they differ from each other during the sampling process, as well as their qualities. To tackle this uncertainty, another line of works seeks to learn controllable image captioning models. For example, | 44 control the style of the image captions with learned sentiment or personality representations like "factual" or "humorous". "positive" or "negative", etc. [15] controls the syntactic structure of the image captions through part-of-speech tags. [10] focuses on different image regions to generate region-specific image captions. However, these methods rely on additional tools or annotations to supervise the learning of modes. More critically, this also restricts their modes within a pre-defined domain.
In this paper, we tackle the above problems by learning distinct and representative modes for image captioning through a new Discrete Mode Learning (DML) paradigm. Different from CVAE-based diverse image captioning methods, DMI. learns codebook, which is an embedding matrix consisting of set of mode embeddings that spans a discrete latent space, thus the language patterns encoded in different modes can be casily evaluated and are more perceptible. Different from previous controllable

---

image captioning methods, DML requires no additional supervision for mode and hence is more convenient use and is not limited to the pre-defined set of control signals.
Specifically, we optimize a dual architecture in the proposed DML: Firstly, an image-conditioned discrete variational autoencoder (CdVAE) branch, where an encoder extracts the hidden states for all the reference captions paired with an image, and further quantizes them using their matched mode embeddings in the codebook according to Euclidean distance. For the matching algorithm, we choose Hungarian algorithm instead of the naive nearest neighbor look-up, which we find to be critical for increasing the number of effective modes. Afterward, a decoder is used reconstruct the reference captions in a fully non-autoreegeessive manner, which breaks the sequential dependencies on previous tokens and enforces the decoder to rely purely on the mode embeddings to generate different captions based on the same image feature, leading to more distinct and representative mode embeddings. Secondly, a mode-conditioned image captioning (MIC) branch, which can be simply modified from an existing image captioning model by adding the mode embedding to the original word embeddings as a control signal. During inference, the CdVAE branch dropped, and the MIC branch is used to generate image captions with various language patterns according to the mode embeddings in the codebook.
In the experiments, we evaluate the effectiveness of our proposed DML paradigm by applying it to the widely-used Transformer [46] and the state-of-the-ar AoANet 220,, denoted by Transformer- DML and AoANet-DML, respectively. We observe that some of the modes tend to yield clear language patterns in the generated captions, as shown Figure IT demonstrating that DML has successfully learned perceptible modes without explicit supervision. We also find that our models perform surprisingly well under diversity evaluation (using metrics like SelfCIDEr
(52) and oracle
performance evaluation (on mainstream
inference-based metrics like CIDEr 47). achieving new
ofte-of-the-art results. This shows that our learned modes are not only distinct but also effectively cover the rich modes that appeared within the dataset, which suggests that they are very representative. Moreover, we find that the models trained with DML outperform their original counterparts in terms of quality on some of the modes, meaning that DML can serve as a cost-free plugin for existing image captioning models.
2 Related Works
seeks to generate descriptions based on the given images, which has received lots of attention from the researchers (23] The conventional paradigm of image captioning models [16] [1] 49 mainly consists of two parts: a CNN-based image encoder and an RNN- decoder for caption generation. Based on this diagram, many works 2 20 57
Image captioning. Image captioning 2
20 49 53
25 53], which enforces models to consider more explore the visual relationships by constructing semantic or scene graph, while [18 optimize their models by Reinforcement Learning (RL) and directly use CIDEr (47) to compute the reward. However, these image captioning models mainly consider how to achieve higher evaluation scores, which usually bias the generated captions to an "average" version that contains the common words and phrases in the training corpus only.
introduce the attention mechanism 42
46
14
about the highlighted regions. Besides, to improve the performance,
54 5
Diverse and controllable image captioning. Diverse image captioning aims at learning a model that can generate various captions based on the same image. To this end, CVAE-based models 6 33 50 learn latent space during training and then generate diverse captions by sampling different priors predict diverse captions by using different random noises as inputs accompanied with the given images. Although the diverse image captioning models are able to produce different captions during inference, it is still non-trivial to interpret the underlying conditional distribution, leading to an uncertainty of the model behavior.
from the latent space. GAN-based models [12 24 4
To make the generated captions controllable, some works 7 8 10 13 35 36 introduce an additional
control signal. For example, Mathews et al. (35) provide a sentiment signal for each caption, and seek to control the sentiment of the generated captions. Deng & Ding [13] take the length of the caption as a control signal. Conditioned on different length level embeddings, the model is able to generate length-controolabble descriptions for the input image. However, most of these methods have to rely on additional tools or annotations to supervise the learning of the control signal, which is usually inconvenient to collect and further limits the language pattern within pre-defined domain.
3

---

bags leying yy- lising with bottle wfne. Yel
MIC
Image Encoder
2(0
Caption Decoder
Image Feature
Image x
Codebook
Mode Embedding
A bags laying astan room. ys listng cauch wine Yzi...
Mode Eucoder
(04) (
Masked Decoder
yu-A leying
CdvAE
ys- Liting rooms oanch and bottle of Yei...
Hidden State
Figure 2: Overall of the DML paradigm. The "++" denotes the matching operation by using Hungarian algorithm while the "W" means there is no gradient flow.
3
Method
The overall architecture the proposed Discrete Mode Learning (DML) paradigm is illustrated in Figure It consists of two branches: image-conditioned discrete variational autoencoder (CdVAB) branch, which contains a mode encoder &m and a masked decoder 'Dmi and a mode-conditioned image captioning (MIC) branch consists of an image encoder & and caption decoder D... The two branches are connected through a codebook Q that contains a set of mode embeddings learned through the DML paradigm. During training, the model takes as inputs an image x and its paired reference captions {y: J= and no additional supervision is required. During inference, the CdVAE branch is dropped, and the MIC branch is used generate image captions with various language patterns according to the mode embeddings in the codebook. More details are as follows.
3.1 Discrete Mode Learning
In typical image captioning model, the training objective maximize log p(y;(z) for each yi in {u Ji which is ill- ill-posed since the model is optimized to approach multiple different targets condi- tioned on the same input x, and usually leads this problem by introducing a latent variable 2; to the objective function, i.e., Exxpp(() [log p(y: (x, 2t)], which serves as an explicit indication of mode and drives the model to generate different targets conditioning on different 2,. E.g.. in CVAE-based diverse image captioning models [33 50] 22 randomly sampled from a continuous latent space, which leads an uncertainty of the model behavior. While in some controllable image captioning models [10 15 2; needs to be pre-defined with the help of additional tools or annotations, which is usually restricted to a small set.
mode collapse problem. Previous works alleviate
Unlike these methods, our DML samples latent variables from an embedding matrix Ω E ggxd which we call codebook. It defines discrete latent space, where each entry in Ω corresponds potential mode embedding, and & is a hyper-parameter representing the total number of modes in the codebook. During training, the mode encoder En, is used to extract the representation for each caption yi. denoted by e33),, and match the representation with one of the entries in Ω. The matched entry is then adopted the mode embedding of yi, denoted by q(y)), and the objective becomes:
max log ((ssla(us) where (1v)) (et(c(( Em Em Dm
Here, p(n (13)), æ)) is the conditional distribution of the caption y4 given its mode embedding q(v)) and the paired image x, which is approximated with the masked decoder Dm Mat ch(,)) indicates the matching operation. Note that, there is usually operators like the nearest neighbor look-up. Therefore, we follow [55 to first use the straight- estimator to directly copy the gradients from ((::) to e(v)), and then apply Vector Quantization algorithm to move q(ys) towards e(y:) through mean square loss so as to train the matched codebook entries. The new objective function is
real gradient defined for typical matching
log logg(uu/g(v), x²) + [[sg[(())] - q(x))1 + Blle(u.) sglg(v))113.
(2)

---

where the first term inherited from Eq. ①I is used to optimize En. and Dm- The second term is used train the matched entries in Q2. The last term commitment loss that is responsible for bounding the embedding space of Ω2 sg[:] refers to "stop gradient" and (3 is a hyper-parameter.
3.2 Mode Encoding and Assignment
We adopt stack of Ne transformer encoder layers our mode encoder &m. The input of &m is y: appended with special [MODE] token to the start. The output of &m. e(y;) E R is the hidden state of the [MODE] token from the last layer. When performing the matching process in Eq. I. naive approach is to use the nearest neighbor look-up algorithm, ie..
(v)) 22 where 1 arg min |e(()) 12yl2-
However, we find in the experiments that in the models trained with this assignment strategy, the output of &m quickly converges to the nearby of two or three mode embeddings in Q2 which is a clear sign of mode collapse (see Figure
To avoid this, inspired by the object query assignment in [ we treat the mode assignment of the captions as Bipartite Graph Matching problem and solve it using Hungarian algorithm. Specifically, for all the reference captions {y: paired with an image, we construct a bipartite graph between the output hidden states (e(w:) ) from & and the mode embeddings in Ω. We first pad {es ) to the size of the codebook, k, with Ø (assume n < k). Then, we search for a permutation of elements E 6% with the lowest assignment cost:
arg min Σ |e(vs) - (1)0)22
7E6
and the assigned mode embedding for each ys is q(ys)) = Poyc-- We find this assignment strategy greatly increases the number of effective mode embeddings in 12 (see Figure
3.3 Fully Non-autoregressive Decoder
After getting the mode embeddings g(v:) for each caption ye, we feed it together with the image features from into decoder to generate yı and estimate the conditional distribution p(y: (v))), x) in Eq. (2. In general, sequence generation models are often trained through the autoregressive Teacher Forcing scheme, which aims to maximize the likelihood of the ground-truth token w; given all preceding ground-trutt tokens wy<t- In our setting, the objective function would be:
T
log pl (1)))
- log Bogg(ww/uup (Wyae a))
(5)
where Wt is the t-th token in y, and 7' is the total number of tokens in yi- However, being able to access the previous tokens causes the decoder to ignore the mode embedding when reconstructing yi. since the previous tokens already provide enough information for the mode of y:- As result, the training signals for the mode embeddings could be useless.
Therefore, we propose to use fully non-autoregressive (NAT) objective for the CdvAE branch of DML to facilitate the training the mode embeddings. Specifically, the masked decoder D,, is stack of N₂ transformer decoder layers that takes ((y)), and a sequence of T [MASK] tokens as manner, i.e.
input, and predicts each target token in y in conditionally-indeeeiie
log p(g
Σ
log p(u: [MASK), ((()))
(6)
Apply this equation to Eq. 2 will result in our final objective function for the CdVAE branch. We find this leads to more distinct and representative mode embeddings (see Figure
3.4 Learning MIC Models with DML
So far we have introduced how DML learns the mode embeddings with the CaVAE branch. Here we show how to attach the CdVAE branch to an existing image captioning model to make it aware of the
5

---

mode of the generated captions. Specifically, in a standard secoder-decoder model, the encoder & encodes the information from the image into a sequence of hidden states, then the decoder De attends to the encoder hidden states and predicts the ground-truth tokens in the reference caption ys following the Teach Foreing scheme similar to Eq. (
image captioning
To adapt it into our DML framework, we only need to make a minor modification to the token embedding layer of D. by adding the mode embedding 4(y:) to the word embeddings of each token in yi (i.e. Wjjet in Eq. ) element-wisely, resulting in our MIC branch. The training of MIC and CdVAE performed jointly. using the objective functions in Eq. (2) and Eq. ( respectively. In practice, we find the CdVAE branch is much harder to train than the MIC branch, due to its non- autoregressive prediction manner. To make the convergence speed of the two branches compatible, we let them use different batch sizes, i.e., for each image, the CdVAE branch takes as input all its paired captions {y: )ー= while the MIC branch randomly samples just one caption from (y/ )== Moreover, the gradient from the CdVAE branch will not be back-propagated to the image encoder.
Inference. During inference, the CdVAE branch is dropped, and the inference of MIC follows very similar procedure as the original image captioning model. The only difference is that it requires selecting mode embedding from the codebook and adding it to the input token embeddings.
Experiment
4.1 Dataset and Evaluation Metrics
Dataset. We train and evaluate our method on MSCOCO dataset (28] that contains 123, 287 images and each image is corresponding to at least 5 captions. For a fair comparison, we follow the previous works [32 33 in the area of diverse and controllable image captioning use the m-RNN split (34) of the COCO dataset, which divides the data into 118, 287, 4, 000 and 1, 000 for training, validation and testing, respectively.
Quality evaluation. To assess the quality of the generated captions, we use five widely used evaluation metrics, i.e., BLEU (38), ROUGE (271, METEOR 44) CIDEr (47), and SPICE M.. Moreover, to further evaluate the performance of the generated captions, we employ another CLIP-based [39]
metric namely ClipScore [9], which can assess whether the generated captions are semantically aligned with given images, even when they are totally different from the reference captions.
Diversity evaluation. To investigate the diversity of the generated captions, we use SelfCIDEr [5]. mBLEU, and n-gram diversity (i.e., Div-n (3). All of these metrics evaluate the diversity by comparing the n-gram differences among the generated captions that belong to the same image.
4.2 Implementation Details
Choice of base models. The proposed DML general learning paradigm and we expect it can be casily applied in many existing image captioning models and improve their controllability and diversity. Thus, for the MIC branch, we choose two widely-used and representative architectures as our base models, i.e., Transformer [46] and AoANet [20). denoted by Transformer Monnsforerr-MLL. and AoANet-DML separately. to show the generalization ability DML. Most current SoTA image captioning models, like M2Transformer [II]. XLAN (37 and many vision-language pre-training models, are based on the Transformer architecture. Moreover, the performance of Transformer and AoANet is also competitive with the SoTA models KOO when vision-language pre-training is not performed. Therefore, they are good baseline models to illustrate the performance of our DML paradigm.
Detailed settings. In the CdVAE branch, the number of transformer layers for e,,, and Dm is set to and 2, respectively. We use 12 attention heads and hidden size of 768 for all transformer layers. in Eq. (2) is set to 0.25, following [45]. The number of mode embeddings in Q is set to 64 by default. We convert each image to 100 object proposals by Faster RCNN 40 pre-trained on Visual Genome 222. We train the models for 100,000 iterations with a batch of images and the paired captions. We use Adam W (29) optimizer with a learning rate of 2e-4, and cosine decay it to O. We use the label smoothing of 0. and the gradient clipping threshold of 1.0. We train Transformer-DMI. and AoANet-DML one NVIDIA 3090 GPU with about 10 and 13 GPU hours, respectively.

---

Table 1: Diversity evaluation on the best-5 sentences obtained from consensus re-ranking
Methods
LNFMM 32
COS-CVAE 33
Seq-CVAE B
Transforeer-S
Transfomeer-Di
Div-1 (1)
0.37
0.39
0.33
0.21
0.43
Div-2 (t)
0.50
0.57
0.48
0.29
0.59
SelICIDEr (f)
0.79
0.57
0.83
mBLEU (1)
0.64
0.53
0.64
0.78
0.54
Image
There mas the middle of field playing with frisbee
There is man riding a motomyycle down the street Man yellow motorcycle driving down the road close a person. riding motoreycle on a road
There man skateboard holding
Mode 7
shirt throwing white frisbee
on skateboard with beer on the ground
Mode
A close up of person on a skate board
Mode 43
A close up of with trishee
A young man holding white frusbee
man riding yellow motorcycle with yellow helmet on
A young man holding bottle of water
Mode
on top a green Belld
while standing on skateboard
Mode 58
A man getting ready to throw frisbee
man is sitting motorcycle
A person court with skateboard
Figure 3: Samples of captions that are generated based on different modes, Mode-7, 7, mode-32 and mode-43 tend to generate the captions with certain pattern, i.e., "There is ...", "Man..." ...". and "A close up of ...". Mode-3 is apt to use long sentences while mode-58 is prone to brief sentences.
4.3 Evaluation on Diversity
Quantitative results. In this part, we perform diversity analysis for the proposed DML paradigm based on Transformer-DMe
We compare our model with previous SoTA methods as well as Beam Search (BS) and show the results in Table [ From the table, our DML-based model achieves higher performance most of the diversity evaluation metrics, which demonstrates its effectiveness in learning distinct modes.
We further provide several samples of the generated captions of our Transformer- DML model in Figure 3 We find that the learned modes exhibit clear and distinct language patterns. More specifically, mode- mode-32 and mode-43 tend to follow the patterns "There is ...". "Man ...... and "A close up of ...". respectively. Mode-3 and mode-58 focus more on the semantic complexity of the caption, i.e., mode-58 is likely to generate a brief caption while mode-3 is prone to the complicated one. These results show that for the same image, mode embeddings learned through DML facilitate the model to generate captions in various and comprehensible ways.
Qualitative results.
4.4 Evaluation on Quality
Oracle results. To investigate whether the embeddings in the codebook are able cover the rich modes that appeared within the dataset, we calculate the caption evaluation metrics in the oracle setting consistent with prior works B 33), i.e., taking the maximum score for each quality metric over all the candidate captions for each image. Specifically, we train Transformer-DNy DML and and
AoANet-DML. with codebook sizes to captions generated by all modes. In Table 2 the models with DML obtain the best and second best results on all the evaluation metrics w.z.f both 20 and 100 samples. Morcover, we compare our DML paradigm with the SoTA bascline COS-CVAL by calculating the oracle scores of CIDEr, SPICE and METEOR with different numbers of samples. In Figure the proposed DML consistently outperforms COS-CVAE on all settings. The high gain in quality metrics demonstrates that the proposed DML successfully captures the rich and representative modes in the training corpus.
100, and evaluate the oracle results of
Results of individual mode embedding. To assess the quality of captions generated using different modes, we calculate the evaluation scores of captions generated from some representative modes that are manually selected with distinct patterns. In Table 3, on mode-58, Transformer-DMi. outperforms the original Transformer on all reference-basee metrics. Moreover, the Transformer-DMIL. with different mode embeddings can achieve better or at least competitive performance compared with the

---

Table 2: Comparison with baselines w.z.t oracle performance (i.e., best-1 quality) on COCO dataset. "#Sample" refers to the number of generated captions for each image. The best and the second best results are highlighted with bold and underline, respectively.
Method
#Sample
19
1882
1363
1304
M
S
Div-BS 48
0.837
0.687
0.538
0.383
0.653
0.357
1.405
0.269
POS
15
0.874
0.737
0.593
0,.49
0.678
0,365
1.468
0.277
AG-CVAE
50
0.834
0.698
0.573
0.471
0.638
0,309
1.259
0,244
Seq-CVAL
20
0.870
0.727
0.591
0,445
0.671
0.356
1.448
0.279
COS-CVAE
3
0.903
0.771
0.640
0,500
0.706
0.387
1.624
0.295
AoANct-DML. (Ours)
0.917
0.799
0.682
0.554
0.734
0.418
1.734
0.328
Transformer- (Ours)
0.788
0.526
0.417
0.325
Div-BS
0.846
0.698
0.555
0,402
0.666
0.372
1.448
0.290
POS [S]
0,909
0.787
0.672
0.550
0.725
0.409
1.661
0.311
AG-CVAE 50
0.883
0.767
0.654
0,557
0.690
0,345
1.517
0.277
Seq-CVAE
100
0.922
0.803
0.691
0.575
0.733
0,410
1.695
0,320
I.NMM 32
0.920
0.802
0.695
0.597
0.729
0.402
1.705
0.316
COS-CVAE 83
0.942
0.842
0.739
0.633
0.770
0,450
1.893
0,339
AoANct-DML (Ours)
0.947
0,850
0.752
0.652
0.782
0.479
1.960
0,356
Transformer- DML (Ours)
0.946
0.849
0.750
0.649
0.780
0.474
1,953
0.354
0.47
METEOR
CIDET
SPICE
0.31
0.41
1.6
1.5
0.29
0.38
1.4
COS
0.27
0.35
1.3
DML
0.25
DMI (Durs)
80
Number of sampled captonss
Number of sampled captions
Number of sampled captonss
(a) CIDEr
(b) SPICE
(c) METEOR
Figure 4: Comparison between our DML and SoTA baseline COS-CVAE. We show the oracle results of CIDEr, SPICE and METEOR with different numbers of samples.
original Transformer in terms of ClipScore. We further report the best-performed modes w.t.f. CIDEr, SPICE and ClipScore for AoANet-DML. In Table the results show that the proposed DML can gain higher scores CIDEr, SPICE and ClipScore by choosing the suitable mode embedding. These results suggest that our DML not only generates diverse results, but the results of each individual mode also yield high quality.
Further comparison with COS-CVAE. To further evaluate the effectiveness of our DML paradigm, we run experiments using the UpDown model (a two-layer LSTM with visual attention module) for our MIC branch, which is also the same language generation model used by COS-CVAE. The oracle performance of this model is 1.688 and 1. 1.942 terms of CIDEr for 20 and 100 samples, respectively, which still outperforms the COS-CVAE by large margin. In fact, UpDown is a strong model that achieves compatible performance with a 6-layer Transformer model captioning setting (1.099 CIDEr vs. 1.114 CIDEr on Karpathy's 's test split (21)), which means that two-layer LSTMs may already have enough capacity for the COCO dataset. Moreover, considering that COS-CVAE requires pre-processing step to construct pseudo supervisions with the help of pretrained joint vision- languul embedding model, the proposed end-to-end learning method could be more convenient to use than COS-CVAE.
general image
4.5 Performance Analysis of the CaVAE branch
In this part, we investigate the effect of our mode assignment strategy (Section 3.2) and the fully non-autoregressive (NAT) objective (Section 3.3) used by CdVAE branch of DML. Specifically, we train Transformer models with codebook size of k = 64 on three different settings: 1) DMI w/o NAT objective; 2) DML. w/o Hungarian assignment; 3) the proposed DMI,, and visualize their mode embeddings in the codebook and caption embeddings from the mode encoder output using t-SNE.

---

Table 3: The performance of image captions generated by different modes. For Transformer-DMo we select some representative modes that exhibit clear language patterns or semantic complexities as shown in Figure [ For AoANet-DMLL we show the best-performed modes in terms of CIDEr, SPICE and ClipScore, respectively (highlighted with underline). The original results of Transformer and AoANet are obtained using the code and settings in [0]. The best results of the two captioning methods on each metric are highlighted with bold.
Models
Mode
B@1
B@2
B
B,04
M
ClipScore
Transformer
[G
N/A
0.751
0.589
0.448
0.338
0.558
0.276
1.119
0.207
0.990
0.474
0.871
0.677
0.497
0,359
0,253
0.499
0.247
0.958
0.189
0.967
Transformer DMI.
32
0.665
0.461
0.315
0.213
0.441
0.230
0.840
0.190
1.121
0.658
0.481
0.339
0.238
0.479
0.234
0.909
0.175
1.098
58
0.752
0.590
0.448
0.340
0.589
0.277
1.124
0,208
0.945
AoANet 20
N/A
0.770
0.613
0.476
0.368
0.571
0.283
1.173
0.213
0.969
49
0.762
0.601
0.458
0.356
0.567
0.279
1.157
0.209
0.972
AoANet-DML
0.658
0,461
0.529
0.284
1.131
44
0.747
0.583
0.440
0.339
0.557
0.279
1.123
0.210
1.195
40-
0-
13
-3
DML w/o NAT objective
(b) DML w/o Hungarian assign
(c) DML
Figure 5: Effects of the Hungarian mode assignment and the fully non-autoregressive decoding objective. We visualize the mode embeddings (red "x") and the caption embeddings ("." with other colors) on the test set by t-SNE. The inactivated modes are not shown for clarity.
As shown in Figure a). DML w/o NAT only activates 5 of 64 mode embeddings, iee,, all caption embeddings are assigned to the modes. Moreover, the caption embeddings of different modes are mixed together (zoom in the details) and distributed among the twisted spaces nearby the mode embeddings, which means that the mode embeddings and the caption embeddings may not contain useful information. The low oracle CIDEr score (1.207) also verifies this hypothesis.
In Figure b)), we show that DML w/o Hungarian assignment activates even fewer modes (i.e., only three), but the caption embeddings now have more distinct distribution compared with the result in Figure 5a.). This shows the importance of the fully NAT objective CdVAE. Still, due to the small number of modes, the oracle CIDEr score in this setting is only 1.211, indicating limited diversity.
Lastly, in Figure our DML obtains 29 effective mode embeddings, and the caption embeddings are tightly distributed around their corresponding mode embeddings. Moreover, our model achieves an oracle CIDEr of 1.871. These results demonstrate the effectiveness the Hungarian mode assignment and the fully non-autoregressive objective for learning distinct and representative mode embeddings.
4.6 Ablation Studies on the CdVAE branch
In this section, we provide more analysis on the CdVAE branch of the proposed DML method. Our experiments are based on the Transformer-DMy model with codebook size of 64.
Asymmetric batch size As we mentioned in Section 3.4 the training of the CdVAE branch much harder than the training of the MIC branch. Thus, we adopt asymmetric batch sizes when training them jointly. Moreover, the gradient from the CdVAE branch will not back prop-propag to the image encoder, so that the whole MIC branch is trained with batch size times smaller than that of

---

Table 4: Oracle results of different batch sizes for CdVAE branch and MIC branch. "#effective modes" indicates the total number of modes that have ever been used during the whole training process. "#sampled caps" refers to how many captions per image are sampled to optimize the MIC branch.
Batch size
#sampled caps
B1
B2
B3
B4
R
M
#effective modes
64
0.932
0.826
0.719
0.608
0.761
0.454
1,871
0.342
29
64
0.929
0.820
0.712
0.599
0.756
0.445
1,833
0.336
28
64
0.921
0.805
0.692
0.574
0.743
0.430
1.779
0.331
24
16
0.917
0.799
0.683
0.561
0.737
0.427
1.760
0.327
15
16
0.925
0.810
0.698
0,580
0.749
0.441
1.799
0.335
17
Table 5: Oracle results of different masking strategies for CdVAE branch. "#effective modes" is the total number of modes that have ever been used during the whole training process. "1.0 - 0.0" means we gradually reduce the masking probability from to throughout the training process.
Mask probability
B1
B2
B3
B4
R
M
C
S
#effective modes
1.0
0.932
0.826
0.719
0.608
0.761
0.454
1.871
0.342
29
0.5
0.833
0.675
0.513
0.342
0.633
0.337
1,384
0.262
1.0 - 0.0
0.879
0,740
0.604
0.448
0.685
0.377
1,579
0.294
0.0
0,790
0.619
0.439
0.270
0.590
0.301
1.207
0.228
5
the CdVAE branch. Here, we show how this strategy benefits the training of our Transformer-DMI. model. Specifically, during training. we change the number of sampled captions for the MIC branch as well as the total batch size, and the rest settings are the same as in Section 4.2 The results are
shown in Table
The first row in Table is the performance of our default setting, which yields the best results among all metries, and also obtains larger number of effective modes than other settings. When increasing the number of sampled captions from 1 to 5 for the MIC branch, the performance of Transformer- DML drops clearly. We believe this is due to the over-fitting of the MIC branch caused by the large batch size. When decreasing the total batch size from 64 to 16, we observe clear reduction in the number of effective modes, which means the CdVAE branch not sufficiently trained. Moreover, with such small batch size, the MIC branch will also suffer from under-fitting if the number of sampled captions small, i.e., oracle CIDEr drops from 1.799 to 1.760 when reducing the number of sampled captions from 5 to 2. These results show the importance of balancing the training of the two branches the proposed DML.
Masking strategy of the masked decoder Dm,. When training the CdVAE branch, we use fully non-autoregressive objective, where the input of the masked decoder Dm are all [MASK] tokens. This prevents the model from using the mode information leaked from the ground-truth tokens, and we find it greatly benefits the learning of modes. Here we evaluate the performance of several different masking strategies, including random masking the input caption using fixed probability, and random masking with dynamically changed probability. The results are shown in Table
From the table, the full masking strategy, i.e., mask probability of 1.0, leads to the best performance, which is also the default setting in DMI.. Moreover, introducing any additional information to the CaVAE branch, i.e., mask probability less than 1.0, will severely hamper the learning of modes, where the number of effective modes drops clearly from 29 to less than 10, and the oracle performances also drop significantly.
5 Conclusion
In this paper, we study problem in image captioning that models tend to be biased to generate an "average" caption, which contains the common words or phases only. To tackle this problem, we propose Discrete Mode Learning (DML) paradigm for image captioning. The idea is to explore multiple rich modes the training caption corpus to learn a codebook that contains a set of "mode embeddings", which enables the image captioning models to generate different captions based on various modes. Moreover, the proposed DMI. paradigm can be easily plugged into the existing image captioning models (e.g., Transformer and AoANet) to generate high-quality and diverse captions.
10

---

References
[1] P. Anderson, B. Fernando, M. Johnson, and S. Gould. Spice: Semantic propositional image caption evaluation. In Eur. Conf. Comput. Vis., 2016
[2] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, Bottom-up and top-down attention for image captioning and visual question answering. In IEFE Conf. Comput. Vis. Pattern Recog., 2018.
[3] J. Aneja, H. Agrawal, D. Batra, and A. Schwing. Sequential latent spaces for modeling the intention during diverse image captioning. In Int. Conf. Comput. Vis., 2019.
[4] S. Banerjee and A. Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65 65-72, 2005,
[5] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In Eur. Conf. Cumput. Vis., 2020.
[6] F. Chen, R. Ji, J. Ji, X. Sun, B. Zhang, X. Ge, Wu, Huang, and Wang. Variational structured semantic inference for diverse image captioning. Adv. Neural Inform. Process. Syst. 2019.
[7] L. Chen, Jiang. Xiao, and Liu. Human-likk controllable image captioning with verb-specific semantic roles. JEEE Conf. Comput. Vis. Pattern Recog., 2021.
[8] S. Chen, Q. Jin, P. Wang, and Q Wu. Say you wish: Fine-grained control of image caption generation with abstract scene graphs. In IEEE Conf. Comput, Vis. Puttern Recog., pages
2020.
[9] T. Chen, Z. Zhang, Q. You, C. Fang, Z. Wang, H. Jin, and J. Luo. "r"cuall"o""emotonn" Stylized image captioning with adaptive learning and attention. In Eur. Conf. Comput. Vis., pages 519-535, 2018.
[10] M. Comia, I.. Baraldi, and R. Cucchiara. Show, control and tell: A framework for generating controllable and grounded captions. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[11] M. Comia, M. Stefanini, L. Baraldi, and R. Cucchiara. Meshrd-memory transformer for image captioning. In IEEE Conf. Comput. Vis. Pattern Recog, pages 10578-10587, 2020.
[12] B. Dai, S. Fidler, R. Urtasun, and D. Lin. Towards diverse and natural image descriptions via a conditional gan. In Int. Conf. Comput. Vis., 2017.
[13] C. Deng, N. Ding, M. Tan, and Q Wu. Length-controllable image captioning. In Eur. Conf. Comput. Vis., 2020.
[14] C. Deng, Q. Wu, Q. Wu, F. Lyu, F. Hu, and M. Tan. Visual grounding via accumulated attention. IEFE Transactions Pattern Analysis and Machine Intelligence, 2020.
[15] A. Deshpande, J. Aneja, L. Wang, A. G. Schwing, and D. Forsyth. Fast, diverse and accurate image captioning guided by part-of-speech, In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[16] J. Donahuc, L. Anne Hendricks, Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell. Long-term recurrent convolutional networks for visual recognition and description. In IEEE Conf. Comput. Vis. Pattern Recog,, 2015.
[17] 1. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu. D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Adv. Newral Inform. Process. Syst. 27, 2014.
[18] L. Guo. J. Liu, X. Zhu, X. He, J. Jiang, and H. Lu. Non-autoregressive image captioning with counterfactuals-rrttee critical multi-agent learning. IJCAI, 2020.
[19] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi. Clipscore: A reference-free evaluation metric for image captioning. EMNLP, 2021.
[20] L. Huang, W. Wang, J. Chen, and X.-Y. Wei. Attention attention for image captioning. In Int. Conf. Comput. Vis., 2019.
[21] A. Karpathy and I.. Fei-Fei. Deep -esual-semantic alignments for generating image descriptions. In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
[22] R. Krishna, Y. Zhu, O. Groth, Johnsson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li. D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Int. Conf. Comput. Vis., 2017.
11

---

[23] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. nature, 521((553):436 444, 2015.
[24] D. Li, Q. Huang, X. He, L. Zhang, and M.-T. Sun. Generating diverse and accurate visual captions by comparative adversarial learning. arXiv preprint arXiv: 804,00001,, 2018.
[25] K. Li, Wu, K.-C. Peng, J. Ernst, and Fu. Tell me where to look: Guided attention inference network. In IEEE Conf. Comput. Vis. Pattern Recog., pages 9215 0223, 2018.
[26] X. Li, X. Yin, C. Li, P. Zhang. X. Hu, L. Zhang, L. Wang, H. Hu, L. Dong, F. Wei, et al. Oscar: Object-semantics aligned pre-training for vision- -ainuale tasks. In Eur. Conf. Comput. Vis., 2020.
[27] C.-Y. Lin. Rouge: package for automatic evaluation of summaries. In Text summarization branches out. pages 81, 2004.
[28] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vts., 2014.
[29] 1.
Loshchilov and F. Hutter. Fixing weight decay regularization in adam. 2018
[30] R. Luo.
/oraatrrttttaa ian www.aaaaeeaaaaar/////ee cal htteeelfaarriiillll.ttt.tt
[31] R. Luo and G. Shakhnarovich. Analysis of diversity-accuracy accuracy tradeoff in image captioning. arXiv preprint arXiv:2002.11845 2020.
[32] S. Mahajan, I. Gurevych, and S. Roth. Latent normalizing flows for many-to-man many cross-domain mappings. Int. Conf. Learn. Represent., 2020.
[33] S. Mahajan and S. Roth. Diverse image captioning with context-object split latent spaces. Adv. Neural Inform. Process. Syst. 2020.
[34] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). In Int. Conf. Learn. Represent., 2015.
[35] A. Mathews, L. Xie, and X. Hc. Senticap: Generating image descriptions with sentiments. In AAAI, 2016.
[36] A. Mathews, L. Xie, and X. He. Semstyle: Learning generate stylised image captions using unaligned text. In IEEE Conf. Comput. Vis. Puttern Recog. 2018
[37] Y. Pan, T. Yao, Y. Li, and T. Mei. X-linear attention networks for image captioning. In IEEE Conf. Comput. Vis. Pattern Recog- pages 10971-10980, 2020.
[38] K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting the Association for Computational Linguistics. pages 311-318, 2002.
[39] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[40] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards cal--tmee object detection with region proposal networks. Adv. Neural Inform. Process. Syst., 28, 2015.
[41] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In IEEE Conf. Comput. Vis. Pattern Recog,, 2017.
[42] Schwartz, S. Yu, T. Hazan, and A. G. Schwing. Factor graph attention. In JEEE Conf. Comput. Vis. Pattern Recog,, pages
2019.
[43] R. Shetty, M. Rohrbach, Anne Hendricks, Fritz, and B. Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In Int. Conf. Comput. Vis., 2017.
[44] K. Shuster, S. Humeau, H. Hu, A. Bordes, and Weston. Engaging image captioning via personality. In IEEE Conf. Comput. Vis. Pattern Recog., pages 12516-12526, 2019.
[45] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Adv. Neural Inform. Process. Syst., 2017.
[46] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L.. Kaiser, and I. Polosukhin. Attention is all you need. Adv. Neural Inform. Process. Syst., 2017.
[47] R. Vedantam, C. Lawrence Zitnick. and D. Parikh. Cider: Consensus maasessus-based image description evaluation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 4566 4575, 2015.
12

---

[48] A. Vijayakumar, M. Cogswell, R. Selvaraju. Q. Sun, S. Lee, D. Crandall, and D. Batra. Diverse beam search for improved deseription of complex scenes. In AAAI, 2018.
[49] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In IEEE Conf. Comput. Vis. Pattern Recog., 2015.
[50] L. Wang, A. Schwing, and S. Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. Adv. Neural Inform. Process. Syst., 2017.
[51] Q. Wang and A. B. Chan. Deseribing like humans: on diversity in image captioning. In IEEE Conf. Comput. Vis. Pattern Recog., 2019.
[52] Q. Wang, J. Wan, and A. B. Chan. On diversity in image captioning: Metrics and methods. IEEE Trans. Pattern Anal. Mach. Intell., 2020.
[53] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
[54] X. Yang, K. Tang, H. Zhang, and J. Cai. Auto-encoding scene graphs for image captioning, In JEEE Conf. Comput. Vis, Pattern Recog., 2019.
[55] T. Yao, Y. Pan, Y. Li, and T. Mei. Exploring visual relationship for image captioning. In Eur. Conf. Comput. Vis., 2018.
[56] C. Zhou, J. Gu, and G. Neubig. Understanding knowledge distillation in non-autoregressive machinc translation. In Int. Conf. Learn. Represent., 2019.
[57] I.. Zhou, H. Palangi, L. Zhang, H. Hu, Corso, and J. Gao. Unified vision- laiiual pre-training for image captioning and vqa. In AAAI, 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Please see the abstract and introduction in Section [
(b) Did you describe the limitations of your work? [Yes] Please see the limitations in the supplementary.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see the potential negative societal impacts in the supplementary.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] We have released the URL of the source code in the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see the implementation details in Section 4.2
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Error bars are not reported because it would too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Please see the implementation details in Section 4.2
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Please see Section (b) Did you mention the license of the assets? [N/A]
13

---

(c) Did you include any new assets either in the supplemental material or as URL? [No] The code will be made public upon acceptance.
(d) Did you discuss whether and how consent was obtained from people whose data you' rou're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "uHVOeeylf5", "reviewer": "Reviewer_eCow", "summary": "This paper proposed a diverse image captioning model, namely conditioned discrete variational autoencoder (CdVAE), which is able to learn multiple modes of captions. Plus, to learn better mode embeddings, a masked decoder and Hungarian algorithm are introduced into the training process. Finally, extensive experiments are conducted, showing that the proposed model can generate diverse captions for an input image.", "questions": "The CdVAE branch is separate from the image captioning branch, and CdVAE can be trained using a large-scale dataset composed of texts, so I am wondering about the performance of using a large-scale dataset.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The weakest part of the paper is that the framework seems just combines VQ-VAE and a normal image captioning model, though some findings are interesting, such as directly using VQ-VAE cannot capture many modes and to mitigate this issue, this paper introduces Hungarian algorithm and masked decoder. Another strength of the paper is that extensive experiments are conducted and the performance of the paper is relatively good. Plus, the paper is well written and can be easily followed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "e94c6ND9XPi", "reviewer": "Reviewer_rUVQ", "summary": "This paper presents a strategy to generate a diverse set of captions for image captioning. The proposed method is composed of two parts: (1) CdVAE which learns a discrete code book of mode embedding, and (2) MIC which leans to generate captions conditioned on a mode embedding. The proposed method demonstrates slightly better diversity scores and quality scores compared to previous SoTA COS-CVAE.", "questions": "- The biggest concern the reviewer has is how to fairly compare with SoTA COS-CVAE method as described in the weakness section. It would be really helpful if the author can either: (1) incorporate their method to the same model architecture as COS-CVAE (LSTM), and/or (2) implement COS-CVAE with a transformer model. In either way, we can rule out the possibility of architectural improvement and focus on the contribution of the proposed method.\n\n- Further analyses of the collapsed mode in Fig.5 as described in the weakness section.\n\n- Some ablations are missing such as K and the loss terms.", "limitations": "- Recent VL works typically pre-train a cross-modal transformer model on large image and text corpus and then fine-tune to the downstream target task such as image captioning. It is unknown how the proposed method can be incorporated into the transformer pre-training methods.\n\n- In real-world applications, the users usually only need one caption with the best quality. How does the proposed method search and find the best performing caption among all the modes?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strength\n\n1. The training objectives of CdVAE is very similar to [1], but the reviewer considers the incorporation of an existing method for solving an interesting question in a new setting (eg. image captioning) as a contribution. Nevertheless, it would still be good to describe the difference.\n\n1. The proposed method seems to outperform SoTA COS-CVAE in both diversity and quality (more in the weakness section).\n\n1. The core idea and the proposed method is easy to understand and follow.\n\n1. Fig.5 successfully demonstrates how the proposed training techniques affect the learned codebook of mode embedding.\n\n[1] Van Den Oord, Aaron, and Oriol Vinyals. \"Neural discrete representation learning.\" Advances in neural information processing systems 30 (2017).\n\n### Weakness\n\n1. The comparison against SoTA COS-CVAE is unfair. First of all, the performance improvement in terms of the diversity scores in Tab.1 is not significant. The SelfCIDEr score is also missing. Since COS-CVAE released its code, it would be good to re-run the method and include the number. Secondly, COS-CVAE uses LSTM as the language generation model, which is weaker than the transformer model. Therefore, given the slight performance improvement and the usage of a stronger transformer model in this work, it is hard to conclude that the performance improvement compared to SoTA COS-CVAE is actually coming from the proposed method.\n\n1. In Sec. 4.5, only qualitative results and the oracle CIDEr scores are provided. What about the diversity scores? Furthermore, any discussions or analyses of the collapsed modes? Do they really lead to the same output samples? Even the proposed method has a mode collapse issue. Please provide further studies.\n\n1. In line 208, it says the model embedding is added to the word embedding of each token in $y_i$. Does that mean for each $w_{j<t}$ in Eq.5 when generating the next token $w_t$? \n\n1. In Tab.2, baseline methods can also generate multiple samples by beam search. It would be helpful to include those results for AoANet and Transformer as a baseline comparison.\n\n1. K (total number of modes in the codebook) seems to be an important hyper-parameter. How does the performance vary with respect to K?\n\n1. The ablation study of the loss terms (1) $| \\text{sg}[e(y_i)] − q(y_i)|^2$ and (2) $| e(y_i) − \\text{sg}[q(y_i)]|^2$ in Eq.2 is not provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UnysDwy9Ulj", "reviewer": "Reviewer_Z9St", "summary": "This paper aims to learn distinct and representative modes for image captioning. The paper propose Discrete Mode Learning (DML), which is a good method to improve the diversity of image captioning. The DML optimizes a dual architecture that consists of an image-conditioned discrete variational autoencoder (CdVAE) branch and a mode-conditioned image captioning (MIC) branch.\n", "questions": "1. The common captioning models generate one caption for each image, while this work and some previous related works generate 20 or 100 captions for each image. Which kind is better?\n2. The visual language pre-training models achieve good performance on image captioning recently. Do you think our new methods should apply to those methods? ", "limitations": "There is no potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The experimental results show that the DML works well. \n2. The Discrete Mode Learning (DML) method is reasonable, and the training loss is well designed.\n\nWeaknesses:\n1. The main models used in this paper are Transformer (published in 2017) and AoANet (published in 2019), which are out of date. M2Transformer and XLAN are open sources. And many pre-training models (like Oscar [3r]) are also open source.  \n2. Since the codebook is in latent space, it is hard to control the semantic meaning of a certain mode. It is a weak point of the latent space based method. The diversity comes from different modes but we can not know which mode is needed. For example, people rarely have the patience to choose the needed captions from 100 samples.\n3. This paper uses the m-RNN split, and the most recent image captioning works use Karpathy split [4r]. The results are hard to compare in this case. It will be a contribution if this paper also turns the previous typical works (e.g. COS-CVAE) into the Karpathy split.\n4. Paper writing needs to be refined. \nIn Figure 2, the text of \"y1, y2, y3\" appears three times exactly the same, which does not help to show the main idea.\n\n\n[1r] Marcella Cornia et al. “Meshed-Memory Transformer for Image Captioning” computer vision and pattern recognition (2019).\n[2r] Yingwei Pan et al. “X-Linear Attention Networks for Image Captioning” computer vision and pattern recognition (2020).\n[3r] Xiujun Li et al. “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks” European conference on computer vision (2020).\n[4r] Andrej Karpathy and Li Fei-Fei. “Deep Visual-Semantic Alignments for Generating Image Descriptions” IEEE Transactions on Pattern Analysis and Machine Intelligence (2014).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6CVVx0_CIqw", "reviewer": "Reviewer_g77u", "summary": "The authors consider the mode collapse problem in image caption\ngeneration.  They give a diverse caption generation model that differs\nfrom prior work because 1) it's more controllable than conditioning on\nrandomly-generated latent codes; and 2) the control codes are learned\nvia VAE clustering, rather than pre-specified by styles in the\ntraining data, e.g., \"funny\". Several modeling innovations are key to\nthe end-to-end training working, e.g., a non-autoregressive component,\nand a hungarian method for mode assignment. The authors evaluate their\nmodel on the MSCOCO benchmark, demonstrating that their model 1)\ngenerates diverse and controllable-in-style captions based on\ninterpretable clusters; and 2) performs quite well in the\noracle-conditioned setup according to standard evaluation metrics.", "questions": "- Is mode collapse really occurring with image captioning models, or are the decoding methods the reason why the generated captions seem generic?", "limitations": "I was a bit disappointed by the discussion of potential negative\nsocietal impacts: what does image captioning have to do with\nfinancial scams? I hope the authors can think a bit more about\nautomatic image captioning's potential negative impacts, though\nI do not think this particular work poses much marginal dual use\nrisk.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nI really liked this work! The introduction frames the problem of mode\ncollapse quite effectively, and the story flows well. The modeling\ncomponents are well-motivated, interesting, and clever. The empirical\nresults are strong. The ablations are appropriate. The qualitative\nresults are convincing. I expect that this approach could be useful\nfor tasks beyond image captioning, too, e.g., in cases where there\nisn't one objectively correct generation target, a similar VAE-style\nclustering could come in handy, without additional labeling, to give\ncontrol over generations when the targets implicitly contain different\nstyles.\n\nWeaknesses:\n\n- This may be explored in prior work, but I was hoping for some\n  information about whether or not the observed mode collapse of\n  models is actually P(y|x) mode collapse, or if it is simply a result\n  of the decoding algorithms favoring blander outputs. Is it possible\n  to score the ground truth captions y_i to see if they collapse to\n  the most generic one vs. just relying on the sampled captions, which\n  may be affected by the decoding approach?\n\n- While the ablations clearly demonstrate that the non-autoregressive\n  training objective (/model) works to spread out mode assignments, I\n  wasn't entirely clear how the autoregressive model avoids mode\n  collapse at inference time, given the discussion/experiments about\n  how \"predict the next token\" by itself does indeed lead to mode\n  collapse. I'm not sure how the authors could address this beyond the\n  ablation they already ran, but perhaps, e.g., the attention of the\n  transformer based model weighs the VAE cluster token more highly\n  after NAT training? Could be interesting to look at.\n\n- It would have been nice to see this model run on datasets other than\n  MSCOCO. I wonder if this approach could still work even if there\n  were only one reference at training time?\n\n\nOverall: I really liked this work, and don't have many improvements to\nsuggest beyond discussing the detail about mode collapse vs. decoding\nas discussed above. It's possible that I missed something because I am\nless up-to-date on the diverse captioning literature, but if I didn't,\nI hope this work appears at NeurIPS.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept