Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to learn distinct and representative modes for image captioning. The paper propose Discrete Mode Learning (DML), which is a good method to improve the diversity of image captioning. The DML optimizes a dual architecture that consists of an image-conditioned discrete variational autoencoder (CdVAE) branch and a mode-conditioned image captioning (MIC) branch.

strengths_and_weaknesses: Strengths:
1. The experimental results show that the DML works well. 
2. The Discrete Mode Learning (DML) method is reasonable, and the training loss is well designed.

Weaknesses:
1. The main models used in this paper are Transformer (published in 2017) and AoANet (published in 2019), which are out of date. M2Transformer and XLAN are open sources. And many pre-training models (like Oscar [3r]) are also open source.  
2. Since the codebook is in latent space, it is hard to control the semantic meaning of a certain mode. It is a weak point of the latent space based method. The diversity comes from different modes but we can not know which mode is needed. For example, people rarely have the patience to choose the needed captions from 100 samples.
3. This paper uses the m-RNN split, and the most recent image captioning works use Karpathy split [4r]. The results are hard to compare in this case. It will be a contribution if this paper also turns the previous typical works (e.g. COS-CVAE) into the Karpathy split.
4. Paper writing needs to be refined. 
In Figure 2, the text of "y1, y2, y3" appears three times exactly the same, which does not help to show the main idea.


[1r] Marcella Cornia et al. “Meshed-Memory Transformer for Image Captioning” computer vision and pattern recognition (2019).
[2r] Yingwei Pan et al. “X-Linear Attention Networks for Image Captioning” computer vision and pattern recognition (2020).
[3r] Xiujun Li et al. “Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks” European conference on computer vision (2020).
[4r] Andrej Karpathy and Li Fei-Fei. “Deep Visual-Semantic Alignments for Generating Image Descriptions” IEEE Transactions on Pattern Analysis and Machine Intelligence (2014).

questions: 1. The common captioning models generate one caption for each image, while this work and some previous related works generate 20 or 100 captions for each image. Which kind is better?
2. The visual language pre-training models achieve good performance on image captioning recently. Do you think our new methods should apply to those methods? 
limitations: There is no potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed a diverse image captioning model, namely conditioned discrete variational autoencoder (CdVAE), which is able to learn multiple modes of captions. Plus, to learn better mode embeddings, a masked decoder and Hungarian algorithm are introduced into the training process. Finally, extensive experiments are conducted, showing that the proposed model can generate diverse captions for an input image.
strengths_and_weaknesses: The weakest part of the paper is that the framework seems just combines VQ-VAE and a normal image captioning model, though some findings are interesting, such as directly using VQ-VAE cannot capture many modes and to mitigate this issue, this paper introduces Hungarian algorithm and masked decoder. Another strength of the paper is that extensive experiments are conducted and the performance of the paper is relatively good. Plus, the paper is well written and can be easily followed.
questions: The CdVAE branch is separate from the image captioning branch, and CdVAE can be trained using a large-scale dataset composed of texts, so I am wondering about the performance of using a large-scale dataset.
limitations: N/A
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a strategy to generate a diverse set of captions for image captioning. The proposed method is composed of two parts: (1) CdVAE which learns a discrete code book of mode embedding, and (2) MIC which leans to generate captions conditioned on a mode embedding. The proposed method demonstrates slightly better diversity scores and quality scores compared to previous SoTA COS-CVAE.
strengths_and_weaknesses: ### Strength

1. The training objectives of CdVAE is very similar to [1], but the reviewer considers the incorporation of an existing method for solving an interesting question in a new setting (eg. image captioning) as a contribution. Nevertheless, it would still be good to describe the difference.

1. The proposed method seems to outperform SoTA COS-CVAE in both diversity and quality (more in the weakness section).

1. The core idea and the proposed method is easy to understand and follow.

1. Fig.5 successfully demonstrates how the proposed training techniques affect the learned codebook of mode embedding.

[1] Van Den Oord, Aaron, and Oriol Vinyals. "Neural discrete representation learning." Advances in neural information processing systems 30 (2017).

### Weakness

1. The comparison against SoTA COS-CVAE is unfair. First of all, the performance improvement in terms of the diversity scores in Tab.1 is not significant. The SelfCIDEr score is also missing. Since COS-CVAE released its code, it would be good to re-run the method and include the number. Secondly, COS-CVAE uses LSTM as the language generation model, which is weaker than the transformer model. Therefore, given the slight performance improvement and the usage of a stronger transformer model in this work, it is hard to conclude that the performance improvement compared to SoTA COS-CVAE is actually coming from the proposed method.

1. In Sec. 4.5, only qualitative results and the oracle CIDEr scores are provided. What about the diversity scores? Furthermore, any discussions or analyses of the collapsed modes? Do they really lead to the same output samples? Even the proposed method has a mode collapse issue. Please provide further studies.

1. In line 208, it says the model embedding is added to the word embedding of each token in $y_i$. Does that mean for each $w_{j<t}$ in Eq.5 when generating the next token $w_t$? 

1. In Tab.2, baseline methods can also generate multiple samples by beam search. It would be helpful to include those results for AoANet and Transformer as a baseline comparison.

1. K (total number of modes in the codebook) seems to be an important hyper-parameter. How does the performance vary with respect to K?

1. The ablation study of the loss terms (1) $| \text{sg}[e(y_i)] − q(y_i)|^2$ and (2) $| e(y_i) − \text{sg}[q(y_i)]|^2$ in Eq.2 is not provided.
questions: - The biggest concern the reviewer has is how to fairly compare with SoTA COS-CVAE method as described in the weakness section. It would be really helpful if the author can either: (1) incorporate their method to the same model architecture as COS-CVAE (LSTM), and/or (2) implement COS-CVAE with a transformer model. In either way, we can rule out the possibility of architectural improvement and focus on the contribution of the proposed method.

- Further analyses of the collapsed mode in Fig.5 as described in the weakness section.

- Some ablations are missing such as K and the loss terms.
limitations: - Recent VL works typically pre-train a cross-modal transformer model on large image and text corpus and then fine-tune to the downstream target task such as image captioning. It is unknown how the proposed method can be incorporated into the transformer pre-training methods.

- In real-world applications, the users usually only need one caption with the best quality. How does the proposed method search and find the best performing caption among all the modes?
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors consider the mode collapse problem in image caption
generation.  They give a diverse caption generation model that differs
from prior work because 1) it's more controllable than conditioning on
randomly-generated latent codes; and 2) the control codes are learned
via VAE clustering, rather than pre-specified by styles in the
training data, e.g., "funny". Several modeling innovations are key to
the end-to-end training working, e.g., a non-autoregressive component,
and a hungarian method for mode assignment. The authors evaluate their
model on the MSCOCO benchmark, demonstrating that their model 1)
generates diverse and controllable-in-style captions based on
interpretable clusters; and 2) performs quite well in the
oracle-conditioned setup according to standard evaluation metrics.
strengths_and_weaknesses: Strengths:

I really liked this work! The introduction frames the problem of mode
collapse quite effectively, and the story flows well. The modeling
components are well-motivated, interesting, and clever. The empirical
results are strong. The ablations are appropriate. The qualitative
results are convincing. I expect that this approach could be useful
for tasks beyond image captioning, too, e.g., in cases where there
isn't one objectively correct generation target, a similar VAE-style
clustering could come in handy, without additional labeling, to give
control over generations when the targets implicitly contain different
styles.

Weaknesses:

- This may be explored in prior work, but I was hoping for some
  information about whether or not the observed mode collapse of
  models is actually P(y|x) mode collapse, or if it is simply a result
  of the decoding algorithms favoring blander outputs. Is it possible
  to score the ground truth captions y_i to see if they collapse to
  the most generic one vs. just relying on the sampled captions, which
  may be affected by the decoding approach?

- While the ablations clearly demonstrate that the non-autoregressive
  training objective (/model) works to spread out mode assignments, I
  wasn't entirely clear how the autoregressive model avoids mode
  collapse at inference time, given the discussion/experiments about
  how "predict the next token" by itself does indeed lead to mode
  collapse. I'm not sure how the authors could address this beyond the
  ablation they already ran, but perhaps, e.g., the attention of the
  transformer based model weighs the VAE cluster token more highly
  after NAT training? Could be interesting to look at.

- It would have been nice to see this model run on datasets other than
  MSCOCO. I wonder if this approach could still work even if there
  were only one reference at training time?


Overall: I really liked this work, and don't have many improvements to
suggest beyond discussing the detail about mode collapse vs. decoding
as discussed above. It's possible that I missed something because I am
less up-to-date on the diverse captioning literature, but if I didn't,
I hope this work appears at NeurIPS.
questions: - Is mode collapse really occurring with image captioning models, or are the decoding methods the reason why the generated captions seem generic?
limitations: I was a bit disappointed by the discussion of potential negative
societal impacts: what does image captioning have to do with
financial scams? I hope the authors can think a bit more about
automatic image captioning's potential negative impacts, though
I do not think this particular work poses much marginal dual use
risk.
review_title: null

Meta-review: metareview: The paper tackles the problem of mode collapse in image captioning and provide a method for generating diverse captions. The proposed approach uses a VAE to learn various modes, each of which can produce a different caption, along with various technical innovations to train the model. Experiments with two models on MS COCO demonstrate the effectiveness of the approach. The paper offers useful insights for the challenging and important problem of diverse captioning and will inform future work in this space.  I encourage the authors to make the revisions suggested by reviewers to improve the clarity of the writing and also include adequate justification for the choice of their base models.
confidence: Certain
Decision: Accept