DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training
Authors: Wei Li
Linchao Zhu
Longyin Wen
Yi Yang
Venue: ICLR 2023 poster (2023)
Keywords: Zero-shot captioning
Decoder training
Multi-modal learning

Abstract: Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero-shot transfer capability in many discriminative tasks, e.g., image classification. Their adaptation to zero-shot image-conditioned text generation tasks has drawn increasing interest. Prior arts approach to zero-shot captioning by either utilizing the existing large language models (e.g., GPT-2) or pre-training the encoder-decoder network in an end-to-end manner. However, the large language models may not generate sensible descriptions due to the task discrepancy between captioning and language modeling, while the end-to-end pre-training requires paired data and extensive computational resources. In this work, we propose a simple framework, named DeCap, for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and computation-efficient: 1) it only requires the \textit{text} data for training, easing the burden on the collection of paired data. 2) it does not require end-to-end training. When trained with text-only data, the decoder takes the text embedding extracted from the off-the-shelf CLIP encoder as a prefix embedding. The challenge is that the decoder is trained on the text corpus but at the inference stage, it needs to generate captions based on visual inputs. Though the CLIP text embedding and the visual embedding are correlated, the \textit{modality gap} issue is widely observed in multi-modal contrastive models that prevents us from directly taking the visual embedding as the prefix embedding. We propose a training-free mechanism to reduce the modality gap. We project the visual embedding into the CLIP text embedding space, while the projected embedding retains the information of the visual input. Taking the projected embedding as the prefix embedding, the decoder generates high-quality descriptions that match the visual input. The experiments show that DeCap outperforms other zero-shot captioning methods and unpaired captioning methods by a large margin on the typical image captioning benchmarks, i.e., MSCOCO and NoCaps. We apply DeCap to video captioning and achieve state-of-the-art zero-shot performance on MSR-VTT and ActivityNet-Captions. The code is available at https://github.com/dhg-wei/DeCap.

Published as a conference paper at ICLR 2023
DECAP: DECODING CLIP LATENTS FOR ZERO-SHOT CAPTIONING VIA TEXT-ONLY TRAINING
Wei Li
Linchao Zhu'
Longyin Wen Yi Yang
CCAL, Zhejiang University *ByteDance Inc., San Jose, USA
(weili6, inchan, (angyics]szzjin edu. edu. w wenibytedance.com
ABSTRACT
Large-scale pre-trained multi-modal models (e.g., CLIP) demonstrate strong zero- shot transfer capability in many discriminative tasks, e.g., image classification. text generation tasks has drawn
Their adaptation to zero-shot increasing interest. Prior arts approach to zero-shot captioning by either utiliz- ing the existing large language models (e.g., GPT-2) or pre-training the encoder- decoder network in end-to-end manner. However, the large language mod- els may not generate sensible descriptions due to the task discrepancy between captioning and language modeling, while the end-to-end pre-training requires paired data and extensive computational resources. In this work, we propose a simple framework, named DeCap. for zero-shot captioning. We introduce a lightweight visual-aware language decoder. This decoder is both data-efficient and peppuuattoo-fffccientt 1) only requires the text data for training, easing the burden on the collection of paired data. 2) it does not require end-to-end training. When trained with text-only data, the decoder takes the text embedding extracted from the off-the-shelf CLIP encoder as prefix embedding. The challenge is that the decoder trained on the text corpus but at the inference stage, it needs to generate captions based visual inputs. Though the CLIP text embedding and the visual embedding are correlated, the modality gap issue is widely observed in multi-modal contrastive models that prevents from directly taking the visual embedding as the prefix embedding. We propose a training-free mechanism to reduce the modality gap. We project the visual embedding into the CLIP text em- bedding space, while the projected embedding retains the information of the visual input. Taking the projected embedding as the prefix embedding, the decoder gen- erates high-quality descriptions that match the visual input. The experiments show that DeCap outperforms other zero-shot captioning methods and unpaired caption- ing methods by a large margin on the typical image captioning benchmarks, i.e., MSCOCO and NoCaps. We apply DeCap to video captioning and achieve state- of-the-art zero-shot performance MSR-Vi VTT and ActivityNet-Captions The code is available at (attps: /ttll::///ttttt com/ weeeewweeeDDaaae
INTRODUCTION
The goal of image captioning to automatically generate descriptions for given images. Models (Anderson et al. 2018 Lu et al. 2017, Rennie et al., 2017 Zhang et al.] 2021 Huang et al. (021) trained on hanan-annotated image-text pairs have achieved impressive results on typical image captioning benchmarks. However, due to the small size and limited visual concepts of human- annotated datasets, these models generalize poorly images in the wild (Agrawal al. 2019
Tran et al., 2016 Wu et al. 2018). In this paper, to reduce the reliance on human-annotuun annotated paired data and improve the generalization real-world captioning scenarios, we propose a new zero-shot captioning framework that requires text-only data for training.
Pre-training on web-scale noisy paired data has been demonstrated to be effective in learning ro- bust multi-modal representations (Radford et al.] 2021 Jia et al.] 2021 Li et al.] 2021; Alayrac et al. 2022; Yu et al. 2022a Wang et al. 2022 Zhu Yang 2020) Changpinyo et al. (2021 and
Yi Yang is the corresponding author.

---

Published as a conference paper at ICLR 2023
Wang et al. (2021b) use web-scale image-tex, pairs to train a captioning model and achieve great improvements on MSCOCO (Chen et al. 2015) and NoCaps (Agrawal et al. 2019) through the pretraining-finet mance on MSCOCO, indicating that these methods still rely on human-annotated paired data for
paradigm. However, these models show inferior zero-shot captioning perfor-
fine-tuning. Besides, training with the captioning objective web-scale data is not efficient, e.g., Wang et al. (b) train their model on ALIGN (Jia et al. 2021) and C4 (Raffel et al. 20200) about 1M steps using 512 TPU v3 chips (Jouppi al. 2017).
Instead of directly training a captioning model in an end-to-end manner on web-scale image-text pairs, another line of work (Tewel al. 2022b; Su et al. 2 achieves zero-shot captioning by combining existing pre- pre-trained models. Specifically, they use a pre-trained multi-modal model CLIP (Radford et al., 202T) to guide a pre-trained language model (PI.M,, i.e., GPT-2 (Radford et al. 2019), to generate sentences that match the given image. However, the inference speed of these methods slow because each word generation involves a CLIP text encoder forward. Besides, language models pre-trained various documents from webpages do not match well with caption- ing tasks that aim to describe visual concepts and their relationships a given image, resulting in inferior performance image captioning benchmarks.
In this paper. we propose a new framework, named DeCap, for zero-shot captioning. We aim to decode sensible visual descriptions from the CLIP multi-modal embedding space. We do not use paired image-text data during the decoder pre-training but only leverage the text data. This is more flexible and efficient when the alignment between images and texts became noisier. Our DeCap framework is described below: During pre-training, the text decoder is trained from scratch. The goal is invert the CLIP text encoder, i.e., sentence is first encoded into an embedding by the CLIP text encoder and later reconstructed by our text decoder. The decoder takes the text embed- ding obtained from the CLIP text encoder as the prefix embedding. During zero-shot inference, the difficulty lies in how to obtain prefix embedding that can match the imput image and be well decoded by the decoder. The modality gap phenomenon (Liang et al.] 2022b) is observed in multi- modal contrastive models which prevents us from directly taking the visual embedding as the prefix embedding. |Ramesh et al. (2022) use paired data to learn a model to map the text embedding to corresponding image embedding. Instead of learning model, we propose training- frre mech- anism to project the image embedding into the CLIP text embedding space. Combining the text decoder with the projection mechanism, we generate high-quality descriptions for given images.
Our main contributions are summarized as follows:
(1) We propose new framework for zero-shot captioning. Our DeCap framework contains pre- trained contrastive model (i.e., CLIP) and a lightweight visual-aware language decoder taking the CLIP embedding as input. Though our decoder is trained only on the text corpus, it associate both the visual embedding and the text embedding, thanks to the encoded multi-modal correlation in the CLIP embedding space.
(2) We propose a training-free projection mechanism to reduce the modality gap in CLIP multi- modal embedding space. We incorporate simple support memory containing embeddings of the text corpus in the pre-training stage. We project visual embedding into the CLIP text embedding space via the support memory. Experiments show that our proposed mechanism effectively reduces the modality gap and significantly improves performance.
(3) Extensive experiments demonstrate DeCap can flexibly apply to various captioning scenarios. DeCap outperforms other zero-shot captioning methods by a large margin on image captioning benchmarks MSCOCO and NoCaps. DeCap trained on text-only data outperforms other unpaired captioning methods on MSCOCO and Flickr30k. We apply DeCap to video captioning and achieve state-of-the-art zero-shot results on MSR- STR and ActivityNe:-Catiion.
2
RELATED WORK
CLIP in Captioning. Vision-language models (Radford et al., 2021 Jia et al. 2021 Yang et al., 2022) trained with contrastive loss show impressive ability in many discriminative tasks. However, due to the absence of text decoder during pre-training, these models can not be directly applied to generative tasks, e.g., captioning. Prior work (Mokady et al. 202T: Barraco et al.) 2022 [Shen et al.] 2022) has applied CLIP the image captioning task visual encoder. However, they ignore
2

---

Published as a conference paper at ICLR 2023
Contrastive Loss
Text
Text
Text Emb
Image Emb
Text Decoder
Text Decodar
Aligned
Leccons
Text Encoder
Visual Encoder
Store memory
Training-free
Projection
Text
Visual
Encoder
Text
Image
Wealdy
Text
Image
(1) Contrastive Pre-training (e.g., CLIP)
(2) Decoder Training
(3) Zero-shot Captioning
An overview of our framework. Our method is based pre-trained contrastive model CLIP containing a text encoder and a visual encoder. We first learn a text decoder to generate sentences conditioned on the CLIP text embedding. At inference, training-free mechanism is used to project the image embedding into the text embedding space with the help of a support memory. The projected embedding is further decoded by the text decoder.
Figure 1:
the CLIP encoder and overlook the aligned modti- latent space provided by CLIP. this work, we train a text decoder with text-only data to invert the CLIP text encoder. By leveraging CLIP multi-modal latent space, we apply CLIP captioning tasks without additional pairwise training.
Zero-shot Captioning, Zero-shot captioning aims to generate image/video captions without human- annotated data. Changpinyo al. (202T); Wang et al. (202Tb); |llayrac al.] (2022) train vision- language models on noisy paired image-text data collected from the Web and evaluate down- stream benchmarks without fine-tuning. Another line work achieves zero-shot captioning by combining existing web-scale pre-trained models. ZeroCap (Tewel al. 2022b) combines a multi- modal model (e.g., CLIP) with a PLM (e.g., GPT-2). In each generation step. they use CLIP to guide GPT-2 toward desired visual direction via the proposed CLIP loss. Socratic Models (Zeng et al. al.
2022 use a pre-trained VLM (Gu et al. 2021) to generate prompt templates for GPT-3 (Brown
2020) and then CILP retrieve the closest description to the image from the generated candidates. In this work, we employ CLIP for zero-shot captioning. Different from the above work using PLMs, use text-only data to train decoder from scratch.
Text Reconstruction. Prior work (enng et al. 2019 Laina et al. 2019 Liu et al. 202 b) employ text reconstruction task to train a decoder for unparred//ussuerrises captioning tasks. Lacking well-aligned mudti-modal latent space, most of these methods require complex pseudo-training or adversarial training align the decoder and visual input. [iu et al. (202Tb) construct knowledge graph to correlate the representations of the visual and textual domains. However, this method needs a well-defined knowledge graph and multi-label classification task train the knowledge graph, which is difficult to apply captioning tasks other than medical report generation. Benefiting from CLIP, on the one hand, our decoder can directly associated with visual input by utilizing the aligned cross-modal embedding space of CLIP. On the other hand, our decoder can be trained various text data and applied to various captioning tasks.
METHOD
Our framework is shown in Figure We learn text decoder to convert the CLIP text encoder (Sec. (.I). This text decoder allows us to generate sentences based on the CLIP text embedding. At inference, we propose training-free mechanism to project the image embedding into the text em- bedding space to reduce the modality gap between the text embedding space and image embedding space (Sec. 3.2.I) We introduce more inference strategies for comparison (Sec. 3.2.2.
3.1 TEXT-ONLY DECODER PRE-TRAINING
Previous approaches Tewel et al. 20226 Su et al. 2022 Zeng et al.) 2022) employ PLMs to gen- erate diverse sentences for zero-shot captioning. However, PL.ss trained various documents
3

---

Published as a conference paper at ICLR 2023
from the webpages do not match well with captioning tasks that aim to describe visual concepts and relationships in the given image.
Instead of employing PLM, train text decoder from scratch to invert the CLIP text encoder. Following recent work (Mokady et al, 2021; Wang et al. 2021b), we train our decoder using the prefix language modeling. Specifically, given sentence = /wordd, , prefix language model Pa learns to reconstruct conditioned on the text embedding extracted by a fixed CLIP text encoder. We regard the text embedding prefix the caption. Our objective can be described as:
wordje ) the
Recone (0)=
8 Σ log ([[word:/wodddee (tevt(t)),
where (-emt(-) means mapping sentence to fz-normalized embedding space via the CLIP text encoder. This decoder trained with text-only data in a self-supervised manner brings two benefits. On the one hand, can control the style of the generated sentences by adjusting the source of text-only data. To generate task-spe descriptive captions, we train our decoder on text data from human-annotated image descriptions and web-collected image captions. On the other hand, this text decoder takes CLIP text embedding as the prefix embedding. The CLIP text embedding is optimized be correlated with the CLIP image embedding, making it possible associate the text decoder with visual input without any pairwise training.
3.2 INFERENCE STRATEGIES
we obtain decoder that can generate descriptions conditioned on the CLIP text em- bedding. At inference, the question is how to use the decoder to generate descriptions given the CLIP image embedding. Due to the modality gap between CLIP image embedding space and text embedding space, it is impractical to directly take the CLIP image embedding the prefix embed- ding. Raaess et al. (2022) learn prior model to map the text embedding to a corresponding image embedding. However, this process requires paired data for training. We propose training- mechanism to project the image embedding into text embedding space.
In Sec. 3.1
3.2.1 PROJECTION BASED DECODING (PD)
Assuming that the language model Po is trained on a given text set T {11, 12.., IN,, where N denotes the size of T. To represent the CLIP text embedding space, we maintain support memory M = {m1, m2 .... mNN) ). where m; = Etent(t,). At inference, we aim to generate caption for a given image I. With the help of the support memory M, can project the image embedding into the text embedding space. Specifically, given the image embedding v = FFmmgge ()), we obtain its representation in text embedding space by performing weighted combination of all the embeddings in support memory. To obtain the weights of these text embeddings, the cosine similarity between and m is calculated, scaled by a temperature parameter 7, and normalized by a softmax function. The combined project vector proj is calculated as:
N w m.
N Σ
exp((m, v)/T) Eisn exp((m) (m]v)/T)
V projj
m;,
where w; is the weight of i-th text embedding in support memory. Vproj is combination of CLIP text embeddings that can be used as the prefix embedding. We denote Px(x) as the auto-regressive process of generating a sentence conditioned on the prefix embedding x. The final output can be generated by Po( [
This projection-basseeee based method does not require additional training. It performs well across many datasets and is flexible. The projected vector Vproj can absorb the information from text embeddings in the support memory, thereby generating diverse and accurate descriptions. On the other hand, the text data used for training and stored in support memory can be different. We can select appropriate text data construct new support memory according to the target domain. The image embedding will then be projected into the new text embedding space, enabling DeCap to generalize quickly to new domains without retraining.
4

---

Published as a conference paper at ICLR 2023
3.2.2 DISCUSSION
In order to investigate the impact of our decoder and projection- -baledd----deddd-- mechanism, we have included the following inference strategies for comparative analysis.
1) CLIPRe. We first consider simple retrieval-based approach that does not require a decoder.
This approach is mentioned in (Su et al. (2022). Given the image / and text set T = {41, 22,... (n); CLIPRe retrieves the most relevant texts from based on the image-text similarity measured by CLIP. This process can be formulated as: arg max teT sim sim(Eimage(44 Eteas (t)), where sim de- notes the cosine similarity. In all experiments, we use CLIPRe as our baseline, since can well reflect the zero-shot performance of the original CLIP without the decoder.
2) Visual Decoding (VD). Considering that text embeddings and image embeddings are correlated, simple approach is to directly image embedding the prefix embedding. We refer to this method as Visual Decoding. This process can be formulated as Po( Fimage (T)). However, across the experiments, this method does not achieve satisfying results in most scenarios, indicating that there is modality gap between CLIP image embeddings and text embeddings.
3) Nearest-neighbor Decoding (NND). Another simple method is use the nearest text embedding as the prefix embedding. Specifically, we first calculate the similarity between the image embed- ding Eimage (I) and the text embeddings in M. Then, the nearest text embedding is directly used as the prefix embedding. We refer to this method as Nearest- neighbt Decoding. This process can be formulated as Po arg max sim(Enmage (I), m)). Ideally, NND and CLIPRe should attain similar performance since the decoder learns to recover the origin text conditioned on the text em- bedding. Interestingly, across our experiments, NND achieves better performance than CLIPRe in most scenarios, suggesting that our decode may generate more descriptive sentences. Moreover, find that the performance could be further improved by reconstructing a new text corpus using the decoder. More results and discussions can be found in Appendix B
EXPERIMENTS
We conduct extensive experiments on captioning tasks including zero-shot image captioning, un- paired image captioning. and video captioning. We demonstrate that DeCap can efficiently achieve impressive results in diverse settings. In Sec. 4.1] we focus on zero-shot image captioning without any human annotation. In Sec. 4.2 we focus on unpaired image captioning where the images and the sentences are treated independently. In Sec. 4.3, we further apply DeCap to video captioning we conduct detailed ablation studies for DeCap.
tasks. In Sec.
Implementation Details. We employ frozen pre-trained Vit-B/32 CLIP model. We adopt a 4- layer Transformer (Subramanian et al.) (0188) with 4 attention heads as our language model. The size of the hidden state is 768. By default, we use all the text data in the training set to train the language model from scratch with naive cross-entropy loss. All the text embeddings from the training corpus are stored in the support memory unless specified otherwise. At inference, the temperature in Eq. 2 is set to 1/150 in video captioning experiments, and 1/100 in image captioning experiments. We report the results over four standard captioning evaluation metrics: BLEU @ (Papineni et al.) METEOR (Banerjee & Lavie 2005). CIDEr (Vedantam et al. 2015), and SPICE (Anderson 2016). Additionally, we use CLiP-S (Hessel et al. 2021) and CLIP-S S to measure the text- text similarity and text-image similarity, respectively. The beam search or constrained beam search Anderson et al. 2017) is not used in all our results.
2002
al.
4.1
ZERO-SHOT IMAGE CAPTIONING
In this section, we conduct zero-shot image captioning using webly-collected corpora. Traditional image captioning methods rely on paired obtain and limited in scale and diversity. We consider three webly-collected corpora for DeCap training: (1) CC3M (Sharma et al. 2018) contains three million image-description pairs collected from the web. We only use the text descriptions (CC3M-text) for training. We use one million descriptions randomly sampled from the 3M descriptions to construct the support memory. (2) SS1M is a webly-colleeted corpus specifically designed for MSCOCO caption. Feng et al. (2019) use the name of the eighty object classes in MSCOCO keywords erawl the descriptions from
data for training, which is difficult to
5

---

Published as a conference paper at ICLR 2023
MSCOCO
NoCaps val (CIDEr)
Methods
Pre-training stage
B@94
M
C
S
In
Near
Out
Overall
Changpinyo al
2021
CC3M
29.2
27.5
37.3
29.7
Changinyo al
202T
CC12M
20.7
24.1
41.6
27.1
ZeroCap CLIPRe
CLIP+GPT-2
2.6
11.5
14.6
5.5
CLIP+CC3M-reNt
4.6
13.3
25.6
9.2
23.3
26.8
36.5
28.2
DeCap-VD
CLIP+CC3M-iext
1.2
10.4
8.1
5.8
8.4
8.0
10.2
8,5
DeCup-NND
CLIP+CCEM-tent
5.3
13.7
27.1
9.1
24.2
27.1
37.6
28.8
DeCap
CLIP+CC3M-txt
10.9
49.9
39.7
DeCap
CLIPASSIM
8.9
17.5
50.6
13.1
41.9
41.7
46.2
42.7
DeCap
CLIP+Book Corpus
6.6
12.9
31.9
8.7
26.8
31.8
44.3
33.6
Table 1:
Zero-shot captioning results on MSCOCO Karpathy-test split and NoCaps validation set. (In: in-domain; Near: near-domain; Out: out-of-domain; B@4: BLEU@4; M: METEOR; C:
CIDEr; S: SPICE).
Shutterstock ! resulting in 2,322,628 distinct image descriptions in total. We reuse this corpus and further remove sentences with more than fifteen words, obtaining 978,662 sentences. (3) Book Corpus (Zhu et al.] (0155) a large collection of free novel books. Book Corpus is often used for unsupervised pre-training of language models (Devlin al. 2018)) and we also use it to train our language decoder, but for captioning tasks. The original Book Corpus data is large and many sentences are not visual-related, which makes our decoder training inefficient. practice, we find that the norm of CLIP text embedding can coarsely filter out some sentences that are not related to visual concepts. A sentence with a large norm usually not visual-related. To improve training efficiency, we only keep sentences with lengths less than 15 and norms less than 10 and obtain 6,217,799 sentences for training. We use one million sentences randomly sampled from the training data to construct the support memory. In addition, for the model trained on Book Corpus. We find that DeCap trained on Book Corpus benefits from prompt engineering, whereas DeCap trained CC3M does not. More other prompts, results, and analyses are in Appendix [F
use "Attention! There is/are" as prompt
The following zero-shot captioning methods are compared in this study. Changpinyo al. (2021) train a captioning model on webly-collected paired data and directly transfer it to downstream datasets without fine-tuning. ZeroCap (Tewel et al., 2022b) is training-free zero-shot caption- ing method leveraging CLIP and GPT-2. DeCap also utilizes CLIP but trains a decoder from scratch on webly-collected corpus. Our DeCap uses projection-based decoding (PD) by default. We compare it with another two inference strategies introduced in Sec. 3.22 We denote visual decoding as DeCap-VD VD and neighst decoding as DeCap-NND. All these methods target zero-shot image captioning and do not use human-annotaan annotated data.
Results. Table M shows the zero-shot results on MSCOCO and NoCaps. DeCap attains a new -fate-of-the--rt on all metrics. On NoCaps, models pre-trained webly-collected data achieve better out-of -ft-of-domain results. This is because the webly-collected data contain diverse visual con- cepts. On MSCOCO, DeCap pre-trained on CC3M-text outperforms ZeroCap by 27.5% in CIDEr. DeCap pre-trained on SSIM outperforms ZeroCap by 36% in CIDEr. DeCap trained on SSIM achieves better performance than trained CC3M (CIDEr: 50.6% vs. 42.1%). indicating that the task-specific webly-collected corpus can further improve the performance downstream datasets. Besides. DeCap trained on Book Corpus still achieves better performance than ZeroCap. Notably, both @aCaa-BBooCCrpuu and ZeroCap have not seen caption-related data.
4.2 UNPAIRED IMAGE CAPTIONING
To explore the potential of DeCap in more captioning scenarios, consider the unpaired image captioning setting, where the human-annotated image-sentence pairs are treated as unpaired images and sentences. In Sec. 4.2.II we investigate in-domain captioning where training data and test data come from the same dataset, but the training data are unpaired. In Sec. 4.2.2, we consider the cross-domain situation where training data and test data come from different distributions.
https://ww.shttterrssttt

---

Published as a conference paper at ICLR 2023
Data
MSCOCO
Flickr30K
Method
T. B@4
M
C
S
B B@4
M
C
S
Supervised Methods
BUTD
36.2
27.0
113.5
20.3
27.3
21.7
56.6
16.0
CLIPCap
33.5
27.5
113.1
21.1
Barraco
2022
36.0
27.8
114.9
20.8
CLIP-VL
37.5
28.1
123.1
21.9
Train on unpaired data. Zero-shot inference on image-text pairs
UVC-VI
22.0
21.4
72.3
Feng 2019
18.6
17.9
54.9
11.1
2019
19.3
20.2
61.8
12.9
ESPER-Style
21.9
21.9
78.2
ESPER-Free
6.3
13.3
29.1
ZeroCap"
7.0
15.4
34.5
9.2
5.4
11.8
16.8
6.2
Magic
12.9
17.4
49.3
11.3
6.4
13.1
20.4
7.1
CLiPRe
12.4
20.4
53.4
14.8
9.8
18.2
31.7
12.0
DeCap-VD
5.0
15.5
25.7
9.8
5.8
15.0
13.0
8.2
DeCap-NND
15.3
21.2
62.9
15.8
12.9
17.2
35.2
10.9
DeCap
24.7
25.0
91.2
18.7
21.2
21.8
56.7
15.2
Table 2: In-domain captioning results on MSCOCO and Flickr30K. denotes results from Suu ett al. (2022). "P.", "I." and "T." denote paired data, unpaired image data and unpaired text data, respectively. t: UVC-VI is special approach that requires image-Chinese paired data for training, and we regard it as an unpaired method here because it does not use Image-Engling pairs.
4.2. IN-DOMAIN CAPTIONING
We compare DeCap with supervised methods and other unpaired image captioning methods. (1) Supervised methods: BUTD (Anderson et al.) 2018) is classic method that uses Faster R-CNN (Ren et al. 2013) to extract visual features. CLIPCap (Mokady et al. 2021). CLIP-VL (Shen et al. 2021 and Barraco al. (2022) recent approaches employing CLIP as the visual encoder. (2) Unpaired methods: Laina et al. (2019) and Feng et al. (2019) treat the images and captions from unpaired data. UVC-VI (Liu al. (021a) uses image-Chinese pairs (Wu et al. 2019) for training. (3) (CLIP+GPT2)-basse
the MSCOCO training
methods: ZeroCap (Tewel et al. (022h), Magic (Su al. 2022) and ESPER-Style (Yu al.) 20226) finetune the GPT-2 on captions from the training set. (4) ESPER-Free Yu et al. (022b) uses reinforcement learning to align multimodal inputs language model generations. (5) CLIPRe is retrieval-
baseline. (6) Our DeCap, DeCap-VD and DeCap-NND. Our decoder is trained on captions from the training set, and text embeddings of all the training captions are maintained the support memory.
Results. Table shows the results MSCOCO and Flickr30K, Overall, DeCap outperforms recent unpaired approaches by large margin. Especially on Flickr30K, DeCap competitive with the supervised learning method BUTD. Two conclusions can be drawn: (1) CLIP provides aligned multi-modal representations for captioning tasks. Compared to the unpaired methods that use a visual concept detector to construct multi-modal embedding space, the CLIP- methods could achieve competitive results using only text data. (2) Our decoder and the projection mechanism crucial for high performance. Compared to CLIPRe, DeCap-NND further decodes the nearest- neighbor text embeddings resulting in higher performance, indicating that our decoder can generate more descriptive sentences. DeCap- VDD achieves inferior performance, demonstrating that there is large modality gap between CLIP image embedding and text embedding, demonstrating the necessity our projection mechanism.
4.2.2 CROSS-DOMAIN CAPTIONING
We evaluate the following methods MSCOCO and Flickr30K in the cross-domain setting where the training data and testing data are from different datasets. (1) Zhao et al. (2020) generate pseudo the source domain. (2)
image-text pairs for the target domain using retrieval model trained Magic (Su et al. 2022) finetunes GPT-2 on text data from the source domain. (3) CLIPRe-S uses text data from the source domain as galleries. (4) DeCap trains the decoder text data from the

---

Published as a conference paper at ICLR 2023
source domain. (5) DeCap-TT trains the decoder text data from the source domain and uses captions from the target domain to construct the support memory.
Results. Table 3 shows the results. Unlike the traditional cross-domain method (Zhao et al. 2020 which relies on paired source domain data and requires training on the target domain, recent CLIP- based text-only methods require text-only data from the source domain for training. DeCap signif- icantly outperforms other text-only methods, e.g., Magic (Su et al.) 2022) and CLIPRe-S, on the cross-domain evaluation. Moreover, if the data from the target domain is accessible, DeCap-TT significantly improves the captioning performance (e.g.. CIDEr is improved from 44.4% to 63. (3.1%) without any additional training. simply employs text embedding from the target domain as the support memory. These results demonstrate the strong capabilities of DeCap in cross cross-domain gen- eralization and the effectiveness of our projection-base decoding mechanism.
Data
MSCOCO to Flickr30K
Flickr30K to MSCOCO
Methods
S.T.
T.T.
T.T.
B@4
M
C
Zhao et al. 2020
4
24.1
19.5
52.8
MRB
12.2
17.5
5.2
12.5
18.3
CLIPPR-S
9.8
16.7
30.1
10.3
6.0
16.0
26.5
10.2
DeCap-VD
13.8
13.7
9.4
DeCap-NND
12.0
15.5
28.6
10.1
7.5
15.9
28.0
9.6
DeCap
17.9
18.0
DeCap/TT
17.7
20.2
42.0
13.8
19.7
20.9
63.1
13.9
Table 3: Cross-domain image captioning evaluation. "*" means using CIDEr optimization (Rennie al. 2017). ("S.P.:: Source paired data; "S.T.": Source text data; "T.1.": Target image data; "T.T.": Target text data).
4.3
VIDEO CAPTIONING
In this section, we apply DeCap the video captioning task. We conduct the experiments on MSR- VTT Xu et al., 2016), Activity-Caption (Caba Heilbron et al. 2015). and VATEX (Wang et al, 2019 Notably, we only download 5182 raw test videos out of 6000 VATEX public test videos because some vidcos are unavailable. In Activity-Captions we use ground-trute proposals following Krishna et al. (2017). We apply the same DeCap for video captioning. We consider three different data sources for decoder training: (1) Generic corpus. We train our decoder on Book Corpus which a generic corpus used for unsupervised learning of language models. (2) Image captions. We train our decoder on captions from MSCOCO and CC3M, which are collected
annotated for image captioning tasks. (3) Video captions. We extract the text annotations in the training set of video captioning datasets. The former two can be viewed as the zero-shot video captioning setting without any video-related data for training.
At inference, Specifically, for each proposal, we directly randomly sample & frames f1, f2....f'k from the clip. We use a mean pooling mechanism on the frame-level features extracted by the CLIP image encoder to obtain a video-level feature. In all experiments, k is set to 10.
pooling mechanism on frame-level features obtain video-level feature.
Results. Table 4 shows the results. DeCap trained on image captions outperforms the recent zero-shot captioning approaches on standard captioning metrics and achieves competitive results on CLIP-S and CLIP-SRef metrics. Notably, unlike other methods, DeCap does not directly take CLIP visual-text similarity as the optimization objective. Moreover, DeCap trained on video cap- tions can further improve performance. These results demonstrate that DeCap can easily apply to video captioning with a simple random sampling strategy and temporal mean pooling mechanism.
4.4 ABLATION STUDY
The size of training data. A key question is how much text data we need to train the decoder from scratch. To investigate the effect of training data size, we sample different scale data from MSCOCO. At inference, we use the same support memory (full training set, 560K captions) for all experiments. The results are in Figure (left). Overall, DeCap benefits from a large data size. Compared with training on the full set, the CIDEr score drops from 91.2% 81.5% when using only 1% of data (5.6K captions). The result indicates that DeCap is data-efficient. shows a promising direction in its application in data-limited scenarios.
8

---

Published as a conference paper at ICLR 2023
Methods
Setting
B64
M
CLIP-S
CLIP-
Results on MSR-VTT test
VNS-GRU Chen
2020
Supervised
45.3
29.9
0.739
0.626
SemSynAN (arcc at 2021
46.4
30.4
51.9
0.733
0.619
Triindd on VATEX Chinase
UVC-VI
Wing et 2019
58.9
27.8
44.5
ZemoCap'
2.3
12.9
5.8
0.739
0.710
Tewel al.
2022a
Zero sbot
3.0
14.5
11.3
0.785
0,775
DeCapp BossppBooCCorgg
6.0
12.7
12.3
0.772
0.719
DeCup-
DeCip- C000
14.7
20.4
0.761
0,607
CLIPRe-MSR
10.2
18.8
19.9
0.835
0,852
DcCap-NND-MSR
DcCap-VI-MMSR
MTT text only
13.1
5.9
20.2
16.3
24.4
10.2
0.805
0.766
0.771
0.722
DeCap-MSK
23.1
0.770
Results ActivityNet-
an-teat
Lei
2020
Ressener
Linne Wang erat.
Supervised
12.5
PDVC
11.8
0.750
DeCap-CCIM
Zero-shot
0.7
12.4
0.761
0,814
DeCap COCO
1.1
6.6
15.0
0.727
0.753
CLIPRe-ACT
1.4
8.2
15.1
0.830
0.871
DeCap-NND-ACT
DeCap VD-ACT
ActivityNet- Captions only
1.1
0.682
0.712
DeCip-ATT
2.3
9,4
20.6
0.767
0.797
Reselis on VATEX public tost set
WiTeX (Wang et ala
Supervisel
28.4
21.7
45.1
DoCap-BookCoruus
4.1
9.9
11.8
0.761
0.731
DeCap-CC3M
Zoto-bbot
7.3
12.6
18.4
0.804
0,802
DeCap-COCO
13.1
15.3
18.7
0.769
0.755
CLIPR&-VATEX
11.1
17.0
27.1
0,335
0,877
DeCag-NND-VATEX
DeCap-VD-VATEE
VATEX-Caption text unly
DeCap-VATEX
21.3
20.7
43.1
0.834
0.824
Table 4: Video captioning evaluation results. "1"
denotes the result from Tewel al.
SCCap-BookCorpus DeCap-CC3M, DeCap-COCO, DeCap-MSR, DeCap-ACT and DeCap- DeCap-VATEX VATEX denote the model is trained on text data from Book Corpus, CC3M, MSCOCO,
2022a).
MSR-VTT, Activity-Captions, and DeCap- VATEX, respectively.
DeCap
016
CUPRe
315
11.2
The sbe
Figure 2: Ablation study the training data size (left) and the support memory size (right).
The size of the support memory. To investigate the effect of support memory size, we first train the language model on the full training (560K captions). At inference, we randomly sample different ratio text embeddings as the support memory. The results are in Figure 2 (right). Overall, DeCap and CLIPRe both benefit from a large support memory. Moreover, when using only 1 1% data as the support memory, the performance drops slightly (3.8% performance drop in CIDEr). It indicates that we can maintain relatively small support memory to achieve competitive results with acceptable storage and computation costs. Additionally, we provide filtering strategy to reduce the number of support embeddings in Appendix E, We visualize the support memory and the projection embedding in Appendix G We add an inference speed analysis to Appendix D
5 CONCLUSION
We propose simple framework for zero-shot captioning and introduce a lightweight visual-aware language decoder that is both data-efficient and computation-efficien.. mechanism to project the visual embedding to text embedding space, significantly reducing the modality gap issue. By combining the decoder with the projection mechanism, we significantly outperform existing zero-shot methods, establishing and CedivityNet-Caation In the future, our DeCap framework may be adapted to other zero-shot text generation problems, e.g. visual dialog.
propose training-free
ofate-of-the-att in MSCOCO, MSR-VTT,

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported by National Key R&D 2020AAA0108800. This work is partially supported by the Fundamental Research Funds for the Central Universities (No. 226-2022-00051),
Program of China under Grant No.
REFERENCES
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8948-8957, 2019.
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propo- sitional image caption evaluation. In European conference on computer vision, pp. 382-398. Springer, 2016.
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Em- pirical Methods in Natural Language Processing, pp. 936-945, 2017.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up up and top-down attention for image captioning and visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 6077-6086, 2018.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Manuele Barraco, Marcella Cornia, Silvia Cascianelli, Lorenzo Baraldi, and Rita Cucchiara. The un- reasonable effectiveness of clip features for image captioning: An experimental analysis. Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4662- 4670, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 202.. 2020.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pp. 961-970, 2015.
2015.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3558-3568, 2021.
Haoran Chen, Jianmin Li, and Xiaolin Hu. Delving deeper into the decoder for video captioning. In ECA! 2020, pp. 1079-1086. IOS Press, 2020.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint IrXiv::504 00325,0 2015.
Jacob Devlin, Ming- bidirectional transformers for language understanding. arXiv preprint / 2018.
Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Yang Feng. Lin Ma, Wei Liu, and Jiebo Luo. Unsupervised image captioning. Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 4125-41344 2019.
10

---

Published as a conference paper at ICLR 2023
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabule vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2021.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. CLIPScore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 7514-7528, November 2021.
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi- directional image and text generation. In Proceedings of the 29th ACM International Conference on Mullimedia, pp. 1138-1147 147, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual international symposium on computer architecture, pp. 1-12. 2017.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp- 706-715, 2017.
Iro Laina, Christian Rupprecht, and Nassir Navab. Towards unsupervised image captioning with shared multimodal embeddings. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2019.
Jie Lei, Liwei Wang. Yelong Shen, Dong Yu, Tamara Berg, and Mohit Bansal. Mart: Memory- augmented recurrent transformer for coherent video paragraph captioning. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp.
2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven
Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 0199694-9705 2021.
Chen Liang, Wenguan Wang, Tianfei Zhou, and Yang. Visual abductive reasoning. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15565- 15575, 2022a.
Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Zou. Mind the gap: Un- derstanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv:2203.02053, 2022b.
Fenglin Liu, Xian Wu, Chenyu You, Shen Ge, Yuexian Zou, and Xu Sun. Aligning source visual and target language domains for unpaired video captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021a.
Fenglin Liu, Chenyu You, Xian Wu, Shen Ge, Xu Sun, al. Auto-encoding knowledge graph for unsupervised medical report generation. Advances in Neural Information Processing Systems, 34: 16266-16222 16279, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations, 2018.
Jiasen Lu, Caiming Xiong. Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 375-383, 2017.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0000 1.09344, 2021.
11

---

Published as a conference paper at ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computational Linguistics. pp. 311-318, 2002.
Jesus Merez-Martin, Benjamin Bustos, and Jorge Pérez. Improving video captioning with temporal composition of a visual-syntactic embedding. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp- 3039-3049, 2021.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(140):1-67, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7008-7024, 2017.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565,,265,-7 2018
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit vision-and- lanon-and-lnnguaee tasks? In International Conference on Learning Representations, 2021.
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai- Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can CLIP benefit vision-anddlanguuge tasks? In International Conference on Learning Representations, 2022. /RLLhttps://ppennreee 1d=2f L13H2Way
net fformm?
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models see: Plugging visual controls in text generation. arXiv preprint arXiv:2205.02655. 02665,. 2022.
Sandeep Subramanian, Adam Trischler, Yoshua Bengio, and Christopher J Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. In International Conference on Learning Representations, 2018.
Yoad Tewel, Yoav Shalev, Roy Nadler, Idan Schwartz, and Lior Wolf. Zero-shot video captioning with evolving pseudo-tokens. arXiv preprint 2 2022a.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-tex, gener- ation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17918-17923 2022b.
Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Comelia Carapcea, Chris Thrasher, Chris Buchler, and Chris Sienkiewicz. Rich image captioning in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 49-56, 2016.
12

---

Published as a conference paper at ICLR 2023
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus- image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575, 2015.
Teng Wang, Ruimao Zhang, Zhichao Lu, Feng Zheng, Ran Cheng, and Ping Luo. End-to-end dense video captioning with parallel decoding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6847-6857, 2021a.
Wenhao Wang, Yifan Sun, Zongxin Yang. and Yi Yang. V I: Leveraging vision and vision- language models into large-scale product retrieval. arXiv preprint arXiv:2207. 2022.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and-lanuuage research. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 4581-4591, 2019.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Sim- ple visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2021b.
Jiahong Wu, He Zheng, Bo Zhao, Yixin Li, Baoming Yan, Rui Liang, Wenjia Wang, Shipei Zhou, Guosen Lin, Yanwei Fu, al. Large-scale datasets for going deeper in image understanding. In 2019 IEEE International Conference Multimedia and Expo (ICME), pp. 1480-1485. IEEE, 2019.
Yu Wu, Linchao Zhu, Lu Jiang, and Yi Yang. Decoupled novel object captioner. In Proceedings of the 26th ACM international conference on Multimedia, pp. 1029-1037, 2018.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Uni- fied contrastive learning in image-text-label space. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 19163-19173, 2022.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners image-text foundation models. Transactions Machine Learn- ing Research, 2022a.
net httpr//eeeeeeeaeeeeeeeeeerrreded
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Am- manabrolu, Rowan Zellers, Ronan Le Bras, Gunhee Kim, et al. Multimodal knowledge alignment with reinforcement learning. arXiv preprint arXiv:2205.
2022b.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Puro- hit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00558, 2022.
Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp-
15465-15474, 2021.
Wentian Zhao, Xinxiao Wu, and Jiebo Luo. Cross-domain image captioning via cross-modal re- trieval and model udaptation. IEEE Transactions on Image Processing, 90:1180-1192 2020.
Linchao Zhu and Yi Yang. Actbert: Learning global-local video-text representations. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8746-8755. 2020.
Yukum Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba. and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pp. 19-27, 2015.
13

---

Published as a conference paper at ICLR 2023
A MORE IMPLEMENTATION DETAILS
We employ a frozen pre-trained Vit-B/32 CLIP model as our cross-modal feature extractor. We adopt a lighting 4-layer Transformer (Subramanian et al., 2018) with 4 attention heads as our decoder (hidden state size 768) following the details (Radford et a.. 2019). A linear layer trained with the decoder used project the CLIP embedding from 512 to 768 dimensions. The training data size and
for different datasets are summarized in Table
MSCOCO
FKickr30K
OC3M
SSIM
MSR-VTT
Cetivity-Caation
Book Corpus
Training
560K
30K
Training steps
40K
20K
200K
150K
20K
8K
400K
Warmup steps
2K
2K
2K
2K
2K
IK
2K
Batch
128
Leurning
1cm3
Label smoothing
0.1
Optimizer
AdamW [Lohccillov & Hutter, 2018
with default hyperparamettrrs
Table 5: Training data size and hyper-parameter
B DISCUSSION ABOUT THE RECONSTRUCTION
In Sec. 3.2.2 we introduce the CLIPRe and Nearest-neighbor Decoding (NND) method. Given an image and its CLIP image embedding. both CLIPRe and NND first retrieve the most relative text embedding m, in the support memory according the image-text cosine similarity. CLIPRe then adopts the original sentence t of the m, as the caption. NND uses the decoder to generate a sentence conditioned on m. Ideally, the generated sentence t* should be the same as the original sentence because the decoder learns to reconstruct conditioned on the m,- However, according to the experiments Sec. I.2.1 we find NND outperforms CLIPRe in most metries. To figure out the reason, we conduct the following experiments.
We first train our decoder on the MSCOCO training set with Eq. [ text decoder, we construct new corpus T {14 (..,.., ... original sentence in MSCOCO training set and t* is the reconstructed sentence. We adopt this new corpus as the support memory.
To investigate the effect of the J. where 1" = (((eeezt())) / is the
Table shows the results. The reconstructed dataset improves the performance of CLIPRe on all metrics, especially on CIDEr, from 53.4% to 63.6% (+10.2%) DeCap adopting the new corpus as the support memory could further improve the CIDEr score to 95.1% (+3.9%). The result demon- strates that the sentences generated by method can better describe the images in MSCOCO. We think the reason that our decoding process has a denoising effect, which can remove some out- liers captions in the training Another open question here is whether such denoised dataset can improve the performance of other fully supervised methods. We leave this as our future work.
B@4
M
C
S
CLIPRe
12.4
20.4
53.4
14.8
CLIPRe-recons
14.9 (+2.5)
21.5 (+1.1)
63.6 (+10.2)
16.2 (+1.4)
DeCap
24.7
25.0
91.2
18.7
DeCap-recons
26.5 (+1.8)
24.9 (-0.1)
95.1 (+3.9)
18.6 (-0.1)
Table 6: Results on MSCOCO Karpathy-test split. CLIPRe-recons and DeCap-recons denote using the reconstructed corpus as the support memory.
C PRETRAINING-RINETUIING
An interesting question whether DeCap can benefit from the pretraining pretraining-fmettning paradigm. Table 7 shows the results. Notably, we only text data for training in both pre-training and fine-tuning Compared to training on MSCOCO, the model trained on CC3M achieves better per- formance in the out-of-domain case, improving the CIDEr metric from 25.8% to 48.7% This is
14

---

Published as a conference paper at ICLR 2023
CIDEr
Pre-training data
Fine-tuning data
Memory data
near
our
overall
MSCOCO CC3M
MSCOCO
65.2
47.8
25.8
45,9
CC3M
34.7
35.9
48.7
38.3
MSCOCO
MSCOCO
72.7
61.9
43.9
CC3M
MSCOCO
70.1
60.4
44.5
58.6
Table Results of DeCap on NoCaps validation split. We only use the text data for both pre-training and fine-tuning
because CC3M covers more diverse classes than MSCOCO. By fine-tuning the pre-trained model on MSCOCO, we find that the overall performance is greatly improved, obtaining an overall CIDEr of 58.2%. It indicates that method benefits from the pretraining-fiettuii rectly changing the support memory without fine-tuning, DeCap achieves comparable performance as fine-tuning. It suggests that our method can be easily adapted new domains without training, requiring only some text data from new domains.
paradigm. By di-
D THE INFERENCE SPEED
Table 8 shows the inference speed of DeCap. Decap is 113x faster than ZeroCap. Because DeCap does not involve gradient updates and multiple text encoder forwards during the inference. Besides, the decoder used in DeCap is more lightweight compared to the GPT-2 employed in ZeroCap. It worth mentioning that the time cost of embedding projection is negligible compared to image encoding and text decoding.
Image cnooding
Embodding projection (IM support memory)
Language decoding
All
FPS
(CLIP image encoder)
ZeroCap DeCap
32.68
11285.36 ms
11318.04 ms
0.088
31.75
0.38
68.54
100.67
9.933
Table 8: The inference speed of ZeroCap and DeCap. The experiment is conducted on single Nvidia RTX2080Ti GPU. Both DeCap and ZeroCap do not use the beam search. We report the average time cost of captioning 100 images with batch size
E AN EFFICIENT STRATEGY TO REDUCE THE NUMBER OF SUPPORT EMBEDDINGS
To make DeCap more practical, we provide method that does not degrade DeCap performance but can significantly reduce the number of support embeddings. In the original DeCap, we randomly sample sentences from the training set construct the support memory. However, the semantics between sentences is highly repetitive. A simple but effective method is to filter the features in the support memory according to the cosine similarity. Specifically, given text feature and the existing support memory, if the maximum cosine similarity between the feature and the support memory is greater than a threshold, the feature will not be stored in the support memory. We set the threshold to 0.8 and construct a new support memory with the filtering strategy. Table shows that this strategy can significantly reduce the number of support embeddings from to 0. O. and thus reduce the additional memory cost from 1.02GB to 0.14GB without performance degradation.
Similarity filter
The number of support embeddings
Additional memory cost
CIDEr
False
1.02GB
42.2
False
0.14M (randomly sampled from 1M)
0.14GB
38.2 (-4.0)
True
0.14M (Filtering from 1M)
0.14GB
42.3 (+0.1)
Table 9: The result of filtering strategy. We use the same 1M sentences in this experiment.
15

---

Published as a conference paper at ICLR 2023
F PROMPT ENGINEERING
Prior work (Tewel et al. 2022b Wang et al. 2021b) found that prefix prompt "a picture of improves the quality of decoded captions. We study the effect of the prompt on our special decoder trained with text reconstruction task. We consider two decoders trained on CC3M-texI and Book Corpus, respectively. At inference, we take the "prefix embedding prompts" as the input of the decoder. We test a set of prompts as shown in Table [0] The results show that the decoder trained on Book Corpus benefits from the prompt engineering. while the decoder trained on CC3M hurts from the prompt engineering in most cases. Although CC3M is dataset collected automatically from the Web, well-filtered by some human-designed strategies. Therefore, most of the text data in CC3M caption-related, and the redundant prompt design will destroy its original text structure, resulting in performance degradation. BookCorpus is popular large-scale text corpus, especially for unsupervised learning of language models. Most of the sentences in Bookcorpus are not originally intended to describe pictures. A well-designed prompt can allow the decoder to generate sentences that match the captioning task.
Prompt
DeCap- BookCopus
DeCap-CC3M
None
21.8
42.1 (+0.0)
"A photo of"
20.4 (-1.4)
38.3 (-3.8)
"A picture of" "There is/are"
20.8 (-1.0)
36.8 (-5.3)
27.1 (+5.3)
40.7 (-1.4)
"See! There is/are" "Attention! There is/are"
(+8.6)
40.9 (-1.2)
(+10.))
42.2 (+0.1)
Table 10: Zero-shot captioning results on MSCOCO Karpathy split with different prompts.
G VISUALIZATION OF THE EMBEDDINGS
Figure 3 shows the support embeddings and category embeddings from MSCOCO. The support em- beddings from the clip text encoder are divided into different clusters according to the semantics. In Figure {a), we sample 500 image-text pairs from the MSCOCO training set and visualize their embeddings extracted by the CLIP encoder. As
see, there is clear modality gap between CLIP text embeddings and image embeddings. Figure b)) and Figure c) show that the projection method can effectively reduce this modality gap. Figure d) shows that the projected embedding is close to the embeddings of human-labeled captions in the latent space. Besides, compared to CLIPRe embedding, the projected embedding is more central, indicating that the projected embed- ding absorbs the information of the support embeddings and nicely preserves the visual information.
fire hydrant
girafte
eleghant
Surppoard rd
umprella
kite
skateboard
trishee
bropcoll
narbeg
tennis rocket
support embeddling category embedding
Visualization of support embeddings and category embeddings from MSCOCO in 2D space by t-SNE (Van der Maaten & Hinton 2008). We randomly sample 10,000 embeddings from the support memory for visualization.
Figure 3:
16

---

Published as a conference paper at ICLR 2023
Inagee
Figure 4:
Visualization of embeddings in 2D space by I-SNE. We construct the support memory using text embeddings from MSCOCO training set and randomly sample 500 embeddings from support memory for visualization.
H EXAMPLES OF GENERATED CAPTIONS
We visualize the generated captions of some images from the MSCOCO Karpathy-test split in Figure S We show the captions generated by the DeCap model trained on MSCOCO and CC3M. The captions from DeCap-MSCOCO and DeCap- have visible style differences.
17

---

Published as a conference paper at ICLR 2023
Ground Truth:
Mon stnding in open door of car on desert road.
group of people walking on street by some cars.
A traffic light surrounded by trees near building. traffic light is shown over two street signs
DeCap MSCOCD:
man on vchicle that is desert area
couple women walking street near
vohicle
photography
traffic two
Ground Truth:
bunch zebras stand In some grass
A baseball player squatting down with his mitl rased.
groupp of people wind surting dusk ot beach
MSCocO.
sanding
DeCap-CC3M
rebra the steppe national park- photo
player pitch during game agairst person
person, a surfer, image
Ground Truth:
pands bear with hand to its mouth
smart phone st different angles with windows smart phone shown with a smart phone
blokk and white dog with colar looking out large window dog looking out window and black dog
DeCap MSCOCO:
panda bear sitting enclosure
DeCap-CC3M:
chinese photographed day
smartphone
looking window
digital, new generation media
large laying giant
group people surfing beach
DeCap-MSCOco:
kitchen ccene with stove being cooked
bear resting on a rock in encicsure
pizza with several toppings being cooked pan
DeCap CCEM:
photo of person cooking in the kitchen of
bear, common animal, resting in the medium
piza la cooked in homemade pan,
chosped Ingredients
Figure 5: Generated captions for images from the MSCOCO Karpathy- -test split. and DeCap-CC3M denote DeCap trained on the MSCOCO training set and CC3M training set, respectively.
18


Reviews:
[{"review_id": "uJXNBxKRIBo", "reviewer": "Reviewer_Sz68", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed a zero-shot captioning framework.  To enable the language-pre-trained decoder to generate captions based on visual inputs, the authors project the visual embedding into the text embedding space of CLIP. The authors perform projections with the help of support memory. The projected embedding is then considered as the prefix inputs to the decoder, and ask the decoder to generate captions.\n", "summary_of_the_review": "The paper presents an interesting idea for \"zero-shot\" captioning. The key idea is to use the text embeddings as the support memory to help project visual embedding to the text embedding space of CLIP. The projected embedding is then considered as the prefix inputs to the decoder for generating caption. \n\nOverall, I think the paper is relative positive, and it could be a good reference for future research. The authors present a clever idea to enable such caption capability without training the encoder. Results on multiple benchmarks show that their method achieves better results than other zero-shot or unpaired baselines. \n\nMinor concern is about the term \"zero-shot\". I am not sure if the proposed method is strictly zero-shot, since the decoder have already seen the caption-related text during training. \n\n", "strength_and_weaknesses": "Strengths:\n1. The idea is good and interesting. The use of support memory is interesting.\n2. The paper is well-written, and it addresses practical problems of zero-shot captioning.\n3. The authors demonstrate good results on both image and video captioning benchmarks. \n\nWeaknesses:\n1. The decoder is actually trained using the text from caption datasets. In my opinion, as the model have seen caption-related training data, strictly speaking, it may not be appropriate to call it zero-shot captioning. It would be more convincing if the decoder is pre-trained on generic text corpus, but not image-caption related text corpus.\n2. Although the projection with support memory is encouraging, there is still a noticeable performance gap between supervised and unpaired trained results. These results might indicate that the projections may not keep the original visual information.\n3. While the method could be seen as training-free mechanism, the decoder seems need to be pre-trained using text data. \n4. It will add more values to the paper if the authors provide more in-depth studies of the support memory. Now the authors only discussed about the size of the support memory. For example, what type of embeddings are required as the support memory?\n\nOthers:\n1. There are some latex typos in page 9.\n2. I wonder if the authors can test their method on VATEX, which is a newer, larger video captioning dataset.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "opt0uTqfzKS", "reviewer": "Reviewer_CANm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work aims to utilize the encoded vision-and-language correlation in the trained CLIP embedding space to propose a DeCap method to perform zero-shot captioning. In implementations, this work first constructs an auto-encoder language model to learn to generate the given sentence based on the text embedding encoded by the off-the-shelf pre-trained CLIP. Then, this work further proposes a training-free projection mechanism to project the CLIP's visual embedding into the CLIP's text embedding space, which will be used in the trained auto-encoder language model to generate the final captions. The experiments on the image captioning and video captioning benchmark datasets show that the DeCap can significantly outperform previous methods under comparable settings.", "summary_of_the_review": "Overall, the paper has provided extensive experiments to show that the proposed approach is effective in performing zero-shot captioning. However, the fairness of the experiments should be clarified.\n\n----After Rebuttal---\n\nI've read the rebuttal and the reviews of fellow reviewers. Many thanks for the detailed response. I have no further questions regarding this paper. ", "strength_and_weaknesses": "Strengths:\n1. The explored problem is valuable. The proposed approach is sound. The proposed training-free projection mechanism is interesting.\n2. The proposed approach can significantly outperform previous zero-shot captioning methods.\n\nWeaknesses:\n1. The idea of the proposed approach is similar to the existing work [1]. Meanwhile, the performances achieved by [1] are better than in this work. However, this work neither cites nor compares with [1]. Although it's acceptable that the proposed approach underperforms the previous work [1] in some scenarios, I still recommend the authors discuss the differences between this work and previous work [1] in Introduction and Related Work. It can help the readers better understand the contribution of this work.\n\n2. The fairness of the experiments should be clarified.\n- Compared with previous works, the approach proposed in this work adopts an external strong pre-training model, i.e., CLIP, and several external large-scale datasets, e.g., CC3M, to perform the zero-shot tasks. So I am wondering how many contributions of the achieved best results are brought by the proposed approach, instead of the CLIP and the additional datasets.\n- Could you report the performance of the proposed approach without the CLIP and/or CC3M?\n\n3. Some experiments are missing. \n- Could you perform a detailed ablation study of Equation (2)? That is, what are the performances without m_i^T v, i.e., ∑w_i m_i? What are the performances of P(v_proj) instead of P(v_proj/||P(v_proj)||_2)\n- What are the performances of the fully-supervised DECAP model?\n- Could you visualize the learned representations, e.g., the visual representations, the projected visual representations, and the textual representations? It can better intuitively prove your arguments that the proposed approach can alleviate the modality gap problem.\n\nReference:\n[1] Aligning Source Visual and Target Language Domains for Unpaired Video Captioning. TPAMI, 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "mFG-zJxdt-p", "reviewer": "Reviewer_1suB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problem that the usage of GPT-2 for zero-shot captioning is not reasonable due to the task discrepancy between captioning and language modeling. To this end, this paper proposes the DeCap, which aims to decode sensible visual descriptions from the CLIP multi-modal embedding space. Specifically, the DeCap contains a pretrained contrastive model and a lightweight visual-aware language decoder taking the CLIP embedding as input. Expensive experiments on the MSCOCO, Flickr30k, MSR-VTT, and ActivityNet-Captions datasets show that DeCap can outperform previous state-of-the-art methods.", "summary_of_the_review": "Overall, I think this work is simple but effective, and shows promising results for zero-shot captioning. However, this paper missed some important references and some claims are imprecise. So, I tend to not fully accept the current version of this paper.", "strength_and_weaknesses": "Strength:\n* This paper is well-written and easy to follow. The paper provides sufficient technical details for readers to understand.\n* The results are very promising. The experiments on both zero-shot image captioning and zero-shot video captioning settings show that DeCap can outperform previous state-of-the-art methods.\n* The text reconstruction module in decoder training is simple and effective. And the projection-based decoding is alleviating the modality gap phenomenon.\n\nWeakness:\n* Lack of important references and comparations. The idea of text reconstruction is not the first work in the captioning area. The authors should compare the differences between the DeCap and [1][2].\n* The introduction claims that “the inference speed of these methods is slow because each word generation involves a CLIP text encoder forward”. It is necessary to conduct inference speed experiments to make a claim more convincing.\n* It is not clear how the training-free projection mechanism can be beneficial to the modality gap. I suggest that use T-SNE to visualize the distribution of the v_proj, M, and v, it will be very interesting.\n\n\n[1] Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation. NeurIPS 2021.\n[2] Aligning Source Visual and Target Language Domains for Unpaired Video Captioning. TPAMI 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P7E_WeAFCuA", "reviewer": "Reviewer_oCpD", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied how to use CLIP model for zero-shot captioning, i.e., no human-annotated image-text pairs. Previous works use large language models or pretrain the encoder-decoder network, which may not generate task-specific descriptions or data/computation consuming. This paper proposed a visual-aware language decoder, which (1) uses text data rather than image-text paired data for training, (2) use a memory module to project visual hint to prefix language embedding for inference. Experiments are conducted on image captioning (MSCOCO, Flickr30K) and video captioning (MSR-VTT, ActivityNet-Captions) datasets.", "summary_of_the_review": "Overall, this paper is a good application paper to adapt CLIP for zero-shot captioning. Considering the strengths and weaknesses, I am leaning towards borderline reject.\n\nThe strengths and contributions are (1) its strong performance on zero-shot captioning, (2) the writing quality, (3) the simple but effective idea.\n\nThe weaknesses and concerns are (1) the technical contribution is limited to extra language decoder and text-only data, (2) the analysis of memory bank is not comprehensive, (3) the concern of discarding image data in the image-text paired training data.\n\nI would increase my score if my concerns can be well addressed.\n\n\n================== After rebuttal ======================\n\nAfter reading the authors' rebuttal and other reviewers' comments, I decided to increase my score to 6.\n\nThe authors addressed most of my concerns well, especially the analysis of inference speed, the number of support embeddings, updated claims of motivations and contributions. I hope that the authors could further revise the paper to include these discussion to make the paper more well-motivated and clear.", "strength_and_weaknesses": "Strengths\n\n\\+ The proposed method is simple but effective. Instead of end-to-end training/finetuning or leveraging additional large-scale language model, this work only fine-tune a language decoder on text-only data, which is more efficient.\n\n\\+ Experiments are conducted on both image captioning and video captioning datasets to validate the effectiveness of the proposed method, which is comprehensive and general. The proposed method achieves SOTA performance.\n\n\\+ The paper is well written and easy to follow.\n\nWeaknesses & Questions:\n\n\\- [Major concern] The technical contribution is not strong. Compared to the baseline CLIPRe, the main difference is to include a trainable language decoder. It is not surprising that using an addition module can improve the performance.\n\n\\- [Major concern] An important contribution of this paper is only using text-only data for language decoder training. However, the text data comes from CC3M and SS1M, while the images in the datasets are discarded. It is wired to discarded the images as the image-text pairs can be collected from the web. Also, collecting text-only task-specific descriptions is the same cost/time-consuming as collecting image-text descriptions from the web. It is interesting to know whether the proposed method can achieve better performance using the images to see the performance gap. This helps to clarify the importance of the motivation.\n\n\\- According to Figure 2, the captioning generation relies on a memory bank, whose capacity is over 10^5. Is this volume too large to affect the inference speed? Also, is there any visualization or analysis on the diversity of memory bank? More detailed explanation and analysis on the memory bank are expected. (Typo: \"Figure ??\" in \"The size of the support memory\")\n\n\\- In Section 3.1, the claim that \"adjusting the source of the training data, we can control the style of the generated sentences\" is questionable. Any model can change the generation style by adjusting the training data. It would be great to explain more on this claim.\n\n\\- Should $\\sum w_i * (m^T_i v) m_i$ in Eq. (2) be $\\sum w_i * m_i$?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CfxBQIZvx9", "reviewer": "Reviewer_9VQ6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new pipeline for zero-shot captioning. It first establishes a text decoder to inverse the text embedding from the CLIP to sentences. The authors further develop an embedding projection technique to project the image embedding to a weighted sum of the memorized text embeddings, which can significantly reduce the modality gap. ", "summary_of_the_review": "An interesting study with some novel designs and extensive experiments. Although I appreciate that authors have evaluated their method on a variety of different tasks, I think it still lacks some important experiments, mainly about inference efficiency and sensitivity check. I'll adjust my score after the discussion.", "strength_and_weaknesses": "Strengths:\n\n1. The proposed method is compact yet effective.\n\n2. DeCap outperforms existing work in a zero-shot setting with a noticeable margin. The authors compare algorithms in four different settings: zero-shot captioning, in-domain captioning, out-domain captioning, and video captioning. The results from these experiments prove the superiority of the proposed method.\n\n3. Ablation study provides some useful and interesting discoveries.\n\nWeaknesses:\n\n1. The proposed method seems to cost a lot of time during inference. Authors should provide the memory cost and inference time and compare them with other existing works.\n\n2. The paper should be further proofread.\n\n    - In the ablation study \"The size of the support memory\", the crossref for Figure 2 is missed.\n    - \"does not requires\" -> \"does not require\"\n    - \"ESPER-Free use reinforcement\" -> \"ESPER-Free uses reinforcement\"\n\n3. Is Decap sensitive to the hyper-parameter $\\tau$ is the proposed method? I hope the authors can discuss more the choice of this hyper-parameter.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0GRMYrxlOO4", "reviewer": "Reviewer_MHPt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a method named DeCap, for zero-shot captioning. The method firstly utilize the CLIP text encoder, to train from-scratch a new text decoder that can reconstruct the text sentence input. Thus, the trained text decoder can be attached to a CLIP image encoder to generate captions from visual input. Due to the modality gap between CLIP image encoder and text encoder, the embedding space of images and texts are actually far away. Another contribution from this work is to mitigate the modality gap, by using projection-based decoding. It uses a support memory with text embeddings, and project image embeddings to the support text embeddings and sum up as the condition text embedding for the text decoder. Extensive experiments are conducted on image captioning tasks and video captioning tasks. ", "summary_of_the_review": "This paper proposed a simple idea named DeCap to apply CLIP models to captioning tasks. Due to a few concerns (not useful in practice, lacking in-depth discussions) as mentioned in the previous sections, my initial rating is below the threshold. \n", "strength_and_weaknesses": "Strength:\n\n(1) This paper proposed a very simple and intuitive idea of making use of pre-trained CLIP image and text models for captioning tasks. CLIP was not trained for generative tasks, which has been bridged by a very lightweight text decoder in this work. The demonstration of this idea in the paper is very easy to follow.\n\n(2) Extensive experiments are conducted on both image captioning tasks and video captioning tasks.\n\nWeakness:\n\n(1) The major concern is about the slowness when adopting a large support memory. In section 4.1, the authors mentioned they used one million descriptions randomly sampled from the 3M descriptions to construct the support memory. According to equation 2, this indicates one million calculations for the v_proj embedding vector. It sounds too expensive to be useful in practice. The ablation test in Figure 2 is appreciated, but that doesn't address the concern about the slowness. The speed analysis of the proposed DeCap model is missing.\n\n(2) There lacks a few baselines as discussed in Section 3.2.2 in a few tables. For example, it would be nice to add VD and NND to Table 1, 3 and 4. The reason is that VD baseline doesn't require (expensive) visual embedding projection, if it achieves sometimes competitive results as DeCap, that could be a very strong and solid baseline.\n\n(3) Although the experiments are very broad and cover diverse tasks, the analysis of the proposed method is not thorough. A few questions are still remaining after reading this paper: (a) what would be the results without projection-based decoding (PD)? I can imagine it's not doing great according to Figure 2, but it's nice to add this missing number to make it complete. (b) Are there any other methods you have tried other than PD, to bring the image and text embeddings closer? (c) How does DeCap perform in a supervised or few-shot setup? For example, could DeCap benefit from additional supervised paired image and text (i.e. in the support memory)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster