MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
Year: 2023

Abstract:
Supervised visual captioning models typically require a large scale of images or videos paired with descriptions in a specific language (i.e., the vision-caption pairs) for training. However, collecting and labeling large-scale datasets is time-consuming and expensive for many scenarios and languages. Therefore, sufficient labeled pairs are usually not available. To deal with the label shortage problem, we present a simple yet effective zero-shot approach MultiCapCLIP that can generate visual captions for different scenarios and languages without any labeled vision-caption pairs of downstream datasets. In the training stage, MultiCapCLIP only requires text data for input. Then it conducts two main steps: 1) retrieving concept prompts that preserve the corresponding domain knowledge of new scenarios; 2) autoencoding the prompts to learn writing styles to output captions in a desired language. In the testing stage, MultiCapCLIP instead takes visual data as input directly to retrieve the concept prompts to generate the final visual descriptions. The extensive experiments on image and video captioning across four benchmarks and four languages (i.e., English, Chinese, German, and French) confirm the effectiveness of our approach. Compared with state-of-theart zero-shot and weakly-supervised methods, our method achieves 4.8% and 21.5% absolute improvements in terms of BLEU@4 and CIDEr metrics. Our code is available at https: //github.com/yangbang18/MultiCapCLIP.

heading: 1 Introduction
text: Visual captioning targets to first 1) understand the information of visual inputs, which are typically videos or images, and then 2) produces a corresponding textual sentence describing the visual objects/attributes/relationships. Visual captioning has drawn remarkable attention from natural language
∗Equal contribution. †Corresponding authors.
processing and computer vision fields due to its wide applications, e.g., cross-modal retrieval (Luo et al., 2022; Cheng et al., 2023b) and helping the visually impaired (Çaylı et al., 2021). Currently, visual captioning models based on the encoderdecoder framework (Huang et al., 2020; Liu et al., 2020; Yang et al., 2021; Zhang et al., 2021; Hu et al., 2022; Lin et al., 2022) have achieved tremendous progress in advancing the state-of-the-art. These models are usually trained with full supervision and rely on large-scale humanly-annotated training data (i.e., vision-caption pairs), which needs expensive labeling work. In particular, when it comes to Non-English caption systems, it is challenging to collect and label sufficient visioncaption pairs in a timely manner, which prevents such encoder-decoder models from rapid deployment in different scenarios and languages.
To deal with the shortage of labeled pairs, we propose the MultiCapCLIP - a prompt-based natural language auto-encoder. As shown in Figure 1, MultiCapCLIP only requires textual input for training, and it can conduct zero-shot multilingual visual captioning, including image and video captioning. Therefore, MultiCapCLIP can deal with the situation where the labeled vision-caption pairs are missing. MultiCapCLIP is particularly suitable for new scenarios and languages, improving the practical value of visual captioning.
To implement MultiCapCLIP, we first adopt a pre-trained vision-language model, i.e., CLIP (Radford et al., 2021), as our backbone. CLIP has shown success in correlating the visual and textual modalities into the same latent space (vision-language embedding space) (Tewel et al., 2022b; Su et al., 2022; Zeng et al., 2023). We observe two critical issues for zero-shot visual captioning: the understanding of domain visual knowledge (e.g., objects, attributes, and relationships) and the generation of descriptive sentences in a specific writing style and language. Therefore, we propose a prompt-based
11908
auto-encoder, which introduces the visual concept prompts P to preserve the corresponding domain knowledge and writing styles of zero-shot visual captioning. During training, given the text-only data, we train the model by reconstructing the caption S in the S → P → S auto-encoding pipeline. Since the auto-encoding process reconstructs the same input sentence, the model training needs only unlabeled text data. In the reconstruction process, the model is able to preserve the necessary domain knowledge and the writing styles of visual captioning (Wang et al., 2016; Tschannen et al., 2018). During inference, we can directly take the vision input V as queries to retrieve the domain knowledge preserved in the visual concept prompts and finally rely on the learned writing styles in a specific language in the text decoder to generate visual descriptions in the V → P → S pipeline.
Meanwhile, to further bridge the modality gap between the visual and textual data (Liang et al., 2022), we introduce an augmentation method, including input augmentation and feature augmentation, which can boost the robustness of the model and in turn improve the performance of zeroshot visual captioning. The experiments on four benchmark datasets, i.e., MS-COCO (Chen et al., 2015), MSR-VTT (Xu et al., 2016), VATEX (Wang et al., 2019), and Multi30K (Elliott et al., 2016), show that our approach can accurately and dataefficiently generate visual captions in English, Chinese, German, and French.
Overall, our main contributions are as follows:
• We propose a simple yet effective approach MultiCapCLIP that requires no downstream labeled data to make the first attempt for zeroshot multilingual visual captioning.
• MultiCapCLIP first introduces visual concept prompts to preserve the domain knowledge and then auto-encodes them to learn the writing styles of captioning. After text-only training, our approach can shift from text-to-text generation to vision-to-text generation.
• The out-of-domain and in-domain experiments on image and video captioning across different languages show that our approach trained on text-only data significantly outperforms previous zero-shot/weakly-supervised methods trained on unpaired or partial labeled visual and textual data, setting new state-ofthe-art zero-shot performance.

heading: 2 Approach
text: In this section, we first give a brief review of CLIP, whose vision-language embedding space lays a foundation for our approach. Next, we introduce the framework of the proposed MultiCapCLIP, followed by two key components: concept prompts and textual augmentations.

heading: 2.1 A Brief Review of CLIP
text: CLIP uses two independent encoders to process image and text input separately and then bridges the gap between modalities with contrastive learning. The image encoder ϕv(·) can be a convolutional neural network like ResNet (He et al., 2016) or a vision Transformer like ViT (Dosovitskiy et al., 2021), and it extracts a feature vector for each input image. The text encoder ϕt(·) is based on Transformer (Vaswani et al., 2017), and it outputs a vector representation of the input text. By training two encoders on 400M imagetext data with noisy correspondences under the InfoNCE objective (Oord et al., 2018), CLIP learns a powerful vision-language embedding space that measures image-text similarity well and enables open-vocabulary classification. In this paper, we re-purpose CLIP for zero-shot multilingual visual captioning and always keep ϕv(·) and ϕt(·) frozen.

heading: 2.2 Overview of MultiCapCLIP
text: As shown in Figure 1, MultiCapCLIP consists of visual and textual encoders from CLIP and a trainable Multilingual Language Model (MLM). MultiCapCLIP supports English text1, images or videos as inputs and can produce output in desired language. Specifically, we implement MLM with a stack of Transformer decoder blocks, each of which comprises a masked self-attention layer, a crossattention layer, and a feed-forward layer. Moreover, we add explicit signals in the embedding layer to indicate which language to be generated.
Let denote the text input as S, the vision input as V , and concept prompts as P . Unlike typical visual captioning models that are trained on a vision-text dataset, MultiCapCLIP relies on a text dataset and follows the S → P → S auto-encoding pipeline during training. Based on the semantic alignment characteristic of CLIP’s feature space, MultiCapCLIP uses the V → P → S pipeline for visual captioning during inference. We extend MultiCapCLIP to support multilingual text generation by us-
1The training corpora for CLIP is mainly in English.
ing parallel corpora with (S, T ) pairs, where T denotes the target text in a desired language. In such a case, MultiCapCLIP follows the S/V → P → T translation pipeline.
In the following, we will detail how to extract and leverage P in Section 2.3. Then in Section 2.4, we will introduce an augmentation method to improve the training of MultiCapCLIP.

heading: 2.3 Decoding with Concept Prompts
text: A set of visual concepts is a good embodiment of domain visual knowledge because a visual concept (e.g., “a young girl”) manifest as the explicit clue in the vision input. Given a pure text dataset, we use the spaCy toolkit2 to extract noun phrases and reserve the most frequent 1,000 noun phrases as visual concepts, which are first embedded into a prompt template “{concept}”3 and then fed into the CLIP’s text encoder ϕt(·) to extract l2-normalized concept features C = {c1, . . . , c1000}.
During training, given the text input S, we first encode it into a global feature ft:
ft = Norm(ϕt(S)), (1)
where Norm(·) denotes L2 normalization. Next, we calculate the dot product of ft and C to measure cosine similarities, based on which we obtain soft concept prompts P , a subset of C that includes K concept features most semantically similar to ft. Assuming that the dimension of vectors outputted by CLIP is d, P is in the shape of K ∗d. To prompt MLM, we prefix embeddings of the target text S
2https://spacy.io 3The simplest prompt template “{concept}” produced better performance than other templates like “a concept of {concept}” in our preliminary experiments.
with P to obtain the final input embeddings E:
E = Concat(ω(P ), e(S)) (2)
where ω(·) is implemented as a fully connected layer followed by a layer normalization (LN) (Ba et al., 2016), and e(·) denotes the summation of position, language, and token embeddings for each si ∈ S, followed by LN. The prompt sequence generated by ω(P ) and token sequence generated by e(S) are concatenated together and sent to the text decoder of MLM to regenerate the input sequence S. Considering that ft may contain information supplementary to P , we do not discard ft. We first feed the projected feature f = ω′(ft), where ω′(·) has the same structure as ω(·) but shares no parameters, into the text decoder of MLM. Then we calculate the cross attention between f and E. We train the model with a cross-entropy loss:
L = − |S|∑
i=1
log pθ(s = si|S<i, P, ft), (3)
where pθ(·) is MLM’s predicted distribution over a vocabulary and θ denotes all trainable parameters.
During inference, we process the vision input V in a similar manner, except that we use CLIP’s image encoder ϕv(·) to obtain V ’s vector representation fv and obtain relevant concept prompts P based on (averaged) image-concept similarities. Given the previously generated text S<i, the prediction of the next token is based on the following probability distribution:
pθ(s|S<i, P, fv). (4)

heading: 2.4 Training with Augmentations
text: Our MultiCapCLIP’s ability of shifting text-to-text generation to vision-to-text generation is built on
the assumption that the paired vision-text data is well aligned in the vision-language embedding space of CLIP. However, Liang et al. (2022) demonstrated that there exists modality gap in CLIP-like models and such gap has a significant impact on model generalization ability. To this end, inspired by denoising auto-encoders (Vincent et al., 2008), we propose to train MultiCapCLIP with augmented text features f ′t . Here we consider both the input augmentation (IA) and the feature augmentation (FA). Specifically, IA replaces the source text S with a semantically similar one S′ to obtain f ′t :
f ′t = Norm(ϕt(S ′)), (5)
where S′ ∼ XS and XS = {S, S′1, . . . , S′N−1} denotes the candidate set of S. For simplicity, we use ϕt(·) to measure similarities among text in the dataset and select the most similar N−1 text to construct XS for each S. Since we sample text from XS with uniform probability, there will be 1/N probability that the input text keeps unchanged. As for FA, we follow Li et al. (2021) to add Gaussian noise n ∼ N (0, ϵ) into text features. Hence, Eq. (5) can be further extended to:
f ′t = Norm(Norm(ϕt(S ′)) + n). (6)
During training, we replace ft in Eq. (1) with f ′t in Eq. (6) to encourage the model to learn more robust latent representations.

heading: 3 Main Experiments
text: In this section, we first introduce the datasets, metrics, settings of the experiments; Then, we provide the out-of-domain and in-domain results of our approach for zero-shot visual captioning.

heading: 3.1 Experimental Setups
text: Datasets. As shown in Table 1, we use three benchmark datasets under CC BY 4.0 licence in
this section: MS-COCO (Chen et al., 2015), MSRVTT (Xu et al., 2016), and VATEX (Wang et al., 2019). We apply the Karpathy and Fei-Fei’s (2015) split to MS-COCO and follow the official split of MSR-VTT for English captioning. Besides, VATEX is a multilingual video captioning dataset that contains parallel English-Chinese captions. We use it for Chinese captioning4. In Section 4, we will further use the Multi30K dataset (Elliott et al., 2016) for German and French caption generation.
Metrics. Following the common practice in the literature, we report BLEU@4 (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), ROUGE-L (Lin, 2004) and CIDEr (Vedantam et al., 2015) for video captioning, and additionally measure SPICE (Anderson et al., 2016) for image captioning. All metrics are computed by Microsoft COCO Evaluation Server5 (Chen et al., 2015).
Settings As shown in Table 2, we conduct the out-of-domain and in-domain experiments. 1) Outof-Domain Experiments are performed by training the model on the text-only data of A dataset, and then evaluating on the B dataset. 2) In-Domain Experiments are conducted by training the model on the text-only data of A dataset, and then evaluating on the A dataset.
4The official splits of VATEX is 25,991: 3,000: 6,000. However, some video clips are no longer available, resulting in the splits 25,006: 2,893: 5,792. Besides, VATEX does not provide Chinese captions of the test set (https://eric-xw. github.io/vatex-website). We construct new validation and test sets from the original validation set.
5For non-English captioning, we do not report METEOR and SPICE metrics because their implementations consider synonym matching and named entity recognition in English.
Baselines Since previous works can not generate zero-shot multilingual visual captions directly, we implement a zero-shot CLIP-based model: ZSCapCLIP, which is trained on text-only data with the same architecture as our MultiCapCLIP but without our proposed concept prompts and text augmentations. To observe the gap between zero-shot and fully-supervised methods, We also implement CapCLIP trained on vision-caption pairs.
Implementations. Following the previous works in zero-shot captioning (Tewel et al., 2022b; Su et al., 2022; Zeng et al., 2023), we adopt the CLIP (ViT-B/16 variant) (Radford et al., 2021) as our image encoder and text encoder, and adopt a randomly initialized Transformer-BASE (Vaswani et al., 2017) as our language decoder. We adopt the same vocabulary as BERT / multilingual BERT (Devlin et al., 2019) for English / non-English captioning. We use the Jieba toolkit6 to segment Chinese sentences. We select the hyperparameter K from values {4, 8, 16, 32}, N from values {5, 10, 20} and ϵ from values {0.01, 0.1, 1.0} according to the CIDEr performance on the validation split, and set K = 16, N = 5, ϵ = 0.01 for all datasets and settings except that ϵ = 0.1 for in-domain experiments on MS-COCO. During training, we apply label smoothing (Szegedy et al., 2016) of 0.1, use batches of 32 samples and AdamW (Loshchilov and Hutter, 2019) with L2 weight decay of 0.01 to train models for 10 epochs. We set the learning rate fixed to 1e-4 with 10% warm-up iterations when training on text-only data. During inference, we use beam search with a beam size of 3 to generate captions.
6https://github.com/fxsjy/jieba

heading: 3.2 Out-of-Domain Results
text: In this section, we evaluate the zero-shot multilingual captioning performance of our approach under out-of-domain settings. We can notice from Table 3 that our zero-shot model MultiCapCLIP achieves competitive performance on three datasets across English and Chinese. Although SGM (Honda et al., 2021) and R2M (Guo et al., 2020) perform better than our model on CIDEr and SPICE metrics on MS-COCO, they require the large-scale image datasets for training and use a larger training corpus (2.3M sentences) than ours (130K sentences). While the previous methods do not target nonEnglish caption generation, our MultiCapCLIP gains obvious relative improvements against the CapCLIP on VATEX Chinese captioning. The outof-domain results show that our approach is able to generate multilingual visual captions without any labeled vision-caption data, which could have the potential to promote the application of visual captioning for low-resource language applications.

heading: 3.3 In-Domain Results
text: For comparisons, we further consider state-of-theart fully-supervised and large-scale pre-trained models and models under the unpaired setting, i.e., both vision and text data of the target dataset are utilized for training independently, leaving their pairing annotations unused. As shown in Table 4, our approach significantly outperforms previous unpaired/zero-shot competitors by up to 4.8% BLEU@4, 3.9% ROUGE-L, and 21.5% CIDEr scores in MS-COCO English captioning. When it comes to MSR-VTT English captioning and VATEX Chinese captioning, our MultiCapCLIP surpasses ZS-CapCLIP by a large margin under the CIDEr metric, e.g., an absolute improvement of
Settings Methods MS-COCO (English) MSR-VTT (English) VATEX (Chinese)
B@4 M R-L C S B@4 M R-L C B@4 R-L C Fu lly -
Su pe
rv is ed VATEX (2019) - - - - - - - - - 23.4 46.0 39.4 MAD+SAP (2020) 37.0 28.1 57.2 117.3 21.3 41.3 28.3 61.4 48.5 - - - Oscarbase (2020) 36.5 30.3 - 123.7 23.1 - - - - - - - OpenBook (2021) - - - - - 42.8 29.3 61.7 52.9 - - - ClipCap (2021) 33.5 27.5 - 113.1 21.1 - - - - 28.3∗ 49.5∗ 51.3∗ CapCLIP∗ (2021) 32.3 27.7 55.4 109.5 20.7 42.9 29.8 62.3 54.5 29.7 49.8 51.0 CaMEL (2022) 39.1 29.4 58.5 125.7 22.2 - - - - - - - LEMONbase (2022) 40.3 30.2 - 133.3 23.3 - - - - - - - SwinBERT (2022) - - - - - 45.4 30.6 64.1 55.9 - - - CLIP-DCD (2022a) - - - - - 48.2 31.3 64.8 58.7 - - - MV-GPT (2022) - - - - - 48.9 38.7 64.0 60.0 - - - GIT (2022a) 44.1 31.5 - 144.8 24.7 53.8 32.9 67.7 73.9 - - -
W ea
kl y-
Su pe
rv is
ed
UIC (2019) 18.6 17.9 43.1 54.9 11.1 - - - - - - - IC-SME (2019) 19.3 20.2 45.0 61.8 12.9 - - - - - - - Graph-Align (2019) 21.5 20.9 47.2 69.5 15.0 - - - - - - - IGGAN (2020) 21.9 21.1 46.5 64.0 14.5 - - - - - - - TSGAN (2021) 18.9 18.2 43.3 55.2 11.3 - - - - - - - USGAE (2022c) 17.1 19.1 43.8 55.1 12.8 - - - - - - - SCS (2022) 22.8 21.4 47.7 74.7 15.1 - - - - - - -
22.5% on MSR-VTT. These results prove the effectiveness of MultiCapCLIP in zero-shot multilingual visual captioning. Nevertheless, there still exists performance gaps between MultiCapCLIP trained on text-only data and existing state-of-the-art fullysupervised models trained on full vision-text data.

heading: 4 Analysis
text: In this section, we conduct several analyses to better understand our approach.

heading: 4.1 Semi-Supervised Visual Captioning
text: To further prove the effectiveness of our approach, we fine-tune MultiCapCLIP with partial labeled vision-caption data of downstream datasets. To this end, in Figure 2, we evaluate the performance
of MultiCapCLIP with respect to the increasing amount of labeled data. Specifically, we randomly sample a small portion of training images/videos and use the resulting vision-caption pairs for finetuning. We repeat this process by three times and report the average performance. For a fair comparison, we also train CapCLIP (Section 3.1) with the same amount of pairs. As we can see in Figure 2, for both in-domain or a out-of-domain corpus, MultiCapCLIP consistently outperforms CapCLIP with different ratios of training data. It is worth noting that the fewer the labeled vision-caption pairs, the larger the margins. In detail, under the extremely low label setting, e.g., 0.1% of paired data on MSR-VTT (only 6 videos), our approach under the in-domain setting significantly surpasses the
CapCLIP by 23.1% absolute BLEU@4 score. It further proves the effectiveness of our approach, which can relax the reliance on the vision-caption annotations. We can make use of available unpaired text-only data as a solid basis for multilingual visual captioning tasks.

heading: 4.2 Quantitative Analysis
text: In this section, we analyze the contributions of each component in our approach.
Ablation Study We conduct the ablation study on the out-of-domain and in-domain settings using MS-COCO dataset (Chen et al., 2015). As shown in Table 5, each component in our proposed approach can boost the performance over all metrics, verifying our arguments and the effectiveness of our approach. In particular, setting (a) shows that the introduced prompts can improve the base model with absolute gains up to 5.9% and 13.3% CIDEr scores under out-of-domain and in-domain settings, respectively. Settings (b,c) show that either input augmentation (IA) or feature augmentation (FA) respectively boost performance, indicating the importance of bridging the modality gap between the visual and textual data and in turn, boosting the robustness of the model and improving the performance of zero-shot visual captioning. Moreover, by comparing the results of (b) and (c), we observe that FA brings more improvements under the in-domain setting whereas IA is better under the out-of-domain setting. This indicates that structure noises are more suitable to bridge the modality gap between vision and text data from the same domain. From another perspective, we need a more com-
plex feature adaptation method for out-of-domain transfer. Since the IA and FA can improve the performance from different perspectives, as shown in setting (d), combining them can lead to the most prominent improvement across all metrics. Moreover, compared with (d), our full model in the setting (g) can still gain improvements under most metrics, especially the CIDEr metric, showing that concept prompts benefit visual captioning by generating more accurate details.
Effect of K As shown in Table 5 (e-h), when we set the number of prompts K to 16, the model substantially performs the best. For other K values, when K < 16, the performance is improved with an increasing K due to more adequate guidance signals to the model. However, when K > 16, we can observe saturated or impaired captioning performance, possibly because retrieving more prompts do not include additional useful clues and introduce irrelevant noises to the model.
Concept Type Other than prompting the model with noun phrases (Section 2.3), we also consider the effect of verbs. As shown in Table 5, setting (g) surpasses settings (i) and (j) at most cases, i.e., using verb-based prompts degrades performance. We speculate the reason is that the vision-language model we used (i.e., CLIP) can recognize salient objects more accurately than human actions.

heading: 4.3 Robustness Analysis: Extensions to More Non-English Languages
text: We adopt the Multi30K dataset (Elliott et al., 2016) to further evaluate in-domain performance on German and French image captioning. As shown in
Table 6, our full model again outperforms the base model by a large margin, proving the effectiveness of concept prompts and text augmentations.

heading: 4.4 Qualitative Analysis
text: In this section, we give some visualization results and examples to better understand our approach.
Visualization To verify the effect of our method on representation learning, we use t-SNE (van der Maaten and Hinton, 2008) to visualize the features produced by ZS-CapCLIP and our MultiCapCLIP in Figure 4, which shows that our approach can bridge the modality gap between visual and textual inputs during training and obtain a blended distribution, leading to a more robust shift from text-to-text generation to vision-to-text generation.
Examples In Figure 3, we compare our model trained with out-of-domain corpora with CapCLIP trained on full in-domain supervision. As we can see, our model can generate accurate keywords, e.g., “sand” in (a), “tire” in (c), and “helmet” in
(d), which can be attributed to the useful clues of concept prompts. However, there exist noises in the retrieved concepts in some cases, e.g., “a punching bag” in (b), misleading the model to produce wrong details. Besides, in (e), we can observe how the training corpus affect the writing style of the model: the corpus of a video caption dataset (VATEX) makes the model focus on the temporal evolution of events, resulting in a speculative description “the catcher catches the ball”. Overall, our approach can be a solid basis for zero-shot multilingual visual captioning. It requires no vision-caption pairs but generates plausible visual descriptions.

heading: 5 Related Works
text: The related works are introduced from zero-shot learning and visual captioning.
Zero-shot Learning Adapting models to novel tasks with limited labeled data is an important research topic toward general intelligence (Griffiths et al., 2019). Contrastive pre-training is an effective technique to achieve this goal and has revolutionized multimodal research (Hou et al., 2021; Gan et al., 2022; Jin et al., 2022; Cheng et al., 2023a). Specifically for the vision-language field, models such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) learn a shared multimodal embedding space from large-scale noisy image-text pairs, leading to an impressive zero-shot performance on tasks like image classification and vision-text retrieval (Zhang et al., 2022; Luo et al., 2022). Nevertheless, employing CLIP-like models in low-data vision-grounded text generation (i.e., visual captioning) remains challenging.
Visual Captioning As a key vision-language task, visual captioning has achieved tremendous progress under the encoder-decoder framework (Xu et al., 2015) and the “pre-training and fine-tuning” paradigm. Yet, typical visual captioning methods require curated datasets of numerous images or videos paired with descriptions in a specific language, which are costly to collect. To this end, some weakly-supervised approaches are proposed (Feng et al., 2019; Guo et al., 2020; Honda et al., 2021; Ben et al., 2022). These methods require disjoint vision and text data for training and rely on a pre-trained object detector like Faster R-CNN (Ren et al., 2015) to construct weak supervision signals. However, the detectors they use are limited to a pre-defined set of categories. Recently, several works integrate CLIP with large language models (LLMs) like GPT (Radford et al., 2019; Brown et al., 2020) for zero-shot visual captioning (Tewel et al., 2022b; Su et al., 2022; Liu et al., 2022; Zeng et al., 2023). Although effective, these methods suffer from over-parameterization of large LLMs. We instead train a lightweight decoder from scratch. Besides, some concurrent works address zero-shot visual captioning by training CLIP with text-only data (Nukrai et al., 2022; Gu et al., 2022; Li et al., 2023; Yang et al., 2023). What differentiates our work from them is that we consider visual concept prompts that perverse domain visual knowledge.

heading: 6 Conclusions
text: We have presented a data-efficient method dubbed MultiCapCLIP to re-purpose CLIP-like visionlanguage pre-trained models for zero-shot multilin-
gual visual captioning. Our approach reduces the reliance on labeled vision-caption pairs of downstream datasets by auto-encoding concept prompts on text-only data. Extensive experiments on four datasets and four languages confirm the effectiveness of our approach, which can be a solid basis for visual captioning in low-data regimes and lowresource languages.
Limitations
Although the proposed MultiCapCLIP can generate multilingual zero-shot visual captions without any labeled vision-caption training pairs. We still need the independent set of text for training/translating, which may still be difficult to collect for some lowresource languages. This might be alleviated in the future with techniques such as knowledge distillation from publicly-available pre-trained models, e.g., BERT (Devlin et al., 2019). Besides, our approach uses CLIP to measure text-text similarities for retrieving concept prompts and conducting input augmentation during training. Considering that CLIP is optimized by image-text global contrast (Radford et al., 2021) and intra-modal retrieval of such a model is not as well as its cross-modal retrieval (Jia et al., 2021), an improvement direction of our approach is using a vision-language pre-trained model that measures intra-modal and inter-modal semantic similarities well (Yang et al., 2022b).
Ethics Statement
We conduct the experiments on public datasets, which are exclusively about natural images, videos, and captions. These datasets have been carefully pre-processed for the academic study purpose, and therefore do not contain any information that names or uniquely identifies individual people or offensive content. It is noteworthy that our approach inherits the drawback of the pre-trained backbone, i.e., CLIP, which has demonstrated that improper class design used for prompting may raise unwanted biases (Radford et al., 2021). Therefore, careful examination is needed before employing our approach in real-world scenarios to avoid prejudices.

heading: Acknowledgements
text: This paper was partially supported by NSFC (No: 62176008) and Shenzhen Science & Technology Research Program (No: GXWD2020123116580700720200814115301001).

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Please see Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Please see Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Please see Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Please see Section 3.1

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Please see Section 3.1

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Please see Section 3.1

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Please see Section 3.1

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? We carry out a detailed anonymization process. We manually examine the data of widely adopted benchmark datasets. If there exists information that names individual people, we replace it with expressions like "he", "she", and "a person".

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Please see Table 1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Please see Table 1
C 3 Did you run computational experiments? Please see Sections 3 and 4

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Please see Section 3.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Please see Section 3.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Please see Section 4.1

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Please see Sections 2.3 and 3.1
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This might be alleviated in the future with techniques such as knowledge distillation from publicly-available pre-trained models, e. g. , BERT (Devlin et al. , 2019). Besides, our approach uses CLIP to measure text-text similarities for retrieving concept prompts and conducting input augmentation during training. Considering that CLIP is optimized by image-text global contrast (Radford et al. , 2021) and intra-modal retrieval of such a model is not as well as its cross-modal retrieval (Jia et al. , 2021), an improvement direction of our approach is using a vision-language pre-trained model that measures intra-modal and inter-modal semantic similarities well (Yang et al. , 2022b)