Weakly Supervised Vision-and-Language Pre-training with Relative Representations
Year: 2023

Abstract:
Weakly supervised vision-and-language pretraining (WVLP), which learns cross-modal representations with limited cross-modal supervision, has been shown to effectively reduce the data cost of pre-training while maintaining decent performance on downstream tasks. However, current WVLP methods use only local descriptions of images, i.e., object tags, as cross-modal anchors to construct weaklyaligned image-text pairs for pre-training. This affects the data quality and thus the effectiveness of pre-training. In this paper, we propose to directly take a small number of aligned image-text pairs as anchors, and represent each unaligned image and text by its similarities to these anchors, i.e., relative representations. We build a WVLP framework based on the relative representations, namely RELIT1, which collects high-quality weakly-aligned imagetext pairs from large-scale image-only and text-only data for pre-training through relative representation-based retrieval and generation. Experiments on four downstream tasks show that RELIT achieves new state-of-the-art results under the weakly supervised setting2.

heading: 1 Introduction
text: Vision-and-language pre-training (VLP) (Chen et al., 2020; Zhang et al., 2021; Kim et al., 2021; Radford et al., 2021; Wang et al., 2022a) has received increasing attention in recent years for its great success on various vision-and-language tasks, such as visual question answering (Antol et al., 2015), cross-modal retrieval (Plummer et al., 2015), and image captioning (Lin et al., 2014). Different from other foundation models (Bommasani et al., 2021) such as BERT (Devlin et al.,
∗Corresponding authors: Peng Li (lipeng@air.tsinghua. edu.cn) and Maosong Sun (sms@tsinghua.edu.cn)
1Relative rEpresentation-based Language-Image preTraining
2Code is available at https://github.com/THUNLP-MT/ RELIT.
2018) and MAE (He et al., 2022) that only require single-modality data, VLP models rely on largescale aligned image-text datasets (Lin et al., 2014; Sharma et al., 2018; Ordonez et al., 2011; Krishna et al., 2017) to bridge the gap between the two modalities, which requires either extensive manual annotations or heavy data cleaning processes (Lin et al., 2014; Sharma et al., 2018). The natural difficulty of obtaining paired data hinders the scale of cross-modal datasets, while the success of unimodal pre-trained models implies the potential to exploit the unlabeled data for pre-training. Therefore, besides collecting more paired data, it is a worthwhile direction to explore how to utilize lowcost unimodal data with limited cross-modal supervision, i.e., weakly supervised vision-and-language pre-training (WVLP).
The core challenge of WVLP is to establish the connection between the two modalities without using a large number of aligned image-text pairs. Existing works on WVLP (Li et al., 2021b; Zhou et al., 2022; Wang et al., 2022b; Chen et al., 2022) usually address this by taking object tags as anchors as they are in the form of text and cover the information of the image at the same time. They use tags to collect weakly-aligned image-text pairs from unaligned unimodal data for pre-training and achieve competitive results compared to standard VLP models, demonstrating that tags can effectively bridge the gap between the two modalities.
Despite its success, using object tags as anchors suffers from two limitations. First, tags are merely local descriptions instead of a complete representation of the whole image and text. Second, the vocabulary of tags only includes common concepts, making it difficult to represent images with complex semantics (Zhou et al., 2022). These limitations could deteriorate the quality of the weakly-aligned data (and possibly pre-trained models) based on the object tags. Therefore, to further improve the performance of WVLP, we need
8341
to reconsider the choice of the cross-modal anchors and find a better approach to measure the alignment between an image and a text.
Recently relative representation has been proven to be effective in representation learning (Moschella et al., 2022) and zero-shot image classification (Norelli et al., 2022). The main idea is to represent a data point as its similarities to a set of selected data points (anchors). We argue that relative representations can be a good choice for WVLP because (1) they are built on the semantic similarities of well-trained neural network representations rather than on superficial human-designed features like tags and (2) they are modality-invariant by design because they reflect the intrinsic relationships between data points, which naturally enables communication between different modalities.
In this paper, we propose RELIT, a novel relative representation-based WVLP framework. Instead of object tags, we directly use a minuscule amount (compared to pre-training data) of available imagetext pairs as anchors, and create a common relative representation space with respect to the anchors for unaligned images and text. This allows us to estimate the semantic similarity of any image-text pair by calculating their distance in the relative representation space. In addition, we design two relative representation-based data collection methods that can retrieve or generate weakly-aligned image-text pairs from unaligned unimodal corpora. Experimental results prove the effectiveness of relative representations in bridging the gap between image and text modalities. Moreover, our work reveals a promising research direction to establish crossmodal alignments by finding and aligning invariant data structures in different modalities, which may inspire future works on multimodal pre-training.
Our main contributions are as follows:
• We introduce the idea of relative representations in WVLP and demonstrate its superiority over object tags in effectively bridging the gap between different modalities.
• We propose a relative representation-based WVLP framework that can both retrieve and generate weakly-aligned image-text pairs for learning cross-modal representations.
• Extensive experiments on four diverse visionand-language tasks show that our proposed
framework outperforms strong WVLP baselines and further closes the performance gap between WVLP and standard VLP.

heading: 2 Related Work
text: Relative Representations. The concept of relative representations is initially proposed by Moschella et al. (2022). They show that the relative representations obtained from the representation spaces of different models are similar, which enables comparison and alignment between latent embeddings of different learning models. Norelli et al. (2022) explore relative representations in a multimodal scenario to align images and text for zero-shot image classification tasks. Specifically, they use 1.6M image-text pairs to build the relative representation space, which is comparable to the size of the data used in pre-training. To the best of our knowledge, our work is the first that exploits relative representations for weakly supervised cross-modal pre-training.
Weakly Supervised Vision-and-Language Pretraining. Li et al. (2021b) first explore WVLP with unaligned image and text corpora and use object tags directly as pseudo captions for images to bridge the vision and language modalities. Zhou et al. (2022) use tags to retrieve weakly-aligned captions for each image and then apply multi-granular alignment tasks on this retrieved dataset. Wang et al. (2022b) propose the cross-modal CutMix to replace some grounded words with regions that have the same tags, and construct a multimodal view of the text-only sentences for pre-training. Chen et al. (2022) introduce an end-to-end framework with a referring expression matching task. Different from all of these WVLP works that utilize tags as anchors to provide object-level cross-modal alignment signals, our work uses relative representations to capture the overall semantic similarity between each image and text and demonstrates its effectiveness in WVLP.
Data Augmentation. Data augmentation has been extensively employed in various computer vision (Zhang et al., 2018; Cubuk et al., 2018) and natural language processing tasks (Sennrich et al., 2015; Guo et al., 2020). In the area of VLP, Li et al. (2022) augment the noisy web-crawled aligned data by filtering low quality image-text pairs and generating synthetic captions with an image captioner fine-tuned on clean image-text pairs. In this work,
we adopt a similar filter-and-generate process in the construction of weakly-aligned data for WVLP, but our relative representation-based pseudo caption generator is fine-tuned on the text-only dataset.

heading: 3 Method

heading: 3.1 Relative Representations
text: Figure 1a provides an illustration of relative representations. The basic idea is to represent a data point as its similarities to other data points (anchors). In this work, we consider the relative representations with cross-modal anchors, which has been shown its potential in zero-shot image classification (Norelli et al., 2022).
Formally, given a set of M cross-modal anchors A = {a1, a2, . . . , aM} where ai = (x̃i, ỹi) is an image-text pair, x̃i is the image and ỹi is the text. For an image x , a pre-trained image encoder EI is used to calculate the similarity between x and each anchor ai as:
sim(x, ai) = cos(EI(x), EI(x̃i)) (1)
where cos(·, ·) is the cosine similarity, and the relative representation of x is defined as:
rA(x) = (sim(x, a1), . . . , sim(x, aM )) (2)
Similarly, the relative representation of a text y is defined as rA(y) with a pre-trained text encoder ET to compute sim(y, ai) = cos(ET (y), ET (ỹi)).
Since the relationship between data points is objective, the relative representations obtained by different models should be similar, despite their independent representation spaces (Moschella et al., 2022). In other word, an image and its corresponding text should share similar relative representations. This allows us to leverage it to construct weakly-aligned image-text pairs from large-scale unpaired image and text datasets.

heading: 3.2 Weakly-Aligned Image-Text Pairs Retrieval
text: While there are no large-scale aligned image-text pairs available, having a joint input of image and text, even if they are not aligned, is still necessary for WVLP (Zhou et al., 2022; Wang et al., 2022b). To achieve this, inspired by previous work (Zhou et al., 2022), we construct a weakly-aligned imagetext corpus from the unpaired unimodal corpora by retrieving semantically related sentences for each image based on the relative representations.
Figure 1b illustrates the process of our weaklyaligned image-text pairs retrieval method. First we collect a very small amount of image-text pairs as cross-model anchors (denoted by pairs of connected squares in the figure). Note that the number of anchors is negligible compared to the imagetext pairs used in standard VLP, which keeps our method in a weakly supervised setting. Then, for all images and text we compute their relative representations with respect to the anchors, which only involves similarity computation within each modality using unimodal pre-trained encoders. We take the cosine distance between the relative representations of each image and text as their semantic relevance score and retrieve the best matching text with the highest score for each image to construct a weakly-aligned image-text pair.
Specifically, we randomly sample M image-text pairs as anchors A from an aligned image-text dataset (e.g., Conceptual Captions (Sharma et al., 2018)) Dalign (M ≪ |Dalign|). Given unaligned image dataset DI and text dataset DT , we construct a retrieved weakly-aligned image-text pair dataset Dwa = {(x1, ŷ1), . . . , (xN , ŷN )} where N = |DI | and ŷi is the retrieved caption from DT for image xi defined as:
ŷi = argmax y∈DT cos(rA(xi), rA(y)) (3)
We use the off-the-shelf ViT (Dosovitskiy et al., 2020) and Sentence-BERT (Reimers et al., 2019) to encode images and text, respectively.
Our retrieval method with relative representations can effectively improve the quality of retrieved weakly-aligned dataset compared to tagbased retrieval. This is because relative representations tend to capture the overall semantics while tags describe only local information of the image. As a result, our method can better measure the semantic similarities between images and text, especially in cases where tag-based retrieval fails to distinguish between images and text that have different semantics but share the same objects.

heading: 3.3 Pseudo Caption Generation
text: Although relative representation-based retrieval can construct reasonable weakly-aligned imagetext pairs for WVLP, there are still cases where non-relevant text are retrieved. This could happen especially when the unaligned unimodal corpora are collected individually and for some images there are no proper captions in the corpora.
To alleviate this problem, we propose to directly generate pseudo captions for these images. As shown in Figure 2, we first adapt a well-trained text generator to perform conditional text generation given relative representations. Then, since images and text share a common relative representation space, we can directly use this generator to predict the pseudo caption for an image based on its relative representation.
Specifically, given the text-only dataset DT , for each text y ∈ DT , we derive a prefix P ∈ RM×d from its relative representations rA(y) as:
P = [rA(y)]TWr + [ET (ỹ1), . . . , ET (ỹM )]We (4) where ET (ỹi) ∈ RdT is the encoder output of the text in the i-th anchor, Wr ∈ R1×d and We ∈ RdT×d are two learnable projection matrices. We fine-tune a pre-trained GPT-2 model (Radford et al., 2019) to learn to predict y given P, and name the fine-tuned model as Rel2Cap. To further save computational cost, we only consider the entries in P that correspond to the top K anchors with the highest similarities as the model input.
After training, the model can be used to predict the pseudo caption for an image x with low quality retrieved captions by constructing an input prefix P ′ based on the relative representations of the image, i.e., rA(x). The definition of P ′
is similar to Equation 4, except that rA(y) is replaced by rA(x). We define a quality score s(x, ŷ) = cos(rA(x), rA(ŷ)) for each weakly-aligned imagetext pair (x, ŷ) collected both by retrieval and generation, and replace the retrieved pair with the generated one if the latter has a higher quality score.
So far, we have discussed how we collect a
weakly-aligned image-text dataset Dwa from the unpaired unimodal corpora by relative representationbased retrieval and generation. Next, we describe how we use these data for WVLP.

heading: 3.4 Pre-training
text: Model Overview. We use the same model architecture as Chen et al. (2022) that consists of a vision and a multimodal encoder. For each weakly-aligned image-text pair, the image is encoded with the vision encoder and the outputs are fed to the multimodal encoder along with the text embeddings to obtain a multimodal representation. Such an end-to-end framework has been proven
to be more effective compared to others that use region features from external object detectors both in standard VLP and WVLP. We apply three pretraining objectives to learn multimodal representations from the collected weakly-aligned imagetext pairs: masked tag prediction (MTP), masked language modeling (MLM) and image text matching (ITM).
Masked Tag Prediction. This objective aims to learn object-level cross-modal alignment from the image-only data and their detected object tags. Following previous works (Li et al., 2021b; Chen et al., 2022), we randomly mask out the tags with a probability of 15%, and then predict the masked tags conditioned on the image and other unmasked tags. Formally, given the image x ∈ DI and its detected object tags t, the MTP objective is defined as:
LMTP = −Ex∈DI logP (tm|t\m, x) (5)
where tm and t\m represents masked and unmasked object tags, respectively.
Masked Language Modeling. To better fuse between the two modalities, the masked language modeling objective is adopted to learn from the joint image-text inputs from the weakly-aligned corpora. Since the weakly-aligned pairs may contain noise in the retrieved or generated text, we only mask out and predict the noun phrases in the text inspired by (Zhou et al., 2022). The MLM loss is formulated as:
LMLM = −E(x,ŷ)∈Dwa logP (ŷm|ŷ\m, x) (6)
where ŷm and ŷ\m are masked and unmasked text.
Image Text Matching. ITM is a commonly used objective for learning instance-level crossmodal alignment in VLP, which aims to distinguish whether an image-text pair is matched semantically. We random replace the text in half of the image-text pairs with another text to form training input, and define the label of each pair as l ∈ {0, 1} where 1 indicates the pair is a match. The ITM objective is to minimize the binary cross-entropy loss:
LITM = −E(x,ŷ)∈D′wa logP (l|x, ŷ) (7)
where D′wa is the dataset after random replacement. Relative Representation-Guided Training. To further reduce the impact of the noisy image-text pairs in the weakly-aligned dataset, we apply the
quality score s(x, ŷ) of each pair described in Section 3.3 to LMLM and LITM to guide the training to learn more from high-quality data:
LMLM = −E(x,ŷ)∈Dwas(x, ŷ) logP (ŷm|ŷ\m, x) (8)
LITM = −E(x,ŷ)∈D′was(x, ŷ) logP (l|x, ŷ) (9)

heading: 4 Experiments

heading: 4.1 Datasets
text: We follow previous WVLP works (Li et al., 2021b; Zhou et al., 2022; Wang et al., 2022b; Chen et al., 2022) and conduct experiments in two different settings, each containing an image-only dataset and a text-only dataset. The first setting treats images and text from Conceptual Captions (CC) (Sharma et al., 2018) as individually collected unimodal dataset without the alignment information. The second setting uses images from CC and text from BookCorpus (Zhu et al., 2015), which is a more realistic scenario where images and text are gathered separately from different sources.

heading: 4.2 Implementation Details
text: Relative Representations. We randomly select 8, 192 aligned image-text pairs from CC as anchors, yielding relative representations as vectors of 8, 192 dimensions. To save computational cost, inspired by Norelli et al. (2022), we only keep the highest 50 dimensions and set the others to 0.
Weakly-Aligned Data Construction. We implement the retrieval system with the faiss (Johnson et al., 2019) library. For each image we only retrieve the text with the best match score. For RelCap, we fine-tune GPT-2 with a learning rate of 5e5 and a batch size of 1, 024 for 5 epochs on the textonly dataset. We generate 5 pseudo-captions for each image using nucleus sampling with p = 0.9 which proved effective in synthetic caption generation (Li et al., 2022), and rank the results with the quality scores. We also include the weaklyaligned dataset based on tag-based retrieval in the pre-training, as described in Zhou et al. (2022).
Pre-training. We use the same architecture as Chen et al. (2022) which includes a 12-layer SwinTransformer (Swin B-384/32) (Liu et al., 2021) as the vision encoder and a 12-layer Transformer initialized from BERT-base (Devlin et al., 2018) as the multimodal encoder. For object tags, we
utilize the off-the-shelf object detector provided by VinVL (Zhang et al., 2021). We pre-train the model with a total training step of 150k and a batch size of 512. We use an AdamW optimizer (Kingma and Ba, 2014) with an initial learning rate of 3e-5, and the warm-up ratio is set to 10%. The pre-training takes 3 days on 4 NVIDIA A100 GPUs.
Downstream Tasks. We follow previous works and test our pre-trained model on four downstream V+L tasks, including Visual Question Answering (VQAv2) (Goyal et al., 2017), Natural Language for Visual Reasoning (NLVR2) (Suhr et al., 2018), Visual Entailment (VE) (Xie et al., 2019) and image retrieval (Flickr30k) (Plummer et al., 2015). Details of the task settings and the finetuning strategies are in Appendix A.

heading: 4.3 Main Results
text: We first compare our proposed RELIT with previous methods pre-trained with unaligned images and text from CC. Note that these baselines only utilize object tags. Table 1 shows the experimental results on the downstream tasks. Our method outperforms previous WVLP methods on all downstream tasks.
Specifically, RELIT outperforms previous best results by 1.8% on NLVR2 and by 3.8% on the image retrieval task (Flickr30k), both of which benefit from the instance-level cross-modal alignment capability of the pre-trained model (Chen et al., 2020; Zhou et al., 2022). This suggests that our relative representation-based method improves the alignment quality of weakly-aligned image-text pairs compared to previous tag-based approaches, resulting in improved cross-modal alignment capability of the pre-trained model.
When pre-trained with images from CC and text from BookCorpus, as shown in Table 2, our proposed RELIT also achieves the best results on all downstream tasks. This demonstrates that the proposed relative representation-based methods can effectively mine useful cross-modal alignment information for multimodal pre-training from imageonly and text-only data, even if they are collected separately from different sources.

heading: 4.4 Ablation Study
text: We conduct an ablation study to verify the effectiveness of the proposed relative representation-based retrieval and generation. Table 3 shows the results.
All models are pre-trained on weakly-aligned data derived from unaligned CC images and text. As we can see from the table, compared to tag-based retrieved data (Retrv (Tag)), pre-training with relative representation-based retrieved data (Retrv (Relrep)) performs better on downstream tasks. Besides, the model achieves the best results when the generated pseudo captions (Rel2Cap) are included during pretraining. We believe this is because the original CC dataset contains noisy captions, such as alt-texts that do not describe the image contents, which is suboptimal for VLP (Li et al., 2022). In summary, the experimental results demonstrate that both our retrieval and generation methods contribute to the performance of the pre-training.
We also compare the performance of the pretrained models on downstream tasks with and without relative representation-guided training. As shown in Table 4, pre-training with guided training can consistently improve results across all downstream tasks, illustrating that relative representations can be used to detect noise in the weaklyaligned data and guide the model to learn from data with a higher level of alignment.

heading: 4.5 Data Quality
text: We evaluate the quality of different kinds of weaklyaligned data from unaligned CC images and text, and the results are listed in Table 5. We use CLIPScore (Hessel et al., 2021) to measure the overall alignment of all weakly-aligned image-text pairs. As we can see from the table, the data quality of Retrv (Relrep) is significantly higher than that of Retrv (Tag), which again illustrates the superiority of relative representations as cross-modal anchors.
In addition, Rel2Cap further improves data quality by filtering and replacing low-quality pairs in Retrv (Relrep). The analysis of the data quality here is consistent with the analysis of pre-training results in Table 3, and again proves that our relative representation-based methods can produce high quality weakly-aligned data from unaligned unimodal data.
4.6 Effects of Anchor Selection
The number of anchors has a significant influence on the effect of relative representations (Norelli et al., 2022). To verify its influence on the collected weakly-aligned image-text pairs, we test the quality of the data retrieved with different numbers of anchors on the COCO (Lin et al., 2014) dataset. From Figure 3, we can see that as the number of anchors increases, the quality of the retrieved data also improves. In addition, we evaluate
the downstream task accuracy using models pretrained on data with varying numbers of anchors. Specifically, we generate 3 random sets of anchors for each size, and retrieve the weakly-aligned data with different sets of anchors. We pre-train models on each set of the retrieved data with the same hyperparamters, and fine-tune them on the NLVR2 task. The results are shown in Figure 4. In general, the higher the number of anchors, the better the model performance. We use 8, 192 anchors in our final experiments as a trade-off between representation capability and computational cost. However, using more anchors will almost certainly give better results due to better quality of the data, which indicates the scalability of our approach. We leave more exploration on this for future work.
We also conduct experiments to verify the impact of anchor diversity on data quality. Specifically, we considered three sampling methods on the COCO dataset: random, diverse, and nondiverse. The diverse sampling first performs Kmeans clustering on all the data, and selects one anchor from each cluster. The non-diverse sampling uses a greedy algorithm to select k anchors, at each step choosing the data closest to the average of the anchors already selected. Table 6 lists the data quality results obtained with different sam-
pling methods. In general, diverse anchors lead to better quality, while random anchors perform satisfactorily when the number of anchors is large enough. Non-diverse anchors can result in catastrophic data quality.

heading: 4.7 Case Study
text: To explore the reasons for the improvement in data quality, we show two examples of the comparisons between different weakly-aligned image-text pairs in Figure 6. In each example, we provide the ground truth caption of the image and the detected object tags, as well as three weakly-aligned captions. From these two examples, we can see that the captions retrieved by tags do have many of the same tags as the images (underlined in the figure), but are not good descriptions of the images. In contrast, our relative representation-based retrieval and generation methods are able to obtain captions that are more relevant to the overall semantics of the images. Specifically, in the example in Figure 6a, our proposed methods successfully identifies key information in the image such as “golfer”, which is difficult for tag-based retrieval since there are no such tag as “golfer”. The same thing happens to Retrv (Tag) in Figure 6b, which retrieves a caption related to “cat” instead of “lynx”. In this example, our retrieval method recognizes the animal in the image as “cheetah”, which is close but not exactly correct, while our generation method correctly generates a caption related to the correct concept “lynx”. This indicates that our generation method has the ability to generate pseudo captions of better quality when the retrieved ones are not good enough.
In Figure 5 we further visualize the relative representations of the image and two retrieved captions in Figure 6a, which helps understand the effectiveness of relative representations in aligning semantically related image-text pairs. From the figure we can see that the image and our retrieved caption Retrv (Relrep) activate the same group of anchors (i.e., have high similarities with these anchors), which makes them close in the relative representation space. On the other hand, Retrv (Tag) activates a completely different set of anchors, which leads to a large distance between it and the image in the relative representation space. These observations suggest that (1) relative representations are (almost) modality-invariant and (2) relative representations can be utilized to effec-
tively estimate the cross-modal alignment of data in different modalities. These properties of the relative representations make it naturally suitable for WVLP, which is verified in this paper.

heading: 5 Conclusion
text: This paper introduces the idea of relative representations to weakly-supervised vision-and-language pre-training and demonstrates its effectiveness in bridging the gap between the two modalities. We propose a relative representation-based framework that can both retrieve and generate weakly-aligned image-text pairs for pre-training. Experimental results show that our method outperforms all previous tag-based approaches under the weaklysupervised setting. We hope our work will motivate future work in multimodal pre-training.
Limitations
As this work is mainly focused on weakly supervised vision-and-language pre-training, we do not fully explore the factors that may influence the performance of relative representations, such as the use of different unimodal encoders and the source of the anchors. Besides, we only validate the effectiveness of relative representations in a weakly supervised setting, while it remains to be explored whether it is also useful for standard VLP and multimodal learning in other modalities (e.g., audio and video). We will further exploit the potential of relative representations and validate it in more cross-modal learning scenarios in the future.

heading: Acknowledgments
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152, 62236011). We thank Ziyue Wang, Fuwen Luo, Rui Jiao and Zonghan Yang for their advices in paper writing.

heading: A Details of Downstream Tasks
text: Visual Question Answering (VQA) The task of VQA is to answer questions correctly according to the given images. We follow previous works (Yu et al., 2019; Chen et al., 2020) and formulate VQA as a classification task with 3, 192 classes representing the most frequent answers in the dataset. We fine-tune the pre-trained model for 10 epochs with a batch size of 256. We use an AdamW optimizer with a peak learning rate of 5× 10−5. Natural Language for Visual Reasoning (NLVR2) The objective of NLVR2 is to decide if a natural language description is true for a given pair of images. We follow previous work (Chen et al., 2020) to form two image-text pairs as inputs, and concatenate the two [CLS] outputs of the model as the final representation for classification. We fine-tune the model for 10 epochs with a batch size of 128 and a peak learning rate of 2.5× 10−5. Visual Entailment (VE) Given an image and a text hypothesis, the task of VE is to determine whether the image implies the hypothesis. This is formulated as a three-way classification task to predict whether the logical relationship between the image and the text is entailment, neutral or contradiction. For the VE task, we fine-tune the pre-trained model with a batch size of 64 and a peak learning rate of 1× 10−5 for 5 epochs. Image Retrieval (Flickr30k) We follow previous works (Li et al., 2021b; Chen et al., 2022) to conduct the image retrieval task on the Flickr30k (Plummer et al., 2015) dataset. We sample 15 negative image-text pairs for each positive pair by replacing its text with randomly sampled ones. The batch size is set to 512. We fine-tune the model with a peak learning rate of 2.5× 10−5 for 10 epochs.

heading: B Additional Examples
text: In Figure 7, we provide more examples of different kinds of weakly-aligned image-text pairs. From these examples, we can see that our relative representation-based approaches yield higher quality weakly-aligned image-text pairs compared to tag-based retrieval.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4
3 B1. Did you cite the creators of artifacts you used? 4
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? 4
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
7 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. We only use publicly available datasets that are widely used in previous vision-and-language pretraining works. Relevant statistics have been fully discussed and can be easily found in these works.
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We hope our work will motivate future work in multimodal pre-training. Limitations
As this work is mainly focused on weakly supervised vision-and-language pre-training, we do not fully explore the factors that may influence the performance of relative representations, such as the use of different unimodal encoders and the source of the anchors. Besides, we only validate the effectiveness of relative representations in a weakly supervised setting, while it remains to be explored whether it is also useful for standard VLP and multimodal learning in other modalities (e. g. , audio and video). We will further exploit the potential of relative representations and validate it in more cross-modal learning scenarios in the future