Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
Authors: Maurits Bleeker
Mariya Hendriksen
Andrew Yates
Maarten de Rijke
Venue: Accepted by TMLR ()

Abstract: Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shortcut that minimizes contrastive loss.  We introduce synthetic shortcuts for vision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data.  We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features
that represent the shortcut. Hence, contrastive losses are not sufficient to learn task-optimal representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partly reduce shortcut learning when training and evaluating with our shortcut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.

Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning
Maurits Bleeker*
m.j.r.bleckeruuoo.t
University of Amsterdam, Amsterdam, The Netherlands
Mariya Hendriksen"
m.hendriksen@hwv..k
AIRLab, University of Amsterdam, Amsterdam, The Netherlands
Andrew Yates
a.c.gnntes@uvo.nt
University of Amsterdam, Amsterdam, The Netherlands
Maarten de Rijke
m. m.derüjke@wo..r nl
University of Amsterdam, Amsterdam, The Netherlands
Abstract
Vision-language models (VLMs) mainly rely on contrastive training to learn general-purpose representations of images and captions. We focus on the situation when one image is associated with several captions, each caption containing both information shared among all captions and unique information per caption about the scene depicted in the image. In such cases, it is unclear whether contrastive losses are sufficient for learning task-optimal representations that contain all the information provided by the captions or whether the contrastive learning setup encourages the learning of a simple shorteut that minimizes contrastive loss. We introduce synthetic shortcuts for lision-language: a training and evaluation framework where we inject synthetic shortcuts into image-text data. We show that contrastive VLMs trained from scratch or fine-tuned with data containing these synthetic shortcuts mainly learn features sufficient to learn lask-optimal
that represent the shortcut. Hence, contrastive losses representations, i.e., representations that contain all task-relevant information shared between the image and associated captions. We examine two methods to reduce shortcut learning in our training and evaluation framework: (i) latent target decoding and (ii) implicit feature modification. We show empirically that both methods improve performance on the evaluation task, but only partially reduce shortcut learning when training and evaluating with our shorteut learning framework. Hence, we show the difficulty and challenge of our shortcut learning framework for contrastive vision-language representation learning.
1 Introduction
Recent work on understanding the internal mechanisms of representation learning has brought to attention the problem of shorteut learning (Robinson et al., 2021; Chen et al., 2021; Scimeca et al., 2022). While there are multiple definitions of shortcut learning (e.g., Geirhos et al., 2020; Wiles et al., 2022), in this work we define shortcuts as casy-to-learn discriminatory features that minimize the (contrastive) optimization objective but nol necessarily sufficient for solving the evaluation lask. More specifically, focus on the problem of shortcut learning in the relatively unexplored context of vision-language (VL) representation learning with multiple matching captions per image.
Contrastive learning (CL. plays a crucial role in VL. representation learning. Despite the success of non- contrastive approaches, c.g., (Bardes et al., 2022), the dominant paradigm in VL representation learning revolves around either fully contrastive strategies (Faghri et al., 2018; Li al., 2019a; Jia al., 2021;
*Co-first author.

---

Radford et al., 2021) a combination of contrastive methods with additional objectives (Li et al., 2021; Zeng et al., 2022; Li et al., 2022a; Zeng et al., 2022; Li et al., 2023a). It is standard practice in contrastive VI. representation learning to sample batches of image-caption pairs and maximize the alignment between the representations of the matching images and captions (Radford et al., 2019; Jia et al., 2021). Given that the typical VL benchmarks, e-g-; Flickr30k (Young et al., 2014) and MS-COCO Captions (Lin el. al., 2014; Chen et al., 2015), are constructed in such a way that each image is associated with multiple captions, each caption can be seen as different view of the image it describes. Therefore, CL with multiple captions per image can be seen CL with multiple views, where each caption provides different view of the scene depicted in the image.
CL with multiple views, where each view represents a different observation of the same datapoint, has proven to be effective for general-purpose representation learning (Hjelm et al., 2019; Chen et al., 2020a; Tian et al., 2020a). The goal of multi-view (contrastive) representation learning methods to learn representations that remain invariant to a shift of view, which is achieved by maximizing alignment between emboddings of similar views. A core assumption within the multi-view representation learning literature is that lask-relevant information is shared across views whereas task-irrelevant information is not shared, given downstream evaluation task (Zhao et al., 2017; Federici et al., 2020; Tian et al., 2020a; Shwartz-Ziv & LcCun, 2023).
An open challenge in the multi-view representation learning domain concerns learning representations that contain task- relevant information that is not shared among different views, i.e., that may be unique for some views (Shwartz-Ziz Ziv & LeCun, 2023; Zong et al., 2023). In the case of image-caption datasets where each image is paired with at least one corresponding caption, the captions matching the same image do not necessar- ily share the same information as each caption is distinct and may describe different aspocts of the image (Biten et al., 2022). Figure 1 illustrratss the concept of shared vs. caption-specific task-relevant information. The image is accompanied by two captions: 'a couple of boats and red car' (xe,) and 'a couple of boats and between the captions includes 'couple of boats' and 'car'. Cap- tion Xc, provides unique information by describing the car as 'red'. Caption Xc, adds unique contextual details about the location with the phrase 'on a street!. To learn task-optimal representations, it is essential to integrate both the shared and unique information from these captions. Furthermore, given the typical quality of captions of image-caption datasets (Chen et al., 2015), we assume that all information present in the cap- tions is relevant. Hence, cach image-caption pair may contain both shared lask-relevant information, i.e., information shared across all the captions in the tuple, and unique tesk-relevant information, i.c., information not shared with other captions. Therefore, learning task-optimal representations for the image implies learning all task-relevant information that comprises both shared and caption-specifie information.
a street' (Xcy). The shared information
CA :a couple of boats and a red car
CB a couple of boats and car on street
Figure 1: Shared vs. caption-specific infor- mation given an example of one image and two associated captions Xc, and Xc,- The purple color indicates information shared between the image and both captions. The green color indicates task-relevant informa- tion specific for X The blue color indi- cates task-relevant information specific for Xca
Another problem of CL approaches is related to feature suppression. Shwartz- Ziv & LeCun (2023) argue that although contrastive loss functions lack explicit. information-thhorrticall constraints aimed at suppressing non-shared information among views, the learning algorithm benefits from simplifying representations by suppressing features from the input data that are not relevant for minimizing the contrastive loss. Furthermore, Robinson et al. (2021) demonstrate that contrastive loss functions are susceptible solutions that suppress features from the input data. In the case of VL. CL with multiple captions per image where at least one caption contains caption-specifie information, the image representation can never have perfect aligument with all matching captions. This is due to the misalignment that happens when encoding unique information for the
2

---

Latent Variables
Latent Nariables
Image
Captions
Image
Captions
Z
Z
Z-
Z2
XCA
XC
Z3
Z3
XI
XI
XCB
XCB
Zn
Zp
Z+
Z+
(a) Minimal shared information.
(b) Tosk-optinal information.
Figure Synthetic shortcuts in the context of minimal shared and task-optimal information for vision- language representation learning with multiple captions per image. The pupll color represents features shared among the image and all captions (minimal shared information). The yellow color represents caption-specifie features (unique information). The grey color indicates features that are not present in both the image and any of the captions (lask-irrelevanl information). The red color indicates synthetic shortcuts. We demonstrate that while shortcuts exist in both scenarios, minimal shared information also includes information shared among the image and all associated captions, whereas task-optimal information combines both minimal shared information and caption-specific information.
other captions. Therefore, it is unclear whether contrastive methods can learn task-optimal representations, i.., representations that contain all information present in the captions associated with the image, or if they Iearn only the minimal sharod information, i.c., information shared between the image and all captions that are sufficient to minimize the contrastive discrimination objective. An illustration of minimal shared information and task-optimal representation is given in Figure 2.
Motivated by the abovementioned problems, we address the following question:
In the contert of VL representation learning with multiple captions per image, to what extent does the presence of shorleut hinder learning losk-oplimal represenlations?
To answer this question, we investigate the problem of shortcut lcarning for VL representation learning with multiple captions per image. We do this by introducing the synthetic shortculs for vision-language (SVL) framework for adding additional, easily identifiable information to image-caption tuples. The information that add is represented identifiers that are applied both image and caption; these identifiers do not bear any semantic meaning. The identifiers provide additional shared information between the image and captions, which is a subset of the total shared information between the image and the caption. For details and examples of shortcuts, refer to Section 3, where Figure 4 illustrates an example of an image-caption pair with a shorteut added. The synthetic shortcuts framework allows us to investigate how much the encoder model relies on the added shortcut during training and evaluation, and hence how much the relevant information is still captured shortcut solution is available. Overall, our SVL framework allows us investigate the shorteut learning problem in controlled way. We focus on image-caption retrieval (ICR) evaluation task because contrastive losses directly optimize for the ICR evaluation task, which assesses the quality of the learned representations by computing similarity score between images and captions (Radford et al., 2021; Yuksekgonul et al., 2023). To investigate the problem, we run experiments on two distinct models: (i) CLIP (Radford el al., 2019), a large-scale model that we fine-tune; and (ii) VSE++ (Faghri et al., 2018), a relatively small model that we train from scratch. We evaluate the models' performance on the Flickr30k (Young et al., 2014) and MS-COCO (Lin al., 2014; Chen et al., 2015) and benchmarks. The benchmarks are constructed in such way that each image associated with five captions and each caption represents concise summary of the corresponding image.
3

---

Therefore, the contributions of this work are two-fold:
I A framework for investigating the problem of shortcut learning for contrastive vision- language representation lcarning in a controlled way: We introduce the synthetic shortcuts for vision-language framework. The framework enables the injection of synthetic shortcuts into image-caption tuples in the training dataset. We use the framework to investigate and understand the extent to which contrastive VL models rely on shortcuts when a shortcut solution is available. We run our experiments using CLIP and VSE++, two distinct vision-language models (VLMs). We evaluate the models' performance on the Flickr30k and MS-COCO benchmarks. We evaluate the effectiveness of contrastive VL models by comparing their performance with and without synthetic shorteuts. We demonstrate that both models trained from scratch and fine-tuned, large-scale pre-trained foundation models mainly rely on shorteut features and do not learn task-optimal representations. Consequently, we show that contrastive losses mainly capture the easy-to-learn discriminatory features that are shared among the image and all matching captions, while suppressing other task-relevant information. Hence, we argue that contrastive losses are not sufficient to learn task-optimal representations for VL representation learning.
II We present two shortcut learning reduction methods our proposed training and evaluation framework: We investigate latent target decoding (LTD) and implicit feature mod- ification (IFM) using our SVL training and evaluation framework. While both methods improve performance on the evaluation task, our framework poses challenges that existing shorteut reduction techniques can only partially address, as the performance not on par with models trained without. synthetic shortcuts. These findings underline the importance and complexity of our framework in studying and evaluating shorteut learning within the context of contrastive VL representation learning.
2
Background and Analysis
In this section, we present the notation, setup, and assumptions on which base the work. Additionally, we conduct analysis of contrastive VL representation learning with multiple captions per image.
2.1 Preliminaries
Notation. We closely follow the notation from (Bleeker et al., 2023). See Table 3 for overview. Let D be N xx) fxt, 1f- ) }
Each tuple E N contains one image xi and k captions xX,, where S S k. All captions in tuple i E N are considered as matching captions w.r.t. image Xy in the tuple The latent representation of an imago-caption pair from a tuple is denoted as z and z respectively. During training, we sample image-caption pairs from the dataset D and optimize for the evaluation task T. We include all captions in the dataset once per training epoch, hence, each image is sampled & times.
a dataset of N image-caption tuples: D = [xx
Given an image Xy, set of k associated captions K = (xxc, 18 1 and one caption randomly sampled from the set X E K, we define the following representations: (i) z as sufficient representation of the caption Xe that describes the image Xy: (ii) z zi-c
SUF as representation of the image x, sufficient for the caption Xei
(iii) ZXTN representation of the image X, that is minimally sufficient for the caption xe; and (iv) 2 representation of the image Xy that is optimal for the set of captions K given the task T.
In addition, we write SsynSC for a synthetic shortcut, S for the original shared information, i.e., information that does not contain synth shortcuts, S+ for the shared information that includes a synthetic shortcut, and R+ for tesk-relevant information that contains a synthetic shorteut.
In the context of task relevance, we define R and RR as task-relevant and task-irrelevant information, respectively, and C as task-relevant information specific for caption xe-
4

---

Sctup. We work with dual-encoder setup, with an image encoder and caption encoder that do not share parameters. The image encoder sa(.) takes an image Xz as input and returns its latent representation: " == fx())) Similarly, the caption encoder go(-) takes a caption X as input, and encodes the caption into a latent representation: c = go (ze). Both ze and z are unit vectors projected into d-dimensional multi-modal space: ze E IR Z E Rd. For an overview of notation, we refer to Appendix A, Table 3.
Assumptions. Given an image-caption tuple, we assume that each caption in the tuple is distinct from the other captions in the tuple. We also assume that each caption in the tuple contains two types of task-relevant information: (i) shared information, i.e. information shared with other captions in the same tuple, and (ii) caption-specific information, i.e., information that is not shared with the other captions. For simplicity, base our subsequent analysis on tuples where one image X is associated with two captions Xc,, and Xcy² (xy,
However, the analysis described in this section can be extended a case with more than two captions. We treat images and captions as views and define X2, Xc,' and Xen to be random variables of an image and two matching captions, with the joint distribution p(Xx, Xc, xcg). For more details on assumptions and problem definition, we refer to Appendix B.
2.2 Analysis of Contrastive Vision-Language Representation Learning for Multiple Captions per Image
InfoMax. We start our analysis of contrastive VL. representation learning by introducing the InfoMax optimization objective, a typical loss for VL representation learning. The goal of an InfoMax optimization objective, e.g., InfoNCE (van den Oord et al., 2018), is to maximize the mutual information (MI) between the latent representations of two views of the same data (Tschannen et al., 2020). Therefore, the optimization objective equivalent to: maxfo.so [(z; where Zz, = [x7z) and ze : gx(x)).
Minimally Sufficient Image Representation. During training, batches of image-caption pairs sampled. The optimization involves maximizing the MI between the image representation 2y and the matching caption representation ze. Wang et al. (2022) argue that, since all supervision information for one view (i.e., the image) comes from the other view (i.c., the caption), the representations learned contrastively are approximately minimally sufficient. Following (Tian et al., 2020b; Wang et al., 2022), we extend the definition of sufficient representation to VL context and define sufficient caption representations, sufficient image representations, and minimally sufficient image representation.
Definition 2.1 (Sufficient caption representation). Given an image X,, and a set of matching captions {x xcg]; the representation SUP of caption x C is sufficient for image X,, if, and only if, SUF
1(z) 1(<c-1:x, x = 1xxx;x7).
The sufficient caption representation z z contains all the information about image X in caption xe.
Definition 2.2 (Sufficient image representation). Given an image X,, and a set of matching coptions C = fx Xcg h: the representation z ((z)-cixe) 1(xx:xe).
SUF of image x, is sufficient for caption Xc E C if. and only if,
Similarly, the sufficient image representation zi-c contains all the shared information between an image X and a caption xc- Note that a sufficient image representation can be sufficient w.r.t. multiple captions.
SUP
Definition 2.3 (Minimally sufficient image representation). Given an image X1, and set of matching captions C = -xxc,.Xcxh the sufficient image represenlation Z7- of image Xx is minimally sufficient for caption x if, and only if, (IN,)N < I(Z for all zi-c that are sufficient.
Intuitively, = comprises the smallest amount of information about x (while still being sulficient) and, therefore, only contains the information that is shared with caption xc, i.e., the non-shared information is suppressed.
Task Optimal Image Representation. The definition of task-optimal image representation based on the notion of task-relevant information. In the context of VL representation learning with multiple captions per image, define task-relevant information all information described by the matching captions. That.
5

---

H(I)
Shared information
Task-relevant information
H(CA)
HCBB) S= = [xx Cx (Cn)
R = CA
Caption A task-relevant information
Caption task-relevant information
CA = [xx: XC (Cp)) CB = Ixx: x (cc)
We define (x²)) as image information, ((x.,) and H(xc,) as caption information; both captions only describe the information depicted in the image and contain shared and caption-specific information. We further define CA (xx xxe) and CB I(xxix<e xx,) as caption-specific information; S ((xxiXciiXea R = CA + CB + S as task-relevant information.
Figure 3:
shared information; -R Hx, Xcxxxco) task-irrelevant information;
includes both caption-spectie and shared information. Consequently, task-optimal image representation is image representation that is sufficient w.r.t. all matching captions.
Formally, following assumptions from Appendix B.2, define task-relevant. information R all the information described by the matching captions. The task-relevant information can be expressod follows:
R Task-relevant information
H(xz)
H(xx
Xen)
Image information
information
(1)
1(xxi
(ce)
+ 1xxixC (xC))
1(xxi Xxxi xce)
Xca
Ca-specifie
Ca-specific task-relevant information
Shared information
task-relevant information
Similarly, task-irrelevant information -R is the image information not described by the captions. Figure 3 illustrates both definitions.
The multi-view assumption states that task-relevant information for downstream tasks comes from the information shared between views (Shwartz-Ziv & LeCun, 2023). However, in the case of VL reprosentation learning with multiple captions per image, lask-relevant information includes both shared information S and caption-specific information CA and CB (Eq. 1).
Definition 2.4 (Task-optimal image representation). Given an image X,, and set of matching captions C = (xx,,, &c the representation z is task-optimal image represenlation for all matching captions if, and only if, OPT. xx) = ((xxiXe), for all xe E C.
In other words, task-optimal image representations contain all the information that the image shares with the matching captions. Hence, task-optimal image representation is sufficient w.r.t. all matching captions. The information contained in the task-optimal image representation includes both shared and caption-specific information. Therefore, task-optimal image representation never be minimally sufficient image representation w.r.t. to specific caption.
Theorem 1 (Suboptimality of contrastive learning with multiple captions per image). Given image Xx, sel. of matching caplions C= (xc, xc,), ) and contrustive learning loss function LIfoNCE that oplimizes for task T, image representations learned during contrastive learning will be minimally sufficient and will never be task-optimal image representations.
The proof is provided in Appendix C. Rephrasing Theorem 1, given an image and two captions that form two image-caption pairs, Xc,) and (xz; xcg), ) and assuming that contrastive loss optimizes the image encoder to be minimally sufficient w.r.t. caption xe, during a training step, all task-relevant information Cg specific to caption Xcg will be suppressed in Zz. Hence, the resulting image representation will not be optimal for the task T.

---

Theorem 1 'iddccates discrepancy between minimally sufficient representations learned during contrastive training with the InfoNCE. loss and the task-optimal image representations in the context of learning VI. representations with multiple captions per image. Although the InfoMax loss does not have an explicit constraint to compress information, prior work indicates that feature suppression is happening (Shwartz-Ziv & LeCun, 2023; Robinsoo et al., 2021). Hence, we question contrastive loss can be used to learn Lask-optimal image representations in the context of multiple captions per image.
Furthermore, Theorem implies that in the context of contrastive VL representation learning with multiple captions per image, the minimally suffcciett representation, which discards non-shared information, is not the same as the task-optimal representation that comprises both caption-specific and shared information. This suggests that the features learned during contrastive learning might be shortcuts, i.e., easy- lo-deect discriminatory features that minimize the contrastive optimization objective but are not necessarily sufficient for solving the evaluation task. To examine this problem, we introduce synthetic shortcuts framework that allows us investigate the problem of suboptimality of contrastive learning with multiple captions per image in controlled way.
3 Synthetic Shortcuts to Control Shared Information
In Section 2 we show the suboptimality of the contrastive InfoNCE, loss with multiple captions per image. In the case of real-world VL datasets with multiple captions per image, there are no annotations that indicate the information shared between the image and captions and the information specific to each caption. Hence, we cannot directly measure how much of the shared and unique information is captured by the representations.
Synthetic Shortcuts. In this section, we introduce the syn- thetic shortcuts for vision-language (SVL) training and evalu- ation framework. We denote the synthetic shortcuts for image- caplion dala as SyysS-- The purpose of the framework is to introduce additional and casily identifiable information shared between an image and the matching captions that lacks any semantic meaning. The shortcuts we use in this work are rep- resented as numbers that add to images and captions. For images, we add the shortcut number by adding MNIST images as overlay lo the original images. For captions, we append the numbers of the shorteut as extra tokens at the end of the caption.
20
A player up to bat in baseball game. 1 9 2 Figure 4: An image-caption pair from the MS-COCO dataset with a shorteut added to both the image and the caption.
Figure 4 illustrates an example of an image-caption pair with an added shortcut. The example contains an image with the caption: 'A player up to bat in bascball game. 10 1 9 9 2 Here, '1 0 9 9 2' is shorteut added to both the image and the caption. For the image modality, we add the shorteut by overlaying MNIST images at the top of the original image. For the text modality, we append the shortcut as additional tokens at the end of the caption. This identifier provides an additional link between the image and the caption without carrying any semantic meaning related to their content. Additional examples are shown in Figure 6.
If contrastive losses learn task-optimal representations, then the presence of synthetic shortcuts should not negatively impact the evaluation performance, since synthetic shorteuts represent additional information and the remaining task-relevant information is intact. By incorporating synthetic shortcuts into the image-caption dataset, the shared information would include the information that was originally shared and the synthetic shortcut: S+ = S SSynSC. Hence, the task-relevant information would comprise caption-specific information that was originally shared and a synthetic shortcut: R+ CA + CB + shortcut influences the performance negatively, we can conclude that by learning represent a synthetic shortcut the model suppresses other task-relevant information in favor of the shortcut, hence the represent ation
SymSC- If injecting synthetic
7

---

is not task-optimal. The setup is inspired by the "datasets with explicit and controllable competing features," introduced by Chen et al. (2021), but we adapt this setup to the VI. scenario.
For experiments, we use the Flickr30k and MS-COCO image-caption datasets, that consist of image-caption tuples, each image associated with five captions. During training, we sample a batch of image-caption pairs B = {x xx, .......... 18) )..)....... from dataset D, and apply shorteut sampling. We inject the shortcuts in a manner that preserves the original information of the images and captions. Furthermore, we append the shorteut after applying data augmentations to ensure that the shortcut is present in both the images and captions (i.c., the shortcut not augmented away). We refer to Figure for some examples. The training, evaluation, and implementation details of the shortcut sampling are provided in Appendix D.4.
We define the following experimental setups:
No shorteuts: As a baseline, we fine-tune a pre-trained CLIP (Radford et al., 2021) and train VSF++ (Faghri et al., 2018) from scratch on Flickr30k and MS-COCO, without using any shortcuts. The experimental setup for training both models is provided in Appendix D.2 and D.3. The goal of this setup is to show the retrieval evaluation performance without adding any shortcuts for both large-scale pre-trained foundation model and small-scale model trained from scratch.
II Unique shortcuts: We add a unique shortcut to each image-caption tuple E D in the dataset. In this sctup, each image caption pair can be uniquely matched during training by only detecting the shorteut. For each tuple D, we use the number as the number of the shorteut we inject to the image and captions in the tuple. If the contrastive loss learns task-optimal representations, the downstream evaluation performance should not decrease when training with unique shortcuts.
III Unique shortcuts on only one modality: To show that the shortcuts do not interfere with the original task-relevant information (S, CA, and CB) of the images and captions, we create a datasct with only shorteuts either the image or caption modality. Therefore, the shorteut cannot be used by the encoders to match image-caption pair. Hence, expect the encoders to ignore the shorteuts and extract the features from the original data similar to the features learned by the baseline models in experimental setup I.
IV N bits of shorteuts: In this setup, for each image-caption pair in the training batch B, we randomly sample a shorteut number from the range [0, 2"), where is the number of bits. The higher the value of the more image-caption pairs in the training batch will have by expectation unique shorteut, and, the less the model has to rely on S and the remaining task-relevant information to solve the contrastive objective. The goal of this setup is to show that, the more unique (shorteut) information is present per sample in the batch, the less contrastive models rely on the remaining task-relevant information.
It should be noted that the shortcuts we add are independent of the image-caption pairs. However, the goal of the SVL framework is to measure the effect of the presence of additional dasy-to-detect shared information on the learned representations.
Evaluation Method. To show the effect of the injected shortcuts on retrieval evaluation performance, we evaluate both with and without adding the shortcuts during evaluation. When training with unique shortcuts, add unique shorteut to each tuple in the test set as well. When training with shortcuts on either one of the two modalities, we only evaluate without shortcuts to show that training with shortcuts on one modality does not influence performance. When training with bits of shortcuts, we add the shorteut mod (i,n) (modulo) to each tuple in the evaluation set, to make sure we use the same number of shortcuts during evaluation as during training. To facilitate the reproducibility and support further research, we provide the code with our paper."
'https://github http:////ggghaacccccaattttttttte framexork
8

---

shortcuts
Unique shortcuts
CLIP zero-shot
Only shortcuts on the images
N bits of shortcuts eval w/ shortcuts
Baseline no shortcuts
Unique shortcuts eval w/ shortcuts
N bits of shortcuts eval w/o shortcuts
600
500
500
Recall sum
400
400
200
200
100
100
2
5
6
10111213
7
10 12
Flickr30k
M5-COCO
(a) Evaluation results for the CL.IP model when using different shortcut sampling setupe.
600
600
500
500
Recall sum
400
300
300
200
100
100
1011121 1111
0
10 12
Number of bits
Number of bits
Flickr30k
MS-COCO
(b) Evaluation results for the VSE++ model when using different shortcut sampling setups.
Figure Effect of synthetic shortcuts CL.PP and VSE++ performance ICR task. The dotted line represents the maximum achievable recall sum, while the dashed line for CLIP indicates its zero-shot evaluation performance (Best viewed in color.)
4 Synthetic Shortcuts and their Impact on Learned Representations and Evaluation Performance
4.1 Findings
First, train and evaluate both CLIP and VSE++ without shortcuts the Flickr30k and MS-COCO dataset for the image-caption retrieval task baseline. We use the recall sum (i.e., the sum of R@1, R@5, and R@10 for both image-to-text (i2t) and text-to-image (t2i) retrieval) as evaluation metric (see Appendix B.1 for the evaluation task description). We visualize the results in Figure 5. The dotted line (in Figure 5a and 5b) indicates the maximum evaluation score (i.e., 600). For CLIP, we also provide the zero-shot performance of the model, indicated by the dashed line in Figure 5a. When referring to specific results Figure 5, we use the color of the corresponding bar and legend key in brackets in the text.
Based on Figure 5, draw the following conclusions:
When training CLIP and VSE++ with only shortcuts on either the caption modality (in Figure 5, the corresponding bar/legend box colored not observe a drop in evaluation scores for CLIP compared the bascline model (=,, 1, in Figure 5a). For VSE++ we only observe a slight drop evaluation score when training with shorteuts on the caption modality (again mainly for MS-COCO, in Figure 5b). Therefore, conclude that the
or on the image modality ( Figure 5), we do

---

synthetic shortcuts do not interfere with the original shared information S other task-relevant information.
II When training the models with unique shortcuts, observe for both CLIP and VSE++ that when evaluating with shortcuts evaluating without shortcuts (,, in Figure 5) the evaluation score for VSE++ drops to zero and for CLIP below the zero-shot performance. We conclude that with unique shortcuts: (i) both CLIP and VSE++ fully rely on the shortcuts to solve the evaluation task, (ii) VSE++ has not learned any other shared or task-relevant. information other than the shortcuts (since is trained from scratch, only detecting the shortcuts is sufficient to minimize the contrastive loss), and (iii) fine-tuned CLIP has suppressed original features from the zero-shot model in favor of the shorteuts.
in Figure 5), the models obtain a perfect evaluation score. When
III When training the models with N bits of shortculs, we observe for both CLIP and VSE++ that the larger the number of bits we use during training and when evaluating without shortcuts ( in Figure 5), the bigger the drop in evaluation performance. When evaluate with shorteuts
in Figure 5), the evaluation performance improves as we use more bits compared to the bascline without shortcuts = in Figure 5). For VSE++, evaluating without shortcuts ( (,, in Figure 5b) results in a drop to zero when having a large number of bits. For CLIP, the evaluation performance drops below the zero-shot performance. we train with 0 bits of shortcuts (i.c., the shortcut is a constant) we do not observe any drop or increase in evaluation scores for CLIP.
4.2 Upshot
Given the findings based on Figure 5 we conclude that contrastive loss (i.e., InfoNCE) mainly learns the easy-to-detect minimal shared features among image-caption pairs that are sufficient to minimize the contrastive objective while suppressing the remaining shared and/or task-relevant information. If contrastive losses are sufficient to learn task-optimal representations for imago-caption matching, these shortcuts should not adversely impact the evaluation performance. Moreover, the contrastive loss would ouly learn features that are shared among the image and all captions (i.e, S), we should not observe a drop in performance to 0 for the VSE++ model when training with unique shortcuts, since there still a lot of task-relevant information present in S. Especially in training setup where model is trained from scratch fine-tuned small datasets, the easy-to-detect. features are likely not equivalent to all task-relevant information in the images and captions. Hence, we conclude that contrastive loss itself is not sufficient to learn task-optimal representations of the images (and sufficient representations of captions) and that it only learns the minimal easy-to-detect features that are needed to minimize the contrastive objective.
5 Reducing Shortcut Learning
In the earlier section, we have demonstrated that contrastive loss mainly relies on the minimal, easy-to-detect features shared among image-caption pairs while suppressing remaining task-relevant information. In this section, describe two methods that help to reduce shortcut learning for contrastive learning SVL framework: Latent target decoding (Bleeker et al., 2023) and implicit feature modification (Robinson el al., 2021).
5.1 Latent Target Decoding
Latent target decoding (LTD)) (Blecker et al., 2023) is a method to reduce predictive feature suppression (i.e., shortcut learning) for resource-costrraine contrastive image-caption matching, The contrastive objective (i.e., InfoNCE) is combined with an additional reconstruction loss, which reconstructs the input caption from the latent representation of the caption z/c,- We refer to Appendix F.2 for the mathematical definition of LTD. Instead of reconstructing the tokens of the input caption in an auto-regressive manner (i.e., auto-encoding), the caption is reconstructed non-auto-regrest regressigsss,v by mapping the caption representation into the latent space of a Sentence-BERS (Reimers & Gurevych, 2019; Song et al., 2020) and minimizing the distance (i.e., reconstructing) between the reconstruction and the Sentenco-BERT representation of the caption x The assumption is that the larget generated by the Sentence-BERT model contains all task-relevant. information
10

---

in the caption. Hence, by correctly mapping the latent caption representation zic into the latent space of Beeence--RRRT,, the caption encoder cannot suppress any task-relevant information or rely on shorteut solutions. LTD is implemented both as a dual-loss objective (i.e., the contrastive loss and LTD are added up) and as an optimization constraint while minimizing the InfoNCE loss, by implementing the loss as a Lagrange multiplier. For the mathematical definition of LTD, we refer to Appendix E.2.
Experimental Setup. We the LTD implementation and set-up similar to Bleeker al. (2023). We train both CLIP and VSE++ with LTD, implemented either dual loss an optimization constraint. When implementing LTD constraint, try E {0.01, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3} as bound values. Similar to Bleeker et al. (2023), when implementing LTD as a dual loss, we use / = 1 as balancing parameters. We train both with and without unique shortcuts. We do this to show (i) what the performance improvement is compared to using only InfoNCE, and (ii) to what degree LTD prevents full collapse to shortcut features. For each model and dataset, we take the training setup that results in the highest performance on the validation set.
5.2 Implicit Feature Modification
Implicit feature modification (IFM) (Robinson et al., 2021) method, originally introduced in the context. of representation learning for images, that applies perturbations to logits used for guiding contrastive models. IFM perpetuates features that the encoders use during training step to discriminate between positive and negative samples. By doing so, IFM alters the features that are currently used to solve the discrimination task, to avoid the InfoNCE loss to learn shortcut solutions. How much of the features are removed, is defined by a perturbation budget e. IFM is implemented as a dual loss in combination with the InfoNCE loss. For the mathematical definition of IFM, we refer to Appendix E.3.
Experimental Sctup. We apply a similar experimental sct-up for IFM as for LTD. We apply IFM both to CLIP and to VSE++, both with and without unique shortcuts. Similar to (Robinson et al., 2021), we try different perturbation budgets e, we try e {0.05, 0.1, 0.2, 0.5, 1}. In line with the LTD setup, we take the training setup that results in the highest performance on the validation set.
5.3 Method Comparison
Both LTD and TFM aim to mitigate shortcut learning through different approaches. LTD aims to learn all task-relevant information by reconstructing the input captions. contrast, IFM perturbs the discriminative features the latent space of the encoder and does not rely on a reconstruction objective. Overall, both methods represent distinct strategies for improving the robustness and generalization capabilities of VL. representation learning.
In the following section, we present experimental results with LTD and IFM, providing insight into their
effectiveness in mitigating shortcut learning.
6 Experimental Results
6.1 Does Latent Target Decoding Reduce Shortcut Learning?
In Table we summarize the effect of LTD on reducing shorteut lcarning.
For CLIP, for both the Flickr30k and MS-COCO dataset, do not observe an increase in recall scores when fine-tuning with LIIENCE+LLT compared to models that are only fine-tuned with LInOONCE- LTD has originally been proposed for resource-constrainee VL models. We argue that the additional features that LTD can extract are either already present in the pre-trained CLIP model, or not relevant for the evaluation task. However, when fine-tuning with LInGONCE+LID and in the presence of shortcuts in the training data, degradation in recall scores significantly lower than when finc-tuned only with the f This shows that LTD can reduce the suppression of features in favor of the shorteut features when finc-tuning large-scale VL. models.
11

---

Table 1: Mean and variance (over three training runs) recall@k evaluation scores for the Flickr30k and MS-COCO datasets for image-to-text and text-to-image retrieval. We train with loss functions: CIiONCE and LIaIONCE+LTD. We train either with (v) or without (x) shortcuts. For the model trained with LInSNNCE+LTD, provide the typer-parameters of the best-performing model. y) indicates that the best- performing model uses LTD implemented as an optimization constraint with bound n, B indicates that the best- pest-performing model uses LTD implemented as a dual-loss with 3 1.
i2t
t2i
Loss
SyysS
R@1
R@5
R@10
R@1
R@5
R@10
rsum
Flickr30k
CLIP
SOSONCE
x
8.
0..
99.0
72.4
95.8+ 92.1
543.5 1.1
LLLDNNE+LLTD B=1
x
86.5
97.1
98.5
72.4
92.3+
95.9+
542.8+ 0.8
LIOGNCE
57.218.3 8.3
8..0
91.0 1.9
44.9 1.5
7..9
84.2 2.5
436.21 145.0
CILIONCE+LTD, B=1
64.
93.2 8..8
50.7
79.8
88.1
463.6; 17.s
VSE++
CILGNCE x
5..6 1.1
79.8
87.
39.5
69.8
79.4
409.0 4.0
LIDIONCE+LTD, 1) = 0.2
x
54.1 81.1
8816
42.
71.9
81.3
419.61
CIUNNCE
0.1
0.64
1.1
0.1
0.5+
1.0+
3.4
CIUSONCE+LTD. 17 0.05
24.7 51.8
65.6.1.4
20.7= 1of
49.2
02.6+1.27
274.6 1.67
MS-COCO
CLIP
CINONCE LINIONCE LLTD, B=1
x
63.8
06.1
92.3.
46.3
74.8
84.1+0.2
447.5
x
63.810.0-
86. 1..0
92.3 0.0
46.31 0.0
74.710.0
8..
44..410.0
LIaOONCE
13.6
31.542.4
42.2
7.3
22.1+
32.7
149.44 32.7
CINIONCE+LTD, ==1
18.9+01f
41.8+0.1
54.1+0.1
16.5
39.4+0.0f
52.640.11
223.44 0.2
VSE++
CINGNCCE
X
42.2+01
72.7+0.
83.2+ 0.1
30.9
61.2+
73.5+
363.8
LIDSONCE 1) = 0.1
X
43.6
1o16
83.710.0f 13.5
32.4
62.51
370.5 74.7
CINONCE CIoIONCE+LTD, = 0.01
0.0
0.1
0.2
0.0
0.1+
0.2+
0.7
V
3.9
13.7
21.6=09f
3.1=02f
11.0+1.ff
18.1+3.f
71.3+3of
Across the board, VSE++ models trained with the LISANCE loss consistently outperform the LIaONNE loss, both for i2t and t2i retrieval and both when trained either with or without shortcuts, as indicated by higher recall@Ak scores; this is consistent with the findings presented (Bleeker et al., 2023)). For both the Flickr30k and MS-COCO dataset, when trained with the LIaNNEE and with shortcuts present in the training data, the model performance collapses to around 0 in the absence of shorteuts (as we have seen in Section 4). However, when train with shorteuts in the training data and with LIsGNNCELLDD, Flickr30k and MS-COCO, a significant gain in performance. The performance improvement is bigger for Flickr30k than for MS-COCO. In general, the recall scores are still siguificantly lower than training without shortcuts, however, the models do not solely rely on the shortcuts anymore to minimize the contrastive loss and are able during evaluation (in the absence of shortcuts) still correctly match imago-caption pairs with each other. The results in Table show that LTD is able, in the presence of shortcuts in the training data, to guide (small-scale) VL. models that trained from scratch to not only learn the shorteut features that.
observe, for both
12

---

Table 2: Mean and variance (over three training runs) recall@k evaluation scores for the Flickr30k and MS-COCO datasets for image-to-text and text-to-image retrieval. We train with loss functions: CItoNCE and LIMIONCE+IFM- We train either with (V) or without (x) shortcuts. For the model trained with LI we provide the hyper-parameters of the best-performing model.
i2t
12i
Loss
SSn
R@1
R@5
R@10
R@1
R@5
R@10
rsum
Flickr30k
CLIP
LIOONCE
x
86.9
97.4+0.0
98.
0..
92.1
95.6+ 0.0
543.5+ 1.3
CIUSONCE+IFM: 0.05
87.4+0.1
97.4+022
09.1
73.2+
0.0
95.6 0.0
544.99 0.2
FILONCE
5.3 84.6
91.31 0.0
1.2
7..6....8
84.4 0.4
436.7 1. 18.8
CIoSONCE+IPM; == 0.1
78.8+0.f 91.5
95.640.of
58.9+0.1f
84.4+
91.1
495.2 5.71
VSE++
LILONCE
X
52.9
80.5
8..
40.
68.8 0.4
78.9 0.3
409.3
CIDIONCE+IPM: 6 0.05
x
52.4
76.
85.3+
29.1
68.8-40.1
78.2+
400.7
CIntONCE
V
0.1+0.0
0.4+
0.8
0.1+0.0
0.4+
1.0
2.9 0.0
LINGNNCR 0.05
V
0.0 0.6
0.2
0.1
0.5 0.0
1.04 0.0
3.2 0.8
MS-COCO
CLIP
LIUONCE X
63.5
86.0
92.2
46.3
74.74
84.2+ 0.0
446.9
LInSONCE 0.05
X
63.0
86.61
92.61
75.6 47.21
84.5 o.o
449.5 1.7
CINONCR
13.9
32.7
43.
8.8
24.7+0.2
35.5+ 0.5
159.4+
CIdNNCE+IIMM, E 0.05
V
23.4+1.5f 46.5+271
58.2+2.1
17.1+
38.9+0.f
51.3+ 1of
235.54 43.81
VSE++
COONNEE
X
41.7+0..
72.5
83.1
31.3+000
61.1 0.0
73.6 0.0
363.4 0.4
LIMGNCER HIPM: 0.05
X
4..2
70.8
81.610.1
30.8
74.3 61.5
o.o
359.3 1.4k
CIGONCE
0.0
0.1
0.2+
0.04
0.1
0.2 0.0
0.64 0.7
CIuNNCE+IFM E 0.05
V
0.0+
0.1+0.0
0.2+
0.0
0.1+
0.2+ 0.0
minimize the contrastive training objective but also represent other remaining task-relevant features in the data that. not extracted by LIafeNCE
6.2 Does Implicit Feature Modification Reduce Shortcut Learning?
In Table summarize the effect of IFM on reducing shortcut solutions.
For CLIP, observe that LILGNNCE-IM,, when training without shortcuts in the training data, only improves performance for the MS-COCO dataset for the t2i task. However, for both Flickr30k and MS-COCO we observe that, when training with unique shortcuts in the training data, fine-tuning with LITTONCE+IFM results in a significantly lower performance drop in recall score than when fine-tuning with the E Similar LTD, the recall@k scores are still lower than when trained without shorteuts in the training data. We conclude that IFM sufficient to reduce the suppression of features in favor of the shortcut features when fine-tuning large-scale VL. model, as indicated by higher recall@k scores when evaluating without shortcuts.
13

---

For VSE++, both for the Flickr30k and MS-COCO dataset, we do not observe that LIuIONCE+IFM outper- forms the LIiONCE, both with and without shortcuts present in the training data. We even observe that LInfONCE+IFM, when training without shorteuts, results in a decrease in performance across all recall@k metrics. When training with +INSNCE++PM and with unique shortcuts in the training data, the evaluation performance still collapses to around 0. The results in Table 2 show that IFM not sufficient to prevent. models trained from scratch from fully collapsing to the artificial shortcut solutions we introduce in this work (as opposed to LTD).
6.3 Upshot
In this section, we have evaluated two methods for reducing shorteut learning on our SVL framework: LTD and IFM. LTD proves effective in reducing shortcut learning for both CLIP and VSE++. IFM demonstrates its efficacy solely during the fine-tuning of CLIP. These findings indicate that our SVL framework is a challenging and interesting framework to study and evaluate shortcut learning for contrastive VL. models. Morcover, our results show that shortcut learning is only partially addressed by the evaluated methods since the evaluation results are not on par with the results on data lacking synthetic shortcuts.
7
Related work
We discuss related work on multi-view representation learning, vision-language learning, and shortcut learning.
Multi-view Representation Learning. To learn the underlying semantics of the training data, a subgroup of representation learning methods involves training neural encoders that maximize the agreement between representations of the similar views (van den Oord et al., 2018; Hjelm al., 2019; Chen al., 2020a; Radford et al., 2021; Bardes al., 2022). In general, for uni-modal representation learning, data augmentations are used to generate different views of the same data point. One of the core assumptions in multi-view representation learning is that each view shares the same task-relevant information (Sridharan & Kakade, 2008; Zhao et al., 2017; Federici et al., 2020; Tian et al., 2020a; Shwartz-Ziv & LeCun, 2023). However, the optimal view for contrastive self-supervised learning (SSL) (i.e., which information shared among views/which data augmentation is used) is task-dependent (Tian et al., 2020b; Xiao et al., 2021). Therefore, maximizing the mutual information (MI) between representations of views (i.c., shared information) docs not necessarily result in representations that generalize better to down-stream evaluation tasks, since the representations may contain too much additional noise that is irrelevant for the downstream task (Tian et al., 2020b; Tschannen et al., 2020). An open problem in multi-view SSL is to learn representations that contain all tesk-relevant information from views where each view contains distinct, task-relevant information Shwartzz-Ziv & LeCun, 2023), this is especially a problem in the multi-modal learning domain (Zong et al., 2023).
Chen et al. (2021) investigate multi-view representation learning for images using contrastive losses. They demonstrate that when multiple competing features exist that redundantly predict the match between two views, contrastive models tend to focus on learning the easy-to-represent features while suppressing other task-relevant information. This results in contrastive losses mainly capturing the easy features, even if all task-relevant information is shared between the two views, suppressing the remaining relevant information.
Several optimization objectives have been introduced to either maximize the lower bound on the MI between viows and their latent representations (van den Oord et al., 2018; Bachman et al., 2019; Hjelm et al., 2019; Tian et al., 2020a) or minimize the MI between representations of views while keeping the task-relevant information (Federici et al., 2020; Lee et al, 2021). To learn more tesk-relevant information that either might not be shared between views or that is compressed by a contrastive loss, several works proposed additional reconstruction objectives to maximize the MI between the latent representation and input data (Tsai el al., 2021; Wang et al., 2022; Li et al., 2023b; Bleeker et al., 2023). Liang et al. (2023) introduce a multimodal contrastive objective that factorizes the representations into shared and unique information, while also removing task-irrelovant information by minimizing the upper bound on MI between similar views.
lisiunnl Representation Learning. The goal of VL representation learning is to combine information from the visual and textual modalities into joint representation or learn coordinated represen-
14

---

tations (Baltrusaitis et al., 2019; Guo et al., 2019). The representation learning approaches can be separated into several groups.
Contrastive methods represent one prominent category of VL representation methods. The approaches in this group are typically dual encoders. Early methods in this category are trained from scratch; for instance, (Frome et al., 2013) proposed a VI. representation learning model that features skip-gram language model and a visual object categorization component trained with hinge rank loss. Another subgroup of methods uses dual-encoder with hinge- based triplet loss (Kiros et al., 2014; Li et al., 2019a; Lee et al., 2018). Kiros et al. (2014) use the loss for training CNN-RNN dual encoder. Li et al. (2019a) leverage bottom-up attention and graph convolutional networks (Kipf & Welling, 2017) to learn the relationship between image regions. Lee et al. (2018) add stacked cross-attention to use both image regions and words as context.
More recently, contrastive approaches involve transformer- based dual-encoders trained with more data than the training data from the evaluation sct(s). ALBEF (Li et al., 2021) propose to contrastively align unimodal representations before fusion, while X-VLM (Zeng et al., 2022) employs an additional cross-moda encoder to learn fine-grained VL representations. Florence (Yuan et al., 2021) leverages various adaptation models for learning fine-grained object-level representations. CLIP (Radford et al., 2021), scaled-up dual-encoder, is pre-trained on the task of predicting which caption goes with which image. ALIGN (Jia et al., 2021) uses a simple dual-encoder trained on over a billion image alt-text pairs. FILIP (Yao et al., 2022) is a transformer- based bi-encoder that features late multimodal interaction meant to capture fine-grained representations. SLIP (Mu et al., 2022) combines language supervision and image self-supervision to learn visual representations without labels. DeCLIP (Li et al., 2022b) proposes to improve the efficiency of CLIP pretraining using intra-modality self-supervision, cross-modal multi-view supervision, and nearest neighbor supervision.
Another line of work includes learning VL representations using models that are inspired by BERT (Devlin et al., 2019). VILBERT (Lu et al., 2019) and LXMERT (Tan & Bansal, 2019) expand upon BERT by introducing a two-stream architecture, where two transformers are applied to images and text independently, which is fused by a third transformer a later stage. B2T2 (Alberti et al., 2019), VisualBERT (Li et al., 2019b), Unicoder-V VL (Li et al., 2020a), VL-BERT (Su et al., 2020), and UNITER (Chen et al., 2020b) propose single-stream architecture, where single transformer is applied to both images and text. Oscar (Li et al., 2020b) uses caption object lags as anchor points that are fed to the transformer alongside region features. BETT-3 (Wang et al., 2023) adapt multiway transformers trained using cross-entropy loss (Bao et al., 2022).
Another category of methods for learning VL representations are generative mothods, that imply learning VL representation by generating new instances of one modality conditioned on the other modality. For instance, BLIP (Li et al., 2022a) bootstraps captions by generating synthetic captions and filtering out the noisy ones; BLIP-2 (Li et al., 2023a) bootstraps VL representation learning and, subsequently, vision-to-language generative learning. On the other hand, Tschannen et al. (2023) propose to pretrain a encoder- architecture via the image captioning task.
Shortcut Learning. Geirhos et al. (2020) define shortcuts in deep neural networks as "decision rules that perform well on standard benchmarks but fail to transfer to more challenging testing conditions, such as real-world scenarios." " In the context of deep learning, a shorteut solution can also be seen as a discrepancy between the features that a model has learned during training and the intended features that a model should learn to perform well during evaluation. For example, shortcuts might be features that minimize the training objective but are much easier to detect than the intended features that are relevant lo the evaluation task. Shortcut learning can be caused by biases in the dataset or inductive biases in either the network architecture or training objective.
Hermann& & Lampinen (2020) design a dataset with multiple predictive features, where each feature can be used as a label for an image classification task. The authors show that in the presence of multiple features that each redundantly predicts the target label, the deep neural model chooses to represent only one of the predictive features that are the easiest to detect, i.e., the model favors features that are easy to detect over features that are harder to discriminate. Next to that, they show that features that are not needed for a classification task, are in general suppressed by the model instead of captured in the learned latent representations.
15

---

Robinson et al. (2021) show that contrastive losses can have multiple local minima, where different local minima can be achieved by suppressing features from the input data (i.e., the model learns a shortcut by not learning all task-relevant features). To mitigate the shorteut learning problem, Robinson et al. (2021) propose implicit feature modification, a method that perpetuates the fcatures of positive and negative samples during training to encourage the model to capture different features than the model currently relies on.
Scimeca et al. (2022) design an experimental set-up with multiple shorteut cues in the training data, where each shorteut is equally valid w.r.t. predicting the correct target label. The goal of the experimental setup is to investigate which cues are preferred to others when learning classification task.
Latent target decoding (LTD) is method to reduce predictive feature suppression (i.e., shortcuts) for resource-constrained contrastive ICR by reconstructing the input caption in a non-autlo-regressive manner. Bleeker et al. (2023) argue that most of the task-relevant information for the ICR task is captured by the text modality. Hence, the focus is on the reconstruction of the text modality instead of the image modality. Bleeker et al. (2023) add a decoder to the learning algorithm, reconstruct the input caption. Instead of reconstructing the imput tokens, the imput caption is reconstructed in non-autoregreesive manner in the latent space of Sentence-BEii implemented as an optimization constraint and as a dual-loss. Li et al. (2023b) show that contrastive losses are prone to feature suppression. They introduce predictive contrastive learning (PCT), which combines contrastive learning with a decoder to reconstruct the input data from the latent representations to prevent shortcut learning.
(Reimers & Gurevych, 2019; Song et al., 2020) model. LTD can be
Adnan et al. (2022) measure the MT between the latent representation and the input as a domain agnostic metric to find where (and when) in training network relics on shortcuts in the input data. Their main finding is that, in the presence of shorteuts, the MI between the input data and the latent representation of the data is lower than without shorteuts in the input data. Hence, the latent representation captures less information of the input data in the presence of shortcuts and mainly relies on shortcuts to predict the target.
Our Focus. In this work, we focus on the problem of shortcut learning for VL the context of multi-viow VL representation learning with multiple captions per image. In contrast with previous (uni-modal) work multi-view learning, we consider different captions matching to the same image as different views. We examine the problem by introducing a framework of synthetic shortcuts designed for VL. representation learning, which allows us to investigate the problem in a controlled way. For our experiments, we select two prevalent VL models that are solely optimized with the InfoNCE loss: CLIP, a large-scale pre-trained model, and VSE++, a model trained from scratch. We select models that are solely optimized with a contrastive loss, to prevent measuring the effect of other optimization objectives on the shortcut learning problem.
8 Conclusion
In this work, we focus on the shortcut learning problem of contrastive lcarning in the context of vision- language (VL) representation learning with multiple captions per image. We have proposed synthetic shortcuts for vision-language (SVL):: training and evaluation framework to examine the problem of shorteut learning in controlled way. The key component of this framework is synthetic shorteuts that we add to image-text data. Synthetic shorteuts represent additional, easily identifiable information that is shared between images and captions. We fine-tune CLIP and train a VSE++ model from scratch using our training framework to evaluate how prone contrastive VL models are to shortcut learning. Next, we have evaluated how shortcut learning can be partially mitigated using latent larget decoding and implicit feature modification.
Main Findings. We have conducted experiments on two distinct VL models, CLIP and VSE++, and have evaluated the performance Flickr30k and MS-COCO. We have found that when training with unique shortcuts, CLIP suppresses pre-trained features in favor of the shortcuts. VSE++ only learns to represent the shorlcuts, when using unique shortcuts, showing that none of the remaining lask-relevant (both shared and unique) information is captured by the encoders when training a model from scratch. When using n bits of shorteuts, we have shown that the more bits we use, the more the contrastive VL models rely on the synthetic shortcuts. Our results demonstrate that contrastive VL methods tend to depend on easy-to-learn discriminatory features shared among images and all matching captions while suppressing the
16

---

remaining task-relevant information. Next, we have evaluated two methods for reducing shortcut learning our framework of synthetic shortcuts for image-caption datasets. Both methods partially mitigate shortcut. learning when training and evaluating with our shortcut learning framework. These findings show that our framework is a challenging framework to study and evaluate shortcut learning for contrastive VL and underline the complexity of our framework in studying and evaluating shortcut learning within the context of contrastive VL representation learning.
Implications. The implications of our findings are twofold. First, we examine the limitations of contrastive optimization objectives for VL representation learning, demonstrating that they predominantly capture features that are easily discriminable but may not necessarily constitute task- representations. Second, our work contributes a novel framework for investigating shortcut learning problem in the context of VL representation learning with multiple captions per image, providing insights into the extent to which models rely on shorteuts when they are available and how existing shorteut reduction methods are capable of reducing shortcut learning when training with our framework.
Limitations. Some of the limitations of our work are related to the fact that we focused on two specific models, one optimization objective (InfoNCE), and two datasets, and the generalizability of our findings to other VL models, optimization objectives, and datasets warrants further exploration. Additionally, the synthetic shortcuts introduced in this work are not dependent on image-caption pairs. Our training and evaluation sctup shows that, in the presence of shortcuts in the training data, contrastive VL models mainly rely on the easy-to-detect shortcut features, which indicates that the InfoNCE loss cannot learn lasks-optimal representations for VL tasks when multiple captions are used for training. However, it remains unclear to what degree the unique information of the captions is captured by the contrastive loss VL models.
Future Work. We suggest working on the development of optimization objectives that specifically address the shortcut learning problem for VL training with multiple captions per image. We also suggest extending our synthetic shortcuts for image-caption datasets to a framework with unique shortcut information per caption. By having unique shorteut information per caption, it becomes possible to measure how much of the shared/caption--peecffc shorteut information is captured by encoder models. Another future direction includes investigating alternative training strategies or loss functions to further miligale shortcut learning problems. Another promising direction for future work includes the improvement of existing methods or the exploration of novel techniques that address the limitations of existing shorteut reduction methods, potentially through the combination of multiple approaches. Extending the SVL framework to better capture muances and complexities of natural data is another important direction that would facilitate a more comprehensive understanding of the implications of shortcut learning in real-world scenarios and datasets.
9 Broader Impact
This paper motivates and introduces framework for investigating the problem of shortcut learning for contrastive VL representation learning with multiple captions per image in a controlled way. It also examines how two shortcut learning reduction methods perform on the proposed framework. Overall, the framework provides a tool for analyzing and understanding the problem of shortcut lcarning in the context of contrastive VL representation learning; can be used in various settings that require deeper insight into the quality of learned VL representations.
We should be aware that the reliance shortcuts in VLMs poses ethical concerns with potential real-world implications. Models that learn shorteuts may overlook nuanced details in images and text, leading to biased or inaccurate outcomes. Furthermore, the transparency and explainability of VLMs are crucial considerations. Models that rely on shorteuts may make decisions based on features that are not easily interpretable or explainable to users. This lack of transparency can diminish trust in AI systems.
Acknowledgements
We thank Marco Federici and Mathijs Henquet for the discussions on mutual information and feedback on the draft. Additionally, we thank Shashank Gupta and Panagiotis Elstratiadis for helpful feedback.
17

---

This research was supported by the Nationale Politie, Ahold Delhaize, project IDEAS with project number VI. 223.166 of the NWO Talent Programme, which is (partly) financed by the Dutch Re- search Council (NWO), the Hybrid Intelligence Center, a 10-year program funded by the Dutch Min- istry of Education, Culture and Science through the Netherlands Organisation for Scientific Research, .n, project LESSEN with project number NWA.1389. 20. 183 of the research program NWA ORC 2020/21,, which is (partly) financed by the Dutch Research Council (NWO), project ROBUST with project number KICH3.LTT.20.000 which is (partly) financed by the Dutch Research Council (NWO), DPG Media, RTL, and the Dutch Ministry of Economic Affairs and Climate Policy (EZK) under the program LTP KIC 2020-2023, and the FINDHR (Fairness and Intersectional Non-Diserimination in Human Recommendation) project that received funding from the European Union's Horizon Europe research and innovation program under grant agreement No 101070212. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
References
Mohammed Adnan, Yani Toannou, Chuan- Yung Tsai, Angus Galloway, H.R. Tizhoosh, and Graham W. Taylor. Monitoring shortcut learning using mutual information. arXiv preprint orXiv:2206.1800 13034, 2022.
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. In EMNLP, pp. 2131-2140, 2019.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NewrIPS, pp. 15509 15519, 2019.
Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41:423-443, 2019.
Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. VLMo: Unified vision-language pre-training with mixture-of-modaliyyyeeeeee 2022.
NewrIPS, pp.
Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Varfacceimaarine--ovvraaaa regularization for self-supervised learning. Tn TCLR, 2022.
Ali Furkan Biten, Andrés Mafla, Lluis Gómez, and Dimosthenis Karatzas. Is an image worth five sentences? A new look into semantics for image-text matching. In WACV, pp. 2483 2492. IEEE, 2022.
Maurits Bleeker, Andrew Yates, and Maarten de Rijke. Reducing predictive feature suppression in resource- constrained contrastive image-caption retrieval. Transactions on Machine Learning Research, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, pp. 1597-1607, 2020a.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. In NewrIPS, pp- 11834-11845, 2021.
Xinlei Chen, Hao Fang, Tsung- Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint Xrx:::5004.000 2015.
Yen- Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: Universal image-text representation learning. In ECCV, pp. 104-120, 2020b.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülgehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In ACL, pp. 1724-1734, 2014.
Jacob Devlin, Ming- Weig Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACI-HILT, 2019.
18

---

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual-semantic embeddings with hard negatives. In BVCM, pp. 12, 2018.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynop Akata. Learning robust representa- tions via multi-view information bottleneck. In ICLR, 2020.
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. DeViSE: A deep visual-semantic embedding model. NeurIPS, pp. 2121-2129 2013.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shorteut learning in deep neural networks. Nature Machine Intelligence, pp- 665-673, 2020.
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey. IEEE Access, 7:63373 63394, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 778, 2016.
Katherine L. Hermann and Andrew K. Lampinen. What shapes feature representations? Exploring datasets, architectures, and training. In NeurIPS, pp. 9995 10006, 2020.
R. Devon Hjelm, Alex Fedorov, Samuel and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. ICLR, 2019.
Karan Grewal, Philip Bachman, Adam Trischler,
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text. supervision. In JCML, pp. 4904 4916, 2021.
Andrej Karpathy and Fei-Fei Li. Deep visual-semnn semanicc alignments for generating image descriptions. In CVPR, pp. 3128-3137, 2015.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Tn ICLR, 2017.
Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2592 2014.
Kuang- Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pp. 201-216, 2018.
Kuang-Huei Lee, Amurag Arnab, Sergio Guadarrama, John F. Canny, and Ian Fischer. Compressive visual representations. In NewrIPS, pp. 19538 19538-19552, 19552, 2021.
Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-VI. A universal encoder for vision and language by cross-modal pre-training, In AAAI, 2020a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NewrIPS, pp. 9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In TCML, pp. 12888 12888-12900, 2022a.
19

---

Jumnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-iman image pre-training with frozen image encoders and large language models. In ICML, pp. 19730-19742, 2023a.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, pp. 1654-4662, 2019a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. VisualBERT: simple and performant bascline for vision and language. arXiv preprint arXi::1905 03537, 2019b.
Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogério Feris, Piotr Indyk, and Dina Katabi. Addressing feature suppression in unsupervised visual representations. In WACV, pp. 1411-1420, 2023b.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In ECCV, pp. 121-133 137, 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjic Yan. Supervision exists everywhere: A data ellicient contrastive language-image pre-training paradigm. In ICLR, 2022b.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Zou, Louis-Pillippe Morency, and Russ Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. In NewrIPS, 2023.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pp. 740 755, 2014.
Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NewrIPS, pp. 13-23, 2019.
Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. SLIP: Self-supervision meets language image pro-training. In ECCV, pp. 529-544, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, pp. 9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pp. 8748 8763, 2021.
Nils Reimers and Iryna Gurevych. Sentence- BERT: Sentence embeddings using siamese BERT-Networks. In EMNLP-IJCNLP, pp. 3980 3990, 2019.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? In NewrlPS, pp. 4974 4986, 2021.
Luca Scimeca, Soong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shortcut cues will DNNs choose? A study from the parameter-space perspective. In ICLR, 2022.
Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress-sell--upeevisss learning and information theory: review. arXiv preprint ar XrXiv:2304.09355, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Yan Liu. MPNet: Masked and permuted pre-training for language understanding. In NewrIPS, pp. 16857- 16867, 2020.
Karthik Sridharan and Sham M. Kakade, An information theoretic framework for multi-view learning. In COLT, pp. 403 414, 2008.
20

---

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VL-BERT: Pre-training of generic visual-linguistic representations. In ICLR, 2020.
Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from transformers. In EMNI.P-JCNT.P, pp- 5099-5110, 2019.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In ECCV, pp- 776-794 794, 2020a.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NewrlPS, pp. 6827 6839, 2020b.
Yao-Hung Hubert Tsai, Yue Wu, Ruslan Salakbutdinov, and Louis-Philippe Morency. Self-supervised learning from multi-view perspective. In ICLR, 2021.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucie. On mutual information maximization for representation learning. In ICLR, 2020.
Michael Tschannen, Manoj Kumar, Andreas Peter Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. In NewrIPS, 2023.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXit:1807.08744. 2018
Haoqing Wang, Xun Guo, Zhi-Hong Deng, and Yan Lu. Rethinking minimal sufficient representation contrastive learning. In CVPR, pp. 16020 16029, 2022.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammod, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign languago: BEIT pretraining for vision and vision-language tasks. In CVPR, pp. 19175 19186, 2023.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre-Alvis Rebuffi, Ira Ktena, Krishnamurthy Dvijotham, and Ali Taylan Cemgil. A fine-grat fine-grained analysis on distribution shift. In ICLR, 2022.
Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In ICLR, 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. FILIP: Fine-grained interactive language-image pre-training. In ICLR, 2022.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67 78, 2014.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint Xiv:21111 1.111111111111111 11432, 2021.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like mags-of-words and what to do about it? In ICLR, 2023.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In ICML, pp. 25994 26009, 2022.
Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Inf. Fusion, 38:43-54, 2017.
Yongshuo Zong, Oisin Mac Aodha, and Timothy Hospedales. Self-supervised multimodal learning: A survey. arXiv preprint Xrxix:2304, 01008, 2023.
21

---

A
Notation
Table 3: Notation used in the paper.
Symbol
Description
LIfONCE
InfoNCE loss
CILONNE+LLD
Loss that combines InfoNCE and latent target decoding (LTD)
LIN
Loss that combines InfoNCE and implicit feature modification (IFM)
Lrecon
Reconstruction loss
Dataset 'D that comprises N image-caption tuples: D = {x2x (x6, )5=1) J i-th image-
N
D
caption tuple in the dataset D consist out of an image x/ and ht associated captions fxt, 3
B
Batch of image-caption pairs
X1
Image
x
Caption
2I
Latent representation of image X
2
Latent representation of caption X
ZC-I
Latent representation of caption X that is sufficient for image X
zI-C
Latent representation of image Xy sufficient for caption X
Latent representation of image X1 that is minimal sulficient for caption X
OPT "I-K
Latent representation of image Xz that is optimal for set of captions K given task T
Task-relevant information
R
Task-irrelevant information
C
Task-relevant information specific for a caption X
SSySSC
Synthetic shortcut
S
Original shared information
S+
Shared information that includes synthetic shorteut
R+
Task-relevant information that contains synthetic shortcut
fo(-)
Image encoder parametrised by 0; takes image Xy as input and returns its latent representation Zz: 2y fx(x;)
(.)+)
Caption encoder parametrised by Φ; takes caption X as input and returns its latent represen- tation ze: 2 == (ze)
T
Temperature paramater of LINONCE
Perturbation budget for LIFM
Reconstruction bound for LLTD
B Problem Definition and Assumptions
In this work, we solely focus on contrastive VL representation learning. We work in setting where we investigate the problem by fine-tuning large pre-trained foundation model (CLIP, Radford al., 2021) and training resource-constrained image-text method from scratch (VSE++, Faghri et al., 2018). We train and
22

---

evaluate using two benchmark datasets where multiple captions per image are available: Flickr30k (Young et al., 2014) and MS-COCO Captions (Lin et al., 2014). Both datasets come with 5 captions per image. We work in a dual-encoder setup, i.e. we have separate image and caption encoder, which do not share parameters.
B.1 Evaluation Task
The image-caption retrieval (ICR) evaluation task, consists two sub-tasks: image-to-text (i2t) and text-to- image (121) relrieval. In ICR, either an image or a caption used as a query and the goal to rank a set of candidates in the other modality. In this work, we follow the standard ICR evaluation procedure (see, e.g., Faghri et al., 2018; Lee et al., 2018; Li et al., 2019a). The evaluation metric for the ICR task is Recall with k {1, 5, 10}. For t2i retrieval, there is one matching/ ppositive image per query caption (when using the Flickr30k or MS-COCO or dataset). Hence, the Recall@k metric represents how often the correct image is present in the top-k of the ranking. For i2t retrieval, however, there are 5 matching captions per image. Therefore, only the highest-ranked correct caption taken into account when measuring the Recall@k (i.e., in the highest-ranked caption present in the top k). Standard practice select the best model checkpoint during training is to use the recall sum (rsum) as a validation metric. The recall sum the sum of recall 1, and 10, for both i2t and 12i. Therefore, the maximum value of the recall sum is 600.
B.2 Assumptions
Throughout this work, we rely on several assumptions about the problem definition. Our assumptions are defined at the level of image-text tuple. Following Section 2, we formalize the assumptions the case where one image is associated with two captions: (xz, (xex, xcg]).
Assumption 1. Each caption in the tuple contain information that distinct from the other captions in the tuple and all captions and image in the tuple contain shared and unique information:
1(x x xC 0
1(x7iXcA Xcg > 0, I(x7i Xcg xcx)) > O and I(xAA XCe (x. > 0
H(xz XCx: xCn) > 0, Hxc,, X1, Xc,) > O and H Hxxe, X1, x2xx >0. 0. Assumption 2. Task-relemnd information is the combination of all the information shared belween on image and each caption the tuple:
R= I=xxxx; xca) I(xx;xCa (xxa) + Xx (CB))
C Analysis of Contrastive Learning for Multiple Captions per Image
Theorem 1 (Suboptimality of contrastive learning with multiple captions per image). Given an image X set of matching captions C = fxe, Xex ,, and a contrastive learning loss function LIfoNCE that optimizes for lask T, image representations leurned during contrastive learning will be minimal sufficient and will never be task-optimal image representations. More formally, assume that:
(H+) Vi,j E (A, such that + IZZMIN xx,) )=)== = I(xxi (xe,) + (exxix),;
(H2) Ji,j B) with # such that X(xx;Xe, xx,,) > 0.
Then the following holds:
(T2) Bie (A.B) such that 1(z2"T: xc, xc,)
Iz (z7-,, xc,))
Proof. Following Eq. we define a task-optimal representation of an image w.r.l. all matching captions in as:
1PZOPT: xe,xxx,xxxx xen) ) 1xxxixea
xxe)
1(xzi Xcn (xxa)
(X2i xcn)
CA
Cb
23

---

Furthermore, following Definition 2.3, define minimal suficcint representations of image Xx W.r.t. each matching caption in C as combination of caption-specifie and shared information:
MIN (2 xxx)) 1(xx; xex xxnn [xxx x
Ca
T( x(2)= T(x;: CC
CB
Following assumption H2, for at least one caption X E C associated with the image Xy. caption-specifie information is positive. Therefore, we consider two cases:
If caption-specifie information of xca is positive, that is, if [(xzi Xca (xca) 0:
1(xyiXea xce) + (xxx XCB xcx) + (xx2 XexiXes
1XyxxXep xxx)) + 1(xzi Xe, xCe)
(# (#7-0ixe, kcy)
I( xxeg)
[1I--65x xC2)
(2II-CE xCo)
T(Z
MIN
Similarly, if caption-specifie information of Xcn is positive, that is, if 1 Xx 1xca) 0:
[xxiiXca (ce) + I(xxxx XCs xxx)) + (xxxxxxxx
1(xziXes (xcx) [xxx; Xc, xCB)
( xcn))
Xx, Xex Xco) > I ;x,,)
Therefore, show that in a setup where single image is associated with multiple captions, and given at least one caption contains caption-specifie information, image representations learned contrastively w.r.t. associated captions would contain less information than task-optimal image representation: 3 i e OPT,
{A, B} such that I(z - Xx &ce > [22 ; xx,)
D Experimental Setup
D.1 Datasets
Flickr30k consists of 31,000 images annotated with matching captions (Young et al., 2014).
MS-COCO consists of 123,287 images, each image annotated with 5 matching captions (Lin et al., 2014).
The original dataset was introduced for large-scale object recognition.
For both datasets, we use the training, validation, and test splits from (Karpathy & Li, 2015).
D.2 Models
We use CLIP and VSE++. Both consist of image and text encoder that do not share parameters.
CLIP is large-scale image-text foundation model (Radford et al., 2021). The model is pre-trained on a collection of 400 million image-text pains collected from the Web. The encoders are pre-trained using a contrastive loss (InfoNCE) on image-text pairs. The text encoder of consists of 12-layer transformer model, described in (Radford et al., 2019). As for the image encoder, CLIP utilizes various model backbones, such as ResNet (He et al., 2016) and Vision Transformer (Dosovitskiy al., 2021). In this work, we use the ResNet-50 ('RN50') variant of the CLIP image encoder.? The CLIP encoders trained to jointly understand images
2httpe://github. httggithhub..co//pesss//iiii
24

---

and text. Therefore, the learned representations generalize to wide range of different zero-shot (visual) evaluation tasks, such as classification, without task-specific fine-tuning, by using textual prompts.
VSE++ is an image-caption encoder trained from scratch (Faghri et al., 2018). The model features a triplet loss function with a margin parameter a = 0.2. The text encoder is a one-layer gated recurrent unit (GRU) (Cho al., 2014). The available image encoder configurations are ResNot-152 (He et al., 2016) and VGG19 (Simonyan & Zisserman, 2015). In this work, we ResNet-152.
D.3 Training
CLIP. To fine-tune CLIP, we follow (Yuksekgonul et al., 2023). All models are fine-tuned for 5 epochs. We employ cosine-annealing learning rate schedule, with a base learning rale of 2e 5, and 100 steps of up. As optimizer, use AdamW (Loshchilov & Hutter, 2019) with gradient clipping value of 2. For the InfoNCE loss, we use the logit-scale (i.c., temperature 7) from the pre-trained CLIP model and fine-lune the logit-scale end-lo-end along with the rest of the model parameters.
VSE++. The model is trained for 30 epochs using a linear learning rate schedule with a base learning rale of 2e 4. We use the Adam optimizer (Kingma & Ba, 2015) with gradient clipping value of 2. Instead of the triplct loss, we use the InfoNCE loss similar to Radford et al. (2021).
For both models, instead of selecting the best-performing model based on the validation set scores, we use the final checkpoint at the end of training.
D.4 Shortcut Sampling
Our goal is to add the shortcuts in manner that preserves the original information of the images and captions. For the captions, we append the shortcut at the end of the captions. In order to prevent a tokenizer from lokenizing the shortcut into a single loken, we insert spaces between each number of the shortcut. For the images, we place the numbers of the shortcuts at the top of the images, evenly spaced across the entire width of the images (to make sure the shortcut is evenly spaced across the feature map of the image). We always use 6 digits to represent a shortcut. If a shortcut number contains fewer than 6 digits, we fill the remaining positions with zeros for padding. For the MNIST images, always sample a random image from the set of images representing the number that belongs to (also during evaluation), to prevent overfitting specific MNIST images. In Figure 6, we provide four examples of image-caption pairs with randomly added shortcuts. The examples in Figure 6 show (i) how synthetic shortcuts are added to the image and the caption, and (ii) that the shortcuts preserve the original (task-rclovant) information of the images and captions.
E
Optimization Objectives
E.1 InfoNCE
In this work, we use InfoNCE loss, LIfONCE (van den Oord et al., 2018). Given dual-encoder setup, we optimize model in two directions: image-to-text (i2t) and text-to-image (t2i). The loss is defined as follows:
1
exp( exp(z/z6/T)
InfONCE
2
log
B
exp(ztzi/7) + Σソ exp(ziz(/7))
1
exp (2/ /T/
INSONCE
Ci2
log
B
exp(2).2/77 Σjtt exp 2/7/
iEB
2 InfoNNE: +
ci2t
2 _
cti
25

---

3
9
9
6
7
7
(a) Caption: "A bathroom sink with wood finish cabi-
(b) Caption: "A guy in brown shirt has just hit tennis ball. 0 777 114"
nets. 0 9 9 5."
6
a
6
9
a
3
2
20
(c) Caption: "A man in shorts is lying the beach. 0 06 9."
(d) Caption: "A player up bat in baseball game. 0 9 2.
Figure 6: Four random samples from the MS-COCO dataset including shorteuts added both the image and caption.
E.2 Latent Target Decoding
Latent target decoding (LTD) (Blecker et al., 2023) an optimization objective that reduces predictive feature suppression for resoure-constrained VL methods. LTD consists of LIMONCE and reconstruction loss Crecon, which reconstructs the input caption from the latent representation ze-
26

---

In (Blecker et al., 2023), LTD is implemented in two ways. Firstly, dual optimization objective:
LIDONCCE+LT LIfONCE 3Lrecon
Secondly, optimization constraint in combination with gradient descent by using the method of Lagrange multipliers:
max min LING LIGGNNCE + A (Freoon-))
77 -1).
This optimization objective is minimized w.r.t. model parameter, while also being maximized w.r.t. A. The value À is automatically tuned by gradient ascent, such that the reconstruction bound rj is met. In this work, we use both LTD dual optimization objective and an optimization constraint. We select the loss with the highest evaluation scores on the validation set for evaluation.
E.3 Implicit Feature Modification
Implicit feature modification (TFM) (Robinson et al., 2021) a contrastive loss, with an additional perturbation budget IFM perturbs the logits value of the similarity scores between the images and captions, such that the model avoids using shortcut solutions for correct similarity score. IFM subtracts e/t from the positive logit values and adds e/T to the negative logits values.
exp( e)/T)
22
log
IFM
B
exp((z)z2e) c)/T) + Eigs exp((z)2) +c//)²
IEB
exp((2}2' -
IFM
2)
1
log
exp((z?72)) - <)/T) + Ejjt exp( + +)/T))
KEB
1 pr2i
1 -:2
IIFM
2- IIIM
+
2 LFFM
1 II _fIFM + CI
2
Similar to Robinson al. (2021), we combine IFM and the InfoNCE in dual optimization objective.
27


Reviews:
[{"review_id": "AXYoYXQyBO", "reviewer": "Reviewer_nQcp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of creating a synthetic shortcuts framework is quite interesting. It allows researchers to deliberately inject shortcuts into the training data, making it possible to analyze how much vision-language models (VLMs) rely on these shortcuts. This controlled setup is a clever way to explore the weaknesses in current models and provides a clear path to understanding and improving contrastive learning.\n\n2. The authors did a thorough job by testing their methods on well-known benchmarks like Flickr30k and MS-COCO. By evaluating two distinct models (CLIP and VSE++), both from scratch and fine-tuned, they provide a robust set of results that highlight the prevalent issues in shortcut learning. This comprehensive approach ensures that the findings are not model-specific and can be generalized across different VLMs.\n\n3. The introduction of latent target decoding (LTD) and implicit feature modification (IFM) as methods to reduce shortcut learning is another strength. Even though these methods only partially address the problem, they still represent significant steps forward. The authors' honest assessment of the challenges these methods face underscores the complexity of the issue and the need for further research, but it also shows their potential to improve model performance.\n\nWeakness:\n\n1. While the methods introduced for reducing shortcut learning, such as latent target decoding (LTD) and implicit feature modification (IFM), are interesting, they only partially mitigate the problem. The paper acknowledges that these techniques do not bring the models' performance up to the level of those trained without synthetic shortcuts, indicating that more effective solutions are still needed.\n\n2. The reliance on synthetic shortcuts to investigate the problem might limit the real-world applicability of the findings. Synthetic data does not perfectly capture the complexities and nuances of natural data. This means that the insights gained might not fully translate to scenarios where the shortcuts are not artificially introduced but are inherent in the data.\n\n3. The paper primarily focuses on the limitations of contrastive losses in learning task-optimal representations. While this is a valid concern, it might have been beneficial to explore a broader range of loss functions or alternative training strategies. By concentrating mostly on contrastive learning, the paper may miss out on other potential avenues for improving vision-language models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The authors have developed a new framework called \"synthetic shortcuts for vision-language\" that allows deliberate addition of shortcuts to image-caption pairs in training data. This framework helps in studying how much contrastive vision-language models (specifically CLIP and VSE++) depend on these shortcuts when they are available. By testing the models on the Flickr30k and MS-COCO benchmarks, the authors compare their performance with and without these synthetic shortcuts. The results show that both models, whether trained from scratch or fine-tuned from large pre-trained models, tend to rely heavily on these shortcut features, rather than learning the most effective representations for the tasks. This suggests that contrastive losses tend to pick up on the easy, obvious features that are shared between images and their matching captions, ignoring other important information. Therefore, the authors argue that contrastive losses alone aren't enough to learn the best representations for vision-language tasks.\n\n2. The authors have also introduced two methods to reduce shortcut learning in their framework: latent target decoding (LTD) and implicit feature modification (IFM). Both methods improve the models' performance on evaluation tasks, but they still face challenges. The models trained with these methods don't perform as well as those trained without any synthetic shortcuts. This highlights the complexity of the framework and the need for more effective techniques to tackle shortcut learning in contrastive vision-language representation learning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The presentation is generally good but it could be better to provide sample cases to demonsatrate what is unique information in a sample pair, which could make the paper more readable.\n\n2. To address the narrow focus on contrastive losses, it would be beneficial for the authors to investigate other loss functions or alternative training strategies that might mitigate shortcut learning more effectively. Incorporating a comparison of different approaches could provide a more comprehensive understanding of how to achieve task-optimal representations in vision-language models.", "broader_impact_concerns": "The reliance on shortcuts in vision-language models can lead to ethical concerns in real-world applications. Models that primarily learn from shortcuts may fail to capture nuanced and critical details in images and text, potentially resulting in biased or inaccurate outcomes. For instance, in applications such as medical imaging or autonomous driving, reliance on superficial features rather than comprehensive representations could have serious, even life-threatening consequences.\n\nAnother concern is the transparency and explainability of VLMs. Models that rely on shortcuts may make decisions based on features that are not easily interpretable or explainable to users. This lack of transparency can undermine trust in AI systems, particularly in critical applications where understanding the model's decision-making process is essential.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TRf17mojy8", "reviewer": "Reviewer_wmwY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: This paper did a solid theory analysis from the perspective of multi-view learning and mutual information. A synthetic shortcut experiments also report an interesting finding to demonstrate the effect of shortcuts. Extensive experiments are conducted to verify the proposed two methods.\n\nWeakness:\n\n1.The figure1 is not intuitive enough to illustrate the concept of \"shortcuts\". Authors should compare (a) and (b) and give an intuitive explanation about shortcut learning and task optimal learning.\n2.There is no figure to illustrate how synthetic shortcuts are added to the image and text data. It would be better to move Figure 4 to the main body for better experiment understanding.\n3. The description of the proposed two methods is too short (sections 5.1 and 5.2). The insight behind these should be stated more clearly.\n4.It would be better to provide some qualitative analysis, such as giving an example with one image equipped with multiple captions, and stating what is the task optimal representations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of contributions: This paper first finds that contrastive learning in vision-language models tends to capture the \"shortcut\" features rather than the task-optimal features. Then, the paper proposes two methods to reduce the effect of \"shortcut\" features.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness", "broader_impact_concerns": "no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "slekEY33p7", "reviewer": "Reviewer_RZvX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper studies the essential problem of identifying potential caveats for contrastive learning with multiple views of the data points. The findings could be helpful for the community and intriguing for future research.\n\n2. The paper conducts thorough experiments with solid evidence that 1) contrastive learning is prone to shortcut info and 2) some existing methods can effectively prevent that.\n\n3. The paper gives some theoretical analysis showing that contrastive learning cannot learn the optimal task info.\n\nWeaknesses:\n1. The paper proposes a new method that can introduce shortcut info into the dataset. However, the way that the MNIST digits are injected into the image is not well-studied and tested. For example, what if the digits are injected in a way that is less detectable? In other words, there is no knob provided to control the strength of the injection.\n\n2. The paper only evaluates existing methods for solving the shortcut problem. It does not propose new techniques or combine existing methods to improve performance. Also, the improvement for the LTD method is not that significant. It is not well understood why IFM does not give benefits.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the problem of contrastive learning for vision-language models with multiple captions. The paper first shows that, in theory, InfoNCE loss could not capture the whole task information in multiple captions and will be prone to shortcut features in the data. The paper then proposes a way to introduce shortcut information into the dataset and shows that the contrastive learning performance degrades terribly. The paper then evaluates two existing methods that could potentially prevent the model from learning from shortcut information only. It shows that one of the two methods is helpful in reducing learning from shortcuts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Are there ways to improve/combine existing methods to make the training more resilient to the shortcut injection?\n\n2. What are the potential reasons that IFM does not work very well, and is there any empirical evidence to support that?\n\nMinor:\n1. Table last row third column value is unusually large and much larger than the baseline. Are there any explanations for that?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses the significant problem of "short-cut" learning with solid theoretical support and extensive experiments, providing valuable insights to the machine learning community. While it does not propose new or combined methods, its detailed analysis and findings are relevant and interesting. The clear presentation of evidence, including the newly added Figure 1, enhances its contribution.
Decision: Accept (poster)