Reducing Predictive Feature Suppression in Resource-Constrained Contrastive Image-Caption Retrieval
Authors: Maurits Bleeker
Andrew Yates
Maarten de Rijke
Venue: Accepted by TMLR ()

Abstract: To train image-caption retrieval (ICR) methods, contrastive loss functions are a common choice for optimization functions. 
Unfortunately, contrastive ICR methods are vulnerable to predictive feature suppression. 
Predictive features are features that correctly indicate the similarity between a query and a candidate item. 
However, in the presence of multiple predictive features during training, encoder models tend to suppress redundant predictive features, since these features are not needed to learn to discriminate between positive and
negative pairs. 
We introduce an approach to reduce predictive feature suppression for resource-constrained ICR methods: latent target decoding (LTD). We add an additional decoder to the contrastive ICR framework, to reconstruct the input caption in a latent space of a general-purpose
sentence encoder, which prevents the image and caption encoder from suppressing
predictive features. 
We implement the LTD objective as an optimization constraint, to ensure that the reconstruction loss is below a bound value while primarily optimizing for the contrastive loss. 
Importantly, LTD does not depend on additional training data or expensive (hard) negative mining strategies. 
Our experiments show that, unlike reconstructing the input caption in the input space, LTD reduces predictive feature suppression, measured by obtaining higher recall@k, r-precision, and nDCG scores than a contrastive ICR baseline.
Moreover, we show that LTD should be implemented as an optimization constraint instead
of a dual optimization objective. Finally, we show that LTD can be used with different
contrastive learning losses and a wide variety of resource-constrained ICR methods.

Published in Transactions on Machine Learning Research (04/2023)
Reducing Predictive Feature Suppression in Resource- Constrained Contrastive Image-Caption Retrieval
Maurits Bleeker
m.j.r.blecke@@woo.t
University of Amsterdam
Andrew Yates University of Amsterdam
a. a.c.yates@uva..l
Maarten de Rijke University of Amsterdam
m.derijke@uvo..t
Reviewed on OpenReview: https: // optp:vveee.....w...mmoo net/ forum? id= TiXt0grVkn
Abstract
To train image-caption retrieval (ICR) methods, contrastive loss functions are common choice for optimization functions. Unfortunately, contrastive ICR methods are vulnerable to predictive feature suppression. Predictive features are features that correctly indicate the similarity between a query and a candidate item. However, in the presence multiple predictive features during training, encoder models tend to suppress redundant predictive features, since these features not needed to learn to discriminate between positive and negative pairs. While some predictive features are redundant during training, these features might be relevant during evaluation. We introduce an approach to reduce predictive feature suppression for resource-constraiigg an additional decoder to the contrastive ICR framework, to reconstruct the input caption in latent space of general-purpose sentence encoder, which prevents the image and caption encoder from suppressing predictive features. We implement the LTD objective as optimization constraint, to ensure that the reconstruction loss is below a bound value while primarily optimizing for the contrastive loss. Importantly, LTD does not depend on additional training data or expensive (hard) negative mining strategies. Our experiments show that, unlike reconstructing the input caption in the input space, LTD reduces predictive feature suppression, measured by obtaining higher recall@k, r-precision, and nDCG scores than a contrastive ICR baseline. Moreover, we show that LTD should be implemented as an optimization constraint instead of dual optimization objective. Finally, we show that LTD be used with different contrastive learning losses and wide variety of resource-constrained ICR methods.
ICR methods: lalent largel decoding (LTD). We add
1 Introduction
Image-caption retrieval (TCR) is the task of using an image or a caption as a query and ranking a set of candidate items in the other modality. Both the images and captions are mapped into shared latent space by two encoders, which correspond to the two modalities. These encoders usually do not share parameters
and are typically optimized with contrastive loss function (Faghri et al., 2018 Lee et al.
2018 Li al.
2019 Wang et al. 2019 Messina et al., 2020b Chen et al.) 2020a Liu et al. 2020 Wang ct al.
2020 Messinal
et. al. 2020a Jia el al. 2021 Diao et al.) 2021 Yu et al. 2021). How well an ICR method generalizes
beyond the specific training data depends on the features that the method has learned during training. The contrastive loss explicitly learns the similarity between positive (matching) candidates, while pushing away negative (non-matching) candidates during training. In the ideal situation, the contrastive objective optimizes the image and caption encoder such that both encoders extract all relevant. information from the

---

Published in Transactions on Machine Learning Research (04/2023)
small (kitten sitting quilt looking up at the camera.
young doglbbling biting an orange frisbee yard.
277c1
1 1
27z21
I
zi
zzz01 1
Figure Visualization of predictive feature suppression using two examples from the MS-COCO captions (COCO) dataset. zi and zē, are input images and captions, respectively, and z and zē₃ are the latent representations of the image and caption. We use cosine similarity as similarity metric. The objective of contrastive loss is to optimize the similarity scores on the diagonal of the similarity matrix, while minimizing the off- diagonal scores. In this small-scale training setup, if both the image and caption encoder only extract the concepts kitten and dog, the remaining concepts in both the images and captions are irrelevant to predicting the correct matching scores between the images and caplions. The input features that are not needed to predict match between an image and a caption are likely to be suppressed by the encoder. However, these features might be relevant during evaluation to predict a correct match.
caption and image that is needed for matching the positive candidates during evaluation. However, it is not defined upfront what information is needed during evaluation for retrieving the correct item among set of candidates.
Predictive feature suppression. Hermann & Lampinen (2020) show that, in the presence of two predictive features that redundantly predict the output label of the input data, a deep neural model preferentially represents one of the two predictive features while the other feature is suppressed. In this work, we define predictive fealure suppression for ICR the suppression of features by an encoder network during training that would be useful to correctly predict the match between a query and the positive candidate at inference time. For contrastive training tasks, the features that are relevant for matching the query with the positive candidate (i.e., the predictive features) mainly depend on the negative candidates in the training batch. Only optimizing the contrastive InfoNCE loss does not guarantee avoidance of shortcut features that suppress certain (predictive) input features and the learned features mainly depend on the difficulty of the discrimination task (Robinson et al. 2021 Especially in resource-constran input features in the caption and image are redundant for learning the similarity between matching images and captions, duc to the limited number of negative samples available to contrast with. The contrastive optimization ion objective is easy to solve by only using small subset of the predictive input features ures of the images and captions. Suppressing predictive features during training is an undesirable side-effect of contrastive representation learning in a resourco-constrainee training setup, since some of these features might be needed during evaluation to retrieve the matching candidate. In Figure ! we provide a visual example of predictive feature suppression in resource-constrainee contrastive image-text matching setup.
training setup, it is likely that the majority of the
2

---

Published in Transactions on Machine Learning Research (04/2023)
How to prevent predictive feature suppression. To increase the difficulty of a contrastive discrimination task, one can increase the batch size during training in order to increase the probability of having difficult in-batch negative samples (Chen et al.
2020c Qu et al., 2021). It is, therefore, not surprising that most progress on the two widely used ICR benchmark evaluation scts, Flickr30k (F30k) (Young et al. 2014] and
MS-COCO captions (COCO) (Lin et al.] 2014): has recently been made by using large-scale image-lext matching training, mainly in combination with transformer network architectures (Jia et al. 2021 Yuan ct al. 2021). Using more data and larger model architectures improves performance but comes with a significant. extra computational cost, both in terms of data needed for training and the number of parameters that need to be optimized.
The two benchmark datasets for ICR, the F30k and COCO datasets, are relatively small in terms of training samples compared to the training data of sfate-of-the-ars pre-trained ICR or image-text matching methods (Jia ct al. 2021 Yuan et al. 2021). When an ICR method trained from scratch using these benchmark datasets only, for example, in a resourco-constrainee training setup, scaling up the size of a batch is not a fcasible solution to increase performance, due to the limited data size of F30k and COCO or due to the lack of computational resources. Hence, it is important to develop algorithms that can improve the effectiveness of ICR methods in resource-constrrineel training sctup, without relying on more data and more compute to achieve this.
A method to increase the difficulty of the contrastive objective that does not rely on the size of the dataset to reduce predictive feature suppression, is to directly mine hard negative examples for each query over the entire dataset, rather than relying on an increased batch size to include difficult negative examples. The disadvantage of hard-negative mining is that it can be computationally expensive (Chen et al.) 2020b). Moreover, the COCO dataset contains many visually similar images (Parekh et al., 2020); when similar image is mined as hard-negative, it will be considered as negative w.r.t. the query, which may create conflicting and incorrect supervision signals.
The autoencoding paradigm (Hinton & Salakhutdinov 2006) provides an alternative solution to reduce predictive feature suppression by learning latent data representations that contain as much of the important input features as possible. Using the information bottle-neck principle, the encoder should compress the input information into low-dimensional reprosentation while preserving much as possible of the input features. Combining autoencoding with contrastive learning should prevent the image and caption encoder from learning features that are only needed to solve the contrastive optimization objective. Therefore, a logical step is to add a decoder to the learning algorithm that decodes the original input from either the caption or image representation (or both). However, adding a decoder on top of the image representations, as in (Li et al. (220aa), is sub-optimal for the TCR task. The captions provided for each image are already a dense summary of the image; reconstructing every pixel in the image results in image representations that contain too much local information, which is irrelevant for the ICR task. A more natural choice would be to decode the imput caption rather than the image, but adding a decoder on top of the caption representations might not result in reduction of predictive fcature suppression. Strong text decoders can reduce a reconstruction loss by mainly relying on the learned language model (Lu et al.) 2021). ). The input for this decoder (the latent caption representation) can mostly be ignored while correctly decoding the input sequence.
Our proposed solution. To address the disadvantages of current approaches to mitigating predictive feature suppression, viz. (i) high costs (in terms of compute and data), and (ii) reconstruction of the input caption and images in the input space, we introduce latent larget decoding (LTD). For each caption in the training set, we generate lalent largel representation by using a general-purpose sentence encoder. We train an image and caption encoder that can be trained in a resource-constraine setup, using standard contrastive learning objective. Next to that, we add an extra decoder to the learning algorithm. We decode the information of the caption in a latent space of the sentence encoder. Thus, the decoder cannol rely on learning a dataset-specific language model to decode the input, and the caption representation learned by the caption encoder should contain all input features that are needed to decode the latent target. By reconstructing this latent target representation we aim to reduce predictive feature suppression by the caption encoder, which should result in representations that, generalize better to the evaluation task. See Figure 2 in Section 3 for a high-level overview of our LTD method. LTD only requires an additional target representation for each caption and a simple leed-forward decoder network, and can be combined with any ICR method that
3

---

Published in Transactions on Machine Learning Research (04/2023)
uses separate caption and image encoder to compute global representation of the imput data. LTD does not depend on (i) additional training data, (ii) extra manual data annotation or (hard) negative mining, or (iii) significantly more computational resources. In this work, we focus on resouree-onntrrineed ICR methods that are trained from scratch on the F30k or COCO dataset on a single GPU.
If we were to add LTD to the learning algorithm, the overall training objective would become a multi-task loss: a contrastive and reconstruction loss. However, multi-task losses are difficult to optimize (Malkiel & Wolf 2021). The reconstruction loss should serve objective. We also do not want the caption encoder to mainly focus the reconstruction objective, since that can harm the contrastive utility of the reprosentations. Therefore, we implement LTD as an optimization constraint. In this manner, we can larget a specific value for that loss function. The main training objective is to minimize the contrastive loss, given the constraint that the reconstruction loss is below a certain bound value. Similar to (Rezende & Viola 2018 van Rozendaal et al. 2020,, we implement the reconstruction loss constraint using a Lagrange multiplier (Platt & Barr) 1987); the two losses are scaled automatically such that the reconstruction bound is met, while minimizing the contrastive loss.
extra regularizer rather than the main learning
Our main findings.
The proposed constrained-basee LTD reduces predictive feature suppression and improves the generalizability of learned representations, as it outperforms ICR baselines that are only optimized by using a contrastive loss. We measure the reduction of predictive feature suppression by using the standard evaluation metrics for the ICR task.
2. Implementing LTD as a dual loss, opposed to an optimization constraint, does not reduce predictive feature suppression. Our analyses suggest that optimizing the reconst loss only until specific bound value is met, results in better evaluation performance than minimizing the reconstruction loss as a dual loss.
LTD can be used in combination with different contrastive losses, for example, InfoNCE (yan den Oord et al.] 2018) and the triplet loss (Faghri et al. 2018), and it can be combined with wide variety of ICR methods that can be optimized in resource-constraiieed setup, such as VSRN Li
et. al. 2019 and TERN (Messina et. al. 2020b).
Below, we first cover related work, then introduce the proposed LTD method, before presenting our experi- mental setup and discussing the outcomes of our experiments, and concluding.
2 Related work
2.1
Image-caption retrieval
Neural architectures for ICR. We focus on ICR methods that compute global representation for both image and caption. In general, an ICR method consists of two encoders: one to encode the image and one to encode the caption into a latent representation (Faghri et al. 2018 Li et al. 2019 Chun et al. 2021 Jia et al.
2021). Most work on ICR focuses on new network architectures to learn multi muulti-mon modal feature representations. State-of-the-art results have been obtained using graph neural networks Li et al.
2019
Wang et al. 2020
Liu et al.) 2020 Diao et al.
2021 to represent visual relations in scenes as a graph,
or attention mechanisms
to align words the caption with specific regions in the input image (Lee et al.
2018
Chen et al.] 2019
Wang et al. 2019 Chen al. 2020a Yu et al., 2021 Zhang et al. 2022). Li et al.
(009
combine a caption
encoder-decoder with the image encoder to add extra training signals to the learning algorithm. These
methods are only trained and evaluated on the F30k and COCO datasets. Recently, there has been shift
to transformer-base -based (Vaswani et al.) 2017) network architectures for both the image and caption encoder. Messina el al. (2020b introduce Lransformer-based network architecture solely trained for the ICR lask. Since then, several transformer- based methods have been introduced (Lu et al. 2019 Chen et al., 2020d Li ct al., 2020b] 2021 Jia et al. 2021 Li et al. b022); some combine the image and caption encoder into one unified architecture. These methods are all (pre-)traine) trained on a large amount of additional training data and most are trained for the ICR task specifically, but general purprsl- vision-text models.
4

---

Published in Transactions on Machine Learning Research (04/2023)
Hard negative mining. Few publications have looked into the improvement of contrastive optimization for ICR methods. Faghri et al.] (2018) introduce a new formulation of the triplet loss that only considers the hardest negative candidate in the training batch instead of all negative candidates, which significantly improved the evaluation scores on the ICR benchmarks. Since then, many ICR methods (Lee et al.) 2018
Li el al. 2019 Wang el al.] 2020 Liu et al. 2020; Messina el al.
2020blai a Chen et al.) 2020a Diao el al,
2021
Yu et al., 2021) have used this loss function for optimization. Chen et al. (2020b) introduce an offline
hard-negative mining approach for ICR in order to overcome the limitations of in-batch negative mining.
Instead of mining an in-batch negd-neg,,ive, they mine additional negative candidates, for each query, over
the entire training set to learn from so-called harder-to-distinuuss negatives.
One-to-many problem. Chun et al. (001)) focus on the one-to-many problem in ICR. An image can be described by many captions. However, most methods in ICR learn one representation for the image, which should match with a number of different captions. They propose a probabilistic ICR method, where images and captions are represented as probability distributions in a shared latent space instead of a point representation. Although their method does not focus on contrastive optimization, it addresses prodictive feature suppression by learning a distribution over features instead of point prediction of features. Chun et al. [021] also propose the plausible match metric, a heuristic for identifying missing positive examples by finding images that contain similar objects (i.e., plausible matches) and considering these in the evaluation.
Bilen et al. (2022) | propose semantic adaptive margin (SAM). Instead using the binary relevance annotation between images and caption (of the F30k and COCO datasets) for the triplet loss computation, the authors propose an adaptive margin to model the many-to-many relation between images and caption. The standard triplet loss uses a fixed margin parameter SAM dynamically assigns unique distance value to the triplets in the training batch, based on the semantic similarity between an image and caption. In contrast, in this work we do not change the formulation of the contrastive loss. We add an extra optimization objective to the learning algorithm to prevent predictive feature suppression.
Upshot
Unlike most previous work, we do not focus on the network architecture to improve the ICR performance. Similar to (Chun et al.) 2021), we focus on small-scale learning set-ups train an ICR method from scratch to show the strength of our method in resource-constrained setting. Our proposed approach incorporates autocncoding into the learning algorithm in order to reconstruct the input caption to reduce predictive feature suppression.
2.2 Contrastive representation learning
Contrastive learning losses are used to learn discriminative representations of the input data that can be used to contrast positive and negative pairs of information in a latent space. These loss functions have made a big impact in representation learning, whether supervied (van den Oord al. 2018 Chen et al. (020c) or supervised (Radford et al., 2021: Karpukhin et al., 2020). Although TCR is supervised contrastive learning task, some of the theoretical findings about self-supervised contrastive learning apply to supervised settings as well.
Self-supervised contrastive learning. A common approach to learn general-purpose representations in a self-supervised manner, to create two (matching) views of the same (or similar) data point(s) by applying different augmentations (Chen et al.) 2020cb or by splitting the data into parts (van den Oord et al. 2018 (i.e., predicting the future). The two positive views are contrasted with other negative samples in the training batch. The goal is to learn encoders that are invariant under these augmentations and that can discriminate between positive and negative pairs. How well self-supervised representations generalize to different settings, after training, is often assessed using a down-stream evaluation task, such as object classification CChnn et al., 2020ch or speaker identification (van den Oord et al. 2018
Some work examines data augmentation to learn strong feature representations. Good angmentations retain task-relevant information while removing iask-irrelevant nuisances (Tian et al., 2020). The main purpose of removing fask-irrelevant nuisances is to prevent encoders from using this information predictive features
5

---

Published in Transactions on Machine Learning Research (04/2023)
during training. Xiao et al. (0021) show that the features needed to learn good representations depend on the down-stream task. ICR does not depend on augmentations to generate positive and negative pairs. These pairs are given by the annotations of the benchmark datasets (Young et al. 2014 Lin et al.) [2014). The difficulty of the discrimination task (and hence the Iearned features) mainly depends on which candidates are present in the training batch.
The generalizability of contrastive learning methods is also influenced by the number of (hard) negatives present in training batch. In general, the larger the number of in- t--batch negatives, the higher the down-stream evaluation performance (Chen et al.) negatives during training (He et al.) 2020) or on applying hard-negative mining strategies to increase the number of hard negatives in the batch (Lindgren et al. 2021 Xiong et al. 2021). Since we are focusing on a resource-constrained setup in this work, scaling up the batch size to increase the number of (hard) negatives is not feasible solution. Morcover, the COCO dataset contains many visually similar images (Parekh et al., 2020). Mining visually similar images as (hard) negatives will result in a suboptimal supervision signal, which makes hard-negalive mining also not feasible approach to reduce shortcut. feature suppression.
2020c). Some work has focused methods to increase the number of
Shortcut feature representations. Robinson et al. (2021) show that the contrastive InfoNCE loss (van den Oord et al.) 2018) does not guarantee avoidance of shortcut feature representations. Shortcut feature ropresentations are solutions that suppress predictive input features, i.c., a shorteut to discriminate between matching/ non-matching candidates. The features learned by the InfoNCE loss depend on the difficulty of instance discrimination during training. If the instance discrimination task is easy to solve during training, the model will learn shorteut features. Especially in a resource-constrinne ICR training setup, the contrastive objective is easy to solve, since there is only limited number of (hard) negative samples in the training batch, which will result in shortcuts/ prreiiiitee feature suppression.
Feature suppression among competing features. Chen et al. 2021 introduce the notion of feature suppression among competing features. Chen et al. (2020c) show that, for example, the SimCLR method (Chen et al. 2020c) when trained without the crop or color augmentation (which randomly crops or shifts the color distribution of an image), shows significant. drop in performance
the down-stream evaluation task. Apparently, the (color) pixel distribution of the image is a sufficient prodictive feature to match two views of the same image during training. However, these features do not generalize well to a down-stream evaluation task, such as object classification. The desired predictive features of the input image (i.e., the object class and its properties) are suppressed by competing features (i.c., the color distribution of the image). Chen et al. (202 refer to this phenomenon as feature suppression among competing features. Feature suppression among competing features is closely related to work by Hermann Lampinen (2020), who show that in the presence of multiple redundantly predictive features, deep neural models prefer one of the features over the other, while the other feature is suppressed. Chen et al. (2021) add artificially generated features (i.c., MNIST digits) as extra overlay to images. They show that the "easy" predictive features (the MNIST digits) are preferred by a deep neural encoder model over the real predictive features (i.e., the object class) when optimizing with a contrastive learning loss. Chen et al. (2021) conclude that contrastive losses rely on easy-to-detect features that solve the contrastive objective, while suppressing the remaining (partly irrelevant) information.
Predictive feature suppression is a prominent problem in resource-constrainee contrastive ICR. Captions often describe multiple aspects of a scene. However, in a resource-coustrained contrastive setup, only one (or few) of the aspects that are described in the caption is likely to be sufficient to match with the positive candidate (i.e., the image) during training due to the limited number of negative candidate in the training batch. To mitigate this problem of predictive feature suppression for resourco-constrainee contrastive ICR, we need an extra oplimization objective that is independent of the negalive samples in the training batch.
Autoencoding. An approach to reduce predictive feature suppression is autoencoding (Hinton & Salakhut dinov 2006). Autoencoding can be combined with contrastive learning loss and reduces predictive feature suppression without depending on sampling (hard) negative candidates. To learn high-quality text sequence eunedddngs for the dense passage retrieval lask, Lu et al. (2021) add a weak decoder top of document encoder to reconstruct the original document. To make image encoders more robust against shortcut features, Li et al. 2020a) add a decoder on top of the image encoder to decode the input image.
6

---

Published in Transactions on Machine Learning Research (04/2023)
Upshot
To reduce predictive Teature suppression in a resource-constrained ICR lask, introduce lalenl. largel decoding (LTD). LTD reduces predictive feature suppression, without focusing on the difficulty of the contrastive discrimination task. LTD requires neither a large number of negative samples nor hard negative mining strategies. Unlike other methods that reconstruct the input data, we reconstruct the input information of the caption in a latent space instead of the input space.
3 Method
In Table in Appendix A provide overview of the symbols and variables used throughout this work. We start with preliminaries and then discuss the InfoNCE contrastive loss and why autoencoding captions in the imput space is not a solution to reduce predictive feature suppression. Finally, we introduce latent target decoding (LTD) to reduce predictive feature suppression for recourse-constrainee ICR. In Figure we provide an overview of LTD.
Optimization
(at
Pens Gradient/ Backwardpass
Optimized
network
Data flowforward pass
Frozen network
Coeine Simitarity
M
1/3
Target generator
Semantic space
InfoNCE
Multi-modal space
Image
Caption
LTD
encoder
Baseline
encoder
Figure 2: Overview latent target decoding (LTD). The baseline (left) consists of general image-caption retrieval framework with an image and caption encoder. The encoders are trained by using the rastivstive InfoNCE loss. To reduce predictive feature suppression method (right). This extra decoder decodes the information of the input caption in latent space of a general-purpose sentence encoder. The decoder is not used during inference, which we indicate by the dashed line around the model has (-)-
add latent target decoding the baseline ICR
3.1
Preliminaries and notation
3.1.1 Notation
We follow the notation introduced in previous work (Chen et al. 2020c Brown et al. 2020 For the ICR task use multi-modal dataset D {( ......., Fcx)....... J This dataset consists of N image-caption tuples. Each tuple contains depicted in the image. At each training iteration, we randomly sample batch B of image-caption pairs from D. Per image, one of the l captions is sampled per training iteration; together, this image and caption form a posilive (or matching) image-caption pair. Each caption is used once during a training epoch.
image a² and & captions zzz, where 1 S j k, that describe the scene
The image and caption encoder are trained for two tasks: image-to-tezt (12t) and tezt-to-image (t2i) retrieval. Thus, each image and caption in B used as a query q. We denote the matching candidate in the other modality vt. All other candidates in B, in the other modalily, are considered negalive candidates v-. The set of all negative candidates for query q in batch B is S where w- S
3.1.2 Contrastive baseline model
The baseline (BL) ICR framework in this work consists of two encoders. The image encoder fo(-) takes an image at input and encodes this image into latent representation 2 fo(w:)) The caption encoder

---

Published in Transactions on Machine Learning Research (04/2023)
(o(+) takes caption imput and encodes this caption into latent representation zè; == go(r)) The vectors =iz and 2 have the same dimensionality and are normalized on the unit sphere. The encoders are only optimized by minimizing a contrastive learning loss Co Our goal is not obtain the highest possible evaluation performance, but to show the strength of LTD resource-constraing training setups.
3.2
Contrastive loss
To train the image and caption encoder, we use the InfoNCE contrastive loss (van den Oord et al. 2018
Chen
itt al.
2020ch.
The InfoNCE loss popular loss function for self-supervised representation learning Chen
et al.
2020c
He al. 2020) and multi-modal representation learming (Radford al.) 2021 Yuan et al. 2021
Jia al. 2021 y. To keep the notation simple and intuitive, we use q and for the latent representations
computed by the caption and image encoder and not Ecj and Zz. The InfoNCE loss is defined follows:
exp(q" /T) - log exp( algg"ut/1 -+/T)+++++++ Ev-es, ES, explq"'0
con
Σ
(1)
B
JEB
Leon in Eq. [ is minimized when, given query q, the cosine similarity score with the positive candidate e is high (i.e., ss 1), while the similarity scores with the negative candidates w- E S in the batch are as low as possible; T serves as a temperature parameter. The main objective of a contrastive learning loss for the ICR task is learn data representations that can be used to discriminate between similar and dissimilar image-caption pairs. However, there is constraint that enforces the encoders to learn representations that contain all available input information to make this discrimination, which is what we add.
3.2.1 Gradient w.r.t. the input representations
To show some important properties of the InfoNCE loss, we provide the derivative of -Loo w.r.t. the input in Eq. 2 CChnn et al. 2020ch:
exp(q" v//T)
Z(g,v) exp(q"' u+//"""'//////// Ev-es, ES, exp(q= V-/T)
(2a)
aLcon T = (1 - Z(q, v x+))²*)) 1)00 - Σ
2(q, v v0
(2b)
0q
ES
iCoon
(1 - Z(g, 0 D+)aa
(2c)
got
aLcon
dv T - 7(g, )..
(2d)
Z(q, v) returns the similarity score of candidate w.r.t. the query q, normalized by the sum of similarity scores of all candidates in the batch B. The full derivations of the derivative - -Loon are provided in Appendix Based on Eq. 2 we infer the following properties:
1. The update w.r.t. the query q (Eq, 2bj, is a weighted sum over the positive candidate ot and all negatives v S The query representation will be pulled closer to ut, while being pushed away from all x- E S The weight value of each candidate, 7(q. v) (Eq. 2a depends on the similarity score with the query.
2. w+ (Eq. 2cc will be pulled closer to the query representation (and the other way around).
3. All negatives v (Eq. 2d will be pushed away from the query representation (and the other way around).
Without contrasting with negalive candidates, the encoders will learn a trivial solution where latent rep- resentations collapse to a single point in the latent space (Jing et al.) 2021). This means that the learned representation mainly depends on contrasting with negative candidates during training. If the representations o only contain a subset of the predictive input features (which still minimize the contrastive training objective), the query representation q (in the other modality) will be updated to match/ micmmahhhh these representations
8

---

Published in Transactions on Machine Learning Research (04/2023)
The contrastive InfoNCE objective itself does not guarantee that all the predictive features in the input data are learned (Robinson et al. 2021 and mainly relies on easy-to-detect features to contrast between positive and negative pairs (Chen et al. 2021).
Importantly, the query and candidate representations are in different modalities and therefore generated by different encoders. Hence, the update of the query and candidate representations is based on fined representations in the other modality (e.g., the caption encoder can only try to match/not not match with the representations of the image encoder and vice versa). By adding constraint the representations of one of the two modalities, the other modality encoder will follow automatically. Therefore, in order to prevent predictive feature suppression for the caption modality in resource-constraint ICR setting, we add a constraint to the learning algorithm that forces the caption represenlation to be projected into the latent. space of a ceneral-uurpose sentence encoder.
3.3 Autoencoding reconstruction objective
Autoencoding (Hinton & Salakhutdinov 2006) is a natural choice for learning latent data representations that contain most of the important input features without relying on hard negative samples. To reconstruct the input caption from the encoded latent representation zzzz, we introduce a decoder network h-):
3 = h
(3)
The decoder network (-) takes the latent caption representation as input and outputs a reconstruction of the input caption Tgy- To decode the input sequence from the latent. representation, this latent represent ation should be a dense representation of the entire input sequence. The reconstruction loss, Crees of sequence of tokens, Ti..... Fn of length n, is the negative log-likelihood of the input data:
Lree
Σ log p( ztz))
(4)
Based on Eq. 4 it is clear that each predicted token Ia in the sequence is conditioned on: (i) the latent caption reprosentation zCy; and (ii) the already prodicted sequence It-1:1-
As discussed in Section 0 a strong decoder will mainly rely on the learned language model and language patterns decode the imput sequence (Lu et al. 2021I. This implies that the input sequence can be decoded correctly while mainly ignoring zgy; especially when t is large. Therefore, decoding the caption sequence in the input space is not guaranteed to reduce predictive feature suppression.
3.4
Latent target decoding
we argued why the contrastive InfoNCE loss is prone to predictive feature suppression, and in Scction 3.3 discussed why decoding caption in the input space will not prevent predictive fcature suppression. In this section, introduce latent target decoding (LTD). LTD decodes the semantics of the input caption in latent space of general-purpose sentence encoder to reduce predictive feature suppression, which can be used in combination with a contrastive loss. LTD addresses the issues of decoding the caption in the input space. See Figure 2 for a high-level overview of LTD for ICR.
In Section 3.2
For each caption ste, the training dataset we generate Vez. a latent target representation. The vector ye, is a dense vector representation. We assume that this vector contains all the (semantic) information of the caption, captured by general-purpose language encoder. We use our decoding network ha to decode vé, instead of the input caption. By reconstructing vector representation of the caption instead of the original input sequence, the reconstruction is not conditioned on the alrcady predicted sequence of tokens. The latent target decoder assumnes conditional independence of each feature in the latent. larget. Therefore, the decoder cannot rely on conditional (language model) patterns in the data to reconstruct the input semantics. This implics that we force the decoder rely completely on 2 to decode the latent target representation. LTD reduces predictive feature suppression by reconstructing the latent target from the caption embedding. To combine LTD with contrastive loss, it is necessary to compute the similarity score between a global
9

---

Published in Transactions on Machine Learning Research (04/2023)
representation of the entire caption and the image embedding. ICR methods that compute the similarity score by using fragments of the caption and regions in the image camnot be combined with L.T as introduced in this work. If there is no global representation of the caption, it is not possible enforce that all the semantic information from the target encoder will be distilled into the caption representation that is used for computing the similarity score.
3.4.1 Target decoding network
To decode Vés we use a three layer feed-forward decoder network:
he (=cg)) W (wie)o (w()) ));)),
(5)
where a is the ReLU non-linearity; h takes the latent caption representation zty as input and maps it to a reconstruction of the latent target reprosentation VC.
3.4.2 Loss function
To train wws we use the cosine distance between Vey and yčs reconstruction loss Creei
Crec = yč,
yc;
(6)
yC
Minimizing the cosine distance equivalent minimizing the mean squared error of two vectors normalizod the unit sphere (Chen & He 2021 Grill et al., 2020). By introducing an extra loss criterion, the overall training objective becomes dual optimization problem. The dual loss L is defined as follows:
Launt from + BLress
(7)
where B serves as balancing parameter scale the two losses.
3.4.3 Constraint-basel optimization
By adding LTD to the learning framework, we introduce an extra loss component Cree. To effectively minimize Cduat, we have to find the right value for the balancing parameter 3 in Eq. 7 This may require considerable amount of manual tuning, and often specific value for 3 does not generalize to different. training settings. Besides that, Cree is not the main training objective for the ICR tasks. The main reason add Crec to the learning algorithm is to reduce predictive feature suppression caused by solely optimizing the contrastive loss. We therefore argue that implementing LTD as an optimization constraint (Rezende & Viola 2018 van Rozendaal et al. 2020), as opposod to an optimization objective, might be more effoctive. Our goal, then, is to minimize the contrastive loss Loon given the constraint that the reconstruction loss is lower than or equal certain bound value rp:
min Lco subject to Lie S 0,00,0
We can implement this optimization constraint in combination with gradient descent by using the method of Lagrange multipliers:
max min Ling Loon + X ( (Liee -1ne-1). 1). 9.0..
(9)
17
The optimization objective is to minimize Ling w.r.l. the model parameters 0, 1/, ws while maximizing Zym w.r.t. to the multiplier X. The multiplier tuned antomatically by using stochastic gradient ascent with momentum. By optimizing À with stochastic gradient ascent, the two losses will be balanced automatically during training such that the reconstruction constraint is met, while the contrastive loss minimized by gradient descent.
10

---

Published in Transactions on Machine Learning Research (04/2023)
3.4.4 Choice of latent target representation
To generate the latent target ycj. we use a Sentence-BERI transformer model (Reimers & Gurevych 2019 Song et al. 2020) P Sentence-BERT a general-purpose sentence encoder that trained on a large amount. of data capture the semantic input information. Thus, we expect these embeddings be more general than those learn for the resourco-constrainee contrastive ICR task, which makes them suitable choice for the latent largel representalions ycj-
3.5
LTD vs. teacher-student framework
LTD somewhat similar a teacher-student framework used with knowledge distillation (Hinton al, 2015). Indeed, the target generator be
teacher network and the caption encoder in combination with the target decoder student network. However, in contrast with knowledge distillation, the goal of LTD is not to closely mimic a teacher network. Instead, the goal is to learn caption representations that can be used for multi-modal contrastive-bass retrieval while extracting as much of the textual semantic input information of the caption as possible.
4 Experimental setup
We design experiments aimed at showing: (i) reduction of predictive feature suppression by using LTD, with focus on the ICR task; (ii) the advantages of LTD over reconstructing the caption in the imput space; (iii) the benefit of constraint-baser optimization of LTD over dual loss optimization; and (iv) the genoralizability of LTD to different castrvstive losses and resource-constrained ICR methods that use different encoder network architectures. To facilitate reproducibility and further research of our work, we include the code with our paper
4.1 Datasets
For training and evaluating our ICR method, we use the two common ICR benchmark datasets: Flickr30k (F30k) (Young al., 2014) and MS-COCO captions (COCO) [Ln et al. 2014). The F30k dataset contains
31,000 image-caption tuples. We use the train, validate and split from (Karpathy & Fei-Fei) 2015), with 29,000 images for training, 1,000 for validation, and 1,000 for testing. COCO consists of 123,287 imago-caption tuples. We use the train, validate and test split from (Karpathy & Fei- 2015 we do not use the 1k test setup. Both F30k and COCO come with k = captions per image.
We also use the crisscrossed captions (CxC) dataset, which extends the COCO validation and test set with additional annolations of similar captions and images (Parekh et al. 2020), so as to evaluate whether LTD improves the evaluation scores by retrieving semantically similar candidates.
4.2 Implementation details
Unless otherwise specified, we use the following architectures for the target decoder, image encoder, and caption encoder. We use similar network architectures for the image and caption encoder as the ones used [Chun et al. 2021), which are simple network architectures that can be trained using limited amount of training data.
Image encoder. For the image encoder, we use pre-trained ResNet-50 (He et al., 2016) network. We apply average pooling on the last convolutional layer followed by projection head to map the image feature vector into a shared multi-modal latent space; the projection head has two feed-forward layers and a ReLC non-linearity.
Caption encoder. For the caption encoder, we use a bi-directional, single-layer, GRU network (Cho et al.) 2014). We use pre-trained GloVe embeddings (Pennington al.) 2014) as word embeddings. We use similar
11

---

Published in Transactions on Machine Learning Research (04/2023)
projection head as for the image encoder (which does not share parameters) to map the caption embedding into the shared latent space.
Target decoding network. For the target decoding network, we use a three-layer feed-forward network as in Eq. 5 To generate the latent target representations, we use the HuggingFace all- MiniLM L6-v2 Sentenco-BERE BERT implementation. The target decoding network is trained together with the image and caption encoder. The target decoding network is not used during evaluation.
Input decoding network. We compare latent. target decoding (LTD) with inpul target decoding (LTD), which reconstructs the imput caption in the input space (i.e., the input tokens). For ITD, we use a single-layer GRU [Cho et al.) 2014) decoder that reconstructs the input tokens in the caption (as explained in Section We train the word embeddings for ITD from scratch. ITD is optimized with the negalive log-likelihood
3.3
loss (Eq.
Training. Similar to (Chun et al. 2021 we use 30 warm-up and 30 fine-tune epochs, a batch size of 128, and a cosine annealing learning rate schedule with an initial learning rate of 2e- The Lagrange multiplier is initialized with a value of 1, bounded between O and 100, and is optimized by stochastic gradient ascent with a fixed learning rate of 5e-3 and a momentum (to prevent À from fluctuating too much) and dampening value of a = 0.9. When we use Lduat, we set 33 to 1. For the InfoNCE loss, we use a temperature value т of 0.05. Evaluation scores of ICR methods tend to differ depending on the random seed used during training (Rao al. 2022); to improve robustness, we apply stochastic weight averaging (SWA) (Izmailov et al.) 2018); take the average of 5 checkpoints, stored during the last 10% of the training iterations each epoch. For the reconstruction constraint bound 7) try for all experiments several values, E {0.05, 0.1, 0.15, 0.2, 0.25, 0.3). When we apply ITD we use 1) E {0.5, 1, 2, 3, 4, 5, 63. All results are based on the best pest-performing value of 1).
Generalizibility to different network architectures. LTD is a general method that can be combined with any global representation contrastive ICR method. To show that LTD works in combination with different network architectures, we apply LTD with multiple ICR methods that can be trained in a resource-constraingg setup. To cover wide spectrum of network architectures, choose two methods that use different network architectures for the image and caption encoder.
VSRN. The visual semantic reasoning network (VSRN) (Li et al. 2019) consists of an image and caption encoder that both compute a global representation of each imput modality. The caption encoder consists of a single directed GRU, similar to the caption encoder used in (Faghri et al. 2018). The image encoder takes a set of pre-computed regions of interest as input generated by a ResNet- 101 HHe et al., 2016) backbone, pre-trained on visual genome (Krishna et al. 2017). This set of pre-computed visual features is considered fully connected graph of regions in the input image. To perform reasoning the graph of visual features, multi-layer graph convolutional networks (Kipf & Welling] 2017] is used. Finally, to obtain one global representation of the entire image, a GRU is used to aggregate the regions of interest into one single representation. We use the same learning rate schedule and number of training epochs as in [Li et al., 2019 and we use the model implementation as provided by the authors However, we modify the original VSRN model on two points:
(i) We use the same caption encoder as described in Section 1.2 instead of the single directed GRU used for the original VSRN model and use a hidden dimensional of 1024 instead of 2048 for the caption and image representations. The goal of this work is not to show which specific network architectures perform best for the ICR task, but to show the generalizability of LTD in combination with different encoder networks.
(ii) The original VSRN model also comes with an additional caption decoder that decodes the input caption from the visual features. In this work, we investigate the reduction of predictive feature suppression for the general ICR framework, consisting of two encoders optimized by using contrastive loss. If we would add LTD to the original VSRN method, we would have two reconstruction objectives and a contrastive loss. The main reason we use VSRN is for the use of graph convolution networks
12

---

Published in Transactions on Machine Learning Research (04/2023)
in the image encoder to show the generalizability of LTD in combination with different encoder networks. Therefore, we remove this caption decoder from the learning algorithm.
The transformer reasoning network (TERN) Messina et al. 2020b) is transformer--ased ICR method that is solely trained and evaluated on the COCO dataset. TERN consists of a pre-trained BERT (Devlin et al. 2018) caption encoder. The image encoder takes a set of pre-computed regions of interest as input (similar to VSRN) and consists of a stack of four transformer layers. The pre-computed features are only available for the COCO dataset. Next, both the image and caption features are pushed through a stack of shared transformer layers. Although the weights of the last part of the caption and image encoder are shared, there is no (cross) attention between the two modalitics; the reprosentations of the images and captions are still computed independently. The image and caplion CLS token is used as a global representation of both the image and the caption. We use the same learning rate schedule, dropout rate, number of training epochs, and data augmentations as in (Messina et al. 2020b and use the model implementation as provided by the authors
TERN.
In this work, we use ICR methods that are trained from scratch on the F30k and COCO dataset and that are trainable on a single GPU. That does not imply that some of the weights of our encoders are not initialized with pre-trained parameters. However, only use pre-trained weights that are trained uni-modal task(s), and not for image-text matching specifically.
4.3 Evaluation metrics
To mcasure the reduction of predictive feature suppression, evaluate how well the learned encoders generalize to the ICR evaluation lask. The more predictive features the encoders learn to capture, the better these encoders are able to retrieve the correct candidate given query. The standard evaluation metric for ICR is r metric, with k = {1,5, 10}. During training, we evaluate the model after each training epoch on the validation set. Similar to (Faghri et al. 2018 Lee et al. 2018 Li et al. 2019 Chen et al.] 2020a Chun et al. 2021), we select the model with the highest score on the validation set (using the sum all recalk@k scores as & metric) for evaluation on the test set.
Recall@k. For ICR, @ecall@k is implemented as the fraction of how many times matching candidate is present in the top-k ranking (Karpathy & Fei-Fei 2015 Parekh et al.) 2020). For the t2i task, there is only one matching image per query caption. For the 12t task, there are 5 matching captions per image. Only the highest-rankod caption per query is used for calculating the recall scores. When using the CxC annotations, for both i2t and 12i, we take the highest-ranked candidate.
R-precision. When extending the COCO dataset with the CxC annotations, we have one or more matching candidates per query for both i2t and t2i Like Chun al. 20221) also use r-precision (R-P) for evaluation; it mcasures the precision for top-r ranking, where is the number of matching candidates given query.
nDCG. The standard evaluation metric for the ICR task, recall@k, mainly measures if the positive candidate is present in the top of the ranking. However, this does not provide much insight into the overall quality of the ranking. To address the limitations of only using recall@k, Messina et al. (2020b) start using nDCG as an additional evaluation metric for t:i retrieval. However, for t2i retrieval there is only one positive image per query caption. To generate more positive images per caption query, images that have captions with a high overlap with the query caption, are also considered positive. As similarity measurements between the captions, ROUGE-L [Lin) 2004) and SPICE (Anderson et al. 2016) are used. There are multiple re-annotations of COCO available that provide multiple matching images per caption query; soe, for example, (Parckh et al.) 2020p. However, these re-annotations are not used by Messina el al. (2020b) to compute the nDCG scores. To keep the evaluation consistent, we use the same relevance labels as used in (Messina et al. 2020b). The nDCG relevance labels are only available for COCO and not for F30k.
http::///github..co/eeeeet
13

---

Published in Transactions on Machine Learning Research (04/2023)
12t
12i
R@k
R@k
nDCG
Method
Loss
#
R-P
1
5
10
R-P ROUGE-L SPICE
F30k
1.1
BL
con
47.4
75.9
84.8
0.34
33.9
65.2
76.6
1.2
BL + ITD
dual, B=1
45.7
74.0
84.4
0.33
33.7
65.1
75.8
1.3
BL + ITD
Ling; 1l 1--6
36.6
66.8
76.5
0.28
27.8
59.1
71.0
1.4
BL LTD
duatt B=1
46.1
75.3
84.1
0.34
34.0
65.9
77.4
1.5
BL LTD
Llag, 17 0.2
49.6
78.7
86.4
0.37
36.7
68.4
79.3
COCO
2.1.1 BL
Con danat, 1=1
33.7
64.4
76.6
0.24
24.2
53.5
67.0
0.6487
0.5729
2.2.1
BI. ITD
32.7
64.4
76.3
0.24
24.2
53.8
67.6
0.6496
0.5733
2.3.1 BL ITD
Ling, 11=4 1
28.4
59.2
72.2
0.22
22.0
50.4
64.5
0.6424
0.5638
2.4.1 BL + LTD
L danats B=1
34.2
64.7
76.6
0.25
25.0
54.3
67.9
0.6510
0.5756
2.5.1 BL + LTD
Dlag: 170.15 0.15
36.0
66.5
78.1
0.26
26.2
56.2
69.4
0.6531
0.5786
CxC
2.1.2 BL
con
36.1
68.1
80.2
0.22
26.7
57.6
71.0
0.23
2.2.2
BL + ITD
L dunats B=1
35.0
68.0
79.7
0.22
26.6
58.0
71.6
0.23
2.3.2
BL + ITD
Llag; 17-4
31.0
62.9
75.8
0.20
24.6
54.8
68.9
0.21
2.4.2 BL LTD
L duully B=1
36.6
68.1
79.9
0.23
27.6
58.8,
72.0
0.24
2.5.2 BL + LTD
Llag 1) 0.15
38.4
70.4
81.5
0.24
28.9
60.4
73.3
0.25
Table 1: Recall@k, nDCG, and r-precision (R-P) evaluation scores for the F30k and COCO datasets (including the CxC annotations). We evaluate three loss functions Leon Lduat and Liag- We use three methods, the contrastive ICR baseline (BL), BL + input largel decoding (ITD), and BL + latent larget decoding (LTD). Boldface indicates the highest value for each evaluation metric per dataset. *_7 indicates that not possible compute the evaluation score for that datasct cattast/cxppente /experiment since the annotations are not available.
5 Results
In Section 5.1 we compare a contrastive ICR baseline with the same baseline combined with LTD. Next, Section 5.2 we ask if similar results can be achieved with ITD with LTD. In Section 5.3 we investigate the role of the optimization constraint and compare constraint-based LTD with LTD optimized dual loss. Finally, in Section 5.4 we ask whether LTD can be used in combination with different contrastive loss function, and in Section 5.5 we show that LTD can be combined with a wide variety of resource-constraine ICR methods.
5.1 Constrastive ICR baseline vs. baseline LTD
In Table we compare the contrastive ICR baseline, which is optimized by using the contrastive loss Loon defined in Eq. I with the baseline combined with LTD. Based Table [ row 1.1, 1.4, 2.1.1, 2.4.1, and dual loss (Cduat) with , = does not convincingly (or only
2.4.2, 2.4.2, we observe that LTD optimized with a small margin) outperform the baseline ICR method, which is optimized solely in a contrastive manner, in terms of recall@k, nDCCG, and r-precision for both datasets and both i2t and 1.i.
In contrast, when we implement LTD as an optimization constraint, by using Liggs row 1.5, 2.1.5, and 2.2.5, we observe that LTD consistently outperforms the baseline ICR methods on both F30k and COCO for both tasks (12t and 122) with large margin. An increase in recall also comes with increase in the r-precision
14

---

Published in Transactions on Machine Learning Research (04/2023)
Recall sum validation set
Trajectory Lambda
340
10a
LTD Lag: bound 0,05
335
LTD Lag: bound 0.1
10a
Recall Sum
LTD Lag: bound 0.15
330
LTD Lag: bound 0.2
Lambda
Lag: Dual, beta=1
10a
325
Contrastive
100
320
10-1
310
10-2
0.5
1.0
1.5
2.0
2.5
0.0
1.5
2.0
2.5
Training Step
le5
Training Step
145
(a) Trajectory of the recall sum the validation set during training.
(b) Trajectory of X during training for different values of
7. log-scale used for the y-axis.
Reconstruction loss
Contrastive loss over training
Contrastive loss value
2.5
Cosine distance
0.4
2.0
0.3
1.5
0.2
1.0
0.1
0.5
0.0
0.0
0.0
0.5
1.0
1.5
2.0
2.5 1e5
0.0
0.5
1.0
1.5
2.0
2.5
Training Step
Training Step
(c) Trajectory of the reconstruction loss Lree.
(d) Trajectory of the contrastive loss Lcon.
Figure 3: Overview of constraint-based optimization on the evaluation metric and the optimization objectives.
We train Cing with four different values of 17 E (0.05, 0.10, 0.15, 0.20, 0.25). All training steps are to the power of 1e5.
scores and nDCG scores. Hence, features learned by constraint based LTD perform better on the evaluation task, which is indication of the reduction of predictive feature suppression.
5.2 Latent Target Decoding vs. Input Target Decoding
decoding the caption in the input space will probably not result in reduction of predictive feature suppression due to overfitting of the learned language model. To empirically show this, we also implemented a decoder that decodes tokens of the input caption (ITD) to reduce predictive feature suppression (see Section 4.2 for details). Based on row 1.2, 2.1.2, and 2.2.2 in Table [ we conclude that implementing ITD as dual loss does not result in improved recall@k scores, for most values of k compared to the contrastive baseline. Surprisingly, when implement ITD for F30k and 77 6 for COCO, other values of 17 do not yield improvements) the evaluation scores even lower (row 1.3, 2.1.3 and 2.2.3) than when implemented as dual loss. We conclude that: (i) ITD does not reduce predictive feature suppression for ICR, and (ii) implementing ITD as an optimization constraint even hurts performance.
As argued in Section 3.3
optimization constraint (with n/
5.3 The role of the optimization constraint
What the role of the optimization constraint when minimizing Lree and what the effect on the evaluation scores compared to using Ldvat? In Figure 3b plot the trajectory of X for different. values of n E
15

---

Published in Transactions on Machine Learning Research (04/2023)
0.1, 0.2, 0.25) during training the COCO dataset. We also provide (i) the trajectory of the evaluation score (recall sum) over the validation set during training (Figure Bay, (ii) the trajectory of the reconstruction loss for different values of 1) and when optimized without using a constraint (Cavat) (Figure 3c and (iii) the trajectory of the contrastive loss for different values of 1 (Figure 3d Based on Figure observe:
wC
increases until the optimization constraint is met (i.e., the bound 17). The closer the reconstruction loss is to n7 the slower the increase of X. When the reconstruction constraint is met, X decreases to 0 (Figure 3bj.
2. À is positive again when the reconstruction loss becomes higher than the bound v) (Figure 3b line).
purple
3. The reconstruction loss converges to the value of 1) (Figure 3cl. However, it is not possible to meet every value of 17. E.g., 7) = 0.05 is too low to achieve for the model.
4. A lower reconstruction loss does not necessarily result in higher evaluation scores (Figure 3al. E.g., the recall sum is higher for 77 = 0.15 than for 17 = 0.1 or 1 = 0.05.
5. The value and the development of the contrastive loss do not depend much on the value of the reconstruction loss (Figure 3d.. E.g., a model optimized with Leon has the same contrastive loss trajectory as a model that is optimized with Liag and Ldaal. Hence, the contrastive loss on its own does not provide a good indication of the performance on the evaluation task. Similar trajectories of the contrastive loss result in different evalustion scores (hence different learned representations).
When we implement LTD as a dual loss, there is always a gradient from the reconstruction loss w.r.t. the parameters of the caption encoder, until the reconstruction loss is 0. This is not the case when we implement LTD as reconstruction constraint. When the constraint is met, À drops to zero and there is no gradient anymore from the reconstruction loss. We can conclude that a constant gradient from the reconstruction loss does not improve the learned representations of the caption encoder in terms of evaluation scores. The evaluation scores are higher when there is only a gradient until a certain reconstruction bound r is mot.
5.4
Generalizability w.r.t. contrastive loss
In Section 3.2 we argued that the InfoNCE loss prone to predictive feature suppression. popular choice of contrastive loss function for ICR methods is the triplet loss with in-batch hard-negative mining (Faghri ct al.
2018 The triplet loss with in-batch hard-ncgative mining is a special case of the InfoNCE loss, where the number of positives and negalives are each one (Khosla el al. 2020 Therefore, our line of reasoning in
Section 3.2 holds for the triplet loss too.
To show the strength and generalizability of LTD to other contrastive losses, we run the same experiments as in Section 5.1 (only for LTD not for ITD), with the triplet loss instead of the InfoNCE loss as Lcon- To prevent the triplet loss from collapsing to the trivial solution, we added a batch normalization layer after the projection head, for both the image and caption encoder; we
margin value of a = 0.2 (Faghri et al., Li et al. 2019 Messina et al. 2020b). Based on Table D we can observe that the highest recall@k scores also come with the highest r-precision and nDCG scores. Since the main goal of this experiment is to show that LTD can be used in combination with different contrastive losses we, therefore, only evaluate for recall@k.
2018
Table provides the recall@k scores for the F30k and COCO datasets. For both F30k and COCO the triplet loss with constraint-based LTD (see rows 3.1.3 and 3.2.3) results in higher evaluation scores than the InfoNCE loss with constraint-based LTD (see Table rows 1.5 and 2.5). Our goal here is not to identify the best contrastive loss for ICR LTD, but to show the generalizability of LTD to different contrastive losses. Morcover, using the triplet loss as Leon (sce row 3.2.1), results in expected evaluation scores on the COCO dataset (given the reproducibility work in (Bleeker & de Rijke 2022 ). Surprisingly, however, the evaluation scores for the F30k dataset while using the triplet loss as Leon (see row 3.1.1) are lower than expected (when compared to Table [ row 1.1). It unclcar why we observe these low evaluation scores for the F30k datasot when only using the triplet loss as Loon. In contrast, we observe that constraint-based LTD in combination with the triplet loss drastically improves the evaluation scores for the F30k dataset, which shows the strength
16

---

Published in Transactions on Machine Learning Research (04/2023)
of constraint-basee LTD for improving ICR evaluation scores and also making the triplet loss more robust to predictive feature suppression and feature collapsing.
12t
t2i
#
Method
Loss
R@l
R@5
R@10
R@1
R@5
R@10
F30k
3.1.1
BL
Con
12.8
33.2
45.4
11.1
32.6
46.6
3.1.2
BL + LTD
Ldaal 3 = 1
17.1
42.7
56.4
13.1
40.5
55.7
3.1.3
BL + LTD
Llngs 17 0.2
54.7
81.5
88.3
40.8
71.3
80.6
COCO
3.2.1
BL
Con
37.1
67.1
78.2
27.8
56.6
69.3
3.2.2
BL + LTD
Ddual: 3 = 1
37.4
67.8
79.1
28.2
57.2
70.5
3.2.3
BL + LTD
Liogs 1) 0.2 39.1
69.3
80.6
29.6
59.4
72.2
Table 2: Recall@k evaluation scores for the F30k and COCO datascts. For experiments 3.*, we use the triplet loss with in-batch hard-negative mining, defined in (Faghri el al., 2018) instead of the InfoNCE loss vvan den Oord et al. 2018) for Lcon. Boldface indicates the highest value for each evaluation metric per dataset.
5.5 Generalizability w.r.t. network architectures
In this section, we consider whether LTD can be used in combination with different resource-constrained TCR, methods that use different network architectures. To answer this question use LTD in combination with the VSRN and TERN methods.
In line with the observations in Table we observe in Table 3 that for both VSRN and TERN: (i) con- straint-based LTD outperforms the contrastive baseline, and (ii) constraint- LTD results a stronger performance improvement than implementing LTD as dual loss.
In line with the results in (Messina al. 2020b), the VSRN baseline outperforms the TERN baseline in terms of Recall@k. However, the difference in nDCG scores between the two models is relatively small. Although constraint-based LTD out both the baseline and LTD implemented as dual loss, improvements gained by using LTD in combination with VSRN are less convincing than for TERN.
The most consistent improvement in evaluation scores obtained by combining LTD with TERN, which is a fully transformer-based ICR method. This substantially different architecture from the one in Table and VSRN, and such transformer network architectures are the most prominent network architectures these days for multi-modal tasks (Radford et al.) 2021 Chen et al.) 2020d Lu al. 2019 Yuan al. 2021 When only limited amount of training data is available and one wants to make use of (partly) pre-trained transformer networks for multi-modal contrastive learning, constraint-based LTD can help to significantly improve the evaluation scores for ICR.
Furthermore, TERN makes use of a pre-trained BERT DDevli et al. 2018 model as a caption encoder. BERT is a general-purpose text encoder pre-trained on text only. An open question is still why to train a caption encoder, while we use a (frozen) general-purpose sentence encoder to generate the latent targets for LTD; why not the target decoder directly caption encoder? The results in Table 3 show that fine-tuning gureral-purpose language encoder (i.e,, BERT) with contrastive loss caption encoder results in lower evaluation scores than fine-tuning the caption encoder in combination with constraint-basee LTD. This shows that LTD helps to extract features (i.e., not suppressing these features) from the input data that are relevant for the ICR task, that are not captured by either the pre-trained BERT model or by only using the contrastive optimization objective. In Appendix [ we provide three qualitative ranking results for i2t retrieval using TERN and samples from the COCO test set. Based on the examples is clear that a baseline TERN does not represent specifie concepts (i.e., predictive features) that needed to rank the
17

---

Published in Transactions on Machine Learning Research (04/2023)
12t
1.2i
R akk
R@k
nDCG
#
Method
Loss
10
1
5
10
ROUGE-L SPICE
VSRN
F30k
5.1.1
BL
Ceon
60.3
85.6
90.8
44.9
74.0
83.3
5.1.2
BL +
LTD
daunt; 3 =
59.6
86.6
91.6
44.3
74.7
83.5
5.1.3 BL
LTD
Llag: 1) = 0.25
61.9
86.8
92.1
45.3
76.4
84.4
COCO
5.2.1
BL
Con
47.0
76.9
87.0
34.5
65.7
78.1
0.6779
0.6080
5.2.2
BL LTD
d 8
45.9
77.8
87.6
34.6
66.2
78.3
0.6791
0.6088
5.2.3
BL LTD
Llag: 1l 0.15
47.6
79.0
87.8
35.0
66.7
78.9
0.6797
0.6112
TERN
COCO
5.3.1
BL
Loon
41.2
72.6
83.6
31.0
61.9
74.7
0.6648
0.5926
5.3.2
BL LTD
d /
42.3
74.3
84.4
31.4
62.7
75.4
0.6684
0.5993
5.3.3 BL LTD
Llogs 1] 0.2
44.1
74.8
85.7
33.6
64.6
76.9
0.6727
0.6059
Table 3: Recall@k and nDCG evaluation scores for the F30k and COCO datasets using the VSRN and TERN network architectures. Boldface indicates the highest value for each evaluation metric per dataset. indicates that it is not possible to compute the evaluation score for that dataset/experimet since the annotations are not available.
correct captions top of the ranking, while TERN optimized with constraint- constraint-based LTD does represent these predictive features.
6 Conclusion
We have presented latent target decoding, novel approach to reduce predictive feature suppression for contrastive resource-constraing ICR methods. Instead of reconstructing the captions in the input space, LTD reduces predictive feature suppression by reconstructing the input caption in the latent space of a general-purpose sentence encoder. By reconstrueting the input caption, more difficult for the image and caption encoder to suppress predictive features that are not needed solve the contrastive optimization objective.
Main findings. Our results show that constraint-based LTD obtains higher evaluation scores than both contrastive ICR. baseline and LTD implemented as dual loss. This implies that we are able to reduce predictive feature suppression (and hence improve evaluation performance) by using constraint-based LTD, which does not require additional imago-text training data or hard-nogative mining strategics. Furthermore, we show that constraint--ased LTD consistently results in bigger improvement in evaluation scores than implementing LTD as a dual loss. These results suggest that, instead of simply minimizing both the contrastive and roconstruction loss, better evaluation scores can be obtained by only optimizing the reconstruction loss until certain bound value 7 is met. Finally, show that constraint-based LTD can be combined with different contrastive learning losses and wide variety of resource-constrrinee ICR methods.
The results of our work show that in a resource-constrainee setup the evaluation performances of contrastive ICR methods can be substantially improved by using constraint-based LTD, without relying on more training data or hard-negative mining strategics. We, therefore, argue that, in eecource-consrainng setup, LTD should be part of the standard ICR framework to mitigate the problem of predictive feature
Implications.
18

---

Published in Transactions on Machine Learning Research (04/2023)
suppression. Furthermore, we argue that when one uses additional reconstruction objective to reduce predictive feature suppression, this objective should be considered to be implemented as an optimization constraint instead of dual loss.
Limitations. In this work, we use a general-purpose sentence encoder to generate our latent target representation yek- However, we need assume that this latent target representation contains the relevant information of the input caption. Furthermore, the availability of general-purpose sentence encoder is not. always guaranteed (e.g., when working with low-resource languages).
For the ICR task, the predictive features are the features needed to retrieve the positive item from set of candidates. We, therefore, measure the reduction of predictive feature suppression by using the standard ranking evaluation metrics, such as recall@k, r-precision, and nDCG. However, we do not explicitly know which features are causing the observed improvement in the evaluation scores by using LTD.
Future work. We have several directions for future work. First, we plan to examine if the choice of different target generators will result in different ICR evaluation scores. Moreover, we also want to look into generating latent target representations without relying on pre-trained sentence encoder.
Another promising direction for future work is analyses on the exact role of the optimization constraint. In Section 5.3 we examine the role of the optimization constraint when training the image and caption encoder. When the optimization constraint rj is met, X (i.c., the balancing parameter of the two losses) drops to zero and the reconstruction loss no longer provides a gradient (Figure 3b Although À is (close to) zero for the majority of the training after the constraint is met, the evaluation scores on the validation set remain higher than when optimizing with Lauel, with 3 = 1 (Figure Bab. This suggests that constant gradient from the reconstruction loss does not benefit the training process, which is the case if LTD is implemented as a dual loss. We plan future research into the role of the optimization constraint, by trying constraint-based optimization for other multi-task optimization problems.
In this work, we focus on the reduction of predictive feature suppression for resource-constraaned ICR methods. In Section [ we argued that predictive feature suppression is less of an issue for models that are trained with large batch sizes since more information is needed to match the query with the positive candidate (due to large number of negative candidates). However, it remains promising direction for further research to investigate if and how constraint based LTD can be used for either large-scale contrastive image-text representation learning or for fine-tuning. Prominent large-scale image-text matching methods, such as ALIGN (Jia et al. 2021), use noisy image-text pairs scraped from the internet. It is unclear if the target generator will provide useful targets (and hence a training signal) when the caption has a weak relation with its matching image (which is possible for noisy image-text pair). It might be the case that the target generator mainly provides useful supervision signals when using high-quality human-curated datascts, such as F30k and COCO.
Finally, we suggest working on methods to measure which features are responsible for the gained improvement in ovaluation performance. A logical choice would be to use feature attribution methods. However, different feature altribution methods lend to disagree with each other, for both RNN and transformer-basel models (Neely et al., 2021). Therefore, the choice of feature attribution method will influence the analyses of which predictive features are better captured by using LTD. To gain further insights into which features are captured by the learned encoders, recommend developing task-specific feature attribution methods that can measure the reduction of predictive feature suppression directly.
Acknowledgments
We thank the reviewers and the action editor for their valuable comments and suggestions. We thank Maartje ter Hoeve for reviewing the text. This research was supported by the Nationale Politie and the Hybrid Intelligence Center through the Netherlands Organisation for Scientific Research. All content represents the opinion of the authors, which is not necessarily shared or endorsed by their respective employers and/or sponsors.
19

---

Published in Transactions on Machine Learning Research (04/2023)
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. SPICE: semantic propositional image caption evaluation. In ECCV, pp. 382-398, 2016.
Ali Furkan Biten, Andrés Maīla, Lluis Gómez, and Dimosthenis Karatzas. Is an image worth five sentences? A new look into semantics for image-text matching. In WACV, pp. 2483-2492, 2022.
Maurits Bleeker and Maarten de Rijke. Do lessons from metric learning generalize to image-caption retrieval? In ECIR, pp. 535-551, 2022.
Andrew Brown, Weidi Xie, Vicky Kalogeiton, and Andrew Zisserman. Smooth-AP: Smoothing the path towards large-scale image retrieval. In ECCV, pp. 677-694, 2020.
Hui Chen, Guiguang Ding, Zijia Lin, Sicheng Zhao, and Jungong Han. Cross-modal image-text retrieval with semantic consistency. In ACM MM, pp. 1749- 1757, 2019.
Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. IMRAM: iterative matching with recurrent attention memory for cross-modal image-text retrieval. In CVPR, pp. 12652-12660, 2020a.
Tianlang Chen, Jiajun Deng, and Jiebo Luo. Adaptive offline quintuplet loss for image-text matching. In ECCV, pp. 549-565, 2020b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton, A simple framework for contrastive lcarning of visual representations. In ICML, pp. 1597-1607, 2020c.
Ting Chen, Calvin Luo, and Lala Li. Intriguing propertics of contrastive losses. In NeurIPS, pp. 11834-11845, 2021.
Xinlei Chen and Kaiming Hc. Exploring simple siamese representation learning. In CVPR, pp. 15750-15758,
2021.
Cen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: universal image-text representation learning. In ECCV, pp. 104-120, 2020d.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. In EMNLP Workshop on Syntar, Semantics and Structure in Statistical Translation, pp. 103-111, 2014.
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende, Yannis Kalantidis, and Diane Larlus. Proba- bilistic embeddings for cross-modal retrieval. In CVPR, pp. 8415-8424, 2021.
Jacob Devlin, Ming-Wee Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACI-HLT, pp. 4171-4186, 2018.
Haiwen Diso, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image-text 2021.
matching. In AAAI, pp.
Fartash Faghri, David J. Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, pp. 12, 2018.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent A new approach to self- supervised learning. In NearIPS, pp. 21271-21284, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770 770-778, 778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervisod visual representation learning. In CVPR, pp. 9726 9735, 2020.
20

---

Published in Transactions on Machine Learning Research (04/2023)
Katherine L. Hermann and Andrew K. Lampinen. What shapes feature representations? exploring datasets, architectures, and training. In NewrIPS, pp. 9995-10006, 2020.
Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 2006.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint ar Xiv: 02531, 2015.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson 2018.
Averaging weights leads to wider optima and better generalization. In UAI, pp.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pp. 4904-4916, 2021.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. Undorstanding dimensional collapse in contrastive self-supervised learning. ICLR, 2021.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, pp. 3128-3137, 2015.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In EMNLP, pp. 6769 6781, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In NewrIPS, pp. 18661-18073, 2020.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123:32-73, 2017.
Kuang-Huc Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, pp. 212 228, 2018
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq R. Joty, Caiming Xiong, and Steven C. H. Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In NewrIPS, pp. 9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language-image pro-training for unified vision-language understanding and generation. In ICML, pp. 12888-12900 2022.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. In ICCV, pp. 4653 4661, 2019.
Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogerio Feris, Piotr Indyk, and Dina Katabi. Making contrastive learning robust to shortcuts. arXiv preprint arXiv:2012.0996 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Tijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks. ECCV, pp. 121-137, 2020b.
Chin-Yer Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
21

---

Published in Transactions on Machine Learning Research (04/2023)
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, pp. 740-755-- 2014.
Erik Lindgren, Sashank J. Reddi, Ruiqi Guo, and Sanjiv Kumar. Efficient training of retrieval models using negative cache. In NeurIPS, volume 4134 4146, 2021.
Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang, and Yongdong Zhang. Graph structured network for imago-text matching. In CVPR, pp. 10918-10927, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lec. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NewrIPS, pp. 13 23, 2019.
Shuqi Lu, Di Hc, Chenyan Xiong, Guolin Ke, Walced Malik, Zhicheng Dou, Paul Bennett, Tic-Ya Liu, and Arnold Overwijk. Less is more: Pre-training a strong siamese encoder using a weak decoder. In EMNLP, pp. 2780 2791, 2021.
Itzik Malkiel and Lior Wolf. Mtadam: Automatic balancing of multiple training loss terms. In EMNLP, pp. 10713-10729, 2021.
Nicola Messina, Giuseppe Amato, Andrea Esuli, Fabrizio Falchi, Claudio Gennaro, and Stéphane Marchand- Maillet. Fine-grained visual textual aligument for cross-modal retrieval using transformer encoders. ACM Trans. Mullim. Compul. Commun. Aypl.,
2020a.
Nicola Messina, Fabrizio Falchi, Andrea Esuli, and Giuseppe Amalo. Transformer reasoning network for image-text matching and retrieval. In ICPR, pp. 5222 5229, 2020b.
Michael Neely, Stefan F. Schouten, Maurits Bleeker, and Ana Lucic. Order in the court: Explainable AI methods prone to disagroement. arXiv preprint arXiv:2105. 005:220550 2021.
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. In EACL, pp. 2855 2870, 2020.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representa- tion. In EMNLP, pp. 1532 1543, 2014.
John C. Platt and Alan H. Barr. Constrained differential optimization. In NIPS, pp. 612 621, 1987.
Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In NAACI-HITT, pp. 5835-5841 5847, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, pp. 8748-8763, 2021.
Jun Rao, Fei Wang, Liang Ding, Shuhan Qi, Yibing Zhan, Weifeng Liu, and Dacheng Tao. Where does the performance improvement come from? A reproducibility concern about image-text retrieval. In SIGIR, pp. 0727-2737, 2022.
Nils Reimers and Tryna Gurevych. Sentence-BERY Sentence embeddings using siamese bert-networks. Tn EMNLP-IJCNLP, pp. 3980 3990, 2019.
Danilo J. Rezende and Fabio Viola. Generalized ELBO with constrained optimization, GECO. In NewrIPS Workshop on Bayesian Deep Learning, 2018.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanic Jegelka, and Suvrit Sra. Can contrastive learning avoid shorteut solutions? In NewrIPS, pp. 4974-4986, 2021.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tic- Yan Liu. Mpnet: Masked and permuted pro-training for language understanding. In NewrIPS, pp. 16857-16867, 2020.
22

---

Published in Transactions on Machine Learning Research (04/2023)
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In NewrIPS, 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1xXxm:807.0374 2018.
Ties van Rozendaal, Guillaume Sautière, and Taco S. Cohen. Lossy compression with distortion constrained optimization. In CVPR Workshops, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, pp. 5998 6008, 2017.
Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, and Xilin Chen. Cross-modal scene graph matching for relationship-aware image-text retrieval. In WACV, pp. 1497-1506, 2020.
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. CAMP: cross-modal adaptive message passing for lext-image retrieval. In ICCV, pp. 5763 5772, 2019.
Tete Xiao, Xiaolong Wang, Alexci A. Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. In ICLR, 2021.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR, 2021.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Tan Yu, Yi Yang, Yi Li, Lin Liu, Hongliang Fei, and Ping Li. Heterogeneous atlention network for effective and efficient mross-modal retrieval. In SIGIR, pp. 1146 1156, 2021.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, Ce Liu, Mengchen Liu, Zicheng Liu, Yumao Lu, Yu Shi, Lijuan Wang, Jianfeng Wang, Bin Xiao, Zhen Xiao, Jianwei Yang, Michael Zeng, Luowei Zhou, and Pengehuan Zhang.
Florence: new foundation model for computer vision. arXiv preprint 1144 2021.
Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Negative-aware attention framework for image-text matching. Tn CVPR, pp. 15640-15649, 2022.
23

---

Published in Transactions on Machine Learning Research (04/2023)
A
Notation and variables
Symbol
Explanation
con
Symbol for the contrastive loss. In this work we either use the InfoNCE van den Oord et al.
2018) triplet loss with in-batch hard-negative mining (Faghri et al. 2018).
ree
Symbol for the reconstruction loss of the input caption (i.c., decoding loss). In this work we use the negative cosine similarity when using embeddinngs (latent targel decoding) or the log-likelihood when we reconstruct the tokens of the input captions (input torget decoding).
dual
Symbol for the sum of the contrastive loss and the reconstruction loss (i.e., the cuall loss). The reconstruction loss scaled by B.
Ligg
Symbol for the sum of the contrastive loss and the reconstruction loss, where the reconstruction loss is implemented as a Lagrange multiplier optimization constraint.
q
Vector representation of query, either an image or caption.
v, o+ v
Vector representation of candidate. Given a query a candidate is either matching (o+) or not matching (v Candidates are either images or captions.
D
Dataset consisting of N imagencaption tuples; each image E N comes with &o captions. Input image i.
Input caption j that describes image i.
Latent representation image i.
2,
Latent representation of caption j that describes image i.
Reconstruction bound (or threshold). The reconstruction loss is only minimized up to the value of y7.
A
The Lagrange multiplier.
Balancing parameter to balance (or scale) two losses when using the dual loss.
B
Batch with training samples.
S
The set of all negative candidates v², in training batch, given query q. Temperature parameter to scale the logist (i.e., cosine similarity) for the InfoNCE loss. Margin parameter for the triplet loss.
Latent target representation (i.e., semantic embedding produced by a Sentence- BERT /languae encoder) for caption j that describes image
Ve,
Reconstructed latent target representation by the decoder network (i.c. LTD), for caption j that describes image i.
Reconstruction of the input tokens (i.e. ITD) by the decoder network for caption j that describes image i.
fo(-)
Image encoder parameterized by 0. Takes as input atz. Outputs a latent (global) image representation zt.
(+(+)
Caption encoder parameterized by Φh Takes as input mgyy Outputs a (global) latent caption representation =
k-))) Decoder network parameterized by w. Takes as input zgyy Outputs a reconstuccion of the input caption, either V6, (LTD) or Vy (ITD).
Table 4: Overview of the notation and variables used throughout this work.
24

---

Published in Transactions on Machine Learning Research (04/2023)
B Gradient of the InfoNCE loss w.r.t. the query and candidates
We start with the definition of the InfoNCE loss van den Oord et al. 2018 using the notation introduced in Scction for one query candidate pair (q. o*) and a set of negative candidates (Sq):
exp(q"v"
Leon
log exp(q" o+ (T) + ES exp(q"00111
(10a)
-- o /T log explq"0* o+ x+/r///)=- Σ
exp(q= - /T)
(10b)
v ES
Loon-
g-'0/1 /r - log expla"o*/11
Σ exp(q u /T)
(10c)
Next, take the derivative of -foo w.r.b. q (as also provided in (Chen el al. 2020c
0hem =v+/ T
exp(a"o=17) /
expla= at /T)
Σ
(11a)
0q
ES
expla /o /T//"/// /T Σ
exp(d" o- /T/D 17
expl9² o+ /T)
1O/T-
(11b)
expld"' at /T) Ev ESy exp( 9- v /T)
explq" v /T)
Σ
0 /T.
explg Ev ES, exp(q"0001
E
Now let define Z(g,u v) (similar to Eq.
2a in Section 3
exp exp(q"00/1
Z(q v) exp(q" /T)
(12)
En Es, exp(q v- v-/T)
Next, we plug-in 7(g, v) into Eq.
&Lcon vt / /T- 70g,0""(00 u+)00t -T- Σ 7(g,0) )00 /
(13a)
0q
ESq
aLcon -t -Z(g "00
Σ
Z(g, )
(13b)
0q
x ES,
Σ
Zig, )o.
(13c)
V-
In similar way, we can take the derivative of Leco W.r.t. t and w=:
oLcon q/7 7(g, x )g/T
(14a)
Out
aL
Out (1- Z(q,v+)) q.
(14b)
(15a)
Ov-
0Lcon
T
-Z(g.07 x)).
(15b)
Out
25

---

Published in Transactions on Machine Learning Research (04/2023)
Query
Query
Query
Rankings
Rankings
Rankings
BL
BL
BL
ready sat pizza
ciosw up view benans eilting on
1. man citting in boot holding a drink and
glessss satti a large pizza
tob
taddy bear.
the plate.
Close up banana siting a white
sitting a bost with delmistion
3. A person shing at bable with some pirns.
Chilcten ailling the sidw of the boat
The man holding fork ready cat his
aggl yellow
going
getting
are bananas that small
brkige
pizza
pluta
Two people a boat with a big fan on the
back.
BL LTD
BL + LTD
BL LTD
photo food table,
1. white cup and a brown and yellow banana
A group feddy bears socved like soldies gasting off boat.
picture food
A close up of barana next a cup with
Equid.
gesup
J.An taking picture table.
a
The banana is next the coffee mug on the
soldier
landing critt
A man sits in dine photographing his meat
rieneed benona cup up banana ni on
Teddy bears dressed up like coldiers boat.
5.Aview pizza from tablo. with man bahind
Tedely bears dressed scid soidier, arriving
counter
holiday
ledely beees,
covered evergreens, and Santa Clauses.
Figure 4: Three query images from the COCO test set. For each image query show the top retrieved captions by TERN. We compare the TERN baseline (BL) and TERN optimized in combination with constraint-based LTD (BL + LTD). Ground-ruthh captions (i.e., matching) are indicated in green. Captions in red indicate captions that do not match with the query image.
C Ranking examples
In Figure f we provide three query images from the COCO test set and the top 5 retrieved captions by TERN. We compare the TERN bascline (BL) and TERN optimized in combination with constraint-basee LTD (BL LTD). We selected three examples with large difference in precision@5 between the BL and BL + LTD.
For all three examples, it is clear that the baseline ICR methods miss concept (i.e., predictive feature(s)) that is nceded to rank the ground-truth captions in the top 5. In the left example, the best matching captions according to the BL ignore that the man in the image lakes photo. In the middle example, the best matching captions, according to the BL, do not match on the fact that there is a cup/muu. In the right example, the best matching captions do not contain the concept of teddy bears. Clearly, an ICR method optimized combination with LTD able to match images and queries based on more fine- grained features in the images and captions than baseline ICR method.
26


Reviews:
[{"review_id": "B9s2Z1JGI6", "reviewer": "Reviewer_W1M4", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written, and the proposed approach is well motivated and easy to follow. I appreciate the contributions, but hold some concerns with the experimental results. I believe that the current experiment findings do not comprehensively justify the benefits of LTD over vanilla contrastive learning.\n\n## Strengths:\n- The proposed approach LTD is well motivated. The paper explains in detail why feature suppression is an unintended consequence of previous contrastive learning frameworks, and why other methods (such as autoencoding) are insufficient.\n- The proposed approach is implemented as an optimization constraint, which minimizes hyperparameter tuning.\n- LTD is general: it works for both InfoNCE and triplet loss, which most contrastive pretraining methods use.\n\n\n## Weaknesses:\n- I am not sure that the derivations presented in eq 2b, 2c and 2d are correct. I get a slightly different result when computing it. Can you please provide details for the derivatives?\n- It’s suggested in the paper that LTD is agnostic to the choice of the language encoder. Could you show some experiments with other pretrained encoders, and show how the results vary with different models?\n- The architecture used is not competitive with most SOTA contrastive learning methods. In particular, the caption encoder is a single layer GRU, which seems to be significantly worse off in general compared to the larger transformers used in CLIP and ALIGN. It’s unclear if LTD will generalize to stronger contrastive learning setups.\n- The generalization experiments are run on VSRN and TERN, which are fine. However, I’m not sure why a CLIP-like model was not used instead, since that appears to be one of the stronger open sourced baselines used in many contrastive learning tasks.\n- The batch size used in experiments seems to be 128 (from the anonymized code). As mentioned by the authors, most SOTA contrastive learning approaches benefit from large batch sizes (for example, CLIP uses a batch size of 32768). Can you show some form of scaling curve as batch size increases, to motivate that LTD will still be beneficial at larger scales?\n\n\n## Minor issues\n- “Upshot” (page 4 and 6) is a strange (and I believe non-standard) way to phrase the conclusion. Consider removing this header.\n- Missing space on page 8: “(Robinson et al., 2021)and”\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes latent target decoding, a method to improve representations learnt through pretraining with contrastive learning. Typical contrastive losses do not enforce constraints that require learnt representations to contain all available information. LTD is motivated as an optimization constraint which measures the reconstruction quality of an input caption in the latent space of a language encoder. The paper shows the positive effect of LTD on various contrastive learning setups for image-text retrieval.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Critical for acceptance:\n- Show experiments with more competitive contrastive learning setups such as CLIP\n- Include experiments showing the effect of LTD over different batch sizes. It is important to know that the proposed method will still be helpful at scale, and its effects will not be washed out with larger batch sizes (which many SOTA models use) and larger model sizes.\n\n\n## Would strengthen the work:\n- Provide the intermediate steps for derivation of the gradients in eq 2b, 2c, and 2d\n- Show results with sentence/language encoders other than Sentence-BERT\n- Include hyperparameter details into the main text. The authors mention that it is the same as Chun et al., 2021, but certain details (such as batch size) are not found in Chun et al., 2021 either.\n\n", "broader_impact_concerns": "There are no major ethical or impact concerns introduced in this paper. This work is about contrastive training to learn features, and the datasets used are fairly standard in the community.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dosXPnzL2R", "reviewer": "Reviewer_ApZs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n(+) The idea is interesting. Contrastive models tend to focus on a low number of discriminative features  (Robinson et al. 2021). Pushing the representation to keep relevant information for reconstruction should help it avoid relying only on discriminative features. \n(+) The paper suggests an important design choice. The reconstruction loss should require similarity to latent space (of the caption) instead of decoding the tokens directly.\n(+) The authors show two variations for the regularization. That is as a dual loss (calibrated with hyperparameter) and constraint-based that does not require intensive tuning and can instead optimize a multiplier.\n(+) The related work section is comprehensive.\n\n\n\n\nWeaknesses:\nThe evidence that the method does with it intends to do, i.e.,  reducing predictive feature suppression, needs to be stronger. \n\n(W1) Retrieval Metric improvement is not proving cause. The authors did not show any evidence that the reason for the improvement is related to \"reducing feature suppression\": There is no analysis that shows the problem exists. The paper should be more self-contained, and here is no qualitative analysis or metrics related to this specific issue. \n\n(W2) There is no comparison to baseline results. The paper only compares against re-implementations. It limits the ability to get a real sense of how robust the method is. Why not follow the evaluation setups presented by Robinson et al. 2021? \n\n(W3) The resource-constrained setup is not defined or justified. Since the method can be applied to existing architectures, the authors can also fine-tune existing large-scale solutions employing their new loss. Such analysis is not provided.\n\n(W4) In general, contrastive loss has been found to be an efficient way to acquire zero-shot capabilities by training on large-scale data (Radford et al., 2021). It is intresting to show if the new loss idea can improve this capability. It is especially intresting since Radford et al., 2021 argued that they could not achieve the same zero-shot capabilities in the reconstruction setup. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses the Image-Caption retrieval (ICR) task. Commonly the task is solved with contrastive learning. This approach might pick on a few \"predictive features\" (i.e., that allow discriminative learning between positive and negative pairs) during training. The authors propose to have an additional decoder head to reconstruct the input caption embedding (namely, latent target decoding).\n\nMain ideas:\ni) reconstruct the latent space of the caption instead of the caption directly. \nii) use the reconstruction as a constraint instead of a dual optimization\n\nEvaluation is done on image-to-text and text-to-image tasks on the COCO, COCO CxC, and F30k datasets with two different architectures, VSRN and TERN.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "\nOverall the idea is intresting, but the current results only show that the direction is promising, but it is not a thorough study. Changes should answer the weakness. Specifically, the authors should focus on defining the problem - with metrics that assess it and qualitative analysis. The improvements should be on both the proposed metrics and qualitatively. An intresting analysis could be employing this method to large-scale models (e.g., CLIP) with fine-tuning.\n\n\nFurther, it might be a matter of styling, but the writing can be more concise, e.g., these two seances repeating the same thing.\n\"Implementing LTD as dual loss, as opposed to an optimization constraint, does not reduce predictive feature suppression. Our analyses suggest that optimizing the reconstruction loss only until a specific bound value is met results in better evaluation performance than minimizing the reconstruction loss as a dual loss.\" Also, there are some typos, like the word `viz' at the end of page 2.\n\n\n", "broader_impact_concerns": "It may be interesting to study if predictive features can cause undesirable biases, and if this new suggestion fixes it or makes it worse. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QcPTxAHABL", "reviewer": "Reviewer_zvm2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The studied toIs an image worth five sentences? a new look into semantics for image-text matchingpic, i.e. image-caption retrieval, is a popular research task to bridge the cross-modal gap. The motivation is clear and the idea to investigate resource-constrained image-caption retrieval is novel.\n2. The proposed method, i.e., latent target decoding (LTD) is well presented and easy to follow. Genearlly, LTD is reasonable and makes good sense.\n3. The reported results are superior to the compared baselines. Different aspects of the proposed method are analyzed in experiments.\n\nWeankesses and suggestions:\n1. The compared baselines are insufficient. There are many recent image-caption retrieval methods, such as Negative-Aware Attention Framework for Image-Text Matching\", \"Is an image worth five sentences? a new look into semantics for image-text matching\", \"Cross-Modal Image-Text Retrieval with Semantic Consistency\". Introducing and comparing with them in both related work and experimental part is necessary.\n2. Some detailed retrieval resutls are expected. Please give both successful and failed cases and analyze the reasons.\n3. A visualized example would be better to show the motivation.\n4. Therr are many notations and equations. A talbe summarizing the used variables will make it easier for users to understand.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes latent target decoding (LTD) to reduce predictive feature suppression for resource-constrained image-caption retrieval (ICR) methods. An decoder is added to the contrastive ICR framework for the reconstruction of the input caption in a latent space to prevent the image and caption encoder from suppressing predictive features. The LTD objective is implemented as an optimization constraint to optimize the reconstruction loss as well as the contrastive loss. Experiments are conducted on two popular ICR benchmark datasets to show the effectiveness of the propopsed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses above to revise this paper.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While most issues raised during reviewing were addressed, several questions by ApZs were not incorporated or replied to. Minimally, there are some lingering concerns about how to evaluate if the proposed approach improves robustness (either additional evaluations or metrics).  zvm2 also requests some additional visualization of examples.  While additional experiments would be an ideal inclusion for ApZs, minimally, the authors should try to adjust claims or indicate places for future work to address the listed weaknesses/concerns.
Decision: Accept (poster)