Improved Probabilistic Image-Text Representations
Authors: Sanghyuk Chun
Venue: ICLR 2024 poster (2024)
Keywords: Probabilistic embedding
image-text matching
cross-modal retrieval
vision-language
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Image-Text Matching (ITM) task, a fundamental vision-language (VL) task, suffers from the inherent ambiguity arising from multiplicity and imperfect annotations. Deterministic functions are not sufficiently powerful to capture ambiguity, prompting the exploration of probabilistic embeddings to tackle the challenge. However, the existing probabilistic ITM approach encounters two key shortcomings; the burden of heavy computations due to the Monte Carlo approximation, and the loss saturation issue in the face of abundant false negatives. To overcome the issues, this paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) by introducing a new probabilistic distance with a closed-form solution. In addition, two optimization techniques are proposed to enhance PCME++ further: first, the incorporation of pseudo-positives to prevent the negative effect under massive false negatives; second, mixed sample data augmentation for probabilistic matching. Experimental results on MS-COCO Caption and two extended benchmarks, CxC and ECCV Caption, demonstrate the effectiveness of PCME++ compared to state-of-the-art ITM methods. The robustness of PCME++ is also evaluated under noisy image-text correspondences. In addition, the potential applicability of PCME++ in automatic prompt-filtering for zero-shot classification is shown. The code is available at https://github.com/naver-ai/pcmepp.

Published as a conference paper at ICLR 2024
IMPROVED PROBABILISTIC IMAGE-TEXT REPRESENTATIONS
Sanghyuk Chun NAVER Al Lab
ABSTRACT
Image-Texs Text Matching (ITM) task, a fundamental vision- "ision-languag (VL) task, suffers from the inherent ambiguity arising from multiplicity and imperfect annotations. Deterministic functions are not sufficiently powerful to capture ambiguity, prompt- ing the exploration of probabilistic embeddings to tackle the challenge. However, the existing probabilistic ITM approach encounters two key shortcomings; the burden of heavy computations due to the Monte Carlo approximation, and the loss saturation issue in the face of abundant false negatives. To overcome the issues, this paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) +) by introducing a new probabilistic distance with closed-form solu- tion. In addition, two optimization techniques are proposed to enhance PCME+ further: first, the incorporation of pouuddivisseessssss..oee prevent the negative effect under massive false negatives; second, mixed sample data augmentation for proba- bilistic matching. Experimental results MS-COCO Caption and two extended benchmarks, CxC and ECCV Caption, demonstrate the effectiveness of PCME++ compared state-of-the ITM methods. The robustness PCME++ is also eval- uated under noisy image-text correspondences. In addition, the potential applicabil- ity of PCME+ + in automatic prompt-filtering for zero-shot classification shown. The code is available at https: //g thub. com/ naver- aateeerrrrraeee wwtpeeerpa/ppppppp../
INTRODUCTION
Given images and captions, Image-Text Matching (ITM) is the task of retrieving the most relevant images/captions for the given query caption/image (Frome et al., 2013; Young et al., 2014; Kiros
et al., 2014; Faghri et al., 2018; Gu et al., 2018; Lee al., 2018; Huang et al., 2018; Li et al., 2019; Song & Soleymani, 2019; Wehrmann et al., 2019; Wu et al., 2019; Wang et al., 2020; Diao et al., 2021; Chun et al., 2021; Chen et al., 2021; Huang et al., 2021; Biten et al., 2022: Kim et al., 2023; Radford et al., 2021). The applications of ITM include cross-modal retrieval (Faghri et al., 2018) from paired image-caption datasets, such as MS-COCO Caption (Chen et al., 2015), and zero-shot classification (Radford et al., 2021), by treating class labels as text (e.g., "a photo of }")".. Owing to its significant role in image understanding and language comprehension, ITM has emerged as fundamental Vision Language (VL) downstream task. However, this problem inherently suffers from correspondences and sparse annotations of the ITM datasets.
the ambiguity caused by
The nature of image-text matching is many-to-many an image can be described numerous text explanations, and there are a plentiful number of visual scenes to visualize text description. However, simultaneously, our datasets are sparsely annotated. The existing ITM datasets are built by collecting paired image-caption and treating the collected image-caption pairs as the only positives without considering other potential positives in "negative" pairs (Chen al., 2015; Plummer et al., 2015; Sharma al., 2018; 8; Changpinyo et al., 2021; Desai et al., 2021). 1). For example, Chun et al. (2022) showed that the MS-COCO Caption dataset has massive missing positives; 88.2% of caption-to-image positives and 72.1% of amage-to-captinn
positives are labeled as "negative", i.e., false negatives, (FNs). Figure shows an example. While humans judge all images and texts are plausibly matched, the dataset only treats a pair (x) x²) as positive when = This paper argues that the inherent multiplicity and abudant FNs lead to the ambiguity of ITM datasets ($2.1).
This paper aims to design a proper joint embedding space that represents the inherent ambiguity by probabilistic embeddings (Oh et al., 2019), i.e., encoding input to random variable rather than

---

Published as a conference paper at ICLR 2024
posilive mitch negutive mutch
Determinkstic embodding space
mowboarder is Royng through the doing stunts on his snoubourd.
Approsimating the embeskiins same entriiling point
embeldingg will located ckuely the ure sematically same. lmagg (TTM) sapenisos becomes uscetaistelie
aoo,
desemmaistic entelddnn space cante captue uncetarry.
The exsing deseriisisie appecackes enforees to the enboldings different
A person on a snow Bying in down snow
making diflen others vine HNM dibburh they are semanticllly the
Prohahilistic embedding space
Z
Z
Z
If dianbutione lave octten mucch eibbakking has unzertaaitty valaz.
A swrwboord jumpiing the eir.
2.
Ife match betwees and become "ancomwin" the enbediiin uncetaiiry (1c., vurance becones loger)
.A prbubbiiitii embedding spece cagtures uscentaitty of eaisy ITM stperiisiss it nush uncomiis. then the vuriance of enchediing becomes lorger (2) 22)
Image
Match
Caption
itas morch is cenair. then the voriance becomes smaller (2
Figure I: Inherent ambiguity ITM. We assume that deterministic textual embeddings are mapped to the same point 24, ie., 26 Re
2t 2 z z4. as well as the probabilistic textual embeddings Z; = .... Z,
a deterministic vector. Probabilistic embeddings have been introduced for many applications with inherent ambiguity, such face understanding (Shi & Jain, 2019; Chang et al., 2020), 2D-to-3D pose estimation (Sun et al., 2020), speaker diarization (Silnova et al., 2020), video understanding (Park al., 2022b), and composed image retrieval (Neculai et al., 2022). Especially, Chun et al. (2021) investigated the primitive probabilistic approach for ITM, PCME. Although PCME shows reasonable retrieval performances and interesting observations through uncertainty measures, PCME suffers from expensive computations due to Monte Carlo approximation and fast loss saturation under FNs.
Firstly, PCME needs expensive sampling operations for both training and inference. For example, we randomly draw 7 samples for each input, computing the distance between two samples costs O(7 x 7). Furthermore, due the sampling operation, PCME retrieval operation cannot be extended to large-scale efficient retrieval systems, such as FAISS (Johnson et al.. 2019). This issue is solved by introducing a new closed- form sampled distance (CSD), and new objective funetion based on the distance ($2.2). In addition, as the CSD consists of Euclidean distance and the relationship between variance embeddings, we can easily adapt approximated KNN (ANN) to PCME+ ($3.5). Experimental results show that the closed-form distance not only makes the operation efficient but also convergences to better solution by computing an exact solution instead of an approximation.
In addition, this paper proposes two optimization techniques to improve PCME+ + under abundant false negatives (FNs) by introducing two soft label strategics that allow less penalty to potential FN pairs in the dataset: assigning pscudo-posie posiodv-ssiivess (PP) for high-confident samples ($2.3) and mixed sample data augmentation (MSDA) for probabilistic matching ($2.4). Note that applying soft labels to PCME++ is straightforward because PCME+ is based a pair-wise loss function.
PCME+ is evaluated on MS-COCO Caption (Chen et al., 2015) and its extended benchmarks CxC (Parekh et al., 2021) and ECCV Caption (Chun al., 2022) with state-of-the-art ITMM methods ($3.2). +CME+ consistently works better than the comparison methods on the COCO benchmark, especially when the backbone size grows. +CME++ is also evaluated on the noisy correspondence benchmark (Huang et al., 2021), indicating that our method is not only effective for the original task but also holds the potential to address the noisy correspondence problem. Furthermore, this paper shows that the textual uncertainty of PCME++ can be applied to prompt-filtering for zero-shot classification with trained model on large-scale VL datasets, demonstrating the versatility and scalability of our method for wide range of applications ($3.5). Finally, the qualitative advantages of the learned uncertainty +CME++ by capturing dataset uncertainty are shown in $3.4.
This paper shows that FNs lead to inherent ambiguity for VL datasets (Section 2.1) and proposes solve the problem by PCME++ The newly introduced probability distance, named CSD, is more suitable to probabilistic representations as shown in Section 2.2. This paper also proposes soft label optimization techniques: PPs and MSDA in Section 2.3. As PCME++
Contributions.
2

---

Published as a conference paper at ICLR 2024
uses a pair-wise loss function invariant to other samples in mini-batch. it is straightforward to apply the soft labels. The extensive studies on COCO Caption show the effectiveness of PCME++. Impressively, while
take advantage of probabilistic representations, such as interpretability (Figure 3, Figure 4, C.3), PCME+ performed the best in all backbones, especially when scaling-up backhones or under strong noisy correspondence. Finally, the primitive results of applying PCME+ to uncertainty-basee prompt-filtering for zero-shot classification are demonstrated.
2 IMPROVED PROBABILISTIC CROSS-MODAL EMBEDDINGS (PCME++)
2.1 PROBLEM DEFINITION; AMBIGUITY OF ITM DATASETS
Let Ty and T2 be the input image and caption, respectively. For each image text pair, a binary matching indicator met E {0, } denotes whether describes Ze well. This paper argues that the inherent multiplicity and the sparse annotations make myt ambiguous. For example, as shown in Figure 1, _ ("A person jumping up in the air"") are semantically almost the same, hence we may assume that x and xi are 2f if we have a proper !
snowboard flyng in the air down the snow") and 2 ("A person
snowboard
mapped to almost the same embedding point z/e i.e., f(x)) = f(x]) mapping f(.) between the input space and the embedding space. In this case, if x and x are a positive match, x? and æt. should be positive match in the embedding space. However, because our dataset contains only sparse matching relationships (Chun et al., 2022; Parekh et al., 2021). x? and ær are negative match. In other words, in the embedding space, the matching between 2! and zi ( f(x)) Rs [(aī)) becomes ambiguous (i.e., it can be either positive or negative). As shown in Figure I, deterministic embedding space cannot capture the inherent uncertainty originated by the multiplicity and the sparse annotations. The existing deterministic approaches, therefore, rely on the Hardest Negative Mining (HNM) strategy (Faghri et al., 2018), selecting the closest pair as the only negative for computing triplet loss. The HNM strategy enforces sparse positive pairs to be closer than other false negative (FN) pairs, resulting in twisted embedding space that cannot capture the inherent uncertainty of VL datasets. We empirically show that the HNM strategy eventually converges to a suboptimal embedding space when the ambiguity intensifies, i.e., under strong noisy correspondences ($3.2). In contrast, probabilistic embeddings can naturally mitigate the issue by capturing the ambiguity of met with a probability distribution (Kirchhof et al., 2023).
2.2 PROBABILISTIC CONTRASTIVE LEARNING
We first define a visual embedding and text embedding of the given image T, and x, normally distributed random variables, Z, ~ N (pe E,) and Z, ~ (N,,,, Σx), respectively. For simplicity, we assume diagonal covariance matrices and simplify the notations as (pe. oo) and N(4, of). where μ and are D-dimensional vectors. As shown in Figure 1, our purpose is to learn probabilistic embeddings Z., and Zr satisfying the following properties: (a) there exists a proper probabilistic distance between Zr and Z, (b) if the match Mey is certain, then Z, and Z, have small variances. (c) if the match between Te and Zy (myy) is ambiguous, then Z. and Z, have large variances.
We detine closed-form sampled distance (CSD), between two probabilistic embeddings Z, and Z, d( ZZ., Ze) = Ez,, z. IZ, Z, 1 = ||4 - 1m11 + l1o? + +f2l1,,
(1)
where || 11 is p-norm operation. To be self-contained, the full derivation of Equation (1) provided in Appendix A. Equation (1) satisfies most of the properties of metric function (i.e., positivity, symmetry, and triangular inequality) except zero self-distance; d(Z, Z) is 2||o2||1, not zero. Lee, Equation (1) satisfies the condition (a). There are two ways to make Z, and Z, closer/further; making He and μe closer/further, or making d and or smaller/larger. Hence, if we assume fixed fu and H4, we have to decrease o and o, to minimize d(Z,,, Z,); if Z, and Z, are certain positive match (i.e., myt = 1), then a. and o2 will be collapsed to zero (i.e., satisfying the condition (b)), and d(Z,, ( will become Fuclidean distance. On the other hand, if the match between Z, and Z, is ambiguous (i.e., Muy can be either positive or negative), then a and or will not be collapsed to zero, for increasing d(Zz, Z.) for the negative match case; d(Z.,, Ze) also satisfies the condition (c).
CSD has a similar form to Wasserstein 2-distance (WD), infz,.z, Z, EZ. [Z, Ze = 11m--118 ||a, o, Il2, where WD includes the infimum operation. However, WD is not proper probabilistic distance in the matching problem, especially WD cannot satisfy the condition (b). Assume the
3

---

Published as a conference paper at ICLR 2024
scenario when j values are fixed again. In this case, o, and 0j have motivation to be decreased, but they are just enforced to have the same values. Hence, the learned a by WD cannot represent the sample certainty. Figure A.1 shows 2-D toy scenario where CSD satisfies the proper uncertainty conditions while WD cannot. In the figure, red, yellow, and green dots are certain samples, and others uncertain samples. The size of each dot denotes the intensity of the learned values. Here, we observe that Ciammi the average o² value for uncertain/certain samples by CSD are 1.82, while we have 1.04 for WD. More details of the toy experiment are described in Appendix A.2.
CSD is also related to the matching probability (Oh et al., 2019) used by PCME (Chun et al., 2021). where the matching probability cannot be computed in a closed-form but should be computed by an expensive Monte-Carlo approximation. Empirically, PCME+ is 33% faster than PCME for this reason. Furthermore, the PCME loss gives more weight to samples that correctly predict the distance relationships. However, our dataset has abundant FNs (e.g., COCO captions have x8.47 positive images than the "ground-truth" positive images (Chun et al., 2022)), which leads to wrong distance relationship supervision. Hence, PCME can suffer from the gradient saturation problem under abundant FNs. The comparison between CSD and matching probability is discussed in Appendix A.3.
Now, based on Equation (1), the probabilistic matching objective function is defined as NLL loss:
Lmatch- Mu log sigmoid( d(Z.: Ze) + b) (1 m.x) ) log sigmoid(( d(z,, Zo) b), (2)
where muz {0, 1} is the matching indicator between v and a and b are learnable scalar values, following Chun et al. (2021). In practice, Equation (2) can be easily implemented by binary cross entropy (BCE) loss. We compute Luach for all pairs in the mini-batch as contrastive learning objectives, such as InfoNCE (Radford et al., 2021). The overview of the comparisons between our objective function, a standard triplet loss, and batch-wise contrastive loss are shown in Figure A.3.
To prevent the collapse of a (i.e., (VIB) loss (Alemi et al., 2017), LVIB, following Chun et al. (2021). As derived by Oh et al. (2019), LviB can be computed by the KL divergence between the learned distribution and (00
0), PCME+ employs Variational Information Bottleneck
2.3 PSSUDO-POSITIVES (PP) FOR HANDLING NUMEROUS FALSE NEGATIVES
In practice, we use small mini-batch (e.g., 128), which does not guarantee that all confusing samples are observed for each iteration. To tackle the issue, +CME+ employs a simple pseudo-positive (PP) strategy: for positive match (v, (), l' is PP match with l if d( Ze < d(Z., Z.). Using the PPs, we compute the PP matching loss mpscndo-maich using equation 2. The objective function becomes:
mmatth + O.Cessukk match + BLVIB,
(3)
where a and 3 are control parameters of PP matching loss and VIB loss. In the experiments, a 0.1 and 3 = 0.0001 are chosen (Appendix C.3). Pscudo-code for Equation (3) is shown in Appendix A.4.
2.4 MIXED SAMPLE DATA AUGMENTATION (MSDA) FOR PROBABILISTIC MATCHING
MSDA, such as Mixup (Zhang et al., 2018) or CutMix (Yun et al., 2019), shows not only great improvements in empirical performances but also shows good theoretical properties, such as general- ization (Zhang et al., 202 a; Park et al., 2022a) or calibration (Zhang et al., 2021b). MSDA consists of two parts; input mixing (i.e., a generative process generate new mixed sample) and label mixing (i.e., modifying the supervision of the mixed sample). The intensity of the augmentation is controlled by X, usually sampled from pre-defined Beta distribution. Usually, it not straightfor- ward to apply MSDA to metric learning or contrastive learning because their losses are computed in batch-dependent way (Figure A.3 (a) and (b)). On the other hand, as our objective function is computed in pair-wis wise manner (Figure A.3 (c)), it is easier to apply MSDA our objective function. More detailed discussion of why MSDA is not applicable to previous methods is in Appendix A.5.
There are two issues with designing MSDA for probabilistic matching, First, MSDA for the textual modality is not straightforward. Hence, PCME++ only mixes visual inputs using Mixup and CutMix. Second, we cannot directly mix labels because our scenario has no class label. Instead, we let mae smooth in Equation (2), i.e., Mayt E [0, 1]. This approach controls the smooth label by mixing intensity À by setting mut X.
4

---

Published as a conference paper at ICLR 2024
GPO
La-sorm
Patchified image
Linear projection Pasirional embeskling
7, N(p. 0,2) log 0
Image
Visual encoder
Textual encoder
GPO
H
"A grey out is and but"
Tokenize embeedding Positional embeskiing
GPO
log of Z, N(4, 99
Transformer blocks
Figure 2: Architecture overview. We use the same visual and textual backbones CLIP. Each modality encoder encodes ég-normalized mean vector j and the variance vector log o², followed by Generalized Pooling Operator (GPO) (Chen al., 2021), to represent a normally distributed random variable Z (u, )
The overview of the optimization procedure with PPs and MSDA illustrated in Figure A.3 (d). In the experimental results, 25% of mini-batch images are mixed by sampling the mixing intensity À from (2ta 2). For every mini-batch, Mixup or CutMix is randomly chosen for the mixing strategy. The empirical study shows that this strategy is slightly better than the widely-used batch-wise mixing strategy, i.e., randomly mixing the whole mini-batch or using the original mini-batch (Appendix C.3).
2.5 ARCHITECTURE
PCME+ trains visual and textual encoders separately, such visual semantic embeddings CLIP. Each encoder has two heads, μl and log x² heads whose output vectors are D-dimensional. An input mapped normal distribution parameterized by the output of μ and log o² heads.
PCME+ employs a Vision Transformer (ViT) (Dosovitskiy et al., 2021) as the visual backbone and 12-layer 512-wide Transformer (Vaswani al., 2017) the textual backbone, following CLIP. PCME++ duplicates the last transformer layer for μu and log o² heads, e.g., a textual backbone has shared feature extractor with a 11-layer Transformer and μ and log o2 are 1-layer Transformer blocks. The o2 head is randomly initialized, while the head is initialized as the same as the backbone initialization (e.g., from pre-trained model). We empirically observe that using more layers for log x² marginally improves the performances, but we set the number of layers for log 0² head to 1 for computational efficiency. Finally, we employ Generalized Pooling Operator (GPO) for the feature aggregation with the same parameter setting of Chen al. (2021). We observe that GPO brings both training stability and performance improvements. The model overview is illustrated in Figure 2.
EXPERIMENTS
3.1
EXPERIMENTAL PROTOCOL
Three evaluation benchmark datasets are used: COCO Caption (Chen et al., 2015), and its two extended benchmarks, ECCV Caption (EC) (Chun et al., 2022) and CxC (Parekh et al., 2021). Note that they have the same images and captions (z 22) but with different match annotations m The overview of each benchmark for all benchmarks and mAP@ R and R- R-Precision (R-P) for EC are reported. The tables also show "RSUM". the summation of R@ R @5, and R @ full results for each modality and R @@ results are Appendix C.7. the main paper, the averaged scores on each modality are reported. As we focus on the mitigation of the FN problem, we mainly focus on BC mAP@R R, EC R-P, and COCO RSUM. Note that COCO and CxC have abundant FNs, hence their @ metrics could mislead a wrong result (Musgrave al., 2020; Chun et al., 2022).
be found in Appendix B. Following Chun et al. (2022), R@
10 for each modality retrieval
COCO 1K. The
Comparison methods. VSE>o (Chen et al., 2021) is based on a triplet loss and hardest negative mining (HNM) (Faghri et al., 2018). InfoNCE is the CLIP (Radford al., 2021) pre-training objective. PCME (Chun et al., 2021) is primitive probabilistic ITM model with sampling- based matching probability. This paper also evaluates recent methods to tackle false negatives (FNs) of
5

---

Published as a conference paper at ICLR 2024
ITM, DAA (Li et al., 2022a)' and P2RM (Wang et al., 2022), where both of them are additional regularization method on top of the HNM triplet loss, L... VSExc. The optimization hyperparameters VSExo ViT-B/32 validation rsum score.
for all experiments are fixed, such as learning rate, based The hyperparameters of DAA and P2RM are searched in the same setting. As we initialize all models by CLIP pre-trained models, CLIP zero-shot (ZS) is also reported same visual and textual backbones, except probabilistic models; they have an additional log o² head (See Figure 2). All models are trained three times, and the average evaluation metric are reported.
baseline. All models have the
Training details and model selection. PCME+ is initialized with the pre-trained CLIP model (Radford et al., 2021), while newly introduced modules, such as log o² head and GPO are randomly initialized. All models are trained for 25 epochs using AdamP optimizer (Hco et al., 2021). The training details can be found in Appendix B.2. The models are selected by the best validation RSUM following previous works. For the case when the model selection criterion is not possible, this paper also shows the SWA (lzmailov et al., 2018) results for ViT-B backbones (See Appendix B.3).
3.2
COCO ITM RESULTS
Main results. Table I shows the main comparison results of PCME++ and other ITM methods. For fair comparison, PCME++ (SWA) is not compared to the other methods. We first observe that PCME+ consistently outperforms other methods in most evaluation metrics on different backbones, particularly on EC mAP@R, R, R-P, and RSUM. Second, we observe that the scale-up of PCME++ leads to consistent performance increases without hyperparameter tuning, while deterministic counterparts (e.g., VSEx and InfoNCE) suffer from performance drops when scaling up from ViT-B/16 to ViT- L/14. As the backbone becomes more complex and larger, presume that the backbone complexity of deterministic methods is sufficiently large to capture the noisy FNs in the dataset. Morcover, VSExc uses a triplet loss with the HNM, making the effect of FNs more significant (Chun et al., 2022). Meanwhile, the probabilistic methods are designed to handle many-to-many relationships with uncertainty representations (Section 2.1); hence, the effect of the FNs is saturated during training PCME and PCME+ Note that applying PP and MSDA to triplet loss (e.g., VSExx, DAA, and P2RM) is non-trivial because the triplet loss is not designed for taking smooth labels. More detailed discussions can be found in Appendix A.5. Third, we observe that the previously proposed methods to mitigate the FN problem, such as DAA and P2RM underperform than its baseline (VSExc). It shows that the regularization-barl limitation of the deterministic embeddings. The full retrieval results for each modality and R @ scores are reported in Appendix C.7.
deterministic methods cannot solve the FN problem due to the
Noisy correspondence. Table 2 shows the additional comparisons under noisy correspondence (NC), i.., by assuming that the training annotations are noisy. Following Huang et al. (2021), the image-text relationships are randomly shuffled with the probability of 20% and 50%. A specifically designed method for solving the NC problem, NCR (Huang et al., 2021) and DECL (Qin et al., 2022), are also compared with the comparison methods. Following Huang et al. (2021), the model selection criterion is also based on the clean validation rsum as the clean dataset scenario. Here, a for PP loss set to 0.01 and 0.0 for 20% and 50% noise ratio because PPs can be incorrectly estimated under strong NC (Li et al., 2020). However, in Appendix C.2, we can observe that although using weaker PPs shows better by the best model selection by clean validation rsum, eventually suffers from serve overfitting. This paper argues that the best model selection criterion based on the clean validation split should be reconsidered for future works to avoid a wrong conclusion. There are two findings in Table 2. First, the hardest negative mining- based triplet loss (VSEco and DAA) shows vulnerability on strong noisy correspondence, e.g., 50%. Second, although the probabilistic methods, such as PCME and PCME++, are not designed for tackling NC, they successfully handle the scenario.
3.3 ABLATION STUDY
Table 3 shows that all the proposed techniques effectively improve probabilistic ITM. More detailed hyperparameter studies for each optimization (VIB, PP, and MSDA) are Appendix C.3. Table shows the impact of the probability distance on training objective (Equation (2)) by replacing
Note that although DAA argues that it is an "approximation of probabilistic embedding", DAA is not prohabilistic method, but a deterministic regularization method based on similarity.

---

Published as a conference paper at ICLR 2024
Table COCO cross-modal retrieval performances. Comparisons of ITM methods with various backbones ECCV Caption, CxC and COCO Caption. "Prob?" denotes whether a method is probabilistic method or not. Each number is the average between the image-to-text retrieval and text-to lext-to-image retrieval results, and is the average of three different runs. The full numbers and standard errors in Appendix C.7. denotes the re-cvaluated results by the official checkpoints, otherwise, numbers are produced by trained models.
ECCV Caption R-P R@
CxC R@1
COCO
Backbone
Method
Prob?
mAP@R
IK R@I
5K R@!
RSUM
CLIP ZS!
26.8
36.9
67.1
42.0
59.5
40.3
471.9
VSEco
x
40.0
49.5
83.1
57.1
75.5
55.2
536.5
P2RM
x
39.0
48.7
82.0
53.6
73.3
51.7
530.2
DAA
39.2
49.0
82.0
54.8
73.6
52.9
530,9
VIT-B/32
(151M)
InfoNCE
39.0
48.7
81.7
54.9
74.0
53.0
532,6
PCME
V
39.1
48.9
81.4
54.7
73.8
53.0
532.0
PCME+ (p only)
X
39.5
49.1
82.7
57.0
75.3
55.2
536.2
PCME+
40.1
49.7
83.1
56.8
75.4
55.1
537.0
PCME+ (SWA)
40.2
49.8
82.9
56.8
75.5
55.2
537.3
CLIP zst
X
29.3
39.0
71.1
44.3
62.0
42.7
481.0
VSEco
X
41.7
50.6
86.3
62.3
79.1
60.7
547.2
P2RM
39.7
49.5
80.7
54.2
73.8
52.5
532.7
DAA
20.7
30.6
50.2
25.4
43.7
23.4
410,2
VIT-B/16
(150M)
InfoNCE
X
41.1
50.4
84.8
60.9
78.3
59.3
545.5
PCME
41.0
50.3
84.3
59.9
77.8
58.2
544.2
PCME+ (I only)
41.2
50.4
85.7
62.5
79.3
61.0
548.0
PCME+
42.1
51.2
86.5
62.6
79.3
61.1
548.0
PCME+ (SWA)
V
42.2
51.2
86.6
62.9
79.6
61.3
548.5
CL
X
28.0
37.8
72.2
48.1
64.8
46.4
491.6
VSEoo
31.5
46.2
22.7
424.3
VIT-L/14
InfoNCE
35.6
45.8
75.6
48.0
69.5
45.9
520.6
(428M)
PCME
41.2
50.3
86.0
63.4
80.3
61.9
550.4
PCME+
42.1
50.8
88.8
65.9
81.8
64.3
554.7
Table COCO noisy correspondence. Noisy correspondence results using the ViT-B/32 backbone (except
NCR) are shown. NCR scores are re-evaluated by the official weights. As DECL does not provide official weights, we report the scores from the paper. Noise ratio 0% is the same as Table
ECCV Caption
CxC R@1
COCO
Noise ratio
Method
mAP@R
R-P
R@1
IK R
5K R
RSUM
VSExc
37.0
46.3
79.7
53.6
72.0
51.8
518.6
DAA
6.7
12.5
18.5
7.0
15.3
6.0
212.8
InfoNCE
35.9
46.3
76.1
47.8
68.2
45.8
514.6
20%
PCME
37.6
47.6
79.2
50.6
70.3
520.7
PCME+ (ours)
37.7
47.6
80.0
52.2
71.6
50.4
524.6
NCRT
35.9
46.0
78.0
50.6
70.1
518.6
DECL
69.6
49.4
$18.2
VSE
18.0
28.5
43.7
20.7
39.2
19.1
394.1
DAA
0.3
0.8
1.0
0.3
0.8
0.2
20.9
InfoNCE
33.6
44.1
73.0
43.5
64.0
41.4
499.5
50%
PCME
35.2
45.5
75.7
46.3
66.6
44.4
508.0
PCME+ (ours)
35.7
45.8
76.3
47.4
67.6
45.5
511.0
NCR!
34.0
44.3
75.1
47.3
66.8
45.5
508.5
DECL
68.3
46.8
513.5
d(Z,, Z.). For fair comparison, all newly proposed optimization techniques except VIB are omitted. As we already observed in Figure A. we confirm that Wasserstein distance is not proper uncertainty estimate as a training objective. Also, the table shows that PCME+ outperforms PCME in all metrics. I presume it is because the matching probability is approximated value by Monte Carlo approximation; therefore, the distance value will have an approximation gap.
More parameter studics for Table 3 and architecture study can be found in Appendix C.3

---

Published as a conference paper at ICLR 2024
Table Effect optimization methods. Ablation study on VIB, Speudo-Positives (PP) Data Augmentation (MSDA) with VIT-B/32 backbone are shown.
Mixed Sample
ECCV
Caption R-P
CxC R@!
COCO
VIB
PP
MSDA
mAP@R
R@! 1
IK R@]
5K R@
RSUM
X
x
38.9
48.6
82.2
56.7
75.2
54.9
535.9
V
X
39.3
49.0
83.1
56.1
74.5
54.3
534.5
X
V
39.0
48.6
82.7
56.8
75.2
55.0
536.0
X
39.0
82.1
56.4
74.9
54.6
535.5
V
39.6
49.2
82.6
56.3
74.8
54.5
534.8
40.1
83.1
56.8
75.4
55.1
537.0
Table 4: Effect of probability distance on training objective. Results on ViT-B/32 backbone with VIB loss.
ECCV Caption R-P
CxC R@1
COCO
Probability distance
mAP@R
R@l
IK R@1
5K R@!
RSUM
KL Divergence
0.0
0.1
0.0
0.0
0.1
0.0
3.2
IS Divergence
0.0
0.1
0.1
0.0
0.1
0.0
3.2
Wasserstein 2-distance
4.2
9.0
121.1
Expected Likelihood Kernel
36.5
46.0
82.0
56.3
74_1
54.7
529.0
Bhattacharyya distance
39.3
48.9
80.7
53.7
72.5
51.8
524.9
Match probability by PCME
39.1
48.9
81.4
54.7
73.8
53.0
532.0
Proposed (Equation (1))
39.3
49.0
83.1
56.1
74.5
534.5
3.4 UNCERTAINTY ANALYSIS
COCO 1k Image sigma Recall@1
mage-to-teet Recallg31
From Equation (I), define the data uncertainty as ||o ie., the summation of the variance. Based on the data uncer- tainty, Figure 3 shows how the uncertainty captures the biguity of datasets. The average COCO IK R@ @ Is for each modality in each of the 10 uncertainty bins are reported in the figure. We observe that by the uncertainty increased, COCO R@ (the same distribution as the training dataset) is decreased. The results support that the learned uncertainty by PCME+ can capture the inherent ambiguity
80
65
Uncertainty
COCO caption sigma vs. Recail@1
matching annotations.
Recallg1
Figure C.3 shows examples of uncertain images and captions, and their retrieved items. The figure shows that data that can be matched with more samples have higher uncertainty values. As shown in the figure, the retrieved items for uncertain inputs are highly plausible even though the retrieved items not in the COCO ground truth. Section 3.5 and discuss more benefits of uncertainty-aware learning and the learned uncertainty.
Becall8:
Uncertaintl levels
Figure 3: ||o? [1 R@L
Figure shows the t-SNE visualization of ViT-B/32 PCME+ and VSExo in Table 1. As shown in Figure the PCME+ embedding space captures the uncertain captions by enlarging their variances; thereby, each uncertain caption embedding can represent the uncertainty caused by the multiplicity of the dataset. On the other hand, due the HNM strategy, the learned embedding space by VSEoo (Figure right) cannot correctly map three different images and captions with multiplicity. We also observe the same phenomenon in Figure A.1; 1; shows that HNM will ruin the embedding space. More details are in Appendix A.2. We also provide more detailed explanation of Figure in Appendix C.4.
3.5 MORE APPLICATIONS
Large-scale retrieval system. Lack of scalability is a common drawback of probabilistic retrieval systems, i.e., it is difficult apply probabilistic embeddings large-scale retrieval system with billion-scale index. As our CSD (Equation (I)) is the sum of Euclidean distance of µu and the intensity of o² of each input,
easily and efficiently combine PCME+ and approximated KNN (ANN).
8

---

Published as a conference paper at ICLR 2024
airplase a clear blue sky
A small sirplane flying through a blue sky
A lage airliner jet flying through the blue sky
Probubilists mupping
Deteminnsttc mapping
Det. image eubbedding
embelding Proh image emberdding:
Prob, text enibedding
Figure 4: 2D t-SNE visualization of learned embeddings by PCME+ and VSE.o. The area of probabilistic embeddings denotes the uncertainty of cach embedding, i.e., more uncertain sample has larger area.
Appendix C.5 describes the details of the modified ANN, and the comparisons of diverse retrieval strategies. CSD is not only stronger than other probability distances but also more practical.
Uncertainty- based prompt-filerrnn. Zero-shot (ZS)
Table 5: ImageNet (IN) Zero-shot (7S)..
classification is the task of predicting an unseen class during training. Usually, ZS classification is done by converting class information as text sentence (e.g.. "a photo of a cat") and mapping into a shared embedding space with the imput. For image ZS classification tasks, large-scut scale ITM pre-training, such as CLIP, has become standard approach. Despite their usability and general- izability. ZS needs hand-crafted prompt engineering for converting class information into proper text sentences. For example, Radford et al. (2021) I) showed that taking the average of 80 different context prompts improves ImageNet top-1 ZS accuracy by 3.5% over a single prompt ("a photo of {-)"). However, designing the best rest-performing prompts for every novel task is time-consuming.
Model
Prompts
Top-1 Acc
"A photo of {- {-}}"
31.85
InfoNCE
All 80 prompts
35.50
"A photo of {-}"
30.43
All 80 prompts
34.22
PCME+ Top- K certain prompts
34.22
Best top- K for each class
41.82
This paper investigates the potential of PCME++ for automatic prompt engineering using the learned text uncertainty: First, the uncertainties of prompts for each class are computed, (e.g., "a photo of cat", "a photo of many car" car"..... ...), and then the most uncertain text prompts are discarded; this process computed directly on the ImageNet validation set (i.e., it is not a true "ZS". See Appendix C.6 for the more detailed discussion). Table shows a study on the proposed simple automatic prompt-filtering For the experiment, ViT-B/16 models using InfoNCE loss and PCME+ ++ are trained on CC3M Sharma et al. (2018), 12M (Changpinyo et al., 2021) and RedCaps (Desai et al., 2021), for 32 epochs with 1024 batch size. "Top-K certain prompts" denotes that every class the filtering, and "Best top- for each class" denotes the best top-K for each class are chosen, egg., "terrace" needs all 80 prompts, while "pencil case" only needs Top-1 certain prompt while other uncertain 79 prompts are discarded. More examples of the selected prompts are shown in Figure C.4. With this simple strategy, the ZS performance is increased with a significant gap (30.43 - 41.82). The full description of the ZS experiment is provided in Appendix C.6.
the same top-K for
4 CONCLUSION
This paper brings attention to the importance of the FNs in ITM training datasets; a deterministic method will fail to capture the inherent ambiguity of the dataset, especially for large backbone. such as ViT-L. This paper addresses the inherent ambiguity of ITM tasks by probabilistic embedding with a novel closed-form probability distance and new matching objective for efficiency and effectiveness. The proposed method, PCME++, is further enhanced by employing a ppeudo-positive strategy and a mixed sample data augmentation strategy, thanks to the pair-wise loss function design. Experimental results demonstrate the extensibility of PCME+ to various applications, such as image- text cross- modal retrieval, mitigating noisy correspondences, automatic prompt-fitterngg for zero-shot classification, and understanding the inherent ambiguity of a dataset. Beyond performance, the learned uncertainty by PCME+ shows high interpretability of the datasets as well as the controllability by the users when the rejection of the retrieved items is required.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
I would like to thank NAVER AI Lab colleagues for valuable discussions, including Sangdoo Yun, Wonjae Kim, Jiyoung Lee, Dongyoon Han, Bycongho Heo, Tackyung Kim, Song Park and Jung-Wor Ha. NAVER Smart Machine Learning (NSML) platform (Kim et al., 2018 8) is used for the experiments. I would like to express my sincere gratitude my adorable cat, Seul Park, who served the model for Figure 2, and my wife, Song Park, the delightful companion who brightened my research journey.
SOCIETAL IMPACT
This work aims to learn better image-text representations based on a probabilistic approach. As shown in the experiments, PCME++ has the potential to improve the interpretability and the controllability of learned representations by providing an additional degree of freedom the users. Accordingly, PCME++ shares the potential impact of developing general image-text representations with better interpretability and controllability. For example, as shown by Radford et al. (2021), visual-textual representations trained on large-scale web dataset often suffers from biases in the web; PCME++ can both mitigate or enhance the biases using its interpretability and controllability.
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In Int. Conf. Learn. Represent. 2017. httpstps: :ttttt::://ppeeeeeee /ttp:///w/p//eeeee/////e net forum?
id=HyxQzBceg.
Ali Furkan Biten, Andres Mafla, Lluís Gómez, and Dimosthenis Karatzas. Is an image worth five sentences? a new look into semantics for image-text matching. In JEEE/CVF Winter Conf. App. Comput. Vis., pp. 1391-1400, 2022. 1
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. In Adv. Neural Inform.
Process. Syst., 2021. 20
Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen Wei. Data uncertainty learning in face recognition. IEEE Conf. Comput. Vis. Pattern Recog., pp. 5710-5719, 2020.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web- scale image-text pre-training recognize long-tail visual concepts. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3558-3568, 2021. 1, 9, 24
Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 15789- 15798, 2021. 5, 16, 19, 20
Xinlei Chen, Hao Fang. Tsung-Ye Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00335, 2015. 1, 5,
Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio De Rezende, Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In IEEE Conf. Comput. Vis. Pattern Recog., 2021. 1, 2, 4, 5, 16, 19, 29
Sanghyuk Chun, Wonjae Kim, Song Park, Minsuk Chang Chang, and Seong Joon Oh. Eeev caption: Correcting false negatives by collecting machine- wwwaaandn-h--rrriiiii for ms-coco. Fur. Conf. Comput. Vis., 2022. 1, 2, 3, 4, 5, 6, 18, 19,21
image-caption associations
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. RedCaps: Web-curated image-text data created by the people, for the people. In NewrIPS Datasets and Benchmarks, 2021. 1, 9. 24
Haiwen Diso, Ying Zhang, Lin Ma. and Huchuan Lu. Similarity reasoning and filtration for image-text matching. In Proceedings of the AAAI Conference Artificial Intelligence, 2021.
10

---

Published as a conference paper at ICLR 2024
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Int. Conf. Learn. Represent. 2021. URL https: https::/openreetteeeeee /wpp//ppeeeeeeeeeeeee ew net. forum? id=YicbFdNTTy...
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. VSE++: Improving visual-semantic embeddings with hard negatives. In Brit. Mach. Vis. Conf., 2018. 3,
Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Marc' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep visual-semantic embedding model. In Adv. Neural Inform. Process. Syst., pp. 2121-2129, 2013. 1
Jiuxiang Gu, Jianfei Cai, Shafiq R Joty, Li Niu, and Gang Wang. Look, imagine and match: Improving textual-visual cross-modal retrieval with generative models. In IEEE Conf. Comput. Vis. Pattern Recog-, pp. 7181-7189, 2018. 1
Byeongho Heo, Sanghyuk Chun, Seong Joon Oh, Dongyoon Han, Sangdoo Yun, Gyuwan Kim, Youngjung Uh, and Jung-Woo Ha. Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights. In Int. Conf. Learn. Represent, 2021. 6, 19
Yan Huang, Qi Wu, Chunfeng Song, and Liang Wang. Learning semantic concepts and order for image and sentence matching. In JEEE Conf. Comput. Vis. Puttern Recog. pp. 6163-6171, 2018.
Zhenyu Huang, Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, hua wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Adv. Neural Inform. Process. Syst., 2021. URL https: // //openreview..o www.eewwwwteeoeemeeeeme iiddiddsso sadd-s920 290-992yhhh 92yWC11 7wJ. 1, 6, 20
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openelip, July 2021. URL https: / / //oiiiioo /tt./i.ooggd///// wtgg/0..5000 5281/ zerodo. 514554433 143773. If you use this software, please cite it as below. 19, 24
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Conference on Uncertainty in Artificial Intelligence, 2018. 6, 20
Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang. Jiaxing Zhang, Tetsuya Sakai, and Yujiu Yang. Map: Multimodal uncertainty-aware vision-language pre-training model. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 23262-23271, 2023. 20
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-5477 2019.
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 3128-3137, 2015. 18
Dongwon Kim, Namyup Kim, and Suha Kwak. Improving cross-modal retrieval with set of diverse embeddings. In Proceedings the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 23422-23493 2023. 1
Hanjoo Kim, Minkyu Kim, Dongjoo Seo, Jinwoong Kim, Heungseok Park, Soeun Park, Hyunwoo Jo, KyungHyun Kim, Youngil Yang, Youngkwan Kim, et al. Nsml: Meet the mlaas platform with real-world case study, arXiv preprint arXiv:1810.0995: 2018. 10
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Aision-and-languall transformer without convolu- tion region supervision. In Inf. Conf. Mach. Learn., 2021. 20
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Int. Conf. Learn. Represent., 2015. 15
11

---

Published as a conference paper at ICLR 2024
Michael Kirchhof, Enkelejda Kasneci, and Seong Joon Oh. Probabilistic contrastive learning recovers the correct aleatoric uncertainty of ambiguous inputs. International Conference on Machine Learning, 2023. 3, 29
Ryan Kiros, Ruslan Salakhutdinov. and Richard Zemel. Unifying sesual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539. 2014. 1
Kuang-Huek Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Eur. Conf. Comput. Vis., 2018. 1
Hao Li, Jingkuan Song, Lianli Gao, Pengpeng Zeng, Haonan Zhang, and Gongfu Li. differentiable semantic metric approximation in probabilistic embedding for cross-modal retrieval. Advances in Neural Information Processing Systems, 35:1 1934-119499 2022a. 6
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels semi- supervised learning. In International Conference on Learning Representations, 2020. URL https: ftttps::/cpenreviie .et Forum? (errummiddHJJRgaaV.t.. 6
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022b. 20
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image-text matching. Int. Conf. Comput. Vis., pp. 4654-4662. 4664, 2019. 1,
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Eur. Conf. Comput. Vis., 2014. 18
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018. 20
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. Eur. Conf. Comput. Vis., 2020. 5, 18, 19
Andrei Neculai, Yanbei Chen, and Zeynep Akata. Probabilistic compositional embeddings for multimodal image retrieval. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 4547-4557, 2022. 2
Seong Joon Oh, Kevin Murphy, Jiyan Pan, Joseph Roth, Florian Schroff, and Andrew Gallagher. Modeling uncertainty with hedged instance embedding. In Int. Conf. Learn. Represent. 2019. 4, 29
Zarana Parekh, Jason Baldridge, Daniel Cer, Austin Waters, and Yinfei Yang. Crisscrossed captions: Extended intramodal and intermodal semantic similarity judgments for ms-coco. Conference of the European Chapter of the Association for Computational Linguistics, 2021. 2, 3, 5, 18, 19
Chanwoo Park, Sangdoo Yun, and Sanghyuk Chun. unified analysis mixed sample data augmentation: A loss function perspective. In Neural Information Processing Systems (NeurIPS), 2022a.
Jungin Park, Jiyoung Lee, Ig-Jae Kim, and Kwanghoon Sohn. Probabilistic representations for video contrastive learning. In IEEE Conf. Comput. Vis. Puttern Recog., pp. 14711-14721, 2022b. 2
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phene torasseeeeeeeeeeee correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference computer vision, pp. 2641-2649, 2015.
Yang Qin, Dezhong Peng, Xi Peng. Xu Wang, and Peng Hu. Deep evidential learning with noisy correspondence for cross- cross-modal retrieval. In Proceedings of the 30th ACM International Conference Multimedia, MM '22, pp. 4948 4956, 2022. doi: 10. 10.11455 00.11453353311133555 1113549911113399922.2.22
12

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Int. Conf. Mach. Learn.. pp. 8748-8763. PMLR, 2021. 1, 4, 5, 6, 9, 10, 19, 20, 24
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-22 2015. 24
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Association for Computa- tional Linguistics, pp. 2556-2565, 2018. 1, 9, 24
Yichun Shi and Anil K Jain. Probabilistic face embeddings. In IEEE Conf. Comput. Vis. Pallern Recog., pp. 6902-6911, 2019. 2
Anna Silnova, Niko Brummer, Johan Rohdin, Themos Stafylakis, and Lukas Burget. Probabilistic embeddings for speaker diarization. In Proc. Odyssey 2020 The Speaker and Language Recognition Workshop, pp. 24-31, 2020.
Yale Song and Mohammad Soleymani. Polysemous visual-semantic embedding for cross-modal retrieval. IEEE Conf. Comput. Vis. Puttern Recog., pp. 1979-1988, 2019.
Jennifer J Sun, Jiaping Zhao, Liang-Chich Chen, Florian Schroff, Hartwig Adam, and Ting Liu. View-invariant probabilistic embedding for human pose. In Eur. Conf. Comput. Vis., 2020. 2
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008. 22
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Adv. Neural Inform. Process. Syst., pp. 5998-6008, 2017. 5
Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, and Lin Ma. Consensus-award visual-semantic embedding for image-text matching. In Eur. Conf. Compus. Vis., 2020. 1, 20
Zheng Wang, Zhenwei Gao, Xing Xu, Yadan Luo, Yang Yang, and Heng Tao Shen. Point to rectangle matching for image text retrieval. In Proceedings of the 30th ACM International Conference Multimedia, pp. 4977-4986, 2022. 6
Frederik Warburg, Marco Miani, Silas Brack, and Soren Hauberg. Bayesian metric learning for uncertainty quantification in image retrieval. arXiv preprint arXiv:2302.01333, 2023. 29
Jonatas Wehrmann, Douglas M Souza, Mauricio A Lopes, and Rodrigo C Barros. Language-agnostie visual-semantic embeddings. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 5804-5813, 2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure in timm. arXiv preprint arXiv:2 10.00476, 2021. 16
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In IEEE Conf. Compus. Vis. Pattern Recog.. pp. 6609-6618, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Association for Computational Linguistics, 2:67-78, 2014.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Int. Conf. Comput. Vis.. 2019. 4
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In Int. Conf. Learn. Represent,, 2018.
13

---

Published as a conference paper at ICLR 2024
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In Int. Conf. Learn. Represent, 2021a. la. 4
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, and James Zou. When and how mixup improves calibration. arXiv preprint arXiv:2102.0622 (22.0. 2021b.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. In IEEE Conf. Comput. Vis. Pattern Recog., 2021c. 20
APPENDIX
More additional materials are included here. More details of PCME++ are described in $A, including the full derivation of the closed-form probabilistic distance ($A.1), the toy experiments (SA.2), comparisons between PCME and PCME++ ($A.3), pseudo-code of PCME+ ($A.4), and why applying PPs and MSDA is non-trivial to other methods ($A.5). In experimental protocol details section (SB), the benchmark dataset details ($B.1) I), hyperparameter, resource details ($B.2) and SWA details ($B.3) are shown. Additional experimental results ($C), including comparisons with state-of-the-ar (SC.1), the analysis of greudo-positivve ($C.2) the full ablation studies ($C.3), the I-SNE visualization details and related discussions ($C.4) the comparisons of different retrieval strategies ($C.5), automatic prompt-filtering experiments (CC.6), and the full experimental results and error bars (SC.7) are presented. Finally, discussions and limitations of PCME++ are in $D.
A METHOD DETAILS
A.1 DERIVATION OF THE CLOSED-FORM PROBABILITY DISTANCE
In this subsection, the full derivation of Equation (1) is shown. We first show two simple well-known lemmas and conclude the full proof using them.
Lemma 1. Let X and Y be independent normally distributed random variables where X Nux, Σx) and Y ~ N(yy, D.). Then, the subtraction between X and Y is another normal distribution, i.e., (X - Y) N(xx -uy, Ex +Ey).
Proof. Let Φx (u) = explit μx - 1 Ext) be a characteristic function of normally distributed random variable X. Using the fact that -Y N(-uy, Ey), we can compute the summation of øx (x(u) and (u) as follows:
1 Φx (x)y(u)= = exp(it px 2t Ext) bY / 1, Evt) = exp(itT(uxx (ux- - (EEx(xxxxx + Ey )t).
(A.1)
Hence, X - Y is another normal distribution, N(ux - py, Ex + Ex).
Lemma 2. Let X ~ (N, E). Then 112 = 11 + Ir(E).
Proof. We first re- parameterize a random variable as X = + SZ. where S is the square root matrix of Σ, i.e., SST Σ, and Z is standard normal distribution. Note that S always exists because a positive semi-definite by definition. Using E[Z] = O, the property of Frobenius norm ||ll tr(A) and the property of trace tr( (AB AA) have:
F x X 2 = Ez || 13 + 2p57 + T 5Z = || Ez T S S2
(A.2)
| + TT TT) = 11 + tr(S S EZ 11 2 + tr(E).
Proposition 1. Let X and Y be independent normally distributed random variables where X ~ Hy
N(px,Ex and Y ~ NNy, Then we have | Y|| = ||xx
tr(2 Er).
Proof. By combining Lemma and Lemma the proof is completed.
14

---

Published as a conference paper at ICLR 2024
Epoch HardesttinlleWarrgnnees
Epoch implettaargnnoss
1-
1-
-3
Epoch WDistanceLoss
Epoch 500 CosedSDLoss
3-
2-
-2
-2-
-3
Figure A.1: Toy results for HNM & SUM VSE, Wesserstien distance & &CCME++ The full animation can be found in htttps:/naver-O000 .g github..hh...... po/pppppp/
A.2 TOY EXPERIMENTS
In Section 2.2, a 2-D toy dataset is introduced for comparing various objective functions under inherent uncertainty. The toy dataset has three classes with "confusing samples" between classes, ie., confusing sample randomly can be either class A or class B. The number of confusing samples are 30% of the total data points. To synthesize the samples, a centroid is randomly chosen for each class. Using the centroid, each sample is randomly drawn from μ 0.1 N I). 500 samples are drawn for each class and 150 samples of them are chosen as "confusing samples", i.e., there are 1500 samples with 1050 certain samples and 450 confusing samples. Then, log of each sample is randomly drawn from U (1.5,, 1.5) where 2/ is a uniform distribution. In summary, the dataset has 350 confident samples for class 1, 2 and 3; 150 confusing samples for class (1, 2), (2, 3) and (3, 1).
To show the effects of different probabilistic distances, the samples are directly updated by the objective function Equation (2) with different metrics, i.e., sample (j. σ)) is directly updated by Equation (2). The dataset is directly optimized using Adam optimizer (Kingma & Ba, 2015) with learning 0.02 during 500 epochs. The mini-bath size is set to 128. The same loss function with PCME+ (i.e., Equation (2)) is employed while the probabilistic distance d(Z., Z,) is chosen from either CSD (Equation (1)) or Wasserstein distance. VSE models are also trained with the hardest negative mining and without the negative mining (i.e., using the summation of triplet distances). The last snapshots of each model are shown in Figure A. The animated learning progress of each method can be found in https:/ : wwnnanvvraaaaa aii gi with i pcmepp/.
As described in Section 2.2, we desire the learned uncertainty can capture the data uncertainty, i.e., we expect that certain samples have small o², while uncertain samples have large o², After training, we observe that the average o² for certain samples and uncertain samples by PCME+ + are 1.68 and 3.05, respectively. On the other hand, Wasserstein distance shows 2.69 and 2.80. respectively. The result and other experiments on large-scale datasets (Section 3.3) support that PCME++ is proper probability distribution to capture uncertainty, while Wasserstein is not.
15

---

Published as a conference paper at ICLR 2024
A.3 COMPARISONS WITH PCME AND PCME+ OBJECTIVE FUNCTIONS
As both PCME and +CME+ + aim to learn probabilistic embeddings, they share the nature of the probabilistic embeddings: train encoder that maps an input mean vector and variance vector and train the encoder by maximizing the negative loglikelihood (NLL) using the extracted distributions, namely min Σ log pmm] Zt). PCME and PCME+ optimize different NLLs where PCME is based on "matching probability" and PCME++ is based on binary cross-entropy.
We recall the definition of matching probability of PCME (Chun al., 2021):
p(m[a,, Fe) = Ez.,z, sigmoid(-all2a 2,112
Σ
sigmoid (-all= -ell== (A.3)
J2
Roy
where J is the number of samples 2, and 24. PCME directly optimized the negative log- lieeliokl
Myt log sigmoid (-all zt l2 + b) + (1 - -mat) log
sigmoid (al + zell2 + b) (A.4)
On the other hand, PCME+ optimizes the NLL using binary cross-entropy loss (Equation (2)):
p(m| = sigmoid(-
11, 2.11 +).
Equation (A.4) and Equation (2) share similar formulation, but the position of the expectation is different. As the expectation is located at the outside of sigmoid, Equation (A.3) cannot be computed in closed-form solution, but our distance can. PCME+ has two benefits over PCME: (1) our form has a closed-form solution, while PCME cannot. (2) our form can be naturally adopted into the binary cross entropy loss function, which is known to be stable and perform well in large-scale training (Wightman et al., 2021).
Furthermore, pointed out in Section 1, The computational cost of PCME depends the number of MC samples J, because it needs to compute OJ²²) pairwise distances between all samples. When we use the same setting of the paper (J = 8), we observe that + 25 epoch training takes 106,311 sees (1 day and 5 hours), while PCME 25 epoch training takes 141, 41,6994 secs (1 day and 15 hours) on a single V100 GPU. Overall, PCME needs 33% more training time compared to PCME+ Note that if we increase the sampling size J, the gap becomes larger. Another issue of the PCME sampling is that we need more memory size when computing the Monte Carlo approximation for larger sampling size. Overall, PCME needs more forward time than PCME++ (33% more), and more memory size than PCME+ (on average, 18% more, but is not a rigorous comparison because PCME has higher peak memory usage).
A.4 PCME-+ PSEUDO-CODE
Figure A.2 shows the PyTorch style pseudo-code of PCME+ +. Note that μ and a are extracted from the augmented inputs, such MSDA (Section 2.4) and SizeAugment (Chen et al., 2021).
A.5 WHY IS IT NON-TRIVIAL TO APPLY MSDA TO PREVIOUS METHODS?
Although this paper does not propose a new MSDA, the contribution of this paper lies in applying MSDA to the relational datasets. For example, applying MSDA classification is straightforward because the mixed sample does not affect the other samples in the mini- relational training objectives, such as triplet loss or contrastive loss, a mixed sample affects the other samples in the batch as well. Especially, the triplet loss is impossible to handle MSDA, because the core concept of MSDA the smooth label, but the triplet loss cannot handle smooth label, because has to construct triplet of the selected sample, the positive sample, and the negative sample. It is non-trivial to define positive and negative samples when the label is smoothed (See Figure A.3 a). For example, assume that we match annotation of v., ta to 0.6 from 0.0. In this case, it is non-trivial to build triplets using this annotation. Moreover, if we introduce mixed samples and mixed labels, the problem becomes more complex. How can we handle Va.b (a mixed image of xa and 27)) and f.. or Vs.a and ta using a triplet relationship. or a pairwise relationship? Therefore, it is non-trivial to apply PPs and MSDA for the triplet- based methods.
However, in the
Similarly. a batch-wist wise contrastive loss, such as InfoNCE, is also little bit tricky to control the effect of smooth labels (See Figure A.3 b) because the mixed samples are combined in the denominator
16

---

Published as a conference paper at ICLR 2024
def compute Lossstte___oeeeeeeeeeee loss mu, v_sig, t_mu, t_sig, matched) :
"*"y_mu, v v_sig: mean and variance for (mixed) images (N by D) t_mu, t_sig: mean and variance for captions (M by D) matched: denoting (i, j) image, caption pair is matched. values are between 0 and (N by M) ***
compute closed-form distance
mu dist (V__mu um.uuqqueeze (1) t m_m unsqueeze (0)) ** 2) sum (-1) sigma_diss ( ((v_sig, unsqueeze (1) t_sig.uusquers (0) sum (-1)
b: learnable affine transform
11
logits
(mu_dist + sigma_dist)
12
13
match loss can be easily computed by BCE loss
match_loss (1ogits matched)
16
compute pseudo-positive (pp) match loss
17
gt_labels, gt indices torch.max max (matched, dim=1) gt_vals - logitsl:, [__indices].diag
18
sorludo_gttindicest pp_matched (gt_labels. unsuueezeee
(logits >- gt_vals)
20
(pseudo_gt_indicces) indiccces]
21
matched (peeddo_tp_eeu___t_____ni st__ndices]] pp_ ap_matched
22
pp si_match_losss BCE (logits, matched)
compute VIB loss
25
v_vib -0.5 (1 torch.log
v_mu 2 v_sig) mean ()
26
v_v = -0.5 + (1 torch.log(t_sigg mu ** 2 t_sig) mean
27
vos_ v_vib + t_vib
final loss, alpha and beta hyperpa remeters
30
return match loss alpha pp match loss beta vib
Figure A.2: PyToreh pseudo-code of PCME+ Here, v_sig and t_sig computed by taking an exponential the output of log heads, BCE denotes binary cross-entropy function.
Samples contributing the loss
Samples contributing to the
A sample group for defining objective function
22
Zr
ZZ
2i' Z
Z: Z
z
z Z
Z
Z
Zi' z
z
7: z: 72
Z+
C.4
D.8
0.6
0.4
Z.
Z.
Z.
2
2.7
(a) Triplet loss
(b) Batch-wise contrastive loss
(c) Pair-wise contmsstiv loss
(d) (c) Pseulo-posilive MSDA
Figure A.3; Comparisons of different objective functions. For given i-th visual embeddings z and j-th textual embedding zi- we illustrate how each sample contributes to different loss functions. (a) Only two image- caption pairs contributed to the loss in each row/column for triplet loss: L. = [|] 2 =tll || =f'l a]+, where L is the loss value of i-th visual feature z and Li is defined similarly. (b) Batch-wise contrastive
loss, such as InfoNCE, is defined for each row/column: L CE
expl -mpp-1=1-=t1l)
where CE denotes
(5 -x,o(((11=
cross-entropy loss. Namely, the entire textual features are used to compute the loss value of i-th visual feature. (c) Pair-wise contrastive loss, such as PCME++, is defined for each image-caption pair: 29 BCE(sigmoid (d(Z., Z,)), where -(:--) CSD (Equation (1)) and is an indicator function. Hence, loss is computed multiple times for each row/column: (uirwise = E.,, L On the other hand, (a) and (b) are computed by Cothers Σ, Ct Σ, C/ (d) As our loss is computed pair-wise, it is straightforward to apply pseudo-positives (PPs) or mixed sample data augmentation (MSDA), while it is not trivial to apply PP and MSDA to other methods as described in Appendix A.5.
17

---

Published as a conference paper at ICLR 2024
Imagex
Captions
Posiive matches
COCO
COCO
Caption
Figure B.1: Difference between COCO 5K, 1K (Chen al., 2015), CxC (Parekh al., 2021) and ECCV Caption (Chun et al., 2022). All matches not illustrated the image are negative. ECCV Caption has separated query sets for each modality, while other datasets use the same images and captions for both query and gallery.
Table B.I: Hyperparameter details
Method
ViT B/32, B/16, I/14 COCO
VIT B/16 CC3M, 12M and RedCaps
Epochs
25
32
Batch size
1,024
Optimizer
AdamP
AdamW W
Initial learning rate
0,0005
0,0005
LR scheduling
Step
linear warmup and cosine
Layer-wise LR decay
0.7
Visual backbone LR decay
0.01
Textual backbone LR decay
0.1
11,32,6 Weight decay
0.9, 0.999, 10
0.9, 0.98, 10
0.0001
0.2
VIB 3
0.0001
10-6
PP
0.1
MSDA CutMix/Mixup A, mix ratio
2/2/25%
-/-10%
Size Augment
X
Embedding dimension
1024
512
Initial a and b
5/5
1/1
Resources and training hours
VIT B/32 V100 (38 hours)
V100 (17 hours)
VIT B/16 V100 (75 hours)
VIT I/14 8 V100 (62 hours)
term of Softmax. On the other hand, the proposed pairwise contrastive loss can directly apply smooth labels because each loss computation invariant the other samples, only the given pairs affect the loss computation as shown in Figure A.3
One of the technical contributions of this paper is allowing smooth labels and mixed samples by designing pairwise loss that is not affected by the other data samples. As shown in Figure A.3 d, each loss computation of + independent of the other pairs, while triplet loss or batch-wise contrastive loss is dependent the relationships of other pairs.
B EXPERIMENTAL PROTOCOL DETAILS
B. MORE DETAILS OF BENCHMARK DATASETS
PCME+ is evaluated on MS-COCO Caption (Chen et al., 2015), a widely used ITM benchmark, containing 123,287 images from MS-COCO (Lin et al., 2014) and five (Karpathy & Fei-Fei, per image. 113,287/5.000 0777550000/500000 images are used for saining/validation/esstin 2015). Although Recall @ k (R @ k) common evaluation metric in COCO Caption, Musgrave
captions
18

---

Published as a conference paper at ICLR 2024
et al. (2020) showed, R@k is often insufficient to measure retrieval performances. Furthermore, recent studies (Parekh et al., 2021; Chun et al., 2022) observed that many COCO Caption negatives actually positives; e.g., Chun et al. (2022) showed that 88.2% and 72.1% positive images and captions are annotated as negative in COCO. In other words, COCO R @@ k; relying on the noisy COCO annotation myt, is not fully reliable.
To mitigate the problem of @@k evaluation, two extended benchmarks, ECCV Caption (EC) (Chun et al., 2022) and CxC (Parekh et al., 2021), are employed for the test split. Both datasets are validated by human annotators; EC contains more plentiful positives than CxC but its queries are the subset of the original COCO Caption; CxC has fewer positives than EC, but its annotations cover the whole COCO test split, and the annotations are less noisy. Note that the original COCO Caption, EC, and CxC have the same images and captions (I.. xy,) but with different match annotations myt-
Figure B.1 illustrates the differences between evaluation datasets. Note that all evaluation benchmarks use the same training dataset described in I..11 The COCO Caption evaluation split consists of 5,000 images and 25,000 captions. COCO 5K uses the full 5,000 images and 25,000 captions where each image has five positive captions and each caption only has one positive image. For evaluation, COCO 5K measures image-to-text retrieval performances by setting 5,000 images as queries and 25,000 captions as galleries, while text-to-image retrieval performances are measured in the opposite way. COCO 1K uses the same positive relationships as COCO 5K, but COCO 1K uses the subset of COCO 5K, i.e. there are 1,000 images and their corresponding 5,000 captions for COCO 1K split. COCO 1K measures the performances by taking an average of five different splits.
CxC (Parekh et al., 2021) and ECCV Caption (Chun et al., 2022) use the same images and captions of COCO I K/5K, but with more positive annotations. CxC uses the entire images and valid 24,972 captions among 25,000 captions (by omitting "I cannot see any image" captions). CxC has more positive annotations than COCO, but there are still many missing positives in CxC because their approach is mostly focused on text similarity, not image-text similarity. On the other hand, ECCV Caption designed for handling false negatives of image-text pairs. ECCV Caption uses the subset of images and captions for the queries, but their retrieval database is the full dataset, i.e. when performing image-to-tex to-tetttttxtt retrieval, the number of query images is 1,261 and the number of gallery captions are 25,000; for text-to-image retrieval, the number of query texts is 1,332 and the number of gallery images is 5,000.
As discussed by Musgrave et al. (2020) and Chun et al. (2022), Recall @ is not an informative metric for measuring retrieval performances in terms of precision. Due to this reason, this paper reports mAP@R R and R-Precision of ECCV Caption the main comparison metrics.
B.2 HYPERPARAMETER AND RESOURCE DETAILS
All models are trained for 25 epochs using AdamP optimizer (Heo et al., 2021) by setting the initial learning rate as 0.0005 and weight decay as 0.0001. The learning rate is decayed by a factor of 0. for the last 10 epochs. Following Chen al. (2021), different learning rate multipliers are applied for the visual backbone (x0.01) and the textual backbone (x0.1). The visual backbone is frozen for 2 epochs, and a linear learning rate warmup is applied for the first epoch after the freezing, Also, layer- wise learning rate decay (LLRD) for each transformer block is applied by 0.7. The batch size is set to 128. Lastly, for the generalizability of GPO, SizeAugment is employed as Chen et al. (2021).
The hyperparameters of PCME++ are set as follows; the affine transform is initialized by a = b = 5 in Equation (2); a for pooudo-ssssives as 0.1; VIB 3 as 0.0001. PCME+ mixes 25% of images in the mini-batch by Mixup or CutMix with a mixing ratio drawn from Beta(2, 2). For comparison methods, The triplet loss margin is set to 0.2 (for VSEco (Chen et al., 202 and the initial softmax temperature for InfoNCE (Radford et al., 2021) is set to 1.0. PCME (Chun et al., 2021) uses the same initialization of PCME+ + for affine transform and VIB, while 8 samples are drawn per input for computing matching probability.
We use different hyperparameters for the large-scale pre-training task in Section 3.5. As our imple- mentation is based openclip (Ilharco et al., 2021), generally follow the official openelip hyperparameters. More details are in Appendix C.6.
Table B.I shows the detailed hyperparameter settings and the detailed GPU resource information.
19

---

Published as a conference paper at ICLR 2024
Table C.I: Comparisons with state-of-the-art models. numbers are reproduced by official weights
We highlight the best scores except expensive retrieval methods, such as BLIP.
Efficient retrieval?
ECCV Caption
CxC
COCO
Method
mAP@R
R-P
RØ1
R@1
IK RØ1
5K R@1
RSUM
CVSE (Wang et al., 2020)
37.4
47.5
76.7
45.8
67.0
43.8
511.1
VSRN (1.i 2019)
51.8
69.5
46.7
NCR (Huang al, 2021)
36.4
46.3
79.9
51.8
71.0
50.0
522.6
VSEoo (BUTDD region) (Chen et al., 2021)
40.5
50.0
82.5
52.4
72.2
50.4
527.5
VSEco (WSL)
42.4
51.4
86.4
60L8
78.3
59.0
545.1
VSEco (B/16, our implementation)
41.7
50.6
86.3
62.3
79.1
60.7
547.2
VILT (Kim al, 2021)
X
34.6
44.3
71.8
53.7
72.8
52.2
528.6
ViaVL (Zhang et al., 2021c)
40.8
49.6
87.8
67.8
82.4
66.4
555.5
BLIP et al., 2022b)
40.5
48.4
91.0
74.3
86.1
73.1
564.4
CLIP Zero shot (L/14) (Radford et al., 2021)
28.0
37.8
72.2
48.1
64.8
46.4
491.6
PCME++ (B/16)
51.2
79.6
61.3
548.5
PCME+ (L/14)
42.1
50.8
88.8
65.9
81.8
64.3
$54.7
B.3 SWA AND MODEL SELECTION
For the evaluation, the best model based on the validation rsum selected. However, the clean validation set is not always achievable; e.g., if the dataset has significant distribution shift. For example, as we know that the COCO Caption is noisy due to lot of FNs (Chun et al., 2022), validating using the noisy annotations could lead to underperforming models. Instead of the best validation model selection, we can apply SWA (Izmailov et al., 2018) where does not need an additional optimization process the existing training procedure, but only training weight trajectory (i.e., weights per every training epoch) is required. SWA is a weight average method of sufficiently trained weights. SWA aims to achieve flat minima, well as more generalizable and robust solutions (Cha et al., 2021). When we apply SWA to our model, we apply SWA for the last 10 epochs. Although the SWA models are not compared to the other models due the fair comparison issue, strongly encourage the use of SWA for future research.
C ADDITIONAL EXPERIMENTAL RESULTS
C.1 COMPARISONS WITH STATE-OF-THE-ARTS
Table C.I shows the comparisons of PCME++ and -fate-of-the-arte with different backbones. Note that ViLT (Kim et al., 2021), VI. (Zhang et al., 2021c) and BLIP (Li et al., 2022b) need heavy computations to perform retrieval because they have to compute pair-wise similarity for all pairs. For example, they need 05,0000x 25,000) computation budgets for measuring retrieval performances. On the other hand, methods with separated encoders just need ((5,000+25,000 computation budgets, 4,166 times smaller computation budgets compared to expensive retrieval methods. Therefore, the table only highlights the best retrieval performances among efficient retrieval methods for a fair comparison. Note that even in an expensive cross-attention architecture, probabilistic approach can be beneficial, as shown by MAP (Ji et al., 2023). PCME+ is applicable to MAP by replacing the 2- Wasserstein distance with CSD. However, as it is out of scope of this work, the comparison with MAP is omitted in this paper. PCME ++ achieves the best recall scores for all evaluation benchmarks while showing second and third-best ECCV mAP@ @ and R- A-PPrccsiion. One possible explanation is the capability of the backbone architecture. For example, VSEcc with CLIP B/16 backbone shows much better recall scores than VSE.c with WSL backbone, but VSE0o (WSL) shows better mAP@R and R- e-Prrcision than the CLIP backbone. From this observation, we expect that PCME++ can outperform the previous retrieval methods in precision metrics if we train PCME+ using different backbones, such as large-scale weakly supervised learning (WSL) backbone (Mahajan et al., 2018).
C.2 THE EFFECT OF PSEUDO-POSITIVE (PPS)
We evaluate various under various noise ratios (NR). Table C.2 shows two findings: (1) unlike "no noise ratio" case (i.e., Table C.3), if there exist noisy correspondences (NC), PPs can harm the performances. One possible explanation is that PPs can be incorrect if there too many NCs. (2) When we tune a, we obtain the best performances in the 50% NR scenario. However, we observe that model is easily overfitted when weaken the strength of PPs. Figure C. la shows that when
20

---

Published as a conference paper at ICLR 2024
Noise ratio
PP a
EC mAP
EC R-P
EC R@
COCO IK R@I
COCO 5K R@1
RSUM
0
37.9
47.8
79.0
71.9
50.9
526.0
20%
0.01
37.7
80.0
71.6
50.4
524.6
0.1
37.9
47.7
79.7
70.8
49.5
522.4
0
35.7
45.8
76.3
67.6
45.5
511.0
50%
0.01
35.2
45,3
75.1
66.1
43.4
506.8
0.1
34.4
44.6
75.0
65.7
44.0
503.9
Table C.2: different noisy ratio results with varying PP a. a = 0.1 equals "PCME-- (ours)" in Table 2.
MAPER
@x
1230
Weaker FPs (smaller
shows overliting after
32
Weaker (smaller a) shows butaar parformmncc
sacurh traiiig pooche
800
NR190) wt..1)
(RL20) 1)0.1)
NRL50) o1
NRU20)
al NRUS0) W|31
200
26
(a) Overfitting happens with weaker PP under NCs.
(b) Number of the PPs.
Figure C.1: The effect of Pseudo-Positives (PPs) during training.
a = the best performed score is better than a = 0.1, but at the end of the training, a = 0.1 shows weaker overfitting. As the current evaluation criterion is based on the validation- model selection, even though PPs can prevent overfitting, PPs cannot be directly helpful for achieving the best performances under NCs. In other words, PPs helpful for preventing overfitting and gradient vanishing (Section 2.3); thereby, when there is less noise (i.e., no noise ratio scenario). PPs can improve performances with gap, especially for handling false negatives (FNs) well (See mAP@ scores of 2nd and 4th row in Table and Table C.3).
The number of PPs during training illustrated in Figure C. We observe that # PPs 16, 185 the first iteration, but it has converged to around 260 after 1 epoch.
C.3 MORE ABLATION STUDIES
Table C.3: Pseudo-positii a ablation study.
ECCV Caption
CxC R@1
COCO
mAP@R
R-P
R@D
IK R R@!
5K R@D
RSUM
0.1
40.2
49.8
83.1
56.5
75.1
54.8
536.0
0.5
40.0
49.5
83.1
56.7
75.4
55.0
536.8
2
40.1
49.7
83.0
56.5
75.1
54.8
535.8
5
40.3
49.9
83.1
55.7
74.7
53.9
534.9
10
40.2
49,9
82.5
54.5
73.7
52.6
531.9
PP and MSDA. Table C.3 shows the ablation study for pseudo positive a. The table shows that PCME++ is not very sensitive to the choice a. We choose a 0.1, which shows the second-best EC mAP@R and COCO recall measures. Table C.4 shows the ablation study for the mixed sample data augmentation design choice. The design choice for PCME+ shows the best performance.
VIB. The parameter study of VIB 3 is provided in Table C.5. In the table, the average uncertainty quantification ||o I1. the retrieval performances, and the correlation coefficient between the uncertainty and the average @ 1, p. by varying VIB 3 from 0 to 0.001 (where x1 0.0001) are reported. = 1 means that uncertainty and R@ are perfectly negatively correlated, i.e., a higher uncertainty then the uncertainty quantity and the recall
shows lower recall (what expect). If p performance are not correlated. Note that although we chose RSUM for performance comparisons due to the space limitation, we observe similar phenomenon for any other metric.
21

---

Published as a conference paper at ICLR 2024
Table C.4: MSDA ablation study.
ECCV Caption
CxC
COCO
Mixup À
CutMix À
Mix ratio
in-batch?
mAP@R R
R-P
R@
R@1
1K R@1
5K R@1
RSUM
2
0
25%
37.3
47.3
79.6
$1.9
71.7
50.0
$25.5
0
25%
37.5
47.6
79.2
50.5
70.9
48.8
523.4
1
50%
X
39.7
49.5
81.8
55.2
74.5
53.5
$34.0
25%
39.9
49.6
82.3
55.5
74.6
53.8
534.4
2
2
25%
X
40.0
49.6
82.8
55.8
74.5
54.0
534.4
1
50%
39.9
49.6
82,7
55.4
74.4
53.7
534.1
2
2
25%
40.1
49.7
82.9
56.5
75.0
54.7
535.9
Table C.5: VIB ablation study.
denotes the paper's choice (0.0001).
ECCV Caption R-P
CxC R@1
COCO
VIB 8
mAP@R
R@
IK R@
5K R@!
RSUM
||||||
P
39.3
49.0
83.1
56.1
74.5
54.3
534.5
2x 2x10-1
0.76
X0.1
39.8
49.4
83.2
57.1
75.6
537.4
0.87
x0.2
39.7
49.2
83.6
57.1
75.6
55.3
537.4
2.2
0.91
x0.5
39.7
49.3
82.9
56.7
75.5
55.0
537.2
4.2
0.92
x1
40.1
49.7
83.1
56.8
75.4
55.1
537.0
7.1
0.94
40.0
49.6
82.6
56.7
75.4
55.0
536.7
0.95
x5.0
40.1
49.7
83.2
56.1
74.8
54.3
535.5
23.1
0.91
x10.0
40.1
49.7
83.0
55.4
74.6
53.6
534.4
37.6
0.9
Table C.6: Impact architecture design choice. Details are the same as the previous tables.
ECCV Caption
CxC R@1
COCO
layers for log 2
GPO
mAP@R @ R
R-P
R@1
1K R@I
5K R@
RSUM
37.4
79.2
51.0
70.4
49.2
$21.8
40.2
49.7
83.2
56.6
75.3
54.8
$36.5
40.0
49.6
83.3
57.0
75.5
55.3
537.1
Table C.5 shows three observations. (1) In terms of performance, there exists sweet spot between |
x0.1 and x2, but the performance drop relatively not significant (cf. RSUM VSEco and PCME are 536.5 and 532.0, respectively). (2) The average uncertainty quantity || [ is increased by the 8 value. It supports that VIB regularizes probabilistic embeddings from variance collapse (a - 0). (3) if we did not use VIB (3 0), the correlation between uncertainty and recall is the smallest (-0.76), while a proper choice of B improves p. e.g., -0.92 (x0.2, x0.5). 00.55 (x1, x2). Our choice x (i.e., 0.0001) shows the reasonable RSUM score (537.0, while the best is 538.2) and the second best (-0.94, while the best is -0.95).
Architecture. Table C.6 shows the architecture ablation study: (1) GPO improves overall perfor- mances; (2) if we use more complex log o² head, ECCV Caption metrics are slightly improved by capturing ambiguity caused by FNs well. However, the performance improvements are marginal, and shows inferior R@ scores than shallower log a² head. Therefore, PCME+ uses the number of layers for the log o² head
C.4 T-SNE VISUALIZATION DETAILS
For Figure 4, I-SNE (Van der Maaten Hinton, 2008) respectively applied for PCME+ and VSEcc embeddings extracted from the COCO Caption test split. Then, three images and their corresponding captions (the same colored image and caption are a "positive" pair; otherwise, a pair is negative) in each embedding space are illustrated in the figure.
The purpose Figure is to show how the learned embedding space by PCME can successfully capture the inherent ambiguity in the dataset. Qualitatively, PCME+ embedding space captures the
22

---

Published as a conference paper at ICLR 2024
Table C.7: Effect of inference methods. compare mean-only inference and probability distance-based inferences using our ViT-B/32 SWA model. Each number is the average of different three runs.
ECCV Caption
CxC R@
COCO 5K R@!
Method
Prob?
mAP@R
R-P
R@!
R@1
RSUM
Mcan only
x
40.2
49.8
83.5
56.9
75.2
55.2
536.3
2-Wasserstein
40.2
49.9
83.0
56.6
75.2
54.8
535.9
CSD (ours)
40.2
49.8
83,6
57.2
75.6
55.5
537.3
FAISS (Meany only)
x
40.1
49.7
83.5
56.4
74.7
54.6
531.2
FAISS + re-ranking
40.1
49.7
83,2
56.6
74.8
54,8
531.7
ambiguity of the many-to-many relationships despite the false negatives. However, the figure be misleading whether the purpose of PCME++ is to make "overlap" between two distributions.
The overlap between the two distributions itself is not directly related to the uncertainty. Conceptually, the overlap between two Gaussian distributions can be represented as the Bhattacharyya coefficient (or Bhattacharyya distance). Here, we recall the CSD's main property (b) discussed in Section 2.2: if the match mye is certain, then Z, and Z, have small variances. As discussed in Section 2.2, a similarity function, that measures whether two distributions are the same or not, cannot satisfy the property because it cannot handle the case when the distributions have the same but larger a. There is no motivation to reduce the size of a using the distance. As shown Table 4, Bhattacharyya distance is not effective probabilistic distance for learning probabilistic embeddings much as CSD. On the other hand, the learned embedding space by CSD-trained PCME+ + is a reasonable probabilistic space. Table C.7 shows that even though function of retrieval, the overall precision- precesion-based retrieval performances are almost preserved; it means that the probabilistic space learned by PCME+ is sound metric space of Wasserstein distance.
Wasserstein distance as the similarity
Finally, instead of focusing on the overlap between two distributions, we focus on how CSD can learn the embedding space shown in Figure 4. We recall the properties of our desired probabilistic embedding space: (1) if the correspondence between a given image and caption is certain (i.e., they certainly positive
negative), then the variance of each instance should be small, (2) if the correspondence is uncertain (i.e., the match is sometimes positive and sometimes negative. It can be happened due to the false negatives in the dataset as shown in Figure I and Section 2.1), then the variance of each instance should be large. As mentioned before, CSD can give proper supervision for the desired embedding space. For example, let's approximate the plane figures in Figure 4 to a visual embedding and the plane captions to textual embedding & (because they have very high semantic similarity). In this case, by choosing different image-caption pairs, the supervision between and t can be either positive negative because training dataset only has one-to-one positive correspondences. In this case, our objective function enforces the CSD between the matches to be larger until the penalty for the positive supervision and the negative supervision are balanced. As shown in Figure 4, the visual embeddings and textual embeddings are aligned into almost the same place by making the y distance closer but penalizing the uncertain supervision by enlarging a.
C.5 COMPARISONS OF DIFFERENT RETRIEVAL STRATEGIES
A modified ANN for PCME+ is two-step process. First, an Euclidean distance-based index system for y built as usual, while o² saved into key-value storage. Then, K items are retrieved by performing ANN on the μ index. Lastly, the retrieved items are re-ranked by computing the summation of the μ distance and o² value of the retrieved items.
Table C.7 shows the comparisons of different retrieval strategies using PCME+ + B/32 model. "Mean only" denotes the retrieval strategy only using μ vectors, without o. "2- "aWssessttenn and "CSD" denote that each probabilistic distance is used for the retrieval. In the table, observe that mean-only retrieval shows sufficiently good performances, but using CSD improves the overall performance.
This paper additionally shows the approximated KNN (ANN) results using FAISS. First, FAISS search index using μ vectors is built. Then, ANN is performed on the FAISS index to get the ranked list. Finally, the ranked list is re-ranked by CSD. Here, CSD can be efficiently computed by storing
23

---

Published as a conference paper at ICLR 2024
gallery a into fast key-value storage, such Redis. As shown in the table, ANN can be efficiently and effectively applied to PCME+ with a reasonable computation-perfonne performance tradc-off.
C.6 DETAILS OF AUTOMATIC PROMPT-FLTERING BY +
For the experiments, a randomly initialized ViT-B/16 is trained by InfoNCE loss and PCME+ loss on Conceptual Caption 3M (Sharma al., 2018), 12M (Changpinyo et al., 2021) and RedCaps (Desai et al.. 2021) using hyperparameters in Table B.I. The implementation is based on openclip (IIharco et al., 2021) software. For a fair comparison, the PCME++ model has the same architecture as the vanilla CLIP model except for the additional uncertainty head (as described in Figure 2). For the stable training, the original CLIP loss and the PCME+ loss are used at the same time; solely using PCME++ loss also converges but shows much worse ZS performance. Note that we cannot apply PPs and MSDA for this experiment due to the additional CLIP loss. We also set longer warmup steps than the original setting (x5 for PCME+ The pre-trained models are evaluated on the ImageNet (Russakovsky et al., 2015) zero-shot (ZS) classification task. Specifically, 80 prompts provided by CLIP (Radford et al., 2021) (shown in the below) are used for the ZS classification. In Table 5, "A photo of a { denotes that only "A photo of a { prompt is used for the zero-shot classification, used for computing text embeddings and the
while "All 80 prompts" denotes that all 80 prompts average text embedding is used for the zero-shot classification.
80 base prompts.
photo of (l., a bad photo of (),, a photo of many 1) a sculpture (h law resclution photo of the (1., rendering of
of (),, photo of the hard a D. graffili of a U.., a bad photo of the (hr a cropped photo of the [ (1-, a Lattoo of a Do the eabroidezed 1),, photo of hard see hhe bright photo a (hi, photo of clean (h., photo of dirty 1)., a dark photo of the (l., drawing of = ()., a photo of my Dh the plastic (l-, a photo of the cocl ()., a close-up photo of (1., a black and white photo of the (I-, a painting of the (h.r painting of a (1-, a pixelated photo of the (). soulpture of the Uh, a bright photo of the { (1rr a cropped photo a (hr plastie U.. a photo of the dirty (),, a ipeg corrupted photo of a ()., a blurry photo of the (l., a photo of the 1h. a good photo of the (he, a rendering of the 1h. a ( in a video game., à photo of one 1h doodle of à (h.r close up photo of the (hi, the origami 1). the () in video game., a sketch of a 1). a doodle of the 1),, origami (l., a low resolution photo of 1)., the toy ()., randition of the ()., photo of the clean [)., photo of a large (1., a rendition of D.. a photo of a nice 1)., a photo of a weird ()., a blurry photo of a 13. cartoon U.. art of Uhr aketch of the (hr, embroidered (1 a pixelated photo of
1 ltap of the (1-, a Opeg corrupted photo of the UI., a good photo of a Uo a plushte (h,,
a photo of the nice (h.r photo of the small ()-, a photo the weird (h., the cartoon 1:.
art of the { the, a drawing of the ()., photo of the large (h., a black and white photo of a
1)., the plushia (1., a dark photo
[)., stop of 1)., graffici of the 1),, a toy D., itap of my ( (1., a photo of a ccol (1., photo of small ()., tasteo of the 1).
This paper explores the potential of PCME++ for automatic prompt-filtering with simple uncertainty- based filtering. First, the prompts for every class are sorted by their uncertainty, i.e., ||a|| Then, uncertain prompts are filtered out, and the remaining prompts are used for 7S classification. Here, two strategies are tested. First, the same top-K uncertain prompts for all classes are filtered. As shown in Figure C.2a, this strategy slightly improves the overall performances, but it only shows a marginal improvement against the "all" baseline (+0.04%). To further improve the uncertainty- the strategy with different top-K for different prompts is also explored. As shown in Table this strategy shows very effective performance improvement against the baseline (+5.42%). Figure C.2b shows the detailed population of the best top-K K filtering per class. Here, the classes whose accuracy is 0% are omitted. Interestingly, we observe that 10% of classes (105) show the best ZS performances when all 80 prompts are used. On the other hand, about half of the classes (499) show the best performance when more than 35 prompts are filtered out.
filtering,
Figure C.4 shows the examples of the filtered prompts and the ImageNet validation images. Interest- ingly, can observe that the keywords "cropped" "close-up" are selected for the Jack-o lantern class due to the low-quality images of the class. On the other hand, a generic class, such as rapeseed, shows various prompts to cover its visual diversity.
This primitive study on uncertainty- based prompt-filtering has a limitation. This study has no validation split, iee,, the best top-K prompt for each class is directly searched from the validation
24

---

Published as a conference paper at ICLR 2024
filtered prompts ZS performances
100
0.34
Top-1 accuracy
0.32
80
Population
0.30
60
0.28
40
0.26
0.24
zhoo-shot accuracy
20
40
60
filtered prompts
filtered prompts
The Top-K filtering results.
(b) The best Top-K for each class.
Figure C.2: Automatic wrompt-filtering results. (a) shows the ImageNet (IN) zero-shot (ZS) results when prompts filtered by the same top-K K for every class. Applying all the same top-K filtering does not improve ZS performances. (b) shows the population best top-K filtering for all classes. Here, 906 classes among 1,000 classes show the best performance when using less than prompts.
Query image
Top-5 Retrieved captions
Query text:
Asseree frait dlepley or frad surker.
| 9.73e
poior walbing
Top-5 Retrieved images
polor valling zone
polor monding on large rech di close up of polar bear on rock firsctioon A polar beat valking olong the rocks.
oyos
GT captions
Query
vilce coedy
1 99.60 GT Imare
[= 1.19c-4
slowe polor boor meik formastion polar boar yossingg while monding meck.
Top-5 Retrieved imarzs
oponing unile standing
open
Query image
Top-5 Retrieved captions
Query vanian Alsoe evericaal
lall 9.43-3
moceboandrr flirs shrough sir musstiin landocape:
Retrieved
Imaage
muwbeard.
moreoondd jaspise
junping
rloge hoard. somdbosod.
jampieg
GT captions
Query text: Anvered froi dlepley frail markst.
9.73e
lla 1.13
A mosooaade is fring through the air deing mont his mow board.
A man with glovet, sogplei and our the air on mosebordd A monboanier orhhome during a trick stop mostinis.
Top-S Retrieved imagess
GT Tmaze
person war morebarding through the oir Mhy grab their boord.
membbande doing a trick ofter jamp.
Retrieved captions
Query text:
Tap-5 Retrieved imagee
Assrive of boteess os aivng veries broaniig
Augg shat gewe.rvlond
GT Imaage
Sevenal eranges are loving ander fow banasau.
A busce Bavaas top of oranges.
Closeap warias crnugs and bonomas in pile.
4 pile of crange riming ander pia of honowes.
Apiie fall sliced oranges noit banch of basanos.
captions
Query text: Blacter oaamu rattalrs, asticrr imseer duplond endter Top-5 Retrieved images
I 9.24e-5
Clonap of varios ononges and bunonas in pile. Senwal erasger are lasing ander fos basases. A bauc banonar
Image
bumanns
rimiig
bovanes
Figure C.3: Example of images and captions with high uncertainty.
split. Searching for the best top-K for each class without directly tuning split using strong probabilistic pre-trained image-text representations will be an interesting future research direction.
C.7 FULL EXPERIMENTAL RESULTS
The image-to-text and text-to-image R@5 and @@ 10 results are shown in Table C.8 and Table C.9. The full experimental results, including separated image-to to-gett-ttxttt and text-to-image retrieval results for the main table and standard errors, are included in Table C. and Table C.I.I The full experimental numbers for all experiments can be found in https: / /naver-aaa ai. github io/ hoppemppp..
25

---

Published as a conference paper at ICLR 2024
Table C.8: Image-to-text retrieval R@5 and R@ results.
CxC
COCO
Backbone
Method
R@5
R@ 10
IK R@5
IK R@ 10
5K R@5
5K R@ 10
VSE
87.1
93.4
96.7
98.8
85,4
92.2
P2RM
85.5
92.2
96.1
98.6
83.5
90,9
DAA
86.75
93.25
96.45
98.85
84.95
92.0
ViT-B/32
InfoNCE
87.3
93.4
96.5
98.9
85.9
92.3
(151M)
PCME
87.5
93.5
96.6
98.7
85.8
92.3
PCME+ only)
88.5
94.2
97.0
99.0
87.1
93.2
PCME+
88.4
94.0
97.0
99.0
87.0
93.0
PCME+ (SWA)
88.5
97.0
87.1
92.9
VSEeo
91.1
95.6
97.8
89,9
94.8
P2RM
86.0
92.7
84.3
91.5
DAA
53.9
67.1
76.8
87.6
49.9
62.7
VIT-B/16 (150M)
InfoNCE
90.9
95.8
97.8
99.3
89.7
94.9
PCME
90.5
95.4
97.7
99.3
89.2
94.5
PCME+ (μ only)
91.5
95.8
90.3
95.1
PCME+
91.3
95.7
97.9
99.3
90.1
95.0
PCME+ (SWA)
91.5
95.9
97.9
99.3
90,4
95.1
VSE
58.8
72.6
82.2
91.4
55.7
69.4
VIT-L/14
InfoNCE
82.8
91.7
95.3
98.6
80.2
90.0
(428M)
PCME
91.8
95.9
98.1
99.4
90.7
95.2
PCME+
93.4
96.8
98.5
92.2
96.2
Table C.9: Text-to-image retrieval R@5 and R 10 results.
CxC
COCO
Backbone
Method
R@5
R@ 10
IK R@5
IK R@ 10
5K R@s
5K R@ 10
VSE
77.7
86.5
92.2
96.7
75.5
84.8
P2RM
77.2
86.2
92.3
96.8
74.9
84.4
DAA
76.9
86.1
91.9
74.7
84.2
VIT-B/32
InfoNCE
77.3
86.5
92.3
96.9
75.1
84.7
(15IM)
PCME
77.3
86.4
92.1
96.9
75.0
84.6
PCME+
78.5
87.1
92.8
97.1
76.5
85.4
PCME+ (SWA)
78.6
87.3
92.8
97.1
76.5
85.5
VSE0o
82.0
89.5
94.2
97.5
80.3
88.2
P2RM
78.7
87.3
76.6
85.7
DAA
50.3
62.5
73.4
85.1
47.1
59.1
VIT-B/16
InfoNCE
81.3
89.1
94.0
97.7
79.5
87.7
(150M)
PСME
80.9
88.9
93.9
97.7
79.1
87.5
PCME+
82.0
89.7
80.3
88.3
PCME+ (SWA)
82.1
89.7
94.4
97.8
80.4
88.4
VSE
46.4
61.1
74.2
42.9
57.1
VIT-L/14 (428M)
InfoNCE
73.6
84.2
91.3
96.4
71.0
82,3
PCME
82.7
90.2
94.5
97.8
81.1
88.8
PCME+
84.0
90.8
95.1
98.1
82.6
89.7
26

---

Published as a conference paper at ICLR 2024
Table C.10: Image-to-text retrieval full results. P2RM VIT-B/32 result has standard emror because failed train multiple P2RM due its instability. Full numbers also can found https: /github.
com/ naver-aaaaaaa a.1 pcmpp..
ECCV Caption R-P
CxC
COCO
Backbone
Method
mAP@R
R@I
R@
5K R@)
VSE
32.3 (d:0.)
43.3 (20.1)
77.2 (#0.4)
63.8 (d:04)
82.0 (20.3)
62.3 (2055)
P2RM
30.2 (
41.7 ())
72.2 (-)
58.1 ())
78.6 (-)
56.6
DAA
31.2 (d:0.1)
42.3 (2:0.))
75.9 (20.3)
61.5 (d:.3
79.8 (2003)
59.8 (2:0.)
VIT-B/32
InfoNCE
31.2 (4-0.3)
42.3 (4-0.13
75.4 (+-L))
61.8 (40L)
80.3 (4-06)
60.1 (#-0.2)
(151M)
PCME
31.2
42.3 (4:0.0
74.9 (#00.))
61.5 (4066)
80.1 (H0.2)
59.9 (#0.6)
PCME+ (μ only)
32.1 (4:022)
43.1 (4:0.2)
77.4 (#:144)
64.1 (40.5)
82.0 (4:02)
62.5 (#0044
PCME+
32.3 (1:0.2)
43.4
(1:03))
76.6 (2:0.6)
63.5 (1:05)
81.6 (1:0.4)
62,1 (2006)
PCME+ (SWA)
32.5 (d:0.2)
43.6 (2:0.))
76.3 (2:0.)
63.4 (b:.3)
81.8 (1:0.6)
62.0 (2:0.5
VSEx
34.4 (d:.1)
44.8
(1:0.2)
81.2 (::0.))
69.4
(2:2)
84.9 (d:0.4)
68.0 (2:0.1)
P2RM
30.6 (d:0.))
42.2
(2000)
72.9 (d:2.2)
58.5 ($0.4)
78.3 (#:04)
56.8 (2:03)
DAA
12.4 (d:0.)
22.4 (2:02))
40.3 (2:02)
26.4 (b:04)
46.1 (#:07)
24.3 (20.3)
VIT-B/16
InfoNCE
33.7 (1:0.))
44.4 (1:0.1)
79.7 (100.)
68.2 (1:06)
84.3 (1:0.7)
66.8 (1:0.5)
(150M)
PCME
33.2 (1:0.3)
44.0
(1:0.4)
79.1 (200.)
66.8 (1:0.)
83.6 (1:0.3
65.3 (2:0.6
PCME+ (μ only)
34.0 (1:0.))
44.5
(1:0.))
80.9 (1:0.6)
69.6 (108)
85.3 (1:0.)
68.4 (1:0.7)
PCME++
34.5 (d:0.1)
45.1
(2:0.))
81.6
69.9 (b:3)
85.3 (2:0.1)
68.7 (2:04)
PCME+ (SWA)
34.6 (d:0.))
45.2 (2:0.))
81.8 (::08)
70.3 (2:0.)
85.6 (2:0.)
69.0 (2:0.)
VSE0o
15.7
27.2
39.7
28.9
51.2
27.4
VIT-L/14 (428M)
InfoNCE L/14
27.8
39.6
53.9
75.9
51.9
PCME
34.1
44.5
81.5
70.7
86.5
69.5
PCME+
35.4
45.3
84.0
73.3
87.9
71.8
Table C.11: Text-to-image retrieval full results. Full numbers also can be found in coon/nver--aa a1 http///veraaa/peeeeeeeeeee ai/pemepp.
ECCV Caption R-P
CxC
COCO
Backbone
Method
mAP@R
R @
R@!
1K R@1
5K R@)
VSE
47.7 (4:2))
55.9 (b0.))
88.6 (2:09)
49.0 (#26))
67.9 (#22)
46.9 (:26
P2RM
47.6 ())
55.5 ()
89.5
48.4
67.5 6)
46.3 ()
DAA
47.3
55.7 (10.0)
88.1 (10.3)
48.2 (100)
67.4 (10.))
46.1 (20.0
VIT-B/32
InfoNCE
46.8 (1:0.))
55.1 (1:0.)
88.0 (1008)
48.0 (1:23)
(1002)
46.0 (10.3)
(151M)
PCME
47.1 (db0.))
55.5 (2:02)
(22:5)
48.0 (ba.)
67.6 (d:0.))
(20.1)
PCME+ (a only)
46.8 (d:n.)
55.0 (20.3)
88.0 (2:8)
50.4 (b:3)
69.3 (2:01)
48.4 (203)
PCME+
47.8
55.9 (E0.1)
89.5 (2:22)
50.1 (ba1)
69.2 (2:01)
48.1 (20.1)
PCME+ (SWA)
47.8
56.0 (8:0.2)
89.5 (4:0.2)
50.2 (40.1
69.3
48.3 (#-0.1)
VSEco
49.1
56.5
(8:0.2)
91.3 (::0.4)
55.3 (4:02)
73.3 (1-0.)
53.4
(#50.3)
P2RM
48.8 (1:03)
56.8 (1:04)
88.5 (2:02)
50.0 (2:01)
69.2 (1001)
48.1
(2:01)
DAA
29.0 (1:.3)
38.9
(1:03)
60.0 (2:1.0)
24.3 (1:04)
41.3
(1:04)
22.4 (2004)
VIT-B/16
InfoNCE
48.5 (1:02)
56,3 (2:01)
89.9 (2:02)
53.6 (1:03)
72.3
(2:0.)
51.7 (2:03)
(150M)
PCME
48.7 (142))
56.5 (b:02)
89.5 (bb.1)
53.1 ($09)
72.0
(E:06)
51.2 (::0.)
PCME+ (p only)
48.5 (4:03)
56.3 (2:01)
90.4 (#0.7)
55.4 (3:33)
73.4
(10.2)
53.6 (É:03)
PCME+
49.7 ((0.))
57.2 (d:0.)
91.4 (60.6)
55.2 (1:02)
73.4
(b0.1)
53.4 (2002)
PCME+ (SWA)
49.8 (1:0.)
57.2 (1:0.2)
91.4 (210.7)
55.5 (1:0.2)
73.5
(1:0.))
53.6 (2:0.2)
VSE
35.8
19.7
37.9
18.0
VIT-L/14 (428M)
InfoNCE L/14
43.4
52.1
82.1
42.1
63.1
39.9
PCME
48.2
56.0
90.5
56.1
74.1
54.3
PCME+
48.6
56.3
92.5
58.9
75.8
57.1
27

---

sculpture Published as a conference paper at ICLR 2024
Label: Bookcase, selected prompis:
close-up
Label: Rapeseed, selected prompis: 1).
photo up phote
game.
sculptuze of a [h. azt of the Uh. the (F in vides game. Lattoo of the 1t. graffili of the 1). the plushia (1. tatton of drawing of (b. drawing the []. sketch the (t. clese-up photo 13. art of D. a photo of a clean (). plushie (). close-up photo of a 1). photo of the clean {). rendering of the photo
large U). rendering 1 sketet of Uh. photo of the 13.
cropped photo of the (h. rendition of
1). graffici of []. rendition of the (1.
photo of the small (). photo of ().
dirty 1). a photo a 1h. a photo of nany 1:
photo
dirty (). a photo
cropped photo of 1). photo of a large ().
black and white photo a 1), painting of the of small (h. photo the painting of (). black and white photo of
photo the nice
().
the (). a low resolution photo of a ().
selected prompts: photo of the clean of the cropped photo (1. clase-up photo the (). a photo a clean 1.. a cropped photo of 1
Label:
dark photo of the (1. dark photo of (1. photo
doodle the
cool ().
doodle of 1). low resolution photo of
the (). a photo of the hard to see ().
blurry photo of the (h. photo of weird blurty phote of {
Figure C.4: Example ImageNet images and the prompts by the uncertainty-basee automatic prompt-filtering
28

---

Published as a conference paper at ICLR 2024
D LIMITATIONS AND DISCUSSIONS
Normal distribution with diagonal covariance would be insufficient? One can argue that the uncertainty modeling power of +CME+ can be improved by relaxing the diagonal covariance condition. However, Oh et al. (2019) showed that if the dimensionality of the embedding space and the number of "hidden features" are the same (e.g.. if an image the combination of two digits, then the number of potential latent features for each input is two), then the diagonal covariance condition can sufficiently capture the inherent uncertainty of the dataset. practice, we use very high dimensional embedding space (e.g.. 1024) that can sufficiently capture complex relationships between features. Also, in practice, if we relax the condition, the dimensionality of the log o² head output should be about 1M (= 1024 X 1024), which will require expensive computational budgets and large memory.
Additional sampling is still required if other density functions. The proposed probabilis- tic distance is defined in distribution-fres Ez..z. IZ, Z, 1. However, the closed-form solution (CSD) is specifically designed for normally distributed embeddings. If one needs probabilistic embeddings with different distributions, such as von Mises-Fisher distribution (Kirchhof et al., 2023) or Laplacian distribution (Warburg et al., 2023). CSD is no longer applicable. Instead, we can adapt any distribution to PCME++ by using a Monte Carlo approximation, i.e., by computing XX
|| 11. where of Zy and 2 Ze This change will share the expen-
sive computation issue of previous approaches (Oh et al., 2019; Chun et al., 2021), but the additionally introduced techniques in +CME++ for mitigating the loss saturation issue (i.e., pseudo-positive and MSDA) will still be effective. Applying other probabilistic densities PCME+ and discovering the effect of different distribution choices will be interesting future work.
How does uncertainty help learning image-text representations? As shown in the main experi- ments, the probabilistic approach is helpful for improving the retrieval performances, but the gaps are not significant (e.g., Table I shows that in ViT-B/32, the gap between VSE>c and PCME+ ++ + with SWA is not significant). However, shown in larger backbone experiments (ViT-B/16 and ViT-L/14) and noisy correspondence experiments (Table 2), PCME++ shows more generalizable performances compared the existing state-of- the-art ITM methods with the same backbone. Furthermore, as shown in Section 3.4 and Section 3.5, the learned uncertainty by PCME+ shows high interpretability of the datasets as well as the controllability by the users when the rejection the retrieved items is required. Thus, I believe that the uncertainty-awe aware learning paradigm and the learned uncertainty will be helpful for image-text matching problems and downstream tasks, such as zero-shot classification.
29


Reviews:
[{"review_id": "SoKXLUGm4r", "reviewer": "Reviewer_XAP9", "summary": "This paper presents an improved Probabilistic Cross-Modal Embeddings (named PCME++) by introducing a new probabilistic distance, incorporates of pseudo-positives to prevent the loss saturation problem, and introduce the data augmentation technique. Experimental results demonstrate the effectiveness of the proposed model.", "questions": "1. Figure 2 is not easy-understood. Please make interpretations about it and formulate the comparison losses.\n\n2. The motivation of this paper is \"PCME suffers from expensive computations due to Monte Carlo approximation and fast loss saturation\". However, the comparison or analysis about computation costs is not presented in the experiments.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The representation is good and easy to follow.\n\n2. The robustness of PCME++ is evaluated under noisy image-text correspondences.\n\n3. The most of the figures in this paper is informative, especially Figure1 and Figure5.", "weaknesses": "1. This work is deeply coupled with PCME( Chun et al. (2021)), which might limiting the inspiration and extensibility for other work.\n\n2. As shown in Figure 5, the visualization could show the uncertainty of learned embeddings of visual features and textual features. However, the proposed closed-form sampled distance (CSD) could only simply measure the uncertainty but not the area of the overlap between two modality. I hope the authors could make more analysis.\n\n3. Some techniques such as Mixup (Zhang et al., 2018) or CutMix (Yun et al., 2019) are proposed in the previous works, and the loss functions are shared with PCME( Chun et al. (2021)). Please clarify your contribution and improvements.\n\nIf my concerns are solved, I would like to raise my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfLza2zNec", "reviewer": "Reviewer_RNqC", "summary": "- This paper discusses some inherent issues with existing Image-Text Matching (ITM) methods and datasets, and provides analysis from the probabilistic perspective,\n\n - To solve the aforementioned issues, the authors propose Improved Probabilistic Image-Text Representations (PCME++) by introducing a closed-form probability distance to reduce the sampling costs of PCME and adopting two training techniques including pseudo-positive matches and mixed sample data augmentation. \n\n - Extensive experiments across abundant VL datasets show the effectiveness of PCME++. Moreover, primitive results on extending PCME++ to other scenarios are shown.", "questions": "- In Table 1, experimental results on different backbones are provided. While methods like VSE and InfoNCE show performance degradation, PCME and PCME++ show consistent improvements. Does this phenomenon relate to specific advantages of probabilistic modeling? (and How)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Serving as its major motivation, this paper analyzes the many-to-many nature and other inherent issues in the Image-Text Matching task. Focusing on this fundamental Vision Language (VL) downstream task, this work addresses shortcomings of previous uncertainty-based methods such as PCME. \n\n - This presentation of this work is clear and easy to follow. In particular, the figures, such as Figure 2, are informative in illustrating important contexts.\n\n - The authors provide solid experimental results to support the effectiveness of PCME++. In addition to the evaluations on MS-COCO, CxC, and ECCV Caption with sota ITM methods, extensions to noisy correspondence and uncertainty-based prompt-tuning are conducted, which further demonstrate the advantage of PCME++.", "weaknesses": "- About the advantage of PCME++, the authors mentioned the advantage of PCME++ when scaling-up backbones, which might be lacking further discussions. (See Questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t2LbFiexRg", "reviewer": "Reviewer_36mP", "summary": "This paper presents an improved probabilistic representation to address issues including the presence of false negatives and sparse annotations in image-text matching. \nSpecifically, authors introduce two optimization techniques to enhance many-to-many matching, including pseudo-positives to prevent the loss saturation and mixed sample data augmentation for probabilistic matching", "questions": "presented in weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- well organized and easy to follow. \n- the paper focuses on two important issues in image text matching: many-to-many matching and sparse annotations.", "weaknesses": "- Presented as Table 1 and 2, when ViT-B/32 was chosen as the backbone, the performance of PCME++ is inferior to VSE\\infnite. \nAdditionally, the performances of P2RM and DAA published in the original paper actually won PCME with a large margin, while their performances presented here are lower than PCME. \nThe authors should make necessary explanations on these issues. \nMost importantly, could the proposed PP and MSDA be applied to P2RM and DAA to enhance the many-to-many performance? If not, please make the necessary explanations; if yes, pls make comparisons. \n\n- The datasets used in this submission including CxC and ECCV both come from MSCOCO, and thus the proposed method needs further verification on larger datasets like vision-language pretraining methods (CC3M dataset). \nThe authors are also recommended to make comparisons against several recent works focusing on uncertainty modeling [A] and noisy correspondence[B]. \n\n\t\n[A] MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model, CVPR 2023.\n[B] Deep Evidential Learning with Noisy Correspondence for Cross-modal Retrieval, ACM MM 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)