Discovering Clone Negatives via Adaptive Contrastive Learning for Image-Text Matching
Authors: Renjie Pan
Jihao Dong
Hua Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Image-text matching
Contrastive learning
Vision and language
Multimodal learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We define "clone negatives" and propose Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters with a modulating anchor to dynamically strengthen the compactness and address clone negatives, even in weakly-supervised settings.

Abstract: In this paper, we identify a common yet challenging issue in image-text matching, i.e., clone negatives: negative image-text pairs that semantically resemble positive pairs, leading to ambiguous and sub-optimal matching outcomes. To tackle this issue, we propose Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters along with a modulating anchor to dynamically strengthen the compactness between positives and mitigate the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without the need for explicit training, allowing for progressive tuning and advanced in-batch supervision. Extensive experiments across several tasks demonstrate the effectiveness of AdaCL in image-text matching. Furthermore, we extend AdaCL to weakly-supervised image-text matching by replacing human-annotated descriptions with automatically generated captions, thereby increasing the number of potential clone negatives. AdaCL maintains robustness in this setting, alleviating the reliance on crowd-sourced annotations and laying a foundation for scalable vision-language contrastive learning.

Published as a conference paper at ICLR 2025
DISCOVERING CLONE NEGATIVES VIA ADAPTIVE CON- TRASTIVE LEARNING FOR IMAGE- TEXT MATCHING
Renjie Pan, Jihao Dong, Hua Yang"
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University Shanghai Key Lab of Digital Media Processing and Transmission, Shanghai Jiao Tong University {ripan21, dong haao hyang @ juu. edu.ccr
ABSTRACT
In this paper, we identify common yet challenging issue in image-text matching, i.e., clone negatives: negative image-text pairs that semantically resemble positive pairs, leading to ambiguous and sub-optimal matching outcomes. To tackle this issue, we propose Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters along with a modulating anchor to dynamically strengthen the compactness between positives and mitigate the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without the need for explicit training, allowing for progressive tuning and advanced in-batch supervision. Extensive experiments across several tasks demonstrate the effectiveness of AdaCL in image-text matching. Furthermore, we extend AdaCI. to -supervsd image-text matching by replacing suman-annotated descriptions with automatically generated captions, thereby increasing the number of potential clone negatives. AdaCL maintains robustness in this setting, alleviating the reliance on crowd-sourced annotations and laying foundation for scalable dision-languse contrastive learning.
INTRODUCTION
Image-text matching Fang al. 2023 Qu et al. 2021: Huang al. 2019 Shi et al. 2019 Chen et al., 202I) has become de facto the most fundamental multimodal task in recent years, which involves searching for images with text queries (text-image) and the retrieval of sentences using images (image-text). Existing image-text matching methods usually learn a robust cross-modal representation through various fusion paradigms, such object-level or global-level aligning(Zhane
et al.
2022; Wang et al., 2022a; Cheng et al. 20222] Li et al. 2022b Qu et al. 2021 Chen et al.
1
Wang et 2020a Wu et al. 2019 Li et al. 2019; Lee al. 2018 Dong et al. 2024 Song 2024). On the other hand, contrastive learning(L Li et al. 2022a Radford et al. 2021| He
et al.
al.
2020] Oord et al., 2018: Zhang Lu 2018) has been widely adopted as the objective, which
brings similar (or positive) samples closer in the latent space while pushing dissimilar (or negative) samples apart. Among them, Triplet ranking loss (TRL) et al. 2019 2018 Faghri al. 2017)
and Contrastive loss (CI)(Li et al., 2021 Radford et al. 2021; He et al. 2020) have witnessed the
success of Issio-llnguage contrastive learning. Specifically, TRL preserves the relative distance between in- in-batch samples through manually-set fixed margin. CL is defined as the cross-entropy similarity between each image-text pair. In self-supervised learning and multimodal learning, InfoNCE and its variants(Oord et al.) 2018: Ma & Collins 2018 Liang et al.) 2024 Cheng et al. 2020; Zhang & Lu 2018) are the prevalent forms of CL.
of the
Despite these advancements, current contrastive learning for image-text matching faces inherent challenge: During an in-batch learning, it is common for image-text pairs to possess semantically related text annotations and similar visual cues, thereby rendering them practically indistinguishable (reflected by their closing similarity scores). As shown in Figure I [a). To is a typical hard negative for 41, which has been widely discussed (Pan et al.) 2023a Wang et al. 2020b Zhang et al. (022). It is noteworthy that apart from the key semantic "two skyscrapers" T3 is cemanticall--conissteee with I, Although matching J and T3 indeed correct, is no doubt a sub-optimal result because
Corresponding author.

---

Published as a conference paper at ICLR 2025
T A crowd people in running outfit runs a murathon with two akyscrapers background.
7
People running race while being filmed by person on
A group people rurning marathon or race across the city.
Points and
Adaptie
OA
Algrmmnt for and under TRl and CI
O4 :Alggment selected anchor
Most
Alggmment
Algring process under under
13
Adapive Contrastive Learning process
Hard Negative
Clone Negative
(a) Observational issuc: Clone negatives
(b) Geometrical illustration
Figure 1: (a) A case of "hard negatives" and "clone negatives". Compared with gold label (Ti). clone negative (Ta) is defined as a "sub-optimal" text sample. (b) Geometrical illustration of Adaptive Contrastive Learning, which adaptively enlarges the distance between positives and clone negatives.
the ground- truth annotation Ti exhibits more precise and fine- seee-graied semantics, demonstrating a closer correspondence with I. Different from false negatives which should be considered as "true", we refer to samples like T3 as clone negatives, which is quite challenging for matching in such scenarios. Distinguishing clone negatives through cross-modal fusion is highly intractable since clone negatives are somehow "positive". Also, the limited prior works primarily focus on negative mining strategies for single modality (Wang al. 22024] Robinson et al.) 2020 Yang et al. (2021). or learning noisy correspondence (Han et al. [224]] Huang et 2021). Image-texi matching continues to predominantly rely on vanilla TRL or CL by setting a fixed margin during training. making
tough to actively address the issue of clone negatives since it has already pre-defined the label of all
the image-text pairs (positive or negative).
In this work, we propose Adaptive Contrastive Learning (AdaCL) to exploit reliable clone negatives and distinguish them effectively in image-text matching. Instead of the fixed learning pattern of TRL and CL, specifically add two margin parameters, serving scaling and shifting factors, to synchronously enhance the compactness of positives while introducing the supervision the clone negatives. To progressively tune the margin parameters, we propose to select a modulating anchor from each in-batch similarity score based on Gaussian discriminant analysis without explicit training. In this way, anchor effectively reflects the intensity of in-batch clone negatives, and imposes substantial penalty through the margin parameters, thereby adaptively enlarging the distance between positives and clone negatives. Comprehensive experiments conducted on image-text matching, noisy correspondence learning, CLIP pre-training, and text-based person search demonstrate the effectiveness and robustness of AdaCL.
Moreover, AdaCL can be applied to weakly-supervised image-text matching with structured and standardized text annotations. We propose to increase the number of potential clone negatives by substituting the original textual descriptions with automatically generated captions using image captioning methods. We verify AdaCL by four distinct captioning tools, and observed that AdaCL can also learn robust cross-modal semantics. This not only demonstrates the effectiveness of AdaCL in distinguishing clone negatives, but also mitigating the need for crowd-sourced annotations, laying foundation for future dision-languan contrastive learning. To sum up, the main contributions of this paper include:
We clucidate the issue of clone negatives: candidates, and highlight the challenge they pose to existing contrastive learning for image- text matching.
but suboptimal matching
We propose Adaptive Contrastive Learning (AdaCL), introducing two margin parameters that dynamically propagates the semantics of clone negatives during training, enhancing the compactness of positives and mitigating the impact of clone negatives.
AdaCL demonstrates superiority in both various downstream tasks and our proposed weakly- supervised image-tex matching, which uses automatically generated captions annotations. Extensive experiments prove the robustness of AdaCL, alleviating the reliance on manual annotations.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Image-Text Matching. The general process of Image-text matching can be attributed to two key components: (1) cross-modal feature fusion, and (2) learning objectives. The former can be categorized to global representation fusion and region-level fragment fusion. Global representation fusion extracts holistic feature from two modalities, and then mapping them to a joint visual-textual alignment where the similarity is calculated(Zhang & Lu 2018: Ma et al.] 2013). Region-level fragment fusion captures local content of two modalities to enhance the cross-modal interaction(L] et al., 2017a Lee et al. 2018; Cheng et al. 2022 Diao et al. [02T] Pan et al. (02T). Apart from this,
graph-structured methods (Liu et al. 2020 Wang et al. |020a;] Li ct al. 2022b) also leverage nodes to represent region-level matching. further exploiting the instance relationships. The learning objectives
of image-text matching can be categorized to contrastive loss (CL) (Li et al. 2022a Radford et al.)
2021 He al. [020)] Oord al.||201 Zhang & Lu 2018) and triplet ranking loss (TRL) (Wang et al. 2019: 2018). For CL, ZZhang & Lu (2018); Wang et al. (2020b) proposed to minimize the
distance between the projection distributions and the normalized matching distributions together with a classification objective. TRL is widely adpoted by existing methods (Wang et al. 2018 Pan
ctt al., 2223a| Qu al., 2021). Several studies have also sought to address false negatives by reducing sampling We et al. [2023a] Zhang ct al. 2022] Li ct al. 2023c), which is essentially "passive" approach. However, such methods cannot handle clone negatives due to the fundamental differences between false negatives and clone negatives, as the latter are of significant learning value. Also, both TRL and CL are inherently limited by a manully-set margin, which fails to consider the sample discrepancy of clone negatives. To address this challenge, pivotal solution lies in the ability to adaptively emphasize the learning of these samples.
Contrastive Learning. Contrastive learning has emerged powerful paradigm in various fields, such as self-supervised learning He et al.) 20200] Grill et al. [2020). representation learning Radford
ct al. 2021. and recommendation systems et al. 2024 Lin et al., 2024 2025 Shan et al. 2025). It aims to learn discriminative features by contrasting positive pairs against negative pairs (Schroff et al. 2015 Oord et al., 2018] Zhang & Lu 2018). The foundation of contrastive learning can be traced back to metric learning, where the goal was to learn a distance function that better represents the underlying structure of the data (Bromley al., 1993; Taigman al. 2014] Wang et al. 2017: Grill et al. 2020: Caron et al. 2020). Seminal works such as SimCLR (Chen et al. 2020b) and MoCo (He et al. 2020) established the foundation for contrastive learning in computer vision. The former introduced a simple yet effective approach using data augmentation and large batch size, while the latter proposed a momentum encoder to maintain a consistent dictionary of negative samples. In parallel to its success in computer vision, contrastive learning has also shown promise in multimodal learning (Radford et al. 2021 Li et al. 2022a 2021), which employed a contrastive objective to align text and image representations in shared latent space. By pretraining on large-scale text-image pairs, remarkable zero-shot transfer capabilities can be achieved. Despite its success, the requirement for high-quality datasets is still a significant hurdle for many real-world applications(Zhang et al. 2024b Young et al. 2014) filling with clone negatives, which hinder a broader adoption of contrastive learning.
ADAPTIVE CONTRASTIVE LEARNING
Our aim is to synchronously enhance the compactness between positive samples and calibrating the supervision of clone negatives. We begin by revisiting vanilla contrastive learning, followed by our proposed Adaptive Contrastive Learning (AdaCL) and the anchor selection methdology. By replacing TRL and CL with AdaCL, both the matching performance and robustness have been greatly improved, demonstrated by supervised and weakly-supervised image-text matching on various baselines.
3.1 PRELIMINARIES
Architecture of contrastive learning. The prerequisites for vision-language contrastive learning include textual and visual modalities. Given text description with a words and the pairwise image, the textual and visual backbone are denoted as (-) and F.).). Each word vector is mapped to the common space through fully connected layer. F. generates sequence of patches to extract visual feature. The ann-ggrined representation of both modalities can be denoted as: w = FT) e Rax
and FT) E RDDD. The similarity score of certain image-text pair can be denoted as
3

---

Published as a conference paper at ICLR 2025
Adoptite Contrastie Learsing
g+
Max:
PUIC)
Seet
funine
sim
GB4 P(C)))
people
F
marathon
runsing
Min: Sat
P(S(C)
Senu uegatives
Sixi negatives
uegatives
Figure 2: Illustration of Adaptive Contrastive Learning (AdaCL), Salient negatives and Reference clone negatives selected by Salient Score (Sec. (.3). AdaCL dynamically adjusts two introduced margin parameters based on GDA, progressively regulating the distance between positives and clone negatives (Sec. 3.2). Unlike vanilla CI., AdaCL. enhances robustness by adapting parameters the intensity of clone negatives. Additionally, AdaCI. features plug-and-play design, demonstrating excellent adaptability and compatibility across various tasks (Sec.
= = (v) gt. For clarity and readability, we use go(+) and g. represent optional cross-modal operations (such as fusion or connector) between visual and textual representation, where D is the dimension of the joint embedding space.
Training objective. Vanilla contrastive learning relies on fixed learning pattern to narrow the distance between positive samples. Specifically, the softmax normalized similarity for image / and its corresponding text Ti can be denoted as:
Pı(1) = (xp(s(1,T.)/' Σ exp(s(I,T,)/r))
where the denominator consists of M samples, i.e., one positive and M negatives. Noted that momentum memory bank is not used, then M equals to the mini-bate size N. 7 is fixed temperature parameter controlling the overall supervision, Pc(T) represents the probability assigning the corresponding ground truth T,. Then, cross-entropy loss is adopted:
C = EI-D (()(
N E ve(T) log(p:(()))))
where y(!) is a one-hot distribution where the ground-truth is set by 1. Equation 2 pulls positive image-text pairs closer while pushing away the pre-defined negative pairs.
3.2 ADAPTIVE CONTRASTIVE LEARNING
Current image-tex, text matching typically utilize TRL(Wang et al.) 2019 2018) and CLiZhang & Lu 2018, Wang al., 2020b Radford et al. 202T) during training. As mentioned before, is highly intractable distinguish clone negatives, leading to insufficient representation learning. Also, the semantic discrepancy between positives and clone negatives cannot be fully exploited. Based on this fact, AdaCL proposed by introducing two margin parameters that play the role of scaling and shifting factors. The margin parameters are adaptively tuned under the supervision potential clone negatives selected via the distribution of in-batch similarity logits.
For simplicity, we analyze the i-th image-text pair from a mini-batch. order to uncover and enhance the supervision of potential clone negatives, introduce two margin parameters to p: to serve as the scaling and shifting factors:
exp [m (s(I,T)) m2)]
pi(1) exp (m - -m)) + Σド=
M+1
exp [s (II, I]
and m2 are the tuning targets during training. For the tuning process, we strategically select specific anchor that embodies the characteristics clone negatives. anchor serves as a regulation in the overall supervision of in- bat-atchh clone negatives (details in Section 3.3 In order to compute mi and my during each batch learning, two conditions are specifically analyzed. First, the probability of anchor be obtained using Equation 3
4

---

Published as a conference paper at ICLR 2025
Dr. exp [m (anchor - m2)
(4)
exp (mn (anchor m2)] Eanchor
where Eanchor is the simplification of MM+1 semble average probability of potential in-batch clone negatives, its corresponding Pu. reflects the approximate convergence degree of the model to a large extent. Second, if we can control f... then the overall supervision within the batch are well regulated. Therefore, we propose to progressively tune m. and m2 based on Pu- To achieve this, a specific boundary condition of Equation D analyzed: Since cross-entropy loss is represented (same as s in Equation IT] approaches 1 (indicating a high similarity), a relative small mi would unnecessarily penalize the loss function for correct image-text pairs. Based on this fact, we reason out a condition that Pu should be as close to as possible when s(Tu, Tw) approaches 1, to boost distinguishing clone negatives. This condition can be expressed as:
Lk=1,e exp (s( Tx)l- Since anchor represents the en-
L = (fog(pu, when the score of anchor, I.e., s( Tw)
lim Pu R 1. (1)+
(5)
Combining Equation 4 we can approximate the following boundary:
exp [m (1 - m2)]
(6)
exp [m, (1 - m2)1 + Eanchor
where is a small value added to avoid m1 reaches O. With Equation andd Equation 6 we can derive the values of mi and m2 in Equation jiterativvy and update Lade based on the objective format in Equation [ during training. In this way, Lado effectively propagates the semantics of clone negatives via i and m2. The detailed derivation is in A.3 of the appendix.
3.3 ANCHIOR SELECTION
As mentioned above, our aim is to select a salient anchor from the in-batch similarity scores to control the supervision of potential clone negatives. It is intuitive that the image/text with the highest similarity score with its positive text/image is more likely to be an salient negative, while relatively lower score corresponds to a potential negative which is hard to distinguish. To best quantitatively analyze the in-batch similarity, we introduce Salient Score for anchor selection. Specifically, S = {st [=] M++1 refers to the similarity between each in-batch text/image and the 7th image/text, and sú is the positive. Then, the salient score of each in-batch sample defined as the difference between the positive and the average similarity of the negatives. Mathematically. it can be expressed
M+1
Salient Score, 8/s
1 M
Σ
813
Compared with directly choosing the sample with the highest similarity, salient score better reflects the "salient" extent by considering the negative similarity. The image-text pair with the highest salient score is termed as Sai. including 1 positive and M salient negatives. On the contrary, the image-text pair with the lowest salient score is termed as San. including 1 positive and M potential clone negatives.
To predict the potential clone negatives from each mini-batch, straightforward and simple way is to calculate the salient score of the rest in-batch image-text pairs. However, we found that in contrastive learning, similarity scores typically do not exhibit significant numerical differences, thus applying salient score to all the samples may result in over-fitting. Therefore, we revisit and propose to leverage a classical algorithm Gaussian Discriminant Analysis (GDA) for prediction without the need for explicit training. Specifically, the classification probability can be formulated as follows:
C) C)p
exp (ap)
(C 8) (
(8)
exp (a.) + exp (aa)
Here binary classification is adopted, thus C and C represent the pairwise similarity is / is not clone negative. s is the similarity score. The logit function is a, = log(p(s C)p(C)). Therefore. the classifier can be obtained by analyzing the data distribution of clone negatives and its prior
5

---

Published as a conference paper at ICLR 2025
distribution. In GDA, the features typically assumed follow the gaussian distribution with identical covariance, Since the dimension of similarity score is 1, i.., 8 E R. the aforementioned distribution expression be simplified to univariate distribution, where (s C) ~ N (Me. oî).
and (s C) NN (Pes o Combining this assumption Equation 8
can obtain:
p (C
Me de exp
(e-Me) 20
(e-He)"
202
where and ae are the priors of C and Ć respectively. Given that Sain and Sdn comprise the most representative salient negatives and potential clone negatives within mini-batch, we utilize the negatives from Sann and Sen within each mini-batch to represent the empirical means and variances Jes pla de and 0z. The potential in-batch clone negatives are selected based on the criterion:
S' = {s p (C () > p (C (s)s),
(10)
where S* is the estimated clone negative set from mini-batch. Finally, anchor is obtained by the median of AS,, which defined as:
AS (ō i |s As E S"; aachhr == soos 8 median (AS).
(11)
anchor reflects the average discrepancy between all potential clone negatives and their corresponding positive. With anchor and adaptive contrastive learning in Section 3.2 the overall process of AdaCL
demonstrated in Algorithm
Algorithm 1 Adaptive Contrastive Learning
Input:
mini-batch of N image-text pairs, with N positives and N MM negatives.
Output: Cada
1: for each mini-batch do
2: Select in-batch grounding salient negatives and clone negatives;
Sain = {5 E M, it argmax, Salient Score;,
Soin = { E M, = argmin, Salient Score,,
3: Sort in-batch potential clone negatives S and select anchor based on AS;
p (C s) = 1/[1+
exp
(a~ (s-
20
2n
}.
S* {s (c (CI s)), anchor goss 5 median ASS).
4: Obtain the probability of anchor for tuning;
exp[mn (anchor- my))
Pu exp/mi (anchor m2])+ L.mttt
5: Compute and according to Eq. and Eq. 6
mi log By (=-(p_-\(1-p_)) (anchor 1), m2 anchor log( 1-Pu
Pn. Lmaii )
6: Update (I) and Lodoi
(1(1) (xp(m1(s(1.77)) -17] Ee exp(s(1,T,)] Cada = EIND [H((((),
7: end for
3.4 WEPRVVSSUPERVISED MATCHING WITH PSEUDO CAPTIONS
Existing image-text datasets typically consist of detailed text descriptions cover fine-grained representations that appear in the corresponding images. However, in practical applications, image- text pairs are mainly scraped from the Internet. These raw datasets not only contain inherent noises but also exhibit high degree of correlation: for two semantic-distinct images, their corresponding text descriptions may be semantic-consistent. This intrinsic characteristic has been largely overlooked in existing benchmarks. Moreover, research trend has recently been shifting towards prompt- based semi-automatic annotations, which totally differs from current benchmarks.
To validate the applicability of AdaCL in a more general annotation setting, conduct weakly- supervised image-text matching on AdaCL by annotating images with external tools. Specifically, to ensure the generalizability and tool-agnostic nature of AdaCL, employ four off-the-sheif image captioning methods (BLIP(Li et al.) (022a), GIT(Wang et al.) 20226). BLIP-2(Li et al., (023b), and generate textual description for images in Flickr30K training set, termed
CoCa Yu al. 2022))

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons of image-text matching. Baselines with three types of backbone
pared, i.c., ResNer+BiGRRU CMPM (Zhang & Lui, 2018), VSE++(Faghri et al., 2017): Faster
CCNN+BBGGRE Lee al. 2018), IMRAM(Chen
al. 2020a). CVSE Wang
2020al
EGRAFDiao et al. (2021) ). CHANPan an et al. (2023b);
Faster R-CNN+BERT DI et al. 2021
CHAN Pan al. 2023b). CORA Pham et al. 2024). USER(Zhang al. 2024a
Pan
MS-COO0 (5dald (K)
MS-COCO (SK)
Flirkra0K
Metheds
Image Text
Text-IImage
Image Text
Text- Text-Ilmage
Test-IImage
R
R
R
R
R:0-0
R
R
+BNGRU
89.0
54.7
79.2
VSE44
90.0
95.7
92.0
81
50.3
59.4
72.4
52.9
80.5
70.1
70.5
91.2
93.3
61.1
55.0
$4.0
81.2
Paster
+DIGRU
72.7
IMRAM
76.7
10
59.7
74.1
95.6
79.4
87.2
AduCI-IMRAM
78.3
96.3
96.0
72.3
76.1
57.3
88.1
95.0
96.3
90.4
AdaCL-CVSE
79.1
AdHCI-CHAN
-DERT
76.8
AAHCL-CHAN
82.1
82.0
03.3
82.8
91.1
ABRCL-CORA
83.3
K3.9
93.6
USER
85.7
95.8
93.5
97.6
94.4
AGICL-USER
84.9
95.7
98.2
94.6
as pseudo captions. The expression of pseudo captions is structured and standardized (typically in a subject-verb-objet order), where more potential clone negatives are included. Then we replace the original annotations with pseudo captions and train with AdaCL separately. Despite the fact that some pseudo captions may not perfectly match their corresponding images, we refrain from re-correcting or modifying them, to ensure a weakly-supervised training manner. In this way, AdaCI. can be validated on coarsely annotated dataset, maximizing the resemblance to image-text pairs collected from the Internet. We demonstrate the results of BLIP as image captioning tool in Section 4.3
Due to the page limit, the complete results of GIT, CoCa, BLIP-2, and the format of pseudo captions are detailed in A.I and A.2 of the appendix.
4 EXPERIMENTS
4.1
SETUP
Datasets. We evaluate AdaCL. on two image-text matching datasets, (1) Flickr30K et al. 2014) consists of 31,783 images, with a traininggtes//aaiiation split of 29, 29,783/1,000/1,000 (2) MS-COCO(Lin et al. 2014) consists of 123,287 images, with training/test/valiaation split of 113,287/5,00 0,008/8/550005500000 The test sets are divided into MS-CMCO 5-fold 1K (average results of 5 test sets) and MS-COCO 5K (results of 5000 test images).
Architecture. For image-text matching, we use three types of backbones: (1) ResNet He et al., 2016)+BiGRU(Chung et al.) 2014). (2) Faster R-CNN(Ren al. +BGGUU, and (3) Faster B:CNN+BERT et al. 2018). For pre-training, we use CLIP (VTT-B/32), where the embedding size is 512.
Training Details. All experiments are performed on four NVIDIA Tesla V100s. For image-text matching, we use a mini-batch size of 64 and the Adam optimizer. The learning rate is 0.0002 and starts decaying 15% of every 10 epochs after epoch 20. The maximum length of each sentence is a = 32. For Faster R-CNN, the region number is n = 36. The dimension of joint embedding space D is set to 256. We follow (He et al. 2020) to use the momentum memory bank, where the momentum coefficient 2 is set to 0.99, and the size M is 4096. For AdaCL, Pu is set to 0.03, and e = c mı and m2 are initialized to 20 and 0.1 respectively for adaptive tuning.
Evaluation Protocols. Following previous works(Lee et al. 2018)), we adopt Recall@ K (R@K, K=1, 5, 10) as the evaluation metric: Given query text, images ranked based on their similarity the
7

---

Published as a conference paper at ICLR 2025
query text, and search is considered correct if at least one relevant image appears within the top K positions in the ranking.
4.2 RESULTS ON IMAGE- TEXT MATCHING
Results on image-text matching baselines. We choose various competitive image-text matching baselines, and the plug-and aad---nd-pllyy effectiveness of AdaCL is denoted as 'AdaCL-Bascline" From Table it is observed that AdaCL outperforms the original baselines across each backbone type. For MS-COCO, AdaCL-CMPY, achieves absolute improvement of (4.6%, 1.9%, 2. 1%) on (R@) R@5, R @ 10) for image-text matching. For Flickr30K, AdaCI-SCAN achieves absolute improvement of (4.0%, 2.7%. 1.4%) and (2.3%. 2.2%, 1.6%) on (R@) 1, R@5 5, @@ 10) for image-text and text- image matching, demonstrating that AdaCL is applicable without elaborate cross-modal fusion designing. An and A.6,, we demonstrate AdaCL's domain generalization capacity in text-based person retrieval, and its robustness noisy correspondence learning. In A.9 the convergence efficiency of AdaCL is analyzed in detail to better verify its robustness.
more general network architecture
Results on CLIP pre-training. To assess the scalability, we extend the evaluation to pre-training of widely used contrastive learning framework CLIP, under two specific large-scale datasets Conceptual Caption 3M (CC3M) and 12M (CC12M) (Sharma et al. 2018). We make fair comparison by training with 32 epochs under the same experimental settings, and validate the zero-shot image-text retrieval on Flickr30K and MSCOCO. As illustrated in Table 2 AdaCL demonstrates significant improvements under both pre-training datasets, indicating the potential in large-scale contrastive learning framework. In/A.7 A.8 of the appendix, we also report results of AdaCL in zero-shot image classification, and more fine-tuning performance of several other vision-language pre-training methods.
Table 2: Zero-shot retrieval on Flickr30K and Table 3: Effectiveness of AdaCL under weakly- MSCOCO. "Baseline" and "AdaCL" represent supervised matching (marked by "PC"). vanll+a CL" and "CLIP+AdaCE"
Image-Testt
Text-=Inage
Data
Methods
Image-FF Test
Re100
R851
Text -nage
R85
MMICLCMPR
54.7
R&
79.0
R@5
RG
87.5
R@0
41.6
69.4
R@5
R::10
79.2
(Flickr30k)
CMPM (PC)
42.1
68.8
82.5
31.6
57.9
70.2
Baseline
52.5
63.2
18.1
Ada (PC)
46.5
73.4
85.2
24.3
63.5
74.1
CC3M
AdaCT.
39.5
60.8
73.7
25.5
46.9
54.3
SCAN (PC)
SCAN
71.4
55.4
80.2
93.0
97.2
90.6
30.9
36.9
69.7
79.9
81.6
56.8
CC12M
Haseline
49.3
77.3
85.0
35.5
61.8
71.6
AdaCL-SCAN (PC)
59.7
94.6
41.4
73.6
84.0
AdaCI.
$1.0
77.5
87.9
38.4
64.6
74.7
AAIC--CVEE
74.5
91.0
95.2
56.9
83.7
91.0
(MSCOCO)
CVSE (PC)
58.0
87.8
41.5
OC3M
Baseline
1344
52.0
43,3
10.1
25.6
35.7
AdaCL-CVSE (PC)
64.9
$4.4
92.6
47.0
76.4
86.2
AdaCI.
22.5
47.1
60.7
17.8
31.6
AdaCL-DIME
98.9
93.7
CC12M
Baseline
29.3
54.4
65.3
19.0
41.0
52.5
DIME (PC)
62.1
86.7
93.0
44.7
72.9
84.5
AdaCL
34,0
55.6
65.9
25.1
47.3
57.4
AdaCL-DIME
70.8
88.3
93.6
48.7
90.4
Table 4: Study of components. "PC" and "GT" represent trained with pseudo captions weakklyTTable 5: Comparisons of different anchor se- supervised), and with ground truth annotations.
lection methods.
Modules
PC
Gr
Image - Text
Text
Image
Selection
Image
<Text
Text Image
AdaCI.
Anchor
R
R
R
methods (on AS)
51.4
77.3
40.5
06.3
R@1
R@5
R@1
R@5
43.7
69.2
31.0
64.7
Random
64.2
85.9
44.5
75.1
82.5
73.4
Maximum
62.7
85.1
42.7
72.9
/
59.6
85.2
43.1
75.6
87.0
77.3
Minimum
71.8
90.9
50.7
79.
53.7
77.0
41.3
70.4
Ours (Median)
74.2
91.7
53.8
81.1
69.0
88.4
48.8
78.0
91.7
4.3 RESULTS ON WEAKLY-SUPERVISED IMAGE- TEXT MATCHING WITH PSEUDO CAPTIONS
To further validate the applicability of AdaCL in general annotation setting, conduct weakly- supervised learning on Flickr30K by replacing the text annotations of the training set with pseudo captions generated by BLIP et al. 2022a). Table 3 demonstrate the performance of AdaCL on several baselines. It is observed that R 10 of AdaCL (PC) is able to rival the original baselines, indicating that AdaCI. effectively distinguishes clone negatives and exhibits strong adaptability to
8

---

Published as a conference paper at ICLR 2025
0.35
0.20
Booch (4) Visualization of my and my in AdaCI.
(b) (SNE of opoch 10 (CL)
(c)E Epoch 10 (AdaCL)
Figure 3: Analysis of the tuning process and comparison between AdaCL and CL.
automatically annotated text descriptions. The matching capability of AdaCL under an automated annotation setting is in hot pursuit of the fine- fine-grained human annotations, highlighting the potential of AdaCL in weakly- supervss learning. In A.Al of the appendix, we also report results on three more caption tools.
4.4 ABLATION STUDY
In this section, we conduct ablation experiments on Flickr30K to verify AdaCL. For fair comparison, we choose Faster R-CNN BiGRU for feature extractors, and cross attention for alignment learning in all the ablation studies.
Verification of modules. In Table 4 provide detailed verification by setting different learning objectives during training. "AdaCI." T in the 3rd column refers to employing AdaCI. with a random anchor selection method, and "Anchor" in 4th column refers employing both AdaCL and our proposed anchor selection. Meanwhile, two annotation settings, i.e., ground-truth text annotations (GT) and pseudo captions (PC) are compared. The 1s to 4thh rows showcase the results of pseudo captions wweakly-superviserr surpasses both TRL and CL across all metrics by a significant margin, proving that AdaCL further facilitates the learning of pross-modal semantics by adaptive tuning. The proposed anchor selection provides AdaCL with absolute @ boost of 5.2%((--) and 4. (t-). TRL generally exhibits better performance than vanilla CL., suggesting that TRL. is supposed to fit in well with retrieval tasks better than vanilla CL. Mcanwhile, we conduct a comprehensive ablation study on other hyper-parameten including momentum memory bank and batch size. Due to page limit, they are demonstrated in A.10
and 5th to 8th rows are results based on the ground truth. AdaCL
Analysis of anchor selection methods. To verify the rationality of the proposed anchor selection, compare with several canonical selection methods. The role of anchor is guide the tuning process of my and my by propagating the semantic of clone negatives. However, as shown in Table selecting the maximum AS contradicts motivation since its anchor has the lowest likelihood of being representative of clone negatives, which naturally leads to poor matching results. On the contrary, minimum selection aligns well with intuition, while it overlooks the possibility of erroncous reasoning by the model itself. Therefore, an intermediary anchor, which reflects the average logit of clone negatives, proved to be the most effective. In addition, we provide in-depth analysis of the additional arguments involved in anchor selection methodology in A1 of the appendix.
Analysis of adaptive tuning process. Figure 3 a) illustrates the change of the mi and m2 in AdaCL. During training, m2 gradually increases and stabilizes around 0.36, while m, exhibits some oscillations before reaching a stable value of approximately 38.1 around Epoch 10, which exhibits rapid convergence. Figure 3b) and (c) showcase the t-SNE of the fused embedding in Epoch 10. AdaCL exhibits a better clustering performance compared to CL, demonstrating its effectiveness in boosting the convergence of the model during training.
4.5 QUALITATIVE ANALYSIS AND VISUALIZATION
Analysis of clone negatives. To analyze how AdaCL handles clone negatives, we rank the matching results based on similarity logits and compared three distinct learning objectives. As illustrated in Figure the case study reveals that: (1) AdaCL demonstrates superior R@1 compared to CL and TRL, successfully eliminating other potential negatives; (2) The learned representations of AdaCL exhibit better performance in identifying clone negatives. For instance, the similarity logit for "A

---

Published as a conference paper at ICLR 2025
Matching results
Rank 1st
Rank 3rd
A group of people is running marathon scress the city.
A crond of people in ranning outfits russ marathen with two skyscrapers in background. Sim logit: 0.52
People happily walking along street with palm trees the background. Sim logit: 0.45
Image query
CL.
Sim logit: 0.57
A young woman is running a murnthon in light blue tank spandex shorts.
crend of people in ranning outfits runs marathon with two skyscrapers in background. Sim logit: 0.47
A persn carth wrelhheeiig over another person while third ruas behind them.
TRL
Sim logit: 0.51
Sim logit: 0.44
of people runsing marathon with skyscrupers the background.
large crowd people walking down the street.
group people is running marathon across
AdaCL
city. Sim logit: 0.30
Sim logit: 0.46
Sim logit: 0.33
Figure 4:
Matching results and similarity logits of clone negatives under CL, TRL, and AdaCL.
group of people running race or marathon across the city" is merely 0.3. In contrast, both CL and TRL. learn sub-optimal similarity logits for clone negatives, despite the presence of the ground-truth.
Image query
TRI.
AdaCT.
Ad&CL (Pseudo Caption)
Q1: A crond of people running outfits marsthon with skyserspers backgrouad.
group people running race marathon city.
T-shiet speaks a blow horn towardss a group of people.
Figure 5: Attention maps in early training stage (Epoch 10). Blue represents AdaCL trained with pseudo captions.
Visualizations. We further make qualitative analysis by visualizing several cases in the early training stage to verify the effectiveness of AdaCL in capturing visual cues among clone negatives, From Figure 5 AdaCL are capable of exploring region-level instances together with some background information, such as "two skycrapers" in Q1 and "blow homn" Q3. Unexpectedly, we find that the distinguishable features among the clone negatives are also learned by AdaCL. For instance, the de facto distinction between Case 2 and Case is the presence of audiences in Case 2. Even though there is no related key word in Q2 such "audience" or "spectator", AdaCL still precisely captures the spectators in the lower-left corner of the image. Meanwhile, despite the limited textual contents provided by the pseudo captions, AdaCL roughly capture the presence of the crowd in Case 3, whose effectiveness is slightly better than TRL at the same training stage. Please refer to A.12 of the appendix for more visualization and analysis.
CONCLUSION
Clone negatives are common but challenging in image-text matching. To mitigate this challenge, we propose Adaptive Contrastive Learning (AdaCL) that introduces two adaptive margins and modulating anchor to dynamically adjust the compactness between sample pairs and propagates the semantics of clone negatives. The modulating anchor is selected based on the distribution of negative samples without explicit training, allowing for progressive tuning and enhanced in-batch supervision. Furthermore, We extend AdaCL to weakly-supervised image-text matching by substituting human- annotations with automatically generated captions, increasing the number of potential clone negatives. AdaCL demonstrates robustness in both supervised and supervied image-tex matching. Its superiority demonstrates the potential alleviating the reliance on crowd-sourced annotations and lays foundation for vision-language contrastive learning.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This research was partly supported by grants of National Natural Science Foundation of China (NSFC, Grant No. 62171281), Science and Technology Commission of Shanghai Municipality (STCSM, Grant Nos. 20D21 2021SHZDZX0102).
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. nents with random forests. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp- 446-461, Springer, 2014.
discriminative compo-
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verifi- cation using a" siamese" time delay neural network. Advances in neural information processing systems, 6, 1993.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 13:9912-99924, 9924, 2020.
Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. Imram: Iterative matching with recurrent attention memory for cross modal image-text retrieval. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 12655- 2020a.
Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15789-15799 15798, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp. 104-120. Springer, 2020c.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, Zhe Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound mutual information. In International conference on machine learning, pp. 1789-1788. PMLR, 2020.
Yuhao Cheng, Xiaoguang Zhu, Jiuchao Qian, Fei Wen, and Peilin Liu. Cross-modal graph matching network for image-text retrieval. ACM Transactions on Multimedia Computing, Communicutions, and Applications (TOMM), 18(4):1-23 2022.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 248-2558 255. leee, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Haiwen Diao, Ying Zhang, Lin Ma, and Huchuan Lu. Similarity reasoning and filtration for image- text matching. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp.
1218-1226, 2021.
Zefeng Ding, Changxing Ding, Zhiyin Shao, and Dacheng Tao. Semantically self-aligned network for text-to-image part-aware person re-identification. arXiv preprint 10.. 2021.
Jihao Dong. Hua Yang, and Renjie Pan. Exploring interactive semantic alignment for efficient hoi detection with langunel model. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6. IEEE, 2024.
11

---

Published as a conference paper at ICLR 2025
Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. Vse++: Improving visual-semantic embeddings with hard negatives. arXiv preprint WwXiv:1707.05612. 2017.
Bo Fang, Wenhao Wu, Chang Liu, Yu Zhou, Yuxin Song, Weiping Wang, Xiangbo Shu, Xiangyang Ji, and Jingdong Wang. Uatvr: Uncertainty-adaptiv text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 conference computer vision and pattern recognition workshop, pp. 178-178. IEEE, 2004.
Jean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to sell-supervised learning. Advances in neural information processing systems, 13:21221-212235 2020.
Haochen Han, Qinghua Zheng, Guang Dai, Minnan Luo, and Jingdong Wang. Learning to rematch mismatched pairs for robust cross- mooal retrieval. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Yan Huang, Yang Long, and Liang Wang. Few-shot image and sentence matching via gated visual- semantic embedding. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8489-8496, 8496, 2019.
Zhenyu Huang. Guocheng Niu, Xiao Liu, Wenbiao Ding, Xinyan Xiao, Hua Wu, and Xi Peng. Learning with noisy correspondence for cross-modal matching. Advances in Neural Information Processing Systems, 34:29406-299 294199, 2021.
Ding Jiang and Mang Ye. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. In IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Kuang-Hus Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pp. 201-216, 2018.
Haoxuan Li, Yi Bin, Junrong Liao, Yang Yang, and Heng Tao Shen. Your negative may not be true negative: Boosting image-text matching with false negative elimination. Proceedings of the 31st ACM International Conference on Multimedia, pp. 934,,,4,7 2023a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- -mage pre- training for unified vision- language understanding and generation. In International Conference on Machine Learning, pp. 12888-12900. PMLR, 2022a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-iman pre-traine training with frozen image encoders and large language models. In International conference machine learning, pp. 19742. PMLR, 2023b.
12

---

Published as a conference paper at ICLR 2025
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Visual semantic reasoning for image- text matching. In Proceedings the IEEE/CVF international conference on computer vision, pp. 4654-4662 2019.
Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu. Image-texi embedding learning via visual and textual semantic reasoning. IEEE Transactions on Pattern Analysis and Machine Intelligence, (5(1):641-656, 2022b.
Shuang Li, Tong Xiao, Hongsheng Li, Wei Yang, and Xiaogang Wang. Identity-aware textual- visual matching with latent co-attention. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1890-1899, 2017a.
Shuang Li, Tong Xiao, Hongsheng Li, Bolei Zhou, Dayu Yue, and Xiaogang Wang. Person search with natural language description. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1970-1979, 2017b. 7b.
Xiujun Li, Xi Yin, Chunyuan Li, Pengehuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28,
2020, Proceedings, Part XXX 16, pp. 121-137. Springer, 2020.
Zheng Li, Caili Guo, Zerun Feng, Jenq-Neng Hwang, and Zhongtian Du. Integrating language guidance into image-text matching for correcting false negatives. IEEE Transactions on Multimedia. 26: 2023c.
Paul Pu Liang, Zihao Deng, Martin Q Ma, James Y Zou, Louis-Philippe Morency, and Ruslan Salakhutdinov. Factorized contrastive learning: Going beyond multi-view redundancy. Advances in Neural Information Processing Systems, 36, 2024.
Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang. Rella: Retrieval-enhance large language models for lifelong se- quential behavior comprehension in recommendation. In Proceedings of the ACM Web Conference 2024, pp. 3497-3508, 2024.
Jianghao Lin, Xinyi Dai, Rong Shan, Bo Chen, Ruiming Tang, Yong Yu, and Weinan Zhang. Large language models make sample-efficient recommender systems. Frontiers of Computer Science, 19 (4):194328, 2025.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Chunxiao Liu, Zhendong Mao, Tianzhu Zhang, Hongtao Xie, Bin Wang. and Yongdong Zhang Graph structured network for image-text matching. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 10921-10930, 2020.
Lin Ma, Zhengdong Lu, Lifeng Shang, and Hang Li. Multimodal convolutional neural networks for matching image and sentence. In Proceedings of the IEEE international conference on computer vision, pp. 2623-2631 2015.
Xinran Ma, Mouxing Yang. Yunfan Li, Peng Hu, Jiancheng Lv, and Xi Peng. Cross Cross- retrieval with noisy correspondence via consistency refining and mining. JEEE Transactions on Image Processing, 2024.
Zhuang Ma and Michael Collins. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. arXiv preprint arXiv:1809.01812 2018.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained 2013.
visual classification of aircraft. arXiv preprint
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722-729. IEFF, 2008
13

---

Published as a conference paper at ICLR 2025
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 10Xiv:807.00374,, 2018.
Renjie Pan, Hua Yang. Xinxin Yang, Mao Xiaodong, Long Ye, and Da Pan. Attentive feature focusing for person search by natural language. In International Forum on Digital TV and Wireless Multimedia Communications, pp. 266-280. Springer, 2021.
Renjie Pan, Hua Yang, Cunyan Li, and Jinhai Yang. Joint intra & inter-grained reasoning: A new look into semantic consistency of image-text retrieval. IEEE Transactions on Multimedia, 2023a.
Zhengxin Pan, Fangyu Wu, and Bailing Zhang. Fine-grained image-text matching by cross-modal hard aligning network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19275-192844 2023b.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and patlern recognition, pp. 3498-3505. IEEE, 2012.
Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava. Composing object relations and attributes for image-text matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14354-14363, 2024.
Yang Qin, Yuan Sun, Dezhong Peng. Joey Tianyi Zhou, Xi Peng, and Peng Hu. Cross-modal active complementary learning with self-refining correspondence. Advances in Neural Information Processing Systems, 36:24829-2446
2023.
Leigang Qu, Meng Liu, Jianlong Wu, Zan Gao, and Liqiang Nie. Dynamic modality interaction modeling for image-text retrieval. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1104-1113, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances neural information processing systems, 28, 2015.
Joshua Robinson, Ching-Yar Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592, 2020.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 815-823, 2015.
Rong Shan, Jianghao Lin, Chenxu Zhu, Bo Chen, Menghui Zhu, Kangning Zhang, Jieming Zhu, Ruiming Tang, Yong Yu, and Weinan Zhang. An automatic graph construction framework based on large language models for recommendation. arXiv preprint arXiv:2412. 18241, 2024.
Rong Shan, Jiachen Zhu, Jianghao Lin, Chenxu Zhu, Bo Chen, Ruiming Tang, Yong Yu, and Weinan Zhang. Full-stack optimized large language models for lifelong sequential behavior comprehension in recommendation. arXiv preprint arXiv:2501. 13344, 2025.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-2565, 2018
Botian Shi, Lei Ji, Pan Lu, Zhendong Niu, and Nan Duan. Knowledge aware semantic concept expansion for image-text matching. In IJCAI, volume 1, pp. 2, 2019.
Jiayan Song, Renjie Pan, Jun Zhou, and Hua Yang. M-rat: multi colli-grraied retrieval augmentation transformer for image captioning. In Proceedings of the Asian Conference on Computer Vision, pp. 3865-3882, 2024.
14

---

Published as a conference paper at ICLR 2025
Yaniv Taigman, Ming Yang, Mare' Aurelio Ranzato, and Lior Wolf. Deepface: Closing the gap to human-leve performance in face verification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1701-1708. 2014.
Haoran Wang, Ying Zhang, Zhong Ji, Yanwei Pang, and Lin Ma. Consensus-aware visual-semantic embedding for image-text matching. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIV 16, pp. 18-34. Springer, 2020a.
Haoran Wang, Dongliang He, Wenhao Wu, Boyang Xia, Min Yang, Fu Li, Yunlong Yu, Zhong Ji, Errui Ding, and Jingdong Wang. Coder: Coupled diversity- momentum contrastive learning for image-text retrieval. In European Conference on Computer Vision, pp. 700-716. Springer, 2022a.
Jian Wang, Feng Zhou, Shilei Wen, Xiao Liu, and Yuanqing Lin. Deep metric learning with angular loss. In Proceedings of the IEEE international conference on computer vision, pp. 2593-2601, 2017.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100, 2022b.
Liwei Wang. Yin Li, Jing Huang, and Svetlana Lazebnik. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 (2):3944007, 2018.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208. 10442, 2022c.
Yifci Wang, Qi Zhang, Yaoyu Guo, and Yisen Wang. Non-negative contrastive learning, In The Twelfth International Conference Learning Representations, 2024. URL httpttt://pppeeeetiiii
Zhe Wang, Zhiyuan Fang, Jun Wang, and Yezhou Yang. Vitaa: Visual-textual attributes alignment in person search by natural language. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 402 420. Springer, 2020b.
Zihao Wang, Xihui Liu, Hongsheng Li, Lu Sheng, Junjie Yan, Xiaogang Wang, and Jing Shao. Camp: Cross modal adaptive message passing for text-image retrieval. In Proceedings of the IEEE/CVF international conference computer vision, pp. 5764-577777 5773, 2019.
Yiling Wu, Shuhui Wang, Guoli Song, and Qingming Huang. Learning fragment self-attention embeddings for image-text matching. In Proceedings of the 27th ACM international conference on multimedia, pp. 2088 2096, 2019.
Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 1134-1143, 2021.
Shuo Yang, Zhaopan Xu, Kai Wang, Yang You, Hongxun Yao, Tongliang Liu, and Min Xu. Biero: Noisy correspondence rectification for multi-modality data via bi-directional cross-modal similarity consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19883-19892, 2023.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini. and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint 07X19.2005.01917,
2022.
15

---

Published as a conference paper at ICLR 2025
Kun Zhang, Zhendong Mao, Quan Wang, and Yongdong Zhang. Negative-aware attention framework for image-text matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15661-15670, 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 5579-5588, 2021.
Yan Zhang, Zhong Ji, Di Wang, Yanwei Pang, and Xuelong Li. User: Unified semantic enhancement with momentum contrast for image-text retrieval. IEEE Transactions Image Processing, 2024a.
Ying Zhang and Huchuan Lu. Deep cross-modal projection learning for image-text matching. In Proceedings of the European conference on computer vision (ECCV), pp. 686-701, 2018.
Zixin Zhang, Kanghao Chen, and Lin Wang. Elite-evgs: Learning event- 3d gaussian splatting by distilling event-to-video priors. arXiv preprint arXiv:2409. 13392, 2024b.
Aichun Zhu, Zijie Wang. Yifeng Li, Xili Wan, Jing Jin, Tian Wang, Fangqiang Hu, and Gang Hua. Dssl: Deep surroundings-person separation learning for text-based person retrieval. In Proceedings of the 29th ACM international conference on multimedia, pp. 209-217, 2021.
16

---

Published as a conference paper at ICLR 2025
A APPENDIX
The appendix is structured as follows:
provvidss experimental results on weakly-supervised image-text matching with pseudo captions generated by GIT, CoCa, and BLIP-2.
A.2
demonstrates the expression format of pseudo captions generated by the image captioning tools.
A.3
provides the detailed derivation of anchor selection.
A.4
provides the detailed derivation of Equation [
A.5
provides experimental results on text-based person retrieval.
5 A.6
provides experimental results on noisy correspondence learning.
provides experimental results on zero-shot image classification of CLIP pre-training.
A.8
provides experimental results on VLP fine-tuning.
provides efficiency analysis AdaCL.
A.I
provides comprehensive study of other ppper-parmmetes..
A.I
provides an analysis on additional arguments involved in anchor selection.
A.
provides comprehensive visualization results of clone negatives.
provides a discussion of the limitations for this work.
A.1 WEAKLY-SUPERVISED IMAGE-TEXT MATCHING
In the main manuscript, we utilize BLIP et al. 2022a) to generate pseudo captions based on Flickr30K training set. Then, the original text annotations are replaced by the pseudo captions for training to verify the robustness of AdaCL in handling clone negatives. To mitigate concerns about reliance on a specific captioning tool, we conduct a range of complementary experiments, to comprehensively analyze its robustness. Specifically, four captioning tools are selected, i.c., GIT(Wang et al. (0226), CoCa Yu et al., (022), and BLIP-2(Li et al. 2023b). GIT a multi- modal pre-training method that unifies vision- language tasks such as image/ captioning and question answering. CoCa employs a unified transformer architecture to perform both image-text matching and image captioning tasks. CoCa is trained on large-scale image-text pairs and can generate descriptive captions for images while also understanding the relationship between visual and textual content. B is an advanced vision-language model that builds upon its predecessor, BLIP. It introduces a lightweight Querying Transformer (Q-Former) to bridge pre-trained vision and language models efficiently.
The way of generating pseudo captions is unified, i.e., through the zero-shot image captioning results. The maximum length of the pseudo caption is 30. Table demonstrates the matching results of different baselines with AdaCL based on pseudo captions. The experimental settings are the same with the settings the main manuscript.
From Table 6, we can make the following conclusions: By employing different image captioning method, AdaCI. demonstrates matching performance that are within 3% margin. AdaCI. achieves highly competitive performance on the four annotation settings. Therefore, AdaCL is further proved to be applicable to more general annotation settings, where the issue clone negatives well mitigated. The impact of image captioning methods on AdaCL is trivial. More specifically, AdaCL-X' (BLIP) and AdaCL-X (BLIP-2) performs slightly better than AdaCL-X' (GIT) and AdaCL-X (CoCa). We speculate that this is because the pseudo captions generated by GIT and CoCa typically focus on action and instance information of the image gallery, while the pseudo captions generated by BLIP possess general descriptions, often consisting of subject-verb-objer structure with more holistic-level semantics. Overall, the retrieval performance is quite comparable, further verifying the robustness of AdaCL.
A.2 EXPRESSION FORMAT OF PSEUDO CAPTIONS
Regarding pseudo captions, we have presented a subset of caption cases generated by four distinct captioning tools (BLIP, GIT, BLIP-2, and CoCa), as illustrated in Figure Pseudo captions generally provide global and coarse overview of the images, encompassing more potential clone negatives. Among the four methods, GIT produces the most concise captions, while CoCa tends to generate more detailed descriptions. The weakly-supervised image-text matching based on pseudo captions
17

---

Published as a conference paper at ICLR 2025
Table 6: Comparisons of Image-Text Retrieval performance Flickr30K test with pseudo captions generated by four distinct captioning methods. AdaCL-X' (BLIP), AdaCL-X' (GIT), AdaCL- X (CoCa), and AdaCL-X' (BLIP-2) represent AdaCL with respective image captioning methods. Bold is the best performance, while red indicates the margin between the best and worst.
Image Text
Text lmxaeelmage
Methods
R@1
R@5
R@D
R@5
R@
AdACL-CMPM (BLIP)
46.3
72.9
85.8
34.1
63.6
75.0
AdaCL-CMPM (GIT)
44.5 (-1.8)
71.2
84.9 (-0.9)
32.7 (-1.8)
61.0 (-2.7)
73.7 (-1.5)
AdaCL-CMPP (CoCa)
44.8
71.0 (-2.4)
85.1
34.1
63.7
74.6
AdaCL-CMPM (BLIP-2) 2)
45.9
73.4
85.7
34.5
63.3
75.2
AdaCI-SCCAN (BIIP)
59.2
86.9
94.7
41.7
73.2
84.1
AdaCL-SCAN (GIT)
60.0
87.7
94.9
70.9 (-2.3)
83.4 (-1.6)
AdaCI-SCCAN (CoCa)
58.1 (-1.9)
85.2
94.2 (-0.7)
40.2 (-2.0)
72.6
83.6
AdaCL-SCAN (BLIP-2)
58.1
85.2 (-2.5)
94.6
42.2
74.1
85.0
AdaCL-CVSE (BLIP)
64.7
82.6
92,9
47.0
77.5
88.4
AdaCL-CVSE (GIT)
63.6 (-1.9)
80.9 (-1.8)
90.9 (-2.0)
46.2
77.4
88.7
AdaCL-CVSE (CoCa)
63.8
81.4
91.5
45.3 (-1.7)
77.1 (-0.6)
88.0 (-0.7)
AdaCL-CVSE (BLIP-2)
65.5
82.7
92.3
46.3
77.7
88.2
AdaCI--DIME (BLIP)
71.3
88.3
94.9
54.7
82.8
90.4
AdaCL-DIME (GIT)
70.1 (-1.2)
88.0
94.2
55.1
82.3
90.6
AdaCL-DIME (CoCa)
70.8
87.8 (-0.5)
93.7
54.6 (-1.3)
82.0 (-1.4)
90.0 (-1.0)
AdaCL-DIME (BLIP-2)
70.4
88.3
93.7 (-1.2)
55.9
83.4
91.0
poses imposes demands for handling clone negatives, thus rendering this task more challenging, We will release all datasets based pseudo captions to facilitate further research in this domain.
A.3 DERIVATION OF my AND m2 IN ADACL
Revisiting AdaCL, our goal is to progressively tune 1 and m2 based on the anchor. Therefore, we first copy Equation in the main manuscript, i.e., softmax normalized similarity for each image and its corresponding text T with two margin parameters, which can be expressed as:
exp (m (T) -22)
((( exp (sm, m2)] E3
(12)
M+1
1971 exp exp [s(I,
1 Σ v1 vi(1) log((,()))
Lada ET-D [y p(()) -
(13)
The potential in-batch clone negatives are represented as:S* {s pCC) s) |
(C s) }. and
anchor is defined as the median of S The two specific boundary functions of the anchor are defined as:
exp [ma (anchor m2)
p.. exp (mi (anchor m2)] Eamehor
(14)
exp [m (1 m2))
(15)
exp [m, m2)) Eandbor
where Enghor is the simplification of _k=Ikku exp [s( TR)], Eanchor can be obtained through Equation 14 expressed as:
Emncbor I- mi (anchor-ma))
(16)
Pu
Combining Equation
16 and Equation
[5] have:
e mi - m2)
Pa
my (anchor ma) (1 1 -6
(17)
P.n
18

---

Published as a conference paper at ICLR 2025
Image Query
Ground-truth
BLIP
GIT
BLIP-2
CoCa
A band playing chroring with manny people.
A crowd people in hand stage.
Many people cujoying
crowd of people in large concert.
A large crowd of people are guthered in concert.
Masic being played by several individuals
People sitting chairs watching man play musical
People having an indeor concert.
group of people and man playing a violia
group musicians sitting in room with instrumeuts.
sits
Thu who trying rope bail rodeo.
There riding
riding chasing
horse rodeo.
wearing jeans trying stop horse.
man falling horse.
riding horse.
trying catch horse that ruasing
man white striped shirt is trying rope horse.
awaay
crowded sidewalk Asian country.
People walking
People walking
people walking
shope.
walking
A crewd of people dowa the city street.
There are many people walkking down together.
Many people gatleer on large crowd people the
crond people walking
walking down
people running outfits runs marathon with two serapess in the background.
People ranning marathon in with rall buildings.
People are jogsing during the day.
group people running city
group people walking em the sidowalk near a building.
group people ruasing marathou the city,
People running city
Athletes ranning
largo people runsing marathon
group people standing street.
street
A man blue T- shirt speaks into blow horn group people.
People are walkingg in crowded city street.
Cyclists ridlng across the street.
standing street corner with megaphone
group people street.
Many people are chilling building.
People front of with cleck tower.
Tourists tables A crowd people outside building,
A crond people sitting tables frest of building.
sitting tables outside building.
A group of people stund in the park of city, with buildings background.
Peuple standing around fountain in city with rall
Several people are standing in the park.
group of people standing in the park.
group of people standing frout of water fountain.
Figure 6: Cases of pseudo captions by four distinct captioning tools, i.e., BLIP, GIT, BLIP-2, and CoCa.
19

---

Published as a conference paper at ICLR 2025
To simplify Equation 17 we have:
(1 c) - Pu))
e anchor e
(18)
Pu
By taking the logarithm of both sides of Equation 18 we have:
m1 anchor log (1 - c)(1 Pu)
m2 log
(19)
Pu
Then, my can be obtained, expressed as:
E Pu
mi log( (1 <) - Pw)
// (anchor 1);
(20)
which corresponds to Algorithm in the manuscript. Meanwhile, by taking the logarithm of both we have:
sides of Equation 16,
log Eanchor = log 1-fe + mn (anchor -222-
(21)
Simplifying Equation
we obtain m
Du Lamncorr Pu ))) 3/m11
m2 = anchor + log(
(22)
which corresponds to Algorithm in the main manuscript. With Equation 20 and Equation 22 mi and m2 can be computed and updated during each batch training process with the supervision of anchor, facilitating the model to exploit more distinguishable cross-modal semanties among samples compared with the original TRL and CL.
A.4 DERIVATION OF EQUATION
Here we demonstrate the derivation of Equation with Bayes's formula can be expressed as:
To begin with, a K-class classification probability
p(r y O)p(== = i)
exp (f(())
pvu i =1) Ej=1 E/=1((E p(E j)p Σ exp (f)(z))
(23)
In anchor selection of AdaCL., the input variable (a. s) is one-dimensional with binary output variable y E 0, 1 (a.k.a C and C). We aim to predict p(y 1 z). Since GDA assumes that for each class y = 0 and y 1, the input x follows gaussian distribution. This can be expressed as: p( y = 0) = N (a μo, oÿ) and p(x y = 1) N (æ (41 Po- μn and og of are the means and variances of distributions for classes y 0 and y respectively. Thus, the posterior
probability can be expressed as:
N (x 14, o To
plu 0 x)) Nx No: of
(24)
TD N (x
M1, of) m
N (x M1, of) m
ply x) Nx |1, + N (a Po, 03
(25)
Since N (x oi) is the probability density function of Gaussian distribution, we substitute (a-w)² into the above equations, obtaining Equation in the
N (x (2,o9) <2ng= exp
(- (-12211)
manuscript:
p p = 0
(26)
(8-11) )
1
H1 exp
mo 01
20
20
p ( x)
(27)
Hn exp
(
88- Hu
(8
20
20
P (y = 0 2)) and (y = Ix) represent the probability of a similarity score to be a clone negative not, without the need of explicit pre-processing to the dataset training.
20

---

Published as a conference paper at ICLR 2025
Table 7: R@1 Results text-based person search. "DG" stands for domain generalization, and "FT" for fine-tuning on the corresponding dataset.
CUHK PEDES
ICFG-PEDES
RSTPReid
Method
DG
FT
DG
FT
DG
FT
CL
16.3
49.3
15.8
43.5
12.7
30.1
AdaCL
30.5
56.7
27.9
49.0
23.9
41,4
A.5 DOMAIN GENERALIZATION ON TEXT-BASED PERSON RETRIEVAL
To evaluate the robustness of AdaCL a and-play module, seek evaluate its domain generalization capabilities in text- text-based person retrieval. Specifically, we conduct training using Flickr30K and select three mainstream text-based person retrieval datasets, CUHK-PEDES(LLee) al.) 2017b), CEFG-PEDESS(Dng et al. 202T), and RSTPReid et al. 2021) for domain generalization experiments. To ensure fair comparison, it adopt in its paper is the closest to vanilla contrastive learning, and its original paper indeed conducted experiments on two of the datasets. As illustrated Table 7 it observed that AdaCL boosts CMPM by large margin. Especially for DG, the results of AdaCL on each dataset improves by over 10%.
choose CMPM the baseline, the learning objective
Table 8: Fine-tuning results of AdaCL three baselines under ICFG-PEDES.
Method
Text-Image R@
Text-Image R@s
Text-Image R@ 10
CMPM
43.5
65.4
74.2
AdaCL-CMPM
49.0
69.7
79.1
VITAA
51.0
68.8
75.8
AdaCL-VITAA
54.8
74.1
78.6
IRRA
63.5
80.3
85.8
AdaCL-IRRA
64.3
81.1
86.5
Table 9: Fine-tuning results of AdaCI. on three baselines under RSTPReid.
Method
Text-Image R@
Text-Image R@5
Text-Image R@ 10
CMPM
30.1
38.5
59.6
AdaCL-CMPM
41.4
57.0
55.7
VITAA
37.7
60.6
66.5
AdaCL-VITAA
42.6
62.1
69.2
IRRA
60.2
81.3
88.2
AdaCL-IRRA
62.7
81.4
89.0
For fine-tuning in addition to CUHK-PEDES, also validate the performance of AdaCL on ICFG- PEDES and RSTPReid. Three baselines are employed: CMPMGZhang & Lu 2018), ViTAA(Wang al. 2020b). and A & Ye 2023). and compare the effectiveness of incorporating AdaCL as a constraint. The experimental results w/ and w/o using AdaCL are presented in Table and Table It observed that AdaCL demonstrates significant improvements across the three baselines, achieving absolute enhancements of 5.5%. 3.8%, and 0.8% in R 1, respectively. These matching results substantiate the robustness of AdaCL other vision-lante languaee downstream tasks, demonstrating its insensitivity to the diverse dataset distributions (both natural images and person search images), and the choice of baselines.
A.6 NOISY CORRESPONDENCE LEARNING
As mentioned in Section I noisy correspondence learning (NC) (Huang al.) 202 Yang et al.)
2023 Ma al. 2024 Qin et al., (203) focuses on handling negatives by manually introducing noisy labels. Several works classify samples into clean and noisy subsets, followed by rectifier and triplet
21

---

Published as a conference paper at ICLR 2025
ranking loss boost the learning of NC. We further validate AdaCL in such challenging scenarios by plugging in AdaCL and verify its NC effectiveness on Flickr30K using the same pre-proe pre-processing strategy (by shuffling the captions of training images for specific percentage, denoted by noise ratio). The matching results under two noise ratio (20% and 40%) are reported Table T0
Table 10:
Noisy correspondence learning AdaCL. We follow (Huang et al. captions of training images for specific percentage, i.e., noise ratio.
2021 to shuffle the
Image TTex
Text+ -Image
Noise Ratio
Methods
R@1
R@5
R@ 10
R@1
R@5
R@ 10
75.0
93.9
97.5
58.3
83.0
89.0
AdaCL-NCR
75.3
93.8
97.4
61.2
84.1
89.7
BiCro
78.1
94.4
97.5
60.4
84.4
89.9
AdaCL-BiCro
79.6
95.2
97.5
62.7
85.1
91.3
20%
CREAM
77.4
95.0
97.3
58.7
84.1
89.8
AdaCL-CREAM
80.0
95.6
97.4
61.9
86.4
91.3
CRCI.
77.9
95.4
98.3
60.9
84.7
90.6
AdaCL-CRRC.
81.0
96.2
98.5
62.3
84.9
91.7
NCR
68.1
89.6
94.8
51.4
78.4
84.8
AdaCL-NCR
74.7
92.3
96.6
57.8
82.0
87.1
BiCro
74.6
92.7
96.2
55.5
81.1
87.4
AdaCL-BiCro
75.3
93.1
96.2
57.4
82.5
40%
CREAM
76.3
93.4
97.1
57.0
82.6
88.7
AdaCL-CREAM
79.2
95.1
98.3
61.5
86.0
90.2
CRCL
77.8
95.2
98.0
60.0
84.0
90.2
AdaCL-CRCL
80.3
95.0
98.1
61.7
84.4
90.9
We also validate the effectiveness of AdaCL on CC152K. CC152K consists of 150,000 samples from training split of Conceptual Captions (CC) (Sharma al. 2018) for training. 1,000 samples from validation split for validation, and 1,000 samples from validation split for testing. As all image-text pairs in CC are automatically harvested from the Internet, approximately 3%% of the pairs in the dataset are mismatched or weakly matched. This benchmark aligns well with the settings of NC, making it a suitable choice for evaluating AdaCL.
From Table 10 and Table [1 it can be concluded that for noise ratio of 20%, AdaCL achieves notable improvements, particularly in I-T R@! (AdaCL-CRCL. improves from 77.9 to 81.0) and R@1 (AdaCL-NCR improves from 58.3 61.2). For noise ratio of 40%, the trend of improvement remains consistent, although the performance naturally decreases as noise increases. Notably, AdaCL- CRCL demonstrates strong robustness with 1-T R @ improving from 77.8 80.3, even high noise levels. While the baseline results degrade significantly as the noise ratio increases, AdaCL exhibits better resilience, evidenced in AdaCL-NCR (I-T @ only drops from 75.3 to 74.7). AdaCL's robustness is particularly evident in T-I matching, where the decline in performance is less pronounced compared the baselines (AdaCL-CRCL achieves T-I R R@S of 84.4 at 40% noise ratio). Similar to Flickr30K, AdaCL also demonstrates consistent improvements over the baselines on CC152K. The performance improvements of AdaCL on both datasets further support its generalizability and applicability in noisy correspondence learning.
Table 11: Noisy correspondence learning of AdaCL on CC152K.
Image + Text
Text Image
Methods
R @
R@5
R@ 10
R@1
R@5
R@ 10
NCR
39.5
64.5
73.5
40.3
64.6
73.2
AdaCL-NCR
43.2
66.9
74.9
42.5
69.0
76.2
BiCro
40.8
67.2
76.1
42.1
67.6
76.4
AdaCL-BiCro
42.9
66.1
76.0
42.7
68.4
78.7
CREAM
40.3
68.5
40.2
68.2
78.3
AdaCL-CREAM
43.1
69.6
77.2
42.2
70.0
80.2
CRCL
41.8
67.4
76.5
41.6
68.0
78.4
AdACL-CRCL.
42.4
68.0
77.4
41.7
69.3
80.0
22

---

Published as a conference paper at ICLR 2025
A.7 ZERO-SHOT IMAGE CLASSIFICATION OF ADACL IN CLIP PRE- TRAININI
In addition to image-text matching, we also evaluate AdaCL on other pre-training task, i.e., zero-shot image classification. Specifically, validate AdaCL eight common classification benchmarks, which can be divided into (i) general datasets: ImageNer(Deng et al., 2009), CIFAR- (iKzh-s
al.) 2009). SEFAR-100(Krizhewv
et al., 2009), Cellech-1O1(Fii-FFF et al.) 2004)), and (ii)
fine-graiin grained datasets: Food-101( Bossarr et 2014), Flowers-102(Nils (ilsbbaaa & Zisserman 2008).
OxfordPets(Parkhi et 2012). and (GVCAircraiiMaaj al. 2013). The Top-1 accuracy results of "CLIP+ + AdaCL." pretrained on CC3M and CC12M are demonstrated in Table 12
Table 12: Zero-shot image classification of CLIP pre-training under different learning objectives. "Bascline" represents "Cnil+vanilla contrastive learning", and "AdaCL" represents "CLIP+AdaCL". Results under two pre-training settings, i.c., CC3M and CC12M are compared.
Datasets
Data
Model
ImageNet
CIFAR-10
CIFAR-100
Caltech-101 101
Food- 101
Flowers
Aireraft
Pots
Baseline
17.2
71.3
32.1
50.9
10.2
10.8
12.1
CC3M
AdaCL
22.0
77.1
42.2
54.8
12.6
13.3
14,9
1.7
Baseline
32.9
72.5
38.0
74.0
26.5
25.7
46.2
2.6
CC12M
AdaCL
34.8
73.4
43.3
74.7
33.1
25.4
46.7
2.8
observed that AdaCL outperforms CL all the general datasets and most the fine-grained datasets, proving its advantage in recognition tasks. Specifically, in ImageNet, CIFAR-10, CIFAR- 100, the Top-1 accuracy of AdaCL has surpassed vanilla CL by over 5%. It is noteworthy that performance on fine- tinengrinned datasets further verifies AdaCL's capacity in challenging scenarios.
A.8 VLP FINE-TUNING
Apart from CLIP pre- vre-training. we further report the fine-tuning results of AdaCL in several Vision Language Pre-trad fre-training methods (VLP) by fine-tuning them using AdaCL MS MS-COCO (5K). As illustrated in Table [3] AdaCL facilitates matching performance across nearly all metrics under both dual-encoder method (BEIT-3 Wang et al. (2022c)) and fusion-encoder methods (UNITER et al. 2020c), OSCAR et al. 2020), Vin VIVLL/Z VLi/hang et al.) 2021) ) effectively boosting the fine-tuning process. These results further corroborate the robustness of AdaCL across multiple basclines.
Table 13: Results of AdaCL on VLP fine-tuning.
Image Text
Text -Imae
Methods
R@1
R@5
R@ 10
R@1
R@5
R@10
UNITERI
65.7
88.6
93.8
52.9
79.9
88.0
AdaCL-UNITER
67.6
89.0
94.3
55.1
81.2
88.9
OSCAR
70.0
91.1
95.5
54.0
80.8
AdaCL-OSCAR
71.0
92.7
96.3
54.0
80.6
89.1
VLnVL
75.4
92.9
96.2
58.8
83.5
90.3
AdaCL-VinVL
78.7
94.4
96.8
60.4
84,2
91.1
BEIT-3
84.8
96.5
98.3
67.2
87.7
92.8
AdACL-BEIT-
84.4
96.9
98.3
68.6
89.1
i Evaluated by us with official repository.
A.9 EFFICIENCY ANALYSIS
Serving as a plug-and and-alay module, AdaCL does not increase the inference time since it is independent of the cross-modal reasoning module. For training efficiency, add detailed analysis on AdaCL..
23

---

Published as a conference paper at ICLR 2025
The speed of model convergence mirrors the learning efficiency of certain constraint. As shown in Figure AdaCL brings considerable convergence efficiency and retrieval results, which even boost CMPM and DIME to achieve their ultimate results within the first epochs, demonstrating the scalability and efficiency of AdaCI..
Recall Image Tort (30)
Text-image 33
Receligg
CMPM
CMPM
AdACL-CMPM
SCAN
SCAN
AdaCL-SCAN
CVSE
CVSE
DIME
DIME
AdACL-DIME
AdSCL-DIME
Epoch
(3) Convergence efficiency of Image-text retricval
(b) Convergence efficiency of Text-image retrieval
Figure 7: Training efficiency of AdaCL.
A.10 ABLATION STUDIES OTHER PPPER-PARAMETERS
A.10.1
ANALYSIS OF MOMENTUM MEMORY BANK
Since memory bank widely adopted in vision-language contrastive learning, we further analyze AdaCL by varying the memory bank size M, which leads to different number of negative samples. The results in Table 14 reveal that the momentum memory bank yields a modest yet discernible improvement: Among the sizes of 4096, 6144, and 8192, the impact of memory bank is not that significant. This suggests that AdaCL does not excessively rely on the quantity of negative samples for ideal similarity distribution.
Table 14: Effect of different memory bank sizes.
Memory
Image Text
Text- Image
Bank Size
R@1 1
R@5
R@ 10
R@L
R@5
R@ 10
N/A
70.3
90.0
95.5
49.5
77.5
87.3
2048
71.9
90.9
96.7
51.4
78.4
87.3
4096
74.2
91.7
97.9
53.7
81.1
88.2
6144
73.7
91.9
98.2
53.2
79.6
87.8
8192
73.9
91.4
98.0
53.3
80.5
87.8
A.10.2 ANALYSIS OF MINI-BATCH SIZE
Since mini-batch size is correlated with the number of potential anchor candidates for selection, we also investigate the impact of mini-batch size, shown in Table [5] It can be observed that AdaCL exhibits remarkable robustness to variations in batch size settings. Across a range batch sizes from 16 128, the fluctuation in R @ remains within narrow margin of 4%.
24

---

Published as a conference paper at ICLR 2025
Table 15: Effect of different mini-hatch sizes.
Batch Size
Image
>ext
Text >Image
R@1 1
R@5
R@ 10
R@D 1
R@5
R@ 10
16
70.6
89.2
96.5
50.0
79.3
84.8
32
72.9
90.9
97.4
52.1
81.0
87.1
64
74.2
91.7
97.9
53.7
81.1
88.2
128
74.0
91.2
97.6
53,9
81.2
88.0
A.11 ADDITIONAL PARAMETERS IN ANCHOR SELECTION
Here we further analyze the anchor selection methodology. In the main manuscript, we utilize the negatives from San and Setn within each mini batch to represent the empirical means and variances. However, cannot guarantee that all samples in Sain and Sein are exclusively salient negatives and clone negatives. Based on this speculation, we continue to select Top- K similarity scores from Sstn and Satn observational samples for caleulating means and variances. The assumption of this study is higher similarity scores for certain negatives correlate with an increased probability of them being clone negatives. We set K 32 and conduct experiments on CMPM, SCAN, and DIME under Flickr30K, as shown in Table T6 It can be concluded that employing Top-K selection strategy does not result in significant improvement or deterioration matching performance, with fluctuations generally remaining within a 2% range. This observation contradicts our initial hypothesis and intuition. Consequently, we can infer that AdaCL exhibits low sensitivity to the specific values of the empirical mean and variance, which is another minor merit. Given that the Top- K selection explicitly increase computation without yielding significant performance improvements, we have opted to maintain the original calculation method in the main manuscript.
Table 16: Matching results of Top-K selection for empirical means and variances.
Image
Text Image
Methods
R@D 1
R@5
R @ 10
R@D 1
R@5
R@ 10
AdACL-CMPM
54.7
79.0
87.5
41.6
69.4
79.2
AdaCL-CMPM f
54.2
77.8
87.1
41.8
68.1
80.0
AdaCL-SCAN
71.4
93.0
97.2
50.9
79.9
86.8
AdaCL-SCAN t
72.7
93.4
96.5
50.2
79.0
87.1
AdaCL-DIME
82.6
96.3
98.9
63.6
88.4
93.7
AdaCL-DIME
82.4
95.3
98.7
63.7
88.2
93.0
1: Employ Top-K selection.
A. MORE VISUALIZATION OF ADACL
We present comprehensive comparison of CL, TRL, and AdaCL trained with ground-truth annotations and pseudo captions. The visualization results of the early training stage are demonstrated in Figure which include kinds of clone negatives with 11 cases. Based the attention maps, we can summarize the following conclusions: AdaCL captures abundant semantics on highly similar clone negatives. Specifically, case (a) and case (b) demonstrate that AdaCL boosts the exploration of spatial semantics among the images, such as "music being played by several individuals" as well as "is trying to stop horse". which effectively distinguishes clone negatives apart. Additionally, case (c) demonstrates AdaCL'ss ability in capturing background information such as "Asian country" and "a city street" are crucial phrases that are reasoned through AdaCL. Case (d) showeases five examples of urban landscape, demonstrating that AdaCL is able to discover instances that are not explicitly described in the text query. For instance, the unique attribute "spectator" not included in Q8, but AdaCL facilitate learning the corresponding representation, which is highlighted in the attention map. Also, the latent "fountain" is not included in Q11 but reasoned by AdaCL. In this way, AdaCL is proved to achieve comprehensive moos- semanties with its adaptive tuning strategy even when
25

---

Published as a conference paper at ICLR 2025
the quality of textual annotations is not high. This finding presents great potential of AdaCL to handle retrieval with low quality labels.
Furthermore, obtain the attention maps by training with pseudo captions under AdaCL, depicted the last column of Figure [ Due the lack of instance-level information during the training process, we do not expect the results to surpass models trained original annotations. However, AdaCL (Pseudo Caption) manages to capture the approximate cross-modal semantics and pays attention to the fine-grained representation, which outperforms CL and TRL (trained with ground- truth) in most cases. This demonstrates the prospects of AdaCL in the vision-language contrastive learning of automatically annotated image-text pairs.
A.13 DISCUSSION: LIMITATION
In this work, AdaCL is evaluated on (1) image-text matching under Flickr30K, MS-COCO, (2) CLIP pre- tre-training under CC3M and CC12M, (3) captions, (4) text-based person search under CUHK-PEDES, (CFG-PEDES, and RSTPReid. We have not extended AdaCL to an all-round vision-lange asioonlanguuge tasks due to time and computational limitations, which is undoubtedly planned in future endeavor.
image-text matching under pseudo
Also, although AdaCL maintains high convergence efficiency, we acknowledge that AdaCL inevitably introduces additional computation during training with moderate computational overhead of O M) per batch training. We believe this trade-off is acceptable given the context of contrastive learning and pre-training. In future work, will delve into lightweight vision- vision-language learning paradigm.
26

---

Published as a conference paper at ICLR 2025
Query
Image Query
AdsCI.(Pseudo Caption)
playing
cheertng concert with
people.
(a)
being played
by several individuals
while crowd sits and listens.
Q3:
Two who
riding buth are trying rope bull in rodes.
(b)
A man wearing blue jeans trying to stop borse.
Q5:
crowdied sidewalkk an Asian country.
(c)
Q6:
crowd people walking middle of city street.
Q7:
crowd people running outfits runs marathon with two skyserapers background.
Q8:
group of people running marathon the city.
09:
(d)
blue shirt speaks into blow born towards people.
Q10:
Many people are chilling front an old building.
O11: A of people city, buildings in the backggoondd.
Figure 8: Attention maps of clone negative cases in early stage (Epoch 10). "CL", "TRL", and "AdaCL" represent model trained with different constraints. The last column represents AdaCL trained with pseudo captions.
27


Reviews:
[{"review_id": "wWV5i9CGOu", "reviewer": "Reviewer_5tK1", "summary": "This paper addresses a challenge in image-text matching referred to as clone negatives, which are negative image-text pairs that are semantically consistent with the positive pairs, leading to ambiguous and suboptimal matching results. To tackle this issue, the authors propose Adaptive Contrastive Learning (AdaCL), which employs two margin parameters with a modulating anchor to dynamically enhance the compactness between positives while mitigating the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without explicit training, facilitating progressive tuning and improved in-batch supervision. Experiments on two benchmark datasets underscore the effectiveness of AdaCL.", "questions": "The method figure (Figure 2) is unclear, and neither the figure caption nor the text adequately explains what the \"reference clone negatives\" refer to.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces a novel approach to address the issue of clone negatives in image-text matching, which is a pertinent problem in the domain.\n2. The proposed Adaptive Contrastive Learning (AdaCL) framework dynamically adjusts the compactness between positives, showing potential for enhanced performance in image-text matching tasks.", "weaknesses": "1. The experimental results are not convincing. The main contribution of the paper is the improvement of Contrastive Learning, with CLIP being the most representative of this field. To prove the effectiveness of the paper, a comparison within the OpenCLIP framework using larger datasets (e.g., LAION) should have been conducted, rather than experiments on the relatively small-scale COCO dataset.\n2. The proposed method appears to be somewhat complex, which goes against the inherently simple and scalable nature of contrastive learning. Additionally, the paper does not provide an analysis of the introduced hyperparameters or the potential increase in overall complexity.\n3. The authors do not clearly explain why clone negatives pose a problem within the paradigm of contrastive learning. Clone negatives refer to texts that are aligned with the image but are not as accurate as the ground-truth text. Hence, the logits of clone negatives should naturally be lower than those of the ground-truth, which aligns with the basic optimization framework of contrastive learning. The authors should provide more empirical evidence or theoretical analysis demonstrating why standard contrastive learning fails to handle clone negatives adequately.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WaWk6VSLdy", "reviewer": "Reviewer_PFrK", "summary": "This paper focused on solving the task of Image-Text Mathing (ITM) especially the key challenge of clone negatives: the captions that have consistent semantics with the GT sentences, but with less coarse-grained information. To solve this, the Adaptive Contrasvite Learning (AdaCL) method is proposed, which introduces supervision of the clone negatives, and can dynamically penalize the clone negatives. This will enlarge the distance between the positives and the clone negatives.", "questions": "Please refer to weakness above.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors focus on a critical issue of clone negatives in the field of Contrastive Learning under the background of Vision-Language Models and Image-Text Matching Tasks.   \n2. Experiments are well performed, which show effectiveness of AdaCL.", "weaknesses": "1. More pre-trained models and more down-stream tasks could be test to further demonstrate the effectiveness of the proposed method.\n\n2. Spelling error(s): \np2 077 andand\np2 Fig 1(b) T2 -> T3\np1 034 action-level -> global level\np6 274 double curly braces “}” -> just one “}”, delete the last one\n\n3. Sequential adverbs: \np4 205 There are “Two conditions”. I can see the “First”, but where is the “Second”?\nIt should be before p4 214 ”(Second, )to achieve this”. \n\n4. Meaning of subscript: \np4 208 What does the subscript “u” mean? What is its abbreviation? Is it “unique(==specific)”?\nAnd what about p4 210 the “u-” in k∈M_u-? Is it “not unique”? I can’t sure. \n\n5. Inconsistencie(s): \np4 Sec 3.2 one anchor + one pos + M neg ==> Why not all M neg? Because Sec 3.2 and Sec 3.3 are connected. \np5 Sec 3.3 one anchor + one pos + (M-1) neg\np6 Algorithm 1 a mini-batch of N pos + M neg??? ==> Why not one pos + M neg?\n  May be the authors want to say “a mini-batch of N, in which 1 pos and M neg”?\n  So, N==1+M?\n\n6. p5 254 what is the superscript of j in Eq. (8)? Is it the same as Eq. (7) or Eq. (3)?\n\n7. p5 263 Why Eq. (9) is that? Can you give us a concrete deduction? And what about p(C=0|s)? Can you give a specific formulation?  \n\n8. p10 509 How to correctly understand the heat map, since a sentence contain ALL WORDS, and every mentioned instance in the image SHOULD be highlited. So, what does these Attention Map mean indeed? What Information can I get from them? Or what should we expect from them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S0qB56LLPI", "reviewer": "Reviewer_vmnW", "summary": "This paper recognises the clone negative issue in image-text matching, where negative image-text pairs are semantically consistent with the positives. To address this issue, the authors modulate the softmax function of original contrastive losses with anchors that are selected from each in-batch similarity score based on Gaussian discriminant analysis. The proposed method, called Adaptive Contrastive Learning, demonstrates superiority in both supervised and weakly-supervised image-text matching.", "questions": "1. From what I understand, this method is matching images with texts. Then why is FasterRCNN, an object detection model, used in the experiments?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation of this work is clear. The issue of clone negatives is well illustrated and intriguing, which is common and significant in matching images and texts.\n\n2. The proposed method is extensively verified under both supervised and weakly supervised settings, exhibiting superior performance under several benchmarks.\n\n3. The authors provide comprehensive ablation studies and analyses, with both quantitative and qualitative verification.", "weaknesses": "1. The method is not applied to or compared with the recent CLIP-style models, which I believe would strengthen the significance and soundness of this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2gGlzpvCfY", "reviewer": "Reviewer_8azX", "summary": "The authors of this paper propose a new method to improve cross-modal learning by adding a conditioning factor to the loss function to alleviate what the authors call \"clone negatives\". The proposed method is effective from the experimental results. However, the concept of clone negatives is more like the False-negative problem mentioned in some existing studies. For this reason, the concept of clone negatives still needs further discussion.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author's idea of ​solving the false-negative problem is clear.\n2. The experiments are rich and the proposed method is effective.", "weaknesses": "1. The authors proposed a new concept, namely clone negative examples. However, this is actually the false-negative problem, which is not a new topic. For example, there has been similar research in [1,2].\n2. The authors should enrich the caption of Figure 2 to make the core idea clear and concise.\n3. Line325: “Comparisons of Image-text matching” should be “Comparisons of image-text matching”\n4. The 2024 baselines are missing.\n5. Line 954,955: Missing citations.\n6. There are usually three mainstream benchmarks for text-based person search tasks. I suggest the author add the results of RSTPReid. Also, does the authors‘ R@ refer to the Recall accuracy? However, in the text-based person search task, the Rank accuracy is usually used as the evaluation indicator.\n\nReference:\n\n[1] Li H, Bin Y, Liao J, et al. Your negative may not be true negative: Boosting image-text matching with false negative elimination[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 924-934.\n\n[2] Li Z, Guo C, Feng Z, et al. Integrating language guidance into image-text matching for correcting false negatives[J]. IEEE Transactions on Multimedia, 2023, 26: 103-116.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)