Discovering Clone Negatives via Adaptive Contrastive Learning for Image-Text Matching

Authors: Renjie Pan, Jihao Dong, Hua Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Image-text matching, Contrastive learning, Vision and language, Multimodal learning

Abstract:
In this paper, we identify a common yet challenging issue in image-text matching, i.e., clone negatives: negative image-text pairs that semantically resemble positive pairs, leading to ambiguous and sub-optimal matching outcomes. To tackle this issue, we propose Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters along with a modulating anchor to dynamically strengthen the compactness between positives and mitigate the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without the need for explicit training, allowing for progressive tuning and advanced in-batch supervision. Extensive experiments across several tasks demonstrate the effectiveness of AdaCL in image-text matching. Furthermore, we extend AdaCL to weakly-supervised image-text matching by replacing human-annotated descriptions with automatically generated captions, thereby increasing the number of potential clone negatives. AdaCL maintains robustness in this setting, alleviating the reliance on crowd-sourced annotations and laying a foundation for scalable vision-language contrastive learning.

Decision: accept (poster)
Reviewer scores: 3, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730361428832

thread_type:
official_review

<thread_object>
creator: reviewer_8azx

<content>
summary:
The authors of this paper propose a new method to improve cross-modal learning by adding a conditioning factor to the loss function to alleviate what the authors call "clone negatives". The proposed method is effective from the experimental results. However, the concept of clone negatives is more like the False-negative problem mentioned in some existing studies. For this reason, the concept of clone negatives still needs further discussion.

strengths:
1. The author's idea of ​solving the false-negative problem is clear.
2. The experiments are rich and the proposed method is effective.

weaknesses:
1. The authors proposed a new concept, namely clone negative examples. However, this is actually the false-negative problem, which is not a new topic. For example, there has been similar research in [1,2].
2. The authors should enrich the caption of Figure 2 to make the core idea clear and concise.
3. Line325: “Comparisons of Image-text matching” should be “Comparisons of image-text matching”
4. The 2024 baselines are missing.
5. Line 954,955: Missing citations.
6. There are usually three mainstream benchmarks for text-based person search tasks. I suggest the author add the results of RSTPReid. Also, does the authors‘ R@ refer to the Recall accuracy? However, in the text-based person search task, the Rank accuracy is usually used as the evaluation indicator.

Reference:

[1] Li H, Bin Y, Liao J, et al. Your negative may not be true negative: Boosting image-text matching with false negative elimination[C]//Proceedings of the 31st ACM International Conference on Multimedia. 2023: 924-934.

[2] Li Z, Guo C, Feng Z, et al. Integrating language guidance into image-text matching for correcting false negatives[J]. IEEE Transactions on Multimedia, 2023, 26: 103-116.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730533875462

thread_type:
official_review

<thread_object>
creator: reviewer_vmnw

<content>
summary:
This paper recognises the clone negative issue in image-text matching, where negative image-text pairs are semantically consistent with the positives. To address this issue, the authors modulate the softmax function of original contrastive losses with anchors that are selected from each in-batch similarity score based on Gaussian discriminant analysis. The proposed method, called Adaptive Contrastive Learning, demonstrates superiority in both supervised and weakly-supervised image-text matching.

strengths:
1. The motivation of this work is clear. The issue of clone negatives is well illustrated and intriguing, which is common and significant in matching images and texts.

2. The proposed method is extensively verified under both supervised and weakly supervised settings, exhibiting superior performance under several benchmarks.

3. The authors provide comprehensive ablation studies and analyses, with both quantitative and qualitative verification.

weaknesses:
1. The method is not applied to or compared with the recent CLIP-style models, which I believe would strengthen the significance and soundness of this work.

questions:
1. From what I understand, this method is matching images with texts. Then why is FasterRCNN, an object detection model, used in the experiments?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730550144120

thread_type:
official_review

<thread_object>
creator: reviewer_pfrk

<content>
summary:
This paper focused on solving the task of Image-Text Mathing (ITM) especially the key challenge of clone negatives: the captions that have consistent semantics with the GT sentences, but with less coarse-grained information. To solve this, the Adaptive Contrasvite Learning (AdaCL) method is proposed, which introduces supervision of the clone negatives, and can dynamically penalize the clone negatives. This will enlarge the distance between the positives and the clone negatives.

strengths:
1. The authors focus on a critical issue of clone negatives in the field of Contrastive Learning under the background of Vision-Language Models and Image-Text Matching Tasks.   
2. Experiments are well performed, which show effectiveness of AdaCL.

weaknesses:
1. More pre-trained models and more down-stream tasks could be test to further demonstrate the effectiveness of the proposed method.

2. Spelling error(s): 
p2 077 andand
p2 Fig 1(b) T2 -> T3
p1 034 action-level -> global level
p6 274 double curly braces “}” -> just one “}”, delete the last one

3. Sequential adverbs: 
p4 205 There are “Two conditions”. I can see the “First”, but where is the “Second”?
It should be before p4 214 ”(Second, )to achieve this”. 

4. Meaning of subscript: 
p4 208 What does the subscript “u” mean? What is its abbreviation? Is it “unique(==specific)”?
And what about p4 210 the “u-” in k∈M_u-? Is it “not unique”? I can’t sure. 

5. Inconsistencie(s): 
p4 Sec 3.2 one anchor + one pos + M neg ==> Why not all M neg? Because Sec 3.2 and Sec 3.3 are connected. 
p5 Sec 3.3 one anchor + one pos + (M-1) neg
p6 Algorithm 1 a mini-batch of N pos + M neg??? ==> Why not one pos + M neg?
  May be the authors want to say “a mini-batch of N, in which 1 pos and M neg”?
  So, N==1+M?

6. p5 254 what is the superscript of j in Eq. (8)? Is it the same as Eq. (7) or Eq. (3)?

7. p5 263 Why Eq. (9) is that? Can you give us a concrete deduction? And what about p(C=0|s)? Can you give a specific formulation?  

8. p10 509 How to correctly understand the heat map, since a sentence contain ALL WORDS, and every mentioned instance in the image SHOULD be highlited. So, what does these Attention Map mean indeed? What Information can I get from them? Or what should we expect from them?

questions:
Please refer to weakness above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730620803650

thread_type:
official_review

<thread_object>
creator: reviewer_5tk1

<content>
summary:
This paper addresses a challenge in image-text matching referred to as clone negatives, which are negative image-text pairs that are semantically consistent with the positive pairs, leading to ambiguous and suboptimal matching results. To tackle this issue, the authors propose Adaptive Contrastive Learning (AdaCL), which employs two margin parameters with a modulating anchor to dynamically enhance the compactness between positives while mitigating the influence of clone negatives. The modulating anchor is selected based on the distribution of negative samples without explicit training, facilitating progressive tuning and improved in-batch supervision. Experiments on two benchmark datasets underscore the effectiveness of AdaCL.

strengths:
1. The paper introduces a novel approach to address the issue of clone negatives in image-text matching, which is a pertinent problem in the domain.
2. The proposed Adaptive Contrastive Learning (AdaCL) framework dynamically adjusts the compactness between positives, showing potential for enhanced performance in image-text matching tasks.

weaknesses:
1. The experimental results are not convincing. The main contribution of the paper is the improvement of Contrastive Learning, with CLIP being the most representative of this field. To prove the effectiveness of the paper, a comparison within the OpenCLIP framework using larger datasets (e.g., LAION) should have been conducted, rather than experiments on the relatively small-scale COCO dataset.
2. The proposed method appears to be somewhat complex, which goes against the inherently simple and scalable nature of contrastive learning. Additionally, the paper does not provide an analysis of the introduced hyperparameters or the potential increase in overall complexity.
3. The authors do not clearly explain why clone negatives pose a problem within the paradigm of contrastive learning. Clone negatives refer to texts that are aligned with the image but are not as accurate as the ground-truth text. Hence, the logits of clone negatives should naturally be lower than those of the ground-truth, which aligns with the basic optimization framework of contrastive learning. The authors should provide more empirical evidence or theoretical analysis demonstrating why standard contrastive learning fails to handle clone negatives adequately.

questions:
The method figure (Figure 2) is unclear, and neither the figure caption nor the text adequately explains what the "reference clone negatives" refer to.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111147390

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Reply [2/2]

comment:
In addition to image-text matching, we also evaluate AdaCL on more downstream tasks, i.e., text-based person search and zero-shot classification, where the former has been detailed in the manuscript. For zero-shot classification, we have verified AdaCL on eight common classification benchmarks, which can be divided into (i) general datasets: ImageNet[2], CIFAR-10[3], CIFAR-100[3], Caltech-101[4]), and (ii) fine-grained datasets: Food-101[5], Flowers-102[6], OxfordPets[7], and FGVCAircraft[8]. The Top-1 accuracy results of "CLIP + AdaCL" pretrained on CC3M and CC12M are as follows:

|  Pretrain Data  |      Method      |     ImageNet     |      CIFAR-10       |       CIFAR-100       |     Caltech-101      |      Food-101       |  Flowers  |    Pets    | Aircraft    |
|:---------------:|:----------------:|:----------------:|:-------------------:|:---------------------:|:--------------------:|:-------------------:|:---------:|:----------:| :---: |
|      CC3M       |     Baseline     |       17.2       |        71.3         |         32.1          |         50.9         |        10.2         |   10.8    |    12.1    | 1.0    |
|      CC3M       |      **AdaCL**       |       22.0       |        77.1         |         42.2          |         54.8         |        12.6         |   13.3    |    14.9    | 1.7    |
| --------------- | ---------------- | ---------------- | ------------------- | --------------------- | -------------------- | ------------------- | --------- | ---------- | ---------    |
|      CC12M      |     Baseline     |       32.9       |        72.5         |         38.0          |         74.0         |        26.5         |   25.7    |    46.2    | 2.6    |
|      CC12M      |      **AdaCL**       |       34.8       |        73.4         |         43.3          |         74.7         |        33.1         |   25.4    |    46.7    | 2.8    |

AdaCL outperforms CL in all the general datasets and most of the fine-grained datasets, proving its advantage in recognition tasks. Noted that the performance on fine-grained datasets further verifies AdaCL's capacity in challenging scenarios. We want to emphasize that the simple-yet-effective design of AdaCL (without the need of explicit modules or training) verifies its potential in image-text contrastive learning. All the additional experiments have been added to the revised manuscript to strengthen its overall quality.


So far, AdaCL has been validated across (1) multiple image-text matching baselines, (2) CLIP pre-training, (3) weakly-supervised image-text matching. Additionally, AdaCL has demonstrated strong performance on other downstream tasks, including (4) text-based person search and (5) zero-shot image classification. It is noteworthy that our proposed weakly-supervised matching scenario showcases its great potential in training without crowd-sourced annotations, which is another contribution of this paper. As one of the fundamental vision-language tasks, image-text matching serves as the most suitable benchmark for assessing and addessing clone negatives, which is a critical limitation that vanilla contrastive learning frameworks struggle to handle. Through extensive experiments across diverse tasks, we believe the effectiveness, compatibility, scalability, and robustness of AdaCL have been comprehensively validated.

[1] Sharma et al., Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In: Association for Computational Linguistics (2018) 

[2] Deng et al., Imagenet: A large-scale hierarchical image database. In: IEEE Conf. Comput. Vis. Pattern Recog. (2009)

[3] Krizhevsky et al., Learning multiple layers of features from tiny images (2009)

[4] Fei-Fei et al., Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In: IEEE Conf. Comput. Vis. Pattern Recog. (2004)

[5] Bossard et al., Food-101–mining discriminative components with random forests. In: Eur. Conf. Comput. Vis. (2014)

[6] Nilsback et al., Automated flower classification over a large number of classes. In: Sixth Indian Conference on Computer Vision, Graphics & Image Processing (2008)

[7] Parkhi et al., Cats and dogs. In: Int. Conf. Comput. Vis. (2012)

[8] Maji et al., Fine-grained visual classification of aircraft. arXiv:1306.5151 (2013)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732111204533

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Meta Reply [1/2]

comment:
We would like to thank all the reviewers for their thorough reviews and highly valuable feedback and suggestions. We are excited to hear that the reviewers found our work pertinent and meaningful [R1, R2, R3, R4], well-motivated and novel [R2, R3], and experimentally comprehensive [R2, R3, R4].


We also appreciate the suggestions provided, which have strengthened our work. Following their suggestions, we improved our paper accordingly (see latest revised manuscript). Below, we provide an answer to the common point raised about pre-training.

**Pre-training and comparison with CLIP**

In order to validate the pre-training ability of AdaCL, we conduct experiments in contrastive image-text pre-training under two large-scale dataets. Due to the time limit, we report the pre-training results on Conceptual Captions 3M (CC3M), and CC12M datasets [1]. Compared to MSCOCO and Flickr30K, they represent the scalability in larger-scale settings. Specifically, We use ViT-B/32 as image backbone, and set "CLIP + vanilla CL" as the baseline. "AdaCL" denotes pre-training with "CLIP + AdaCL". The following table is the zero-shot image-text matching results on the testset of Flickr30k dataset:


|  Pretrain Data  |      Method      |      I-T R@1       |      I-T R@5       |       I-T R@10       |      T-I R@1       |      T-I R@5       |       T-I R@10       |
|:---------------:|:----------------:|:-------------------:|:-------------------:|:---------------------:|:-------------------:|:-------------------:|:---------------------:|
|      CC3M       |     Baseline     |        26.6         |        52.5         |         63.2          |        18.1         |        39.4         |         49.7          |
|      CC3M       |      AdaCL       |  39.5                   | 60.8                    |  73.7                     |   25.5                  |  46.9                   |   54.3                    |
| --------------- | ---------------- | ------------------- | ------------------- | --------------------- | ------------------- | ------------------- | --------------------- |
|      CC12M      |     Baseline     |        49.3         |        77.3         |         85.0          |        35.5         |        61.8         |         71.6          |
|      CC12M      |      AdaCL       | 51.0                    | 77.5                    | 87.9                      |  38.4                   | 64.6                    |  74.7                     |

The second table is the zero-shot image-text matching results on the testset of MSCOCO dataset:


|  Pretrain Data  |      Method      |      I-T R@1       |      I-T R@5       |       I-T R@10       |      T-I R@1       |      T-I R@5       |       T-I R@10       |
|:---------------:|:----------------:|:-------------------:|:-------------------:|:---------------------:|:-------------------:|:-------------------:|:---------------------:|
|      CC3M       |     Baseline     |        13.4         |        32.0         |         43.3          |        10.1         |        25.6         |         35.7          |
|      CC3M       |      AdaCL       | 22.5                    | 47.1                    | 60.7                      | 17.8                    | 31.6                    |  39.5                     |
| --------------- | ---------------- | ------------------- | ------------------- | --------------------- | ------------------- | ------------------- | --------------------- |
|      CC12M      |     Baseline     |        29.3         |        54.4         |         65.3          |        19.0         |        41.0         |         52.5          |
|      CC12M      |      AdaCL       |  34.0                   | 55.6                    | 65.9                      |  25.1                   |  47.3                   |  57.4                     |

"I-T" and "T-I" represent image-text matching and text-image matching, respectively. From the results above, it can be observed that AdaCL improves CLIP's performance compared to vanilla CL, leading to notable enhancements in both image-text and text-image matching.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper focuses on the clone negatives problem in image-text matching and proposes Adaptive Contrastive Learning (AdaCL), which introduces two margin parameters to dynamically strengthen the compactness between positives. Extensive experiments image-text matching, noisy correspondence learning, CLIP pre-training, and text-based person search, are conducted to prove the effectiveness of AdaCL. The introduced approach might incur extra costs for training. However, given the strong results from the paper, the increase of cost (presented by # samples per second per GPU) is negligible.