Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a method for cross-modal retrieval in the presence of noisy image-text pairings. While there has been a lot of work in learning in the presence of noise in this space, existing works mainly handle the problem of "label" noise - i.e. when semantic categories / labels on the data are available, but are noisy. In contrast, the authors tackle the unsupervised setting where all that is available are image-text pairs harvested from the web. Some of these pairs may be noisy and some may not be. The idea of the paper is straightforward. The authors use a technique commonly used to address noisy labels - i.e. co-teaching. They train two networks, A and B on the same data. After each epoch, the authors collect the similarity scores computed by each of the networks and use it to train a Gaussian Mixture Model. The reason for this is the GMM allows one to calculate the probability that a sample is from a given distribution. They model it with two Gaussians. The idea is that the GMM will fit one to the set of data that has lower similarity (hence noisy) and a separate Gaussian to the one with high similarity (hence clean). They then use the learned similarities predicted by each model to "correct" the labels used for training the other model (for example, treating some data that was considered clean as noisy or vice versa) based on the co-teacher's predictions. Finally, they integrate the similarity / confidence that the model is clean or not into the triplet loss as an adaptive margin, rather than use a static margin. The authors experimentally validate their method on MSCOCO, FlickR30K, and Conceptual Captions and compare against a number of baselines showing reasonable improvement.
main_review: # Strengths
The problem tackled by the authors is a good one - many cross-modal retrieval methods that exist assume some sort of label space is either available, or can be predicted on the data. However, this might not always hold, particularly when one is harvesting web-scale data without labels. Thus, the problem of noisy pairings is important to address. I have done a literature search and haven't found any papers explicitly tackling this problem directly, so I would agree that the problem itself does have some novelty (although, many other papers do learning in the presence of label noise, which is somewhat different).

The approach taken by the authors is interesting, as they train two models, then try to essentially learn based on the distribution of the learned similarities which are the noisy samples and which are the clean ones - this is essentially a clustering, but done using a GMM. The authors don't have any novelty in that regard though, as the GMM is a standard off-the-shelf model. However, they use it to essentially get a per sample score of how likely the sample is noisy or not. The main idea is that these are then used to update the training set of the other model being used for co-teaching. This is a good idea, as it extends co-teaching methods into the cross-modal retrieval space in the absence of any labeled data. 

The authors' way of integrating this knowledge into the model is an interesting one - they essentially use an adaptive margin in the triplet loss. The main idea is that when we are really confident about a sample being clean, we should enforce a large margin. But when we aren't confident the sample is clean or noisy, we should enforce a small margin. This is a good idea, as it allows the model to learn an embedding space where such uncertainty can naturally be captured.

Experimentally, the authors show reasonable gains in the presence of noise. For the MS-COCO dataset, in the most noisy setting (50% pairings), the authors show ~2.1% gain for T-->I and around ~2.9% gain for I-->T on COCO. However, they are often outperformed in the 0% noise setting. The authors show more convincing gains on the Conceptual Captions dataset.

The authors include an ablation where they remove various components of their method, such as various pieces of the co-teaching model and show that each has a contribution to improving the performance of the method.

# Weaknesses
Before getting into specifics of the details of the model, I would like to point out there is an implicit assumption made by the authors that unclean pairs will end up having a lower similarity after the initial training, while clean pairs will have a higher similarity. The authors note throughout the text that models tend to pick up obvious correspondences first, for example. Thus, the assumption is, if samples have a low similarity, they could possibly be unclean (mismatched image-text). I can buy this argument on some datasets, like COCO and Flickr, where there are clear, defined object categories and objects mentioned in the captions. In this case, the model can quickly pick up the correspondences in caption between "car" and the visual object "car" (for example). Thus it makes sense that after the initial training such samples will have a high similarity, while noisy samples will have a lower similarity. However, I would argue this assumption is not always applicable - and **most importantly**, it is not applicable to the problem setting the authors' method is designed to address. Specifically, for datasets like Coco and Flickr or other datasets where there is a strong grounding between image and caption, for these types of datasets there *are* often object labels or categories available, or, if not, they can be predicted using many of the other existing methods for learning cross-modal retrieval in the presence of noisy labels (you can, for example, identify a set of objects in the captions, train a model to classify them, and then do some prediction of labels - and voila, you have a self-supervised and noisy set of labeled data). However, the situation that the authors' method is most applicable to are cases where a label set of objects or classes cannot be readily defined, because obviously if such a label set can be defined there are many baselines that deal with that. So, the authors approach is best attuned to situations where a label set can't be defined, like when harvesting a ton of data from the web, with very loose image-text pairings and an unknown set of classes. If the authors wish to argue their method is applicable to the cases where labels can be defined, then they should show results comparing against those noisy self-supervised label methods. But assuming they don't, then the most important dataset the authors test on is the Conceptual Captions dataset. This dataset does not have a definable set of labels, as it is just a huge set of images and paired text crawled from the web. Again, that's the setting the authors are targeting. However, this type of data exhibits properties that break the authors' assumptions. For example, real world data often features images and text that aren't directly overlapping (i.e. the text isn't really a caption, but just goes with the image) - in other words, the image and text are complementary with one another. In other words, the text and image are related, but not directly duplicative of meaning. This is an important aspect of such data to capture, but the authors approach will actually de-emphasize it. For example, after training, these samples exhibiting this complementarity will have a lower similarity score than the obvious caption-like samples, thus the authors' approach may discard them as noisy. 
There is a recent work that tackles this exact scenario in ECCV20 and doesn't require any labeled data and learns a strong semantic space. I am also interested in hearing the authors thoughts on they can distinguish noisy samples from samples that just have a more indirect relationship between image and text (the so-called complementarity). This is a very challenging problem. Moreover, the authors should have compared against this paper. In my view, the results on FlickR and COCO aren't really the scenario this method is suitable for.
Paper studying complementarity and dealing with indirect relationship of image-text on real world data:
Thomas, Christopher, and Adriana Kovashka. "Preserving semantic neighborhoods for robust cross-modal retrieval." European Conference on Computer Vision. Springer, Cham, 2020.

Again, the authors paper is targeting learning on web-scale data where there are no discernable classes and thus there is noise. Thus, I would expect the authors to compare against using CLIP as a baseline. CLIP is trained on a massive dataset of webly harvested data and presumably there are a lot of image-text pairs in there that are noisy. *Why isn't the solution to noisy image-text pairings just to have a lot more data?* That seems to be the answer CLIP provides and it is the state-of-the-art image-text retrieval model at present. Moreover, anybody can take CLIP and finetune it on a dataset if they feel there is too much domain shift.
The authors should have showcased a result showing why just having a ton of noisy image-text data isn't enough and that they can do better on less data by explicitly managing / modeling the noise. At present, I am not convinced I should reach for the authors' method over just reaching for a model trained on a large noisy dataset. This is a key point the authors should address.
Citation for CLIP:
Contrastive Language–Image Pre- training) (Radford et al., 2021)

Relationship with hard / soft negative mining methods, weighting approaches, and adaptive margin methods:
While the experimental comparison against a few recent retrieval methods is OK, the key point is to evaluate your experimental technique against other methods that do the same thing in different ways. For example, there are many papers that do a soft hard-negative mining technique (see Mithun, 2018 for the soft triplet loss), where they weigh each sample according to the distance. Similarly, there are papers that try to measure properties of the surrounding space and integrate it as a weight into the loss (e.g. Hubness aware loss). Others still try different mechanisms for learning an adaptive margin. I would have liked to see a comparison of why your weighting approach is better or at least a thorough discussion of these in related work.

Moreover, onto your approach, I understand that you use the confidence of the sample to compute the margin, but shouldn't we also consider the confidence of the negative samples as well? For example, if the confidence of the sample is very high then your method would say this sample should have a large margin. But if we are in a batch that has noisy negatives where the confidence is very low, then your method might try to enforce a large margin between the positive sample and the negatives - but those negatives could be noisy and actually shouldn't be that far from the positive. In other words, the margin should take into account the relationship of the positive sample as well as each negative to calculate the adaptive margin.

For the approach, I would also point out that the co-teaching technique used is really an adaptation of what has previously been done in the label space. Other prior work also uses a co-teaching approach to correct noisy data. So, the actual methodology of doing so isn't new, but the application to noisy image-text pairings and the way it is done by integrating it as a weight is new.

# Conclusion
The authors present a reasonable technique for correcting noisy image-text pairings. However, the scenario where the proposed method actually would apply has not been thoroughly explored. Moreover, if the authors wish to argue their approach does apply to scenarios where labels could be defined, they should have compared against noisy or self-supervised labeling methods. I believe the authors should also explain why the approach will not ignore the types of samples emphasized by the ECCV20 paper - i.e. the authors approach might learn to ignore the samples that have an indirect image-text relationship, at the expense of losing semantics from the representation space. Moreover, since they wish to argue web-scale data harvested from the web is often noisy, it isn't clear that the solution to that isn't just to use a lot more web-scale data. In that case, they should have compared against CLIP - which is the standard image-text retrieval method at the moment, is trained on noisy image-text pairings, and is available. In sum, while the approach attempts to solve an important and interesting problem, I am concerned about the evaluation scenarios and the lack of discussion of the issues mentioned.

# Final Comment (Post Rebuttal)
I thank the authors for their very thorough responses. I have reviewed the authors' results on CLIP and I encourage them to include these comparisons in their submission. Given that the performance is superior for some cases, and particularly because the performance is superior on the released model (CLIP) relative to the unreleased model, I would recommend the paper be accepted. I appreciate the authors' discussion of complementarity vs noise. This is an interesting issue that can be discussed in the text. Given the thorough results provided by the authors, I raise my rating to accept.
limitations_and_societal_impact: This is a technical retrieval work and has no dataset contribution. There are no societal concerns beyond any existing cross-modal retrieval model.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper introduces a novel task, noisy correspondence of cross-modal retrieval tasks. The noisy correspondence is similar to the noisy label in object classification tasks, where the noisy correspondence scenario assumes that some image-caption pairs are incorrectly matched. To solve the problem, this paper proposes a Noisy Correspondence Rectifier (NCR), which could be considered as the novel extension of Co-teaching [9] to cross-modal retrieval tasks. Note that directly applying Co-teaching to cross-modal retrieval is non-trivial as the authors discussed in the paper (L105-115), because noisy label methods assume the existence of the prediction score function of the given input, while cross-modal retrieval methods usually do not have such per-sample score function. Hence, the proposed NCR first selects (co-divides) clean and noisy samples by using triplet loss values and the 2-mean Gaussian Mixture Model (GMM). Using the divided samples, NCR co-rectifies the noisy labels by the prediction function, where the prediction function is defined by the value of triplet distances. Using the extracted clean/noisy samples and rectified labels, NCR applies the hardest negative mining triplet loss to train the model, while the margin value is adaptive to the rectified labels. The experimental results show the effectiveness of NCR in terms of robustness to the noisy correspondence, e.g., achieving nearly dropped retrieval performances when 50% of correspondences are incorrect.
main_review: ## Pros

- (+) The introduced noisy correspondence is an interesting and important problem. Although a few previous works [A, B] tackle similar motivation in cross-modal retrieval tasks, as far as the reviewer knows, this paper is the first paper that evaluates synthetic noisy correspondence with state-of-the-art cross-modal retrieval methods.
    - [A] Chun, Sanghyuk, et al. "Probabilistic embeddings for cross-modal retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
    - [B] Wray, Michael, Hazel Doughty, and Dima Damen. "On Semantic Similarity in Video Retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
- (+) The proposed approach is sound and empirically achieves good results under the noisy correspondence scenario. Especially, The proposed Co-teaching-like algorithm is conceptually well-founded and empirically works well.
- (+) Experimental results on different noise levels (0%, 20%, 50%) are very interesting; particularly, existing methods perform poorly when noisy correspondence happens.
- (+) This paper is well-written, well-organized, and easy to follow. I really enjoyed reading this paper.

## Cons

I think this paper has a good problem definition contribution and technical contribution. However, I feel that discussions related to the previous attempts (could be considered as contemporary works) are not enough. Especially if the authors can add discussions related to Chun et al. [A], I think this paper will be stronger submission.

### Precision metric

Musgrave et al. [C] recently showed that "the use of accuracy metrics that are either misleading, or do not a provide a complete picture of the embedding space", in other words, recall scores may lead to a wrong conclusion. Chun et al. [A] tackles the same problem in cross-modal retrieval tasks, and proposed Plausible-Match R-Precision (PMRP) for MS-COCO evaluation. I suggest the authors report PMRP scores [A] to show the effectiveness of the proposed NCR method. As far as the reviewer understood, re-evaluating models with PMRP scores does not need extra training budgets but only needs the evaluation procedure which is relatively reasonable computation during the rebuttal period.

- [C] Musgrave, Kevin, Serge Belongie, and Ser-Nam Lim. "A metric learning reality check." European Conference on Computer Vision. Springer, Cham, 2020.

### Possible related works

As far as the reviewer understood, this paper mainly assumes the underlying dataset has (almost) clean correspondence. However, in practice, the existing widely-used cross-modal datasets, such as MS-COCO already have many noisy correspondences. For example, many images in MS-COCO only contain a big wave and a surfer and their corresponding captions are barely distinguishable (See Figure 4 of [A] as another example). A similar problem, many-to-many correspondence, is actively discussed in Chun et al. [A], which solves the problem by a probabilistic approach. I believe that [A] should be discussed in the main paper because the target problem by [A] is highly related to the proposed noisy correspondence problem. This could be an explanation of why NCR performs better than other methods in 0% noise scenarios in Table 1 and 2. Wray et al. [B] also targeted a similar problem, instance-based retrieval evaluation of video retrieval, and solved the problem by soft labels as rectified soft labels in this paper. Kim et al. [D] also employed soft non-binary labels to solve metric learning tasks. 

- [D] Kim, Sungyeon, et al. "Deep metric learning beyond binary supervision." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

The listed related works do not hurt the novelty of this paper, but I think adding discussions on these related works, the submission will be stronger.

### Impact of vanilla triplet (or warmup)

- In L191-197, the paper mentions that $L_\text{soft}$ uses the hardest negative mining, not the vanilla triplet loss without hard negative mining as "L_w". I wonder about the performance gap between the vanilla triplet and the hardest negative mining triplet while L192-193 says "Although the hard negatives are helpful in improving the performance".
- On the other hand, it seems that the vanilla triplet loss without hard negatives hurt the performance of SGR in CC152k, as shown in Table 2. Why does it happen? The paper assumes that SGR is very sensitive to noisy correspondence (as denoted in L246-249), but there is no explanation of how the vanilla triplet can mitigate the noisy correspondence. 
  - Similarly, warm-up (with the vanilla triplet) seems the most important ingredient of the method (Table 3). It may imply that the vanilla triplet is one of the most important keys to solve the noisy correspondence. I wonder about the performance of warmup only, i.e., the vanilla triplet trained model.

### More related works

While the following related works do not hurt the novelty of this paper, I suggest the authors add more discussions on previous works [A, B] discussing noisy correspondence problems as well. Especially, 


## Comments not affecting the decision score

- L143-L149 can be replaced by citation of the textbook by Bishop, Christopher M (Bishop, Christopher M. "Pattern recognition." Machine learning 128.9 (2006).)
- L164: "On the other hand" appears two times in sequential sentences. It could be a typo.
- L186: It would be good to add VSE++ citation for explaining the hardest negative mining strategy
- Table 3. "02." could be a typo of "0.2"

## Overall comment

This paper introduces an important and undiscovered problem, noisy correspondence. The proposed method motivated by noisy label methods successfully tackles the problem. Although there are some raised concerns, for example, (1) missing precision comparisons in MS-COCO (2) some missing related works (3) study on vanilla triplet without hard negative mining is missing, I think these issues are not a significant issue to reject this paper, but resolving the raised concerns will make the submission stronger. Overall, I vote for accepting this paper.

---

## Post-rebuttal update

I really appreciate the authors addressing my suggestions very well in the response. The precision results (outperforming the baseline with more than +20%) are particularly interesting. I revise my score to clear accept.
limitations_and_societal_impact: One possible limitation of this paper is the lack of discussions on the existing noisy correspondence in the existing cross-modal retrieval methods. I would suggest adding more related works and their discussions in the main paper.

Another possible limitation could be the evaluation metric; Recall@K is too noisy and less informative metric than precision. I suggested the authors testing PMRP proposed by [A] in MS-COCO benchmark for measuring precision performances by the proposed method.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper focuses on cross-modal matching with the presence of noisy correspondence. The author proposes a novel method for learning with noisy correspondence to achieve robust cross-modal matching based on the SGR. In brief, it first splits the data into clean and noisy subsets in the manner of co-teaching and rectifies the labels with an adaptive prediction function. Extensive experiments on three datasets show the effectiveness of the proposed method.
main_review: In brief, this paper did a good job for solving the noisy problem in cross-modal matching. The proposed method is well verified with dense experiments on three benchmark datasets. Here are some detailed comments.
Pros：
1.	The proposed method for learning with noisy correspondence is novel. To perform label refinement in the scenario of a matching problem, the author proposes an adaptive prediction function for label rectifying. Moreover, to incorporate the rectified soft labels in the triplet loss, they propose to recast the labels as the soft margin in the loss.
2.	Experiments on the simulation noisy datasets (COCO and Flickr) and the real-world dataset (Conceptual Captions) show the superiority of the proposed method. Moreover, the ablation study and visualization verify the effectiveness of different modules.
Cons:
1.	In the method, the soft labels are recast as soft margins in the triplet loss. Why are the soft margins computed in the form of Eq. 7? Are there other alternative ways to determine the soft margins?
2.	There is a typo in Eq. 5. S is the similarity margin between the given pair and the mean of negatives. The second minus symbol should be modified as a plus.
3.	The model predictions are based on the average similarity margin of the largest 10% pairs in Eq.6. How to guarantee that the predictions are accurate enough for label rectifying? Why only select 10% pairs?
4.	I’m a bit concerned about training time as the NCR trains two models in the manner of co-teaching.

limitations_and_societal_impact: The authors have discussed the limitation and potential negative impact of the proposed method.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: This paper proposes a noisy label learning framework for cross-modal learning, where the training data consists of wrongly matched modalities. The proposed NCR first divides the data into clean and noisy splits based on the difference of loss distribution. Then NCR proposes to rectify the labels with a prediction function and further recast them into soft margins in the triplet loss. Extensive experiments and comparisons are conducted to verify the effectiveness of the method.
main_review: Overall, the paper is generally well written and easy to read. The main strengths and weaknesses of this paper are summarized as follows.

Strengths：
1.	The experiments are comprehensive. To show the effectiveness of the proposed method, the authors take SGR as an example and provide comparisons with other SOTA methods. The results with varying noise ratios and real-world noisy data demonstrated the effectiveness of NCR. Besides the SGR, experimental results of NCR-SCAN also show the superiority of the proposed method.
2.	the technique presented in the paper is novel. To achieve the robust cross-modal matching, the authors propose to turn the rectified soft labels into soft margins which enforce the true positive pairs closer than the negative by a large margin, while the false positive will has a small margin. 

Weaknesses:
1.	It seems that the proposed method NCR is only applicable to the triplet loss in cross-modal matching as defined in Eq.5-6. How to achieve robustness to other loss formulations like the softmax loss in ALIGN (Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision)?
2.	NCR recast the soft labels as soft margins defined in Eq.7. Why NCR need this procedure for robust matching? Why it could achieve robustness?
3.	Since two models are trained individually in the manner of co-teaching. It inevitably needs more computation time for convergence. And how to obtain the final retrieval results with two models?

limitations_and_societal_impact: The paper provides a comprehensive discussion about the potential negative impact in the Border Impact Statement.
review_title: null

Meta-review: The paper initially received positive reviews from four reviewers (i.e., 6 7 7 8). In the rebuttal, the authors address the concerns from the reviewers very well (e.g., comparison with CLIP), and then all reviewers further raise their scores (to 7 8 9 9). 

After carefully reading the manuscript, comments, and the corresponding response, I agree with the reviewers that this paper makes a significant contribution to the community, namely, revealing a NEW problem (i.e., noisy correspondence/mismatching pairs) in practice for the first time. The work is certainly valuable to the community including but not limited to cross-modal matching because the essence of many tasks such as ReId, visual grounding, and VQA, is exploring the correspondence of paired data. 

Overall, this paper is with strong motivation, a technical sound approach, extensive experiments, and good writing.
Decision: Accept (Oral)