Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies generalization in similarity learning with noisy labeled data focusing on SNNs. Empirical studies were designed to show that dataset topology crucially affects generalization under SLN and PLN, and both the network architecture and the loss function choice can disturb an online/offline correspondence for similarity tasks.
strength_and_weaknesses: This work studies how NNs generalize the concept of similarity in the presence of noise by investigating two phenomena: Double Descent (DD) behavior and online/offline correspondence, which is interesting.

Below are some concerns:
1. The authors stated that it is widely known that larger models (with more parameters) usually obtain better generalization. However, in my point of view, over-parameterized models usually have the overfitting problem, leading to bad generalization ability.
2. What are the limits in `this framework connects under- and overparameterized limits.'
3. I am confused about this sentence `the training dataset size dictates the two regimes instead of the number of parameters'. Do you mean ``the training dataset size instead of the number of parameters dictates the two regimes''?
4. What do you mean by ``online and offline test soft-errors match each other ''?
5. I don't see why DD and online/offline correspondence are two **complementary** approaches in terms of generalization. Can you provide some explanation on the **complementary**?
6. ``DD and online/offline correspondence were mainly applied to classification and regression, but, if true, they should also hold for other tasks such as similarity learning.''. What do you mean by `if true'?
7. What is DIBS? What do the gray shaded areas stand?
8. So many special terms are mentioned without definition, e.g., effective noise, which make it hard for readers to understand this paper.
9.  The density of this graph is defined as $\left|N_{\text {pairs }}\right| /\left(\begin{array}{c}
N \\
2
\end{array}\right)$ ", which tells us how much information we have about the input images. The density is proportional to $1/N$. How do you maximize the information by constructing all possible pairs (increasing N)?
10. In scenario 1, how do you determine $N_{sample}$?
11. Can you provide some theoretical analysis on the generalization in similarity learning with two kinds of label noise under two scenarios in terms of DD and online/offline correspondence?
clarity,_quality,_novelty_and_reproducibility: Clarity: Overall this paper is not well-written. The introduction section makes up half of the whole article, which is too long. The position of the figures is not well designed. I don't see why figures of totally different meanings are combined together. As pointed out above, many sentences are hard to read.

Quality: This work is not theoretically evaluated. The empirical study is a little bit rough.

Novelty: The main claim, i.e., the noise affects the generalization of similarity learning seems to be obvious.

summary_of_the_review: To sum up, the contributions of this paper are limited. It would be better if the authors can provide some theoretical analysis and refine the writing of this work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies how deep networks generalize the concept of similarity in the presence of noise. Specifically, two phenomena are studied (1) double descent behavior and (2) online/offline relations. The double descent behavior also exhibits in the cases of contrastive learning with noise. Besides, the equivalence between online optimization and offline generalization is probed. An extensive empirical study is provided to justify the paper's claims. 
strength_and_weaknesses: **Strengths**
- The research problem is interesting and significant. 
- Experiments are sufficient to support claims. 

**Weaknesses**
- The main contributions of this paper are unclear. 
- Technical novelty is somewhat limited. 
- Both writing and organization should be improved to reach the requirements of a top-tier conference. 


clarity,_quality,_novelty_and_reproducibility: Clarity. At the present stage, there are some concerns that need to be addressed: 
- The paper claims that "SLN outperforms PLN in the overparametrized region in dense datasets". Both SLN and PLN are noise. Does the paper mean that the SLN noise is less harmful in this case?
- For claims "online and ofﬂine test soft-errors match each other for classiﬁcation tasks under certain conditions", is there some evidence for this statement?
- Perhaps, it is better to stress similarity learning but not contrastive learning, since the paper mainly exploits SNNs. 
- What are "reﬂexive, symmetric, and transitive properties"? Could the paper provide more formal definitions for these properties?
- The paper shows that the interpolation threshold (training error = 0) cannot be achieved in some cases. Is this because the network capacity is not large enough, because prior works, e.g., [1] show that deep networks can fully fit any given noisy labels? 
- The double descent phenomenon in learning with label (label-pair) noise is not very exciting. In fact, prior works such as [2] have presented this phenomenon through training dynamics. Intuitively, since clean labels are dominant in noisy classes, the double descent phenomenon exists when we train deep networks on noisy data. 
- Could the paper provide more discussions about why an online/offline correspondence is supported by showing that the bootstrap error is small? It is confusing to me.

Quality. The quality of this paper should be improved, although the presentations in the experimental parts are admirable. 

Novelty. The conceptual and technical novelty is not prominent.

Reproducibility. The reproducibility is good. The paper provides detailed descriptions of implementations.
summary_of_the_review: This paper studies an important problem. Extensive empirical evidence is provided. However, at the present stage, there are still some unclear explanations that need to be addressed. Therefore, the rating is "5" before the rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper empirically investigates the phenomena of double descent and offline-online correspondence (the gap between test errors of offline and online learners, as far as I understood) for similarity learning. The authors introduce two types of dataset conversion procedures from data points to pairs and two types of noise-generating processes to see how these factors affect generalization in similarity learning. These data-generating processes are expected to imitate well actual procedures in contrastive learning with data augmentation and negative sampling. After all, the authors discover for double descent that a type of noise---called pair label noise (PLN)---affects generalization more significantly because it breaks the label "transitivity," making the interpolation/overfitting harder. In addition, the authors reveal that offline learners are more prone to label noise than online learners in the empirical study of offline-online correspondence.
strength_and_weaknesses: ### Strengths

This work extends the concept of double descent from well-studied classification and regression to similarity learning. Many recent deep learning training procedures partly rely on self-supervised pre-training models that leverage contrastive learning, hence it is essential to understand more what kind of regime goes on under the over-parametrized regime in similarity learning. The authors provide one perspective to this answer to reveal that pair label noise has a severe effect on the generalization aspect. This should be an interesting perspective for researchers studying generalization.

### Weaknesses

The main drawback of this paper is the unclearness of the motivation and writing. I provide several writings that could be improved in the "Clarity, Quality, Novelty and Reproducibility" section and focus on the motivation aspect here.

Although research on double descent/interpolation/over-parametrization regime has been gaining attention in the recent community, I could not understand well what is the motivation of this work. In the study on double descent, we are primarily interested in how model performances degrade and improve as the model size increases, and there should not be label noises. Specifically, we usually consider the data-generating process $Y_i = f(X_i) + \\epsilon_i$ for regression and treat $\\epsilon_i$ as "white noise," but do not consider "more drastic (label flipping) noise" to purely focus on the clean generalization aspect. The current study seems to entangle labeling noise and the generalization aspect, so it is not sufficiently clear that the authors would like to focus on generalization or the effect of the noise.

In addition, the authors introduce two types of noise models (SLN and PLN) and two scenarios (sparse and dense connections). However, I did not understand well the reason why the authors introduce them. Are these introduced to imitate the real-world data-generating process in similarity learning? Or, are these introduced to make the data-generating process more scalable (in particular, the sparse connection)? This part makes me puzzled so I'm not sure the authors would like to study/reveal a certain phenomenon or propose a new data sampling procedure.

As for SLN and PLN, if we follow the initial motivation---to study the generalization aspect of contrastive learning with data augmentation and negative sampling---then these noise-generating processes could be overly simple. At least, in real-world scenarios, SLN and PLN could happen simultaneously. Why do we consider these two noises independently?
clarity,_quality,_novelty_and_reproducibility: Here, I make a list of unclear points in the paper. I hope this helps the authors to improve the manuscript.

- (In the abstract, introduction, and several places) The authors say "NNs generalize the concept of similarity" occasionally, which is confusing because the verb "generalize" in the machine learning context should be an intransitive verb. I guess an alternative way to express what the authors initially thought could be "NNs generalize in similarity learning," which causes less misunderstanding.
- (In the abstract) The authors state "SLN outperforms PLN in the overparametrized region," which sounds like the authors propose a new noise model to achieve better performance by injecting noises on purpose, causing a misunderstanding regarding the aim of this paper.
- (In the introduction; 1st paragraph) The connection between the online-offline learning correspondence and double descent is not sufficiently understandable from the current sentences. In particular, the last part "overparameterized models (trained on a finite number of samples) and underparameterized models (trained on very large datasets)" would be ambiguous. In addition, I would appreciate if the authors state slightly more about why the topic drifts to the online/offline correspondence from the double descent in the middle of this paragraph.
- (At beginning of page 2) The sentence "online/offline training compares the network performances varying the quality (diversity) of the dataset" puzzles me. What is the quality (diversity) of the dataset in this context and what is the related concept in the latter part of this paper? Does online/offline training itself vary the quality?
- (At the bottom of page 3) I failed to understand what the phrase "dataset quality and pairs topology" indicates. Does the "pairs topology" refer to SLN vs. PLN?
- (In Section 2) In the data-generating process introduced in the 2nd paragraph of Section 2, the authors choose to generate positive pairs within the same class in a "chain" manner, but why is it? It does not seem to represent a realistic data-generating scenario.
- (In Section 2.1) What is the purpose to introduce two scenarios, sparse and dense connections? It looks like the authors intend to accelerate data sampling with these two new scenarios, but do they represent a realistic scenario? I am not sure whether they are just a proxy model to a real scenario or a proposed sampling method for some goals.
- (In Section 3) The last paragraph of page 7 may be excessively long. It can be split into several pieces because it contains several important topics.
- (In Section 3; the last paragraph of page 7) In the last part, the authors claim "we need to take into account the number of possible configurations" but I failed to understand what is "possible configurations" and for what purpose we *need* to think about it. In addition, I am not sure what the last equation shown in page 7 calculates (partly because of some undefined notations like $N\_{\\mathrm{eq.pairs}}$).
- (In page 8; the latter part of the 1st paragraph in "Online vs. offline") Could you explicate why an online/offline correspondence is supported by *showing that the bootstrap error is small*?

I feel slightly sorry for pointing out structural and grammatical issues too much but could not avoid doing so because these issues hinder me from understanding the main scope of this work. I believe improving these aspects must make the paper more attractive.

In addition, while looking at the experimental results, I am not confident enough that "the PLN peaks appear to be shifted to the right-hand side, hitting that PLN data are harder to interpolate" (stated in the middle of page 7) from Figure 1 (bottom right). Both peaks of SLN and PLN range over similar regions and the difference does not look significant.
summary_of_the_review: I think this paper reveals an interesting contrast between the behavior of SLN and PLN in regard to the double descent phenomena. Nevertheless, several parts of the paper would not be sufficiently clear so the scope of the paper is not easy to grasp. There would be room for improvement in the structure and logical flow of the paper as suggested in "Clarity, Quality, Novelty And Reproducibility" section.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: **Summary**

 This paper presents work to investigate the double-descent phenomena and offline-online correspondence. Two kinds of data transform from individual points to pairs and two noise-generating processes are introduced to see how the factors influence generalization. Empirical studies are provided to show that pair-label noise can influence generalization more significantly. Also, offline learners are more prone to label noise than online learners. 

**Strengths** 
- The research problem is interesting and important. Similarity learning recently attracted a lot of attention from the research community. This paper provides an interesting view to study generalization under similarity learning. 
- Experimental results are plentiful to justify claims. 

**Weaknesses**

- The motivation is unclear. The paper begins with contrastive learning. However, it mainly targets simplified Siamese-type networks that are obviously different from recent popular contrastive learning. 
- Both writing and organization need to be improved. Reviewers and AC find that it is somewhat hard to follow this paper. Although in the rebuttal, the authors provide responses for clarification, there are still remaining concerns by reviewers. The current paper does not seem to be easily improved by superficial modifications. 


justification_for_why_not_higher_score: This paper provides an interesting perspective to understand similarity learning and generalization. Empirical results are provided to support its claims. Before rebuttal, reviewers raise several concerns about this paper. The authors provide responses to them. 

The responses address partial concerns. Reviewers hence become more positive about this paper. However, they still think that this paper cannot reach the acceptance line, particularly due to the unclear motivation and writing that should be promoted. Minor modifications are not enough for acceptance. AC checks the opinions of reviewers and this paper, and agrees with reviewers. Therefore, the recommendation is "reject".
Decision: Reject