Similarity and Generalization: from Noise to Corruption
Authors: Veronica Guidetti
Nayara Fonseca
Venue: Submitted to ICLR 2023 (2023)
Keywords: double descent
online/offline training
generalization
similarity learning
noise

Abstract: Contrastive learning aims to extract distinctive features from data by finding an embedding representation where similar samples are close to each other, and different ones are far apart. We study how NNs generalize the concept of similarity in the presence of noise, investigating two phenomena: Double Descent (DD) behavior and online/offline correspondence.  While DD examines how the network adjusts to the dataset during a long training time or by increasing the number of parameters, online/offline correspondence compares the network performances varying the quality (diversity) of the dataset. We focus on the simplest contrastive learning representative: Siamese Neural Networks (SNNs). We point out that SNNs can be affected by two distinct sources of noise: Pair Label Noise (PLN) and Single Label Noise (SLN). The effect of SLN is asymmetric, but it preserves similarity relations, while PLN is symmetric but breaks transitivity. We find that DD also appears in SNNs and is exacerbated by noise. We show that the dataset topology crucially affects generalization. While sparse datasets show the same performances under SLN and PLN  for an equal amount of noise, SLN outperforms PLN in the overparametrized region in dense datasets. Indeed, in this regime, PLN similarity violation becomes macroscopical, corrupting the dataset to the point where complete overfitting cannot be achieved. We call this phenomenon Density-Induced Break of Similarity (DIBS). Probing the equivalence between online optimization and offline generalization in SNNs, we find that their correspondence breaks down in the presence of label noise for all the scenarios considered.

Under review as a conference paper at ICLR 2023
SIMILARITY AND GENERALIZATION: FROM NOISE TO CORRUPTION
Anonymous authors Paper under double-blind review
ABSTRACT
Contrastive learning aims to extract distinctive features from data by finding an embedding representation where similar samples are close each other and dif- ferent ones are far apart. We study how neural networks generalize in similarity learning in the presence of noise, investigating two phenomena: Double Descent (DD) behavior and online/offline correspondence. We focus on the simplest con- trastive learning representative: Siamese Neural Networks (SNNs). We introduce two representative noise sources that can act on SNNs: Pair Label Noise (PLN) and Single Label Noise (SLN). The effect of SLN asymmetric, but it preserves similarity relations, while PLN is symmetric but breaks transitivity. We find that DD also appears in SNNs and is exacerbated by noise. We show that the density of pairs in the dataset crucially affects generalization. Training SNNs on sparse datasets affected by the same amount of PLN or SLN gives the same performance. On the contrary, using dense datasets, PLN cases generalize worse than SLN ones in the overparametrized region. Indeed, in this regime, PINN similarity violation becomes macroscopical, corrupting the dataset to the point where complete over- fitting cannot be achieved. We call this phenomenon Density-Induced Induced Break of Similarity (DIBS). Probing the equivalence between online optimization and offline generalization in SNNs, we find that their correspondence breaks down in the presence of label noise for all the scenarios considered.
INTRODUCTION
In recent years, several works have studied generalization in neural networks (NNs) and the connection between the classical underparametrizeed regime, where the number of training samples is larger than
the number of parameters in the model, and that of deep learning, where the opposite is usually the
norm. Indeed, the empirical success of overparameterized NNs challenges conventional wisdom in classical statistical learning as it is widely known among practitioners that larger models (with more
parameters) often obtain better generalization: Szegedy al. (2015): Huang al. (2019) Radford
al. (2119)
Two frameworks adopted to study gencralization in regression or classification tasks are Double Descent (DD) and online/offline learning correspondence, which we describe in the following. DD from [Bllinn et al. 2019) connects "classical" and "modern" machine learning by observing that once the model complexity is large enough to interpolate the dataset (i.e., when the training error reaches zero), the test error decreases again, reducing the generalization gap. This pattern has been empirically observed for several models and datasets, ranging from linear models, in Loog al. (020), to modern DNNs, as in Spigler et al. 2211); Nakkiran et al. (2020)). Instead, the online/ofiline learning correspondence of Nakkiran et al. (202)) D. studies the relationship between online optimization and offline generalization. The conjecture, empirically verified on supervised image classification, states that generalization in an offline setting can be effectively reduced to an optimization problem in the infinite-date limit. This means that online and offline test errors coincide if the NN is trained for a fixed number of weight updates. This setup aims to find a connection between under- and overparameterized models: the infinite-data limit (online) sits in the ander-parameterizee region (number of samples > number of parameters), while the finite-data case (offline) corresponds to the overparameterized regime (number of samples < number of parameters). Here, test if this correspondence is also valid for similarity tasks. DD and online/offtine correspondence are two complementary approaches that look at different generalization properties: while DD studies how the

---

Under review as a conference paper at ICLR 2023
network adjusts an increasing number of parameters, online/offline training compares the network performance by varying the dataset size while fixing the number of weight updates. Although these approaches have mainly been applied to classification and regression, if they are associated with some fundamental properties of DNNs, they should also hold for other tasks such as similarity learning-
There are key differences between learning, the relation among features is crucial but not necessarily the features themselves. For this reason, a priori it not possible to predict whether the DD behavior and the online/offline learning correspondence will also occur for similarity problems. To take the first steps towards understanding how DNNs generalize in similarity learning, we export both frameworks to the simplest contrastive learning representative, Siamese Neural Networks (SNNs) from Bromley et al. (1994); Chopra et al. (2005). A Siamese architecture is made of two identical networks sharing weights and biases that are simultaneously updated during supervised training. The two networks are connected by a final layer, which computes the distance between branch outputs. SNNs are trained using pairs of data that are labeled similar different. The task of successfully trained network is to decide if the pair samples belong to the same class. Studying the DD and online/offline correspondence in SNNs and comparing the results with those found in classification problems requires identifying which propertiesccharactersttss of the training set most influence similarity learning. We identified two crucial sources of variability: (i) the effect of noisy data in SNNs, and (ii) the density of pairs in the training set.
discrimination and classification. For similarity
Noise is crucial in understanding generalization it appears in every real-world dataset and may compromise model performance. While DD was also studied in the presence of noise very little (if none) attention was devoted to noise in the online/offline setting. By construction, SNNs can be affected by more complex types of noise than classification problems. This derives from the use of pairwise relations defining a similarity graph. To show the reaction of SNNs to different noise sources, we introduce two representative examples with distinctive properties: Single Label Noise (SLN) and Pair Label Noise (PLN), which we extensively describe in Sec. 2 and illustrate in the top panel of Fig. D As we will show, SINN breaks similar/different pairs balancing but preserves similarity relations. Instead, PLN acts symmetrically on pair labels, but it breaks transitivity and, thus, similarity. Furthermore, show that similarity learning is strongly influenced by the density of pairs in the training set. In particular, we will show how pairs created from populations with different levels of similarity graph density/image diversity, i.e., the average number of different images appearing in set of pairs, give rise to very different learning models. We discuss sparse and dense connections in detail Sec. 2
Our results show that
DD clearly appears in SNNs, regardless of the noise level, phenomenon rarely found in classification problems in the absence of noise.
DD is exacerbated by noise (in line with Nakkiran et 2020a and its shape is affected by the pair training set density. While SNNs trained on sparse datasets show similar DD curves in the presence of SLN and PLN, these become quite distinct when the similarity relations in the training set are dense. Specifically, the interpolation threshold in the presence of PLN requires more parameters and its test error remains higher in the overparameterized region. An example of this behavior is shown in the bottom right plot of Fig. 1
We show that the poor performances of PLN derive from its similarity- bbrakkin nature that manifests when input data are highly connected. We show that the interpolation threshold (training error = 0) cannot be achieved in this scenario, and we derive the analytic formula for the asymptotic training error value in the deep overparametrized regime. We call this phenomenon Density- Induced Break of Similarity (DIBS).
We test the correspondence between offline generalization and online optimization for similarity learning. We study how the architecture and the presence of noisy labels can differently impact these two regimes. We find that the conjecture only holds for clean data.
In the presence of label noise, we find that the online/offline correspondence breaks down for all choices of training settings considered. In particular, the effect of label noise is notably more relevant in the offline case.
Notably, it is known that the DD curve is exacerbated in the presence of random label noise in supervised classification (see, e.g., Nakkiran al. (020aa).
2

---

Under review as a conference paper at ICLR 2023
Single Label Noise (SL.N)
Pairs
Similar
Ankle
Treaut
Denn
Adiio bocet
Pair Label Noise (PLN)
Pairs
v² lahel
Nobe
Different
Different
Difficrent
Different
Nerell
error
5J8 29%
SPARSE
10'
neurons
neurons per layer
DENSE
Train
to
Test
2,N 100
10-
20%
10'
10
neurons par layer
neurons per layar
Figure 1: Top: Pictorial view of SLN and PLN in SNNs. Bottom: Train (left) and test error (right) as a function of model size. We consider 3-layer MLP with ReLU activation function on trained on sparse and dense pairs of MNIST with 10% and 20% effective noise.
1.1 RELATED WORK
In the past few years, much effort has been made to understand how neural networks were able to
generalize classification problems in the presence of noise (e.g.. Li et al. (2019); HHan al. (2019);
Arazo et al. (2019;; Harutyunyan et al. (2020); Song et al. (2020)). Remarkably, the DD behavior al-
lowed to investigate the NÑ behavior as the number of trainable parameters, the evolution time and
dimensionality of the sample vary: Nakkiran al. (2020a); Bodin & Macris (202)); Heckel & Yilmaz
(2020): Pezeshki et al. (202T). Subsequently. other works have produced analytical studies of some of these phenomena: d' d'Accoli et al. (2020a b); Mei & Montanari (2022.. Another complementary tool used to study generalization in classification tasks the online/ofiliee correspondence proposed in Nakkiran et (202T). which focuses on datasets without noise. This study empirically showed that a correspondence between online optimization and offline generalization holds for modern deep NNs trained to classify images. Earlier studies have proposed a similar comparison for linear models focusing on the asymptotic regime of training (see, e.g, Bottou & LeCun (2004 2005
Contrastive learning, introduced by Chopra et al. (2005) Hadsell et al. (006); Oord et (018)) has become one of the most prominent supervised (Khosla la et al. (2020); Gunel et al. (020)) and
self-supervised (Bachman et al. (2019): Tian et al. (2020); He et al. (2020); Chen
2020 MI.
techniques to learn similarity relations of nigh-dimensiona data, producing impressive results in several fields, see e.g. Le-Khac et al. (220)); Jaiswal et al. (202T). Despite its success, Ohri & Kumar (2021): Liu et al. (202 1): Jaiswal et al. (2020): Le-Khaa Khac al. (2020) show that contrastive learning usually requires huge datasets and considerable use of data augmentation techniques. Dealing with augmentation techniques and unlabeled data where negative samples are randomly selected introduces instance discrimination challenges. i.c. the need to find ways to limit the appearance faulty positive and negative samples. Indeed, Robinson al. (20T)) Wang & Liu (202)) show that contrastive

---

Under review as a conference paper at ICLR 2023
loss does not always sufficiently guide which features are extracted. For these reasons several works tackled the problem of discriminating against faulty negatives, as Huynh et al. (2022); Kalantidis et al. (2020) Chuang et al.) (2020); Iscen al. (2018). removing faulty positives and negatives dynamically (see Robinson et al.] (2021 hu et al. 2021)) and creating more robust contrastive setups introducing new losses (see Chuang al. (2022) Morgado al. (202T) or architectural components, Grill et al.
(2020).
2 DATASET CONSTRUCTION
In this section we describe the choices we made to study the dataset features that influence training. i.., the density of the image pairs and the presence of noise. We start by defining the criteria we used to construct the pairs dataset.
No NOISE
PLN
SLN
SPARSE CONNECTIONS
DENSE CONNECTIONS
Pictorial view of data relation appearing in Scenario (top) and 2 (bottom) for two classes of data. Positive pairs are connected by green edges, negative pairs by red edges. Ignored connections and data are in light gray. Gray-shaded areas are examples of transitivity breaking (DIBS).
Figure 2:
Similarity graph. As opposed to classification problems, where the main concerns during dataset creation are class balancing and image diversity, in contrastive learning, we should consider that pair (or group) relations between images define an unoriented similarity graph inside the input space. Calling N the total number of images in the full dataset, the density of this graph, o = J.V / (+). tells how much information we have about the input images. To maximize the information about certain dataset, should construct all possible labeled pairs, (²) 2 but this quickly becomes unfeasible when considering large datasets. For this reason, we construct pairs in way that maximizes the information about similar images (all similar images transitivity-related)) transitivity related)) and scales linearly with N. In practice, we construct closed chains of positive pairs within the same class, c. {=f, ..),..., ( 1h. {= " where n is the total number of images in c. Then, to build negative pairs, each image is connected to randomly chosen one belonging to a different class. If the original dataset classes are balanced, each image appears on average in different pairs (2 times in the positive and 2 times in the negative pairs). Therefore, the total number of pairs is given by Npairs = N x X ne, where n. is the total number of classes Finally, we describe the dataset construction method we used to study how density in the similarity graph affects training.
sparse connections. To train the network in the absence of noise, we first create the pairs using the full dataset. We follow the procedure described at the beginning of this section so that Npains = 2 x N. We then take Nsample balanced pairs (data used to train the model) from the Naairs list to train the NN and repeat this procedure nt, times.
Scenario 1:
Scenario 2: dense connections. In this setup, we create reduced version of the original dataset. Being interested in training the network on Npairs pairs, we select Nieduced = N /pairs/2 images from the original training set. The reduced dataset is balanced so that we have Npairs/ (2n.) images per class. Then, we create our training and test samples using the same prescription described the beginning of this section. We connect adjacent images
Note that this formula holds for dataset with least 2 images per class.
4

---

Under review as a conference paper at ICLR 2023
within the same class and each of them with random image belonging to different class so that we get exactly Npairs pairs that will be automatically balanced. We repeat this procedure n, times.
Pictorial representations of the similarity graph are shown in Fig. 2 where we represent elements belonging to different classes with nodes of different colors (black and orange classes), similarity and dissimilarity relations with green and red vertices, respectively.
Noise introduction. SNNs can be subjected to different types of noise having different properties. To show their impact on the training process we introduce two simple representatives, namely Single Label Noise (SLN) and Pair Label Noise (PLN) which we describe below (see Fig. for an illustration).
Single Label Noise (SLN), Let us consider a dataset with N samples X NND belonging to ne classes and their corresponding labels ys8 (vT 92 VN J. Suppose the classes are uniformly populated. If some label noise is present in the original dataset, this will propagate the training pairs as these are created. If SLN is uniformly introduced across all classes. it will keep the original class balancing on average (over multiple samples). On the other hand, in every single run, statistical fluctuations of uniform distribution introduce some asymmetry in the original class repre- sentative number (see Fig. 2 Finally, in SLN, similarity relations (reflexive, symmetric, and transitive properties) are preserved as mislabeling appears in all pairs containing misclassified image.
{2z, Z2.
Pair Label Noise (PLN). Let now consider a dataset of N pairs XP H*f ... (z). x 2h can be similar (y 1) or different (y² = 0). We construct them that they are balanced (half are similar, half different). Suppose we randomly shuffle some fraction of the total labels. In that case, the noise introduce is symmetric under similar different changes, and it acts democratically on every class of the original dataset. On the other hand, PLN can lead inconsistent relations in the pairs dataset. Indeed, we will show in the following sections, it breaks transitivity and, therefore, similarity.
{<". x with pair labels Y = Lvt ...,,., vh which
As discussed later, these two sources of noise impact how models learn similarity relations in distinct ways. To fairly compare the outcome of the model in the presence of PLN and SLN, we need to ensure that we introduce the same amount input label noise the two setups. We illustrate below how we ensured that the same amount of esfective noise was introduced. Being nte the number of image classes, yf the label of the i-th image, and of the label of the i-th pair of images, we can define the SLN transformation
( yf random(1, ne)
with probability q
and the PLN transformation as
TPLx(T): vp" random(O, 1) with probability q.
As SLN appears in the dataset before pair creation and the pairs are constructed so that the dataset is balanced (half pairs are similar, half are different), the probability of effective pair mislabeling induced by SIN, PSIN (g), given by
9² 2
PS)
(3)
while the probability of effective pair mislabeling coming from PLN, PPLN (ÿ), is
PPIn (g)
(4)
The requirement of having the same amount of effective noise in the dataset (PsLx(g) = PPLN(())
boils down to the following relation between y and qj:
q == vI
(5)
The full derivation of these results and the pseudocodes describing dataset creation are given in the supplementary material in Sec C and B respectively.
5

---

Under review as a conference paper at ICLR 2023
2.1 EXPERIMENTAL SETUP
In this work, we consider two simple Siamese branch architectures. The first one is an MLP with 3 hidden layers having the same width and ReLU activation functions with Xavier uniform initialization, see Glorot & Bengio (010). The second architecture is 4-layer CNN. We also considered two training setups: in one case, we compute the Euclidean distance in the output layer training the network using Contrastive Loss from Hadsell al. (2006), in the other one we compute the cosine similarity training the network using Cosine Embedding Loss (see Section Di the supplementary material for further details). The CNN architecture is based on the model described in Page (2018), it contains three
layers and a fully-connected output layer. The number of filters in each convolution layer scales as [k: 2k, 2 while the MaxPooling is [1, 8]. We fix the kernel size = 3, stride 1 and padding 1. When we train the network using contrastive loss (cosine embedding loss), we set the fully-connected output layer width to (2k).
We test the presence of DD using MNIST, FMNIST (from Xiao et al. 2017) and CIFARTO datasets (from Krizhevsky et al. (2009)) ). To understand the impact of overparameteriiation, we study how training and test errors vary at increasing network width and training time. To do so, we increase the number of neurons per layer in the fully connected architecture and the parameter in the CNN. For all datasets, we consider 6000 training and 9000 test pairs. In every DD experiment, we let the network evolve for 2000 epochs using Adam optimizer with minibatches of size 128 and learning rate À 10 except explicitly stated. All the typer-parameter and the margins were chosen empirically. To see the average effect regardless of the particular choice of images in the dataset and weights initialization, we run 15 evolutions of the network using different training and test samples at each time. In most of the experiments, unless otherwise stated, we considered q = 0.2, i.e., an effective noise of 10%.
DD setup:
Online/offtine setup: Since we cannot reuse samples for the online training, we consider an extended version of the standard MNIST dataset, namely the EMNIST (from Cohen al. (017)). We use the digit section of EMNIST that contains 240,000 training (and 40,000 test) 28 x 28 greyscale pixel images. We train Real World over 40 epochs using 12k pairs that are created considering Sparse and Dense scenarios. The Ideal World is trained once on 480k pairs created using the full training set of 240k samples. We test the models with 9k pairs constructed from the test set and consider Siamese networks with MLP and CNN blocks described in Sec. 2. In order to compare the results on different network architectures we used a comparable total number of parameters, namely, 200 nodes per layer for the MLP cases (total of 237, 337,400 parameters) trained with the contrastive loss (A = 10040 ); and width &s- = 47 (total of 235,611 parameters) for the CNN cases trained with the cosine loss (A 5 x 10 10-5). To provide an estimate of the results regardless of the particular choice of images and network initialization, we run MLP (CNN) experiments 5 (4) times.
All our experiments make use of the Tensor Flow/Keras framework: Abadi al. (205)). Each of the experiments mentioned above was performed in the presence and absence of noise and considering sparse (scenario 1) and dense pairs (scenario 2) in the training set.
RESULTS
DD results. In all experiments, can see the parameter-wise DD, regardless of architecture, loss function, scenario and noise. To support the consistency of our results with previous DD literature, in the supplementary material we also investigate the presence of epochwise DD in SNNs. Both kinds of DD are observed under every experimental condition. This does not happen in classification problems which typically require the presence of noise make the DDs clearly visible (see, for example, Nakkiran et al. 2020a)). As expected, DD becomes more prominent in the presence of noise. In Fig. D we show how the network reacts to different amounts of noisy labels. Our results, showing the average training and test errors together with error bars, can be found in Fig. and in the supplementary material in Sec. G.I In Scenario 1, the input dataset connections are sparse, and PLN and SLN have the same impact on training. This makes sense as there should not be any difference between PLN and SLN effects in the extreme case where every image appears only once in the training set. Instead, Scenario 2 is characterized by dense input connections, and the system behaves differently under SLN and PLN. We experimentally observe that the DD peak location changes in some but not all setups. Specifically, this happens in MLL--ucliddan Distance and CNN-Cosine Loss setups where the PLN peaks appear to be shifted to the right-hand side, hinting that PLN noise is harder interpolate than SIN. No such thing appears using CNN-Fuclidean Distance.
6

---

Under review as a conference paper at ICLR 2023
TRANSITIVITY BREAKING
123
nobb 20%
204-
102
CONFIGURATION
numbar classes
2.12
3.09
COLLAPSED CONFIGURATION
2.8
182
0.16
3335
0.00
622
polse procability
Figure 3: Left: leading transitivity breaking configuration (top) and its collapsed versions (bottom). Right: Analytic (lines) and numerical (scatter points) estimates of the asymptotic training error behavior varying number of classes n., (top) and effective noise (bottom) in the presence of PLN in scenario 2.
We believe that the presence of this shift signifies that the right NN setup is being used (i.e., the natural architecture- function match) Increasing the amount of noise enhances the test errors as expected, but does not induce any significant peak shift, SLN test error tends to be higher in small to medium network sizes. A hint about how this happens given in Fig. ? Indeed, SLN introduces systematic error: a mislabeled image appears to be mislabeled in every pair. Therefore, given that the image features are not going agree with pair labels, the only way the network has classify correctly is by extracting the image from its natural distribution. NNs being continuous functions, this implies that neighborhood of said image must be extracted well, increasing the test error. At higher network widths, the volume of the mislabeled image neighborhood can become arbitrarily small, and the test error is free go down again. In fact, SLN introduces systematic errors that do not compromise the consistency of the similarity graph. On the other hand, PLN stays higher in the deep overparametrized regime. Indeed, randomly changing similarity relations the input dataset, PLN ends up breaking transitivity, making the training set similarity graph inconsistent. Beyond keeping test error high, this inconsistency also implics that the network will never be able to overfit completely: the training error will no longer vanish just by increasing the number of network parameters.
Origin and magnitude of DIBS. We now explain the origin of the phenomenon call Density- Induced Break of Similarity originating from PLN. A similarity relation must satisfy transitivity. We can see transitivity satisfied or violated in the training set by evaluating the consistency of the closed paths in the similarity graph. See for example Fig. inconsistent paths with gray areas. To facilitate this operation in more complex setups, we can also resort to collapsed configurations, i.e., collapsing nodes connected by green vertices (see discussion around Fig. in the supplementary material). The analysis of the collapsed configurations shows that transitivity breaking mainly derives from the configuration in Fig. 3 The asymptotic training error (at leading order) is given by the probability associated to this local triangular configuration, given the dataset construction method explained in Sec. 2 This is given by the probability of having a misclassified equal pair, attached to a correctly classified one. Both pairs should be connected to the same other class of images. Finally, need to take into account the number of possible configurations. These are two and are given by swapping colors between the two pairs attached to
where we highlight some
Thhis intuition is supported by further online/offliee correspondence experiments present in the supple- mentary material.
7

---

Under review as a conference paper at ICLR 2023
different classes. This leads to:
cverect different pair
# conligurations
Neq
Ndiff.pairs
2
x P
x P)
N
pairss
N pairs
The 1
misclassified equal pair
connected same 2 classes
where peqrs refers to the number of equal pairs (similar), Ndiff.paire is the number of different
pairs (dissimilar), and Npairs is the total number of pairs, with Neq.paire Naift paire
Nouics Npaies 2 as we
consider balanced pairs. Therefore, the dominant contribution to the asymptotic training error is given by:
P(1 2(n -1
lim TrainError PLN (P,
(6)
0g +000
where ne is the number of network parameters and P= (PL is the effective amount of noise. In Fig 3 we validate our formula by comparing it with experimental results. There, we consider the FMNIST dataset trained on our MI.P architecture with 500 neurons per layer, using Euclidean distance and contrastive loss. Numerical results (mean and standard error bar) come from 10 runs where we choose different random classes each time. These results show how, in the overparametrized regime, the training eπror changes with the effective noise and with the different number of classes. This analysis shows that the macroscopic presence of transitivity breaking is linked to the presence and number of closed paths in the similarity graph and therefore to the dataset density.
Online vs. offiine learning. We probe the correspondence between offline generalization and online optimization (Nkkkiran et al. (20T)) see Sec. [ for details) for similarity tasks by studying how the training setting and the presence of noisy labels can impact these two regimes. Considering usual training settings (i.e.. natural choices of architecture-loss function match). the conjecture holds for data without noise, regardless of the dataset density. In the presence of label noise, however, we find that the online/offline correspondence breaks down for all choices of training settings considered.
MLP with Contrastive Loss
CNN with Cosine Loss
10% RUN Mral Test
10% P.N id0a 1:z
Dence 10% PLN Real That
Denpe 15% PUN Resl Thy
04-
Test Error
0.4
03-
0.3
500 1000
1500
2000
2500
3000
3500
500
1000
3000
2000
2500
3000
3500
Adam Iterations
Adam Iterations
Figure 4: Ideal vs. Dense Real worlds with 10% PLN. Plots show the Test Errors function of minibatch Adam iterations for a Siamese architecture with MLP (left) branches with 200 nodes per layer and with CNN (right) blocks with width k 47. The architectures details are given in Sec. 2.1
Two representative examples where the conjecture breaks are depicted in Fig 4 There, we show the median test error values on dense dataset of real- and ideal-world scenarios with 10% of PLN trained using MLP (left) and CNN architecture (right). We compare offline and online settings after the same number of training iterations. We observe that while both Ideal and Real test errors are affected by noise, this effect is exacerbated in the Real World scenarios. This can be understood because "fresh" samples bring more diversity to the model, improving generalization even if these new samples have noisy labels. Interestingly, we find that the online/offline correspondence for similarity tasks is influenced by the network architecture and the loss function choice. In particular, Fig T shows that the Real-world scenario for the MLP architecture diverges from the corresponding online case earlier (with less iteration steps) than the CNN case. Nevertheless, independently of the architecture-log matching, the equivalence between online and offline settings breaks down in the presence of label noise for all the scenarios considered. Similar behavior occurs for the sparse case as shown in Sec. G.5 in the supplementary material. There, we also present several additional comparisons between the architectures, losses and noise levels.
8

---

Under review as a conference paper at ICLR 2023
DIBS and modern contrastive learning. The Aimilarity-beaakii nature of PLN in dense datasets should not be underestimated as it may appear in widely employed settings. Modern approaches to supervupe contrastive learning (see the recent reviews of Ohri & Kumar (0221 Liu et al.
(202T); Jaiswal et al. (2020) Le-Khac et al. (2020) heavily rely on data augmentation to learn representations, TTan et al. (020). The massive use of data augmentation, however, may result in partial representation learning (feature suppression) or lead to semantic errors as in Purushwalkam
& Cuptt (2020). Moreover, as exposed in Huynh et al. (2022), if negative pairs are formed by sampling views from different images, regardless of their semantic information, this may lead to the appearance of false- negative pairs, potentially breaking transitivity and compromising the training efficiency. Interestingly, this skewness towards false-negative pairs is the same effect we find studying the asymptotic training error balance with DIBS (see discussion around Fig. 7 in the supplementary material). Despite these problems, data augmentation and random selection of negative samples are Therefore, several works in contrastive learning have focused on controlling the quality of augmented data and mitigating the effects of false negatives (see section 1.1 For this reason, feature extraction in self-supervised contrastive learning is usually affected by pair label noise by construction.
intrinsic to self-supervised methods R
4 DISCUSSION AND CONCLUSIONS
We move the first steps towards understanding generalization in similarity learning focusing on SNNs. To do so, we borrow the frameworks of DD and online/offline correspondence from classification tasks. We show that DD appearance is magnified in SNNs as it appears also in the absence of noise. Notably, we find that noise and the density of pairs in the training set crucially affect generalization. We present two kinds of noise: SLN, preserving similarity relations, and PLN, breaking transitivity. The same noise sources presented in this work can be easily generalized to models where the network input is given by multiple images. Studying DD, we show that comilarity-breakiiin asymptotic generalization performance (large training time) of the network in the overparametrized regime. Moreover, these effects get magnified at increasing training set density, preventing perfect interpolation. Studying the online/offline correspondence, we find that the generalization properties before overfitting time not sensitive to the density of the training set and only depend on noise. In particular, in the presence of noise the online/offline correspondence breaks down and the differences between the real and ideal generalization gap not universal and depend on the training setup.
noise compromises the
Limitations. This is an exploratory work that does not investigate all possible setups which may affect or lead to DD, such as regularization (see Nakkiran et al. (202bb); Mei & Montanari (2022)) )
epoch and sample-wise DD (see, Nakkiran et al. (2020a); Bodin & Macris 2021): Heckel & Yilmaz
(0200: Pezeshki et al. (202T). Moreover, we focus on the under- and over werrppaameetriee regime
without providing quantitative results about the interpolation threshold itself, a A'scolli et al. (2020a b);
Mei & Montanari (2022). This is because, to the best of our knowledge, there is no predefined way of treating SNNs analytically as no proxy model as Random Fourier Features (see Rahimi & Recht (2007)) can be constructed. Indeed, while in classification or regression tasks the output layer size is known, this not true for SNNs. For this reason, we believe that an analytic study of DD in SNNs may require some other approach, and we leave this study for future work.
Outlook. In the majority of modern contrastive learning works, the final graph of similarity relations in the dataset becomes really dense each training step involves multiple images a time. Moreover, from instance discrimination task examples, we know that contrastive learning tends to be affected by faulty positive and negative pair relations. This is the setting were we find that noise crucially impacts generalization. While the technological developments and the applications of contrastive learning kept on expanding during the last years, a fundamental study about how it generalises and reacts to noise is still missing.
'For example, in pretext task, the original image acts as an anchor, its augmentations act positive samples, and the rest of the images in the batch (or in the training data) as negative samples.
5 'Indeed, when two different images belonging to the same class of objects (sharing semantic features) are classified as different, convergence slows-down and semantic information gets lost. This goes under the name of instance discrimination task (i.e., the problem of discriminating pairs of similar points from dissimilar ones), and failing it can he harmful the formation of features useful for downstream tasks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving. Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Femaanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg. Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: www.wr.nssrrff...moommnm low, .ow..rg///.oo.oooo.o............ org/ from tensorflow.org.
Olah,
Software available
Ben Adlam and Jeffrey Pennington. The neural tangent kernel in high dimensions: Triple descent and multi-scale theory of generalization. In International Conference on Machine Learning, pp. 74-84. PMLR, 2020.
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O' O'Conoo,, and Kevin McGuinness. Unsupervised label noise modeling and loss correction. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 312-321. PMLR, 2019. URL http:///proceedings ml wttp.m.pree//////eeeeeeaeee html
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. arXiv preprint arxiv: 1906:0091000 00910, 2019.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine-
learning practice and the classical bias-variance trade-off. In Proceedings of the National Academy of Sciences of the United States of America vol. 116,32 (2019): 15849-15854, 2019. URL
https: // org/ wttp:///......aaaa.aaa.aaa 100..1/77//. 1903070116
Antoine Bodin and Nicolas Macris. Model, sample, and epoch- wise descents: exact solution of gradient flow in the random feature model. Advances in Neural Information Processing Systems, 34, 2021.
epoch-wise
Léon Bottou and Yann LeCun. Large scale online learning. In Sebastian Thrun, Lawrence Saul, and Bernhard Schölkopf (eds.), Advances in Neural Information Processing Systems 16 (NIPS 2003). MIT Press, Cambridge, MA, 2004. URL http:://een: www.oo...ttoooooooaoooooooeotaee
bottou-lecun-2000
Léon Bottou and Yann LeCun. On-line learning for very large datasets. Applied Stochastic Models in Business and Industry, 21(2):137-151, 2005. URL
bottou-lecus 2004a
Olivier Bousquet and André Elisseeff. Algorithmic stability and generalization performance. Ad- vances Neural Information Processing Systems, pp. 196-202, 2001.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. Signature verification using "siamese" time delay neural network. In J. Cowan, G. Tesauro, and Alspector (eds.), Advances in Neural Information Processing Systems, volume 6. Morgan- httprrrppp.ccpppprr////////1/1
Kaufmann, 1994. URL https: http://////ccceeeeeeccee 288ccOff022877ben htg.://0....777eeee449e433333e33
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR'05), volume pp. 539-546 vol. 2005. doi: 10.1..0//.... 202.
Ching Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. De- biased contrastive learning. Advances in neural information processing systems, 33:8765-889 8775, 2020.
10

---

Under review as a conference paper at ICLR 2023
Ching- Yao Chuang, R Devon Hjelm, Xin Wang, Vibhav Vineet, Neel Joshi, Antonio Torralba, Stefanie Jegelka, and Yale Song. Robust contrastive learning against noisy views. arXiv preprint arXiv:2201.04309 040000 2022.
G. Cohen, S. Afshar,, J. Tapson, and A. van Schaikf. EMNIST: an extension of MNIST to handwritten letters. 2017. URL https: /attps://githbb wwomhhosfordee /EMIIMISTTE
Stéphane d' Ascoli, Maria Refinetti, Giulio Biroli, and Florent Krzakala. Double trouble in double descent: Bias and variance (s) in the lazy regime. pp. 2280-2290, 2020a.
Stéphane d' Ascoli, Levent Sagun, and Giulio Biroli. Triple descent and the two kinds of overfitting: Where & why do they appear? Advances in Neural Information Processing Systems, 33:3058-3060 2020b.
B. Efron. Bootstrap methods: Another look at the jackknife. The Anmals of Statistics, 7(1):1-26, 1979. ISSN 00905364. URL http: h/wp://wwwwww... www.ristor.oo wtw.oorrrrrr//taaale//53333
Bradley Efron and Robert Tibshirani. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical science, pp. 54-75, 1986.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings. 2010.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances Neural Information Processing Systems, 33:21271-211 -22222 2020.
Beliz Gunel, Jingfei Du Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. arXiv preprint arXiv:201 1.0000../400 2020.
R. Hadsell, S. Chopra, and LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference Computer Vision and Pattern Recognition (CVPR 06), volume 2, pp. 1735-1742, 2006. doi: 10. 1911109/C...2000... 100.
Jiangfan Han, Ping Luo, and Xiaogang Wang. Deep self-learning from noisy labels. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 5137-5146 IEEE, 2019. doi: 10. 1,110//CCV.2229.0055244
URL pattps://dor org. 10.10000 11111119CCCCCCC wtw.Cccv.....1.0 2019. 00524]
Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference Machine Learning, pp. 1225-1234, PMLR, 2016.
Hrayr Harutyunyan, Kyle Reing, Greg Ver Steeg, and Aram Galstyan. Improving generalization by controlling label-noise information neural network weights. In Proceedings of the 37th International Conference on Machine Learning, ICML. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4071-4081 PMLR, 2020. URL http: /ttp:://rrccees proceedinggs... mlr. prwss seress/vvlii http//arutuuuyan2.....m html
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Reinhard Heckel and Fatih Furkan Yilmaz. Early stopping in deep networks: Double descent and how to eliminate it. arXiv preprint arXiv 10099, 2020.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le. Yonghui Wu, et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural information processing systems, 32: 10,-112.40 2019.
11

---

Under review as a conference paper at ICLR 2023
Tri Huynh, Simon Kornblith, Matthew R Walter, Michael Maire, and Maryam Khademi. Boost- ing contrastive supervvpdd learning with false negative cancellation. Proceedings of the IEEE/CVF Winter Conference Applications of Computer Viston, pp.
2022.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Mining on manifolds: Metric learning without labels. In Proceedings of the IEEE Conference Computer Vision and Patlern Recognition, pp. 7642-7651, 2018.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey on contrastive self-supervised learning. Technologies, 9(1):2, 2020.
Ashish Jaiswal, Ashwin Ramesh Babu, Mohammad Zaki Zadeh, Debapriya Banerjee, and Fillia Makedon. A survey contrastive self-supervised learning. Technologies, 9(1):2, 2021. 2021.
Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, and Diane Larlus. Hard negative mixing for contrastive learning. Advances in Neural Information Processing Systems, 33: 21798 21809, 2020.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in Neural Information Processing Systems, 33:18661-18673 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Phuc H Le-Khac, Graham Healy, and Alan F Smeaton. Contrastive representation learning: A framework and review. IEEE Access, 8: 19999907-199934, 2020.
Junnan Li, Yongkang Wong, Qi Zhao, and Mohan S. Kankanhalli. Learning to learn from noisy labeled data. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA. USA, June 16-20. 2019, pp. 5051-5059, Computer Vision Foundation IEEE. 2019. doi: 10.1109/CVPR.2019 00519. URL http: Learning to Learn From Noisy Labeled Data CVPR 2019 paper.ht..
httphttt////peaccsssss.s thecvf. com com/content CVPR 2019/htm1/Li
Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang. Self- supervised learning: Generative or contrastive. IEEE Transactions on Knowledge and Data Engineering, 2021.
Marco Loog, Tom Viering, Alexander Mey, Jesse H Krijthe, and David MJ Tax. brief prehistory of double descent. Proceedings of the National Academy of Sciences, (17222)::0225-000 2020.
Song Mei and Andrea Montanari. The generalization error of random features regression: Precise asymptotics and the double descent curve. Communications on Pure and Applied Mathematics, 75 (4):667-766, 2022.
Pedro Morgado, Ishan Misra, and Nuno Vasconcelos. Robust audio-visual instance discrimination. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12945- 2021.
Preetum Nakkiran, Gal Kaplun, Yamini Bansal, Tristan Yang, Boaz Barak, and llya Sutskever. Deep double descent: Where bigger models and more data hurt. In International Conference Learning Representations, 2020a. URL
net
Preetum Nakkiran, Prayaag Venkat, Sham Kakade, and Tengyu Ma. Optimal regularization can mitigate double descent. arXiv preprint arXiv:2003.01897, 2020b.
Preetum Nakkiran, Behnam Neyshabur, and Hanie Sedghi. The deep bootstrap framework: Good online learners are good offline generalizers. 9th International Conference on Learning Rep- resentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
httts:://ppentt iew. net. www.:/oororooooooo fot.://ooormmm
Kriti Ohri and Mukesh Kumar. Review on self-supervised image recognition using deep neural networks. Knowledge-Basse Based Systems, 107090, 2021.
12

---

Under review as a conference paper at ICLR 2023
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03748, 2018.
David Page.
How to train your resnet. 2018
URL
https: /tpp:://mytlle.e myrtle.
how train your resnet- architecture/
Mohammad Pezeshki, Amartya Mitra, Yoshua Bengio, and Guillaume Lajoie. Multi-scale feature learning dynamics: Insights for double descent. arXiv preprint arXiv:2/ 2021.
Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. Advances in Neural Information Processing Systems, 33:3407-3418, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Joshua Robinson, Li Sun, Ke Yu, Kayhan Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in Neural Information Processing Systems, 34, 2021.
Hwanjun Song, Minseok Kim, Dongmin Park, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. CoRR, abs/2007.08 .99, 2020. URRLAttps:///xxxxv
Bbs/2007. 08199
Stefano Spigler, Mario Geiger, Stéphane d' Ascoli, Levent Sagun, Giulio Biroli, and Matthieu Wyart. A jamming transition from under-to over wer-paramettrzztioon affects generalization deep learning. Journal of Physics A: Mathematical and Theoretical, 52(47):474001, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pp. 776-794. Springer, 2020.
Feng Wang and Huaping Liu. Understanding the behaviour of contrastive loss. In Proceedings of the IEFF/CV conference on computer vision and pattern recognition, pp. 2495-2504, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms. 2017. URL https://qithut comm
za andoresearch/ wwwhcrafaiiiooooooo mnist
Rui Zhu, Bingchen Zhao, Jingen Liu, Zhenglong Sun, and Chang Wen Chen. Improving contrastive learning by visualizing feature transformation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10306-100 10315, 2021.
13


Reviews:
[{"review_id": "Y5Hlb8-r8EX", "reviewer": "Reviewer_JwMY", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper empirically investigates the phenomena of double descent and offline-online correspondence (the gap between test errors of offline and online learners, as far as I understood) for similarity learning. The authors introduce two types of dataset conversion procedures from data points to pairs and two types of noise-generating processes to see how these factors affect generalization in similarity learning. These data-generating processes are expected to imitate well actual procedures in contrastive learning with data augmentation and negative sampling. After all, the authors discover for double descent that a type of noise---called pair label noise (PLN)---affects generalization more significantly because it breaks the label \"transitivity,\" making the interpolation/overfitting harder. In addition, the authors reveal that offline learners are more prone to label noise than online learners in the empirical study of offline-online correspondence.", "summary_of_the_review": "I think this paper reveals an interesting contrast between the behavior of SLN and PLN in regard to the double descent phenomena. Nevertheless, several parts of the paper would not be sufficiently clear so the scope of the paper is not easy to grasp. There would be room for improvement in the structure and logical flow of the paper as suggested in \"Clarity, Quality, Novelty And Reproducibility\" section.", "strength_and_weaknesses": "### Strengths\n\nThis work extends the concept of double descent from well-studied classification and regression to similarity learning. Many recent deep learning training procedures partly rely on self-supervised pre-training models that leverage contrastive learning, hence it is essential to understand more what kind of regime goes on under the over-parametrized regime in similarity learning. The authors provide one perspective to this answer to reveal that pair label noise has a severe effect on the generalization aspect. This should be an interesting perspective for researchers studying generalization.\n\n### Weaknesses\n\nThe main drawback of this paper is the unclearness of the motivation and writing. I provide several writings that could be improved in the \"Clarity, Quality, Novelty and Reproducibility\" section and focus on the motivation aspect here.\n\nAlthough research on double descent/interpolation/over-parametrization regime has been gaining attention in the recent community, I could not understand well what is the motivation of this work. In the study on double descent, we are primarily interested in how model performances degrade and improve as the model size increases, and there should not be label noises. Specifically, we usually consider the data-generating process $Y_i = f(X_i) + \\\\epsilon_i$ for regression and treat $\\\\epsilon_i$ as \"white noise,\" but do not consider \"more drastic (label flipping) noise\" to purely focus on the clean generalization aspect. The current study seems to entangle labeling noise and the generalization aspect, so it is not sufficiently clear that the authors would like to focus on generalization or the effect of the noise.\n\nIn addition, the authors introduce two types of noise models (SLN and PLN) and two scenarios (sparse and dense connections). However, I did not understand well the reason why the authors introduce them. Are these introduced to imitate the real-world data-generating process in similarity learning? Or, are these introduced to make the data-generating process more scalable (in particular, the sparse connection)? This part makes me puzzled so I'm not sure the authors would like to study/reveal a certain phenomenon or propose a new data sampling procedure.\n\nAs for SLN and PLN, if we follow the initial motivation---to study the generalization aspect of contrastive learning with data augmentation and negative sampling---then these noise-generating processes could be overly simple. At least, in real-world scenarios, SLN and PLN could happen simultaneously. Why do we consider these two noises independently?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Nr2fVrfK34I", "reviewer": "Reviewer_egc1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies how deep networks generalize the concept of similarity in the presence of noise. Specifically, two phenomena are studied (1) double descent behavior and (2) online/offline relations. The double descent behavior also exhibits in the cases of contrastive learning with noise. Besides, the equivalence between online optimization and offline generalization is probed. An extensive empirical study is provided to justify the paper's claims. ", "summary_of_the_review": "This paper studies an important problem. Extensive empirical evidence is provided. However, at the present stage, there are still some unclear explanations that need to be addressed. Therefore, the rating is \"5\" before the rebuttal.", "strength_and_weaknesses": "**Strengths**\n- The research problem is interesting and significant. \n- Experiments are sufficient to support claims. \n\n**Weaknesses**\n- The main contributions of this paper are unclear. \n- Technical novelty is somewhat limited. \n- Both writing and organization should be improved to reach the requirements of a top-tier conference. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NRz5ivcdAWJ", "reviewer": "Reviewer_UiBx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies generalization in similarity learning with noisy labeled data focusing on SNNs. Empirical studies were designed to show that dataset topology crucially affects generalization under SLN and PLN, and both the network architecture and the loss function choice can disturb an online/offline correspondence for similarity tasks.", "summary_of_the_review": "To sum up, the contributions of this paper are limited. It would be better if the authors can provide some theoretical analysis and refine the writing of this work.", "strength_and_weaknesses": "This work studies how NNs generalize the concept of similarity in the presence of noise by investigating two phenomena: Double Descent (DD) behavior and online/offline correspondence, which is interesting.\n\nBelow are some concerns:\n1. The authors stated that it is widely known that larger models (with more parameters) usually obtain better generalization. However, in my point of view, over-parameterized models usually have the overfitting problem, leading to bad generalization ability.\n2. What are the limits in `this framework connects under- and overparameterized limits.'\n3. I am confused about this sentence `the training dataset size dictates the two regimes instead of the number of parameters'. Do you mean ``the training dataset size instead of the number of parameters dictates the two regimes''?\n4. What do you mean by ``online and offline test soft-errors match each other ''?\n5. I don't see why DD and online/offline correspondence are two **complementary** approaches in terms of generalization. Can you provide some explanation on the **complementary**?\n6. ``DD and online/offline correspondence were mainly applied to classification and regression, but, if true, they should also hold for other tasks such as similarity learning.''. What do you mean by `if true'?\n7. What is DIBS? What do the gray shaded areas stand?\n8. So many special terms are mentioned without definition, e.g., effective noise, which make it hard for readers to understand this paper.\n9.  The density of this graph is defined as $\\left|N_{\\text {pairs }}\\right| /\\left(\\begin{array}{c}\nN \\\\\n2\n\\end{array}\\right)$ \", which tells us how much information we have about the input images. The density is proportional to $1/N$. How do you maximize the information by constructing all possible pairs (increasing N)?\n10. In scenario 1, how do you determine $N_{sample}$?\n11. Can you provide some theoretical analysis on the generalization in similarity learning with two kinds of label noise under two scenarios in terms of DD and online/offline correspondence?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject