The Dark Side of Invariance: Revisiting the Role of Augmentations in Contrastive Learning
Authors: Alex Tamkin
Margalit Glasgow
Xiluo He
Noah Goodman
Venue: Submitted to ICLR 2023 (2023)
Keywords: contrastive learning
self-supervised learning
feature suppression

Abstract: What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations are often crucial in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. for digits superimposed on photographs, predicting the class of one vs. the other). We find that Viewmaker Networks, a recently proposed model for learning augmentations for contrastive learning, produce label-destroying augmentations that stochastically destroy features needed for different downstream tasks. These augmentations are interpretable (e.g. altering shapes, digits, or letters added to images) and surprisingly often result in better performance compared to expert-designed augmentations, despite not preserving label information. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models

Under review conference paper at ICLR 2023
FEATURE DROPOUT: REVISITING THE ROLE OF AUGMENTATIONS IN CONTRASTIVE LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that laste-des augmentations can be useful in the foundation model setting, where the goal is to learn diverse, ceneral-purposs representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. for digits superimposed photographs, predicting the class of one vs. the other). We find that Viewmaker Networks, recently proposed model for learning augmentations for contrastive learning, produce label-destroying augmentations that stochastically destroy features needed for different interpretable (e.g. altering shapes, digits,
downstream tasks. These augmentations letters added to images) and surprisingly often result in better performance compared to expert-designed augmentations, despite not preserving label information. To support our empirical results, we theoretically analyze simple contrastive learning setting with linear model. In this setting, label- -esttroing augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.
1 INTRODUCTION
In recent years, foundation models (Bommasani et al. (021) have exhibited remarkable progress on a range of AI tasks (Devlin et al., 2019 Liu et al. |2019: Ramesh et al. |021 Radford al.] [021] Brown et al. 2020 Chowdhery et al. 2022] Hoffmann et al. 2022 Alayrac et al. 2022 Reed et al. 2022). A crucial characteristic of foundation models is that they can be adapted for range of downstream tasks. For example, a foundation model trained on ImageNet should ideally not only perform well at object classification, but should also have learned general features useful for localization, segmentation, and other visual tasks. Indeed, this is borne out by recent work showing the high accuracy of foundation models on a range of downstream tasks (Chen et al. 2020b), as well as range of analysis work showing models learn high-level semantic features including texture, color, pose, and style (Goh et al. 2021).
One popular strategy for training foundation models involves training models to match transformed versions (known as views or augmentations) of the same input. For example, image views might include common data augmentations such as cropping or color jitter (Chen et al.
2020b), while views for speech might include pitch 2021: Park et al. 2019). This family of objectives includes contrastive approaches such as SimCLR and MoCo, as well as non-contrastive approaches such as
modulation spectrogram masking (Kharitonov al.
BYOL and AW (Chen et al.) 2020b; He et al. 2020 Grill et al. 2020 Caron et al. 2020).
Given the central importance of these views for defining the self-supervised task, much work has focused on the question of what views lead to high-quality representations. The prevailing consensus, exemplified by

---

Under review conference paper at ICLR 2023
(Tian et al. 2020), holds that views should be preservigg with respect a downstream task. In other words, because the contrastive loss will produce representations which are invariant to features that vary across views, any information we wish to preserve in the representations should not be altered by such views. As Tian et al. (2020) write: "A good set of views are those that shure the minimal information necessary to perform well at the downstream task."
Here, we question whether this assumption-in particular, with its focus on a single task enough to explain why contrastive foundation models succeed on range of downstream tasks. In Section that the actual choice and application of views in practice does not align with this prevailing consensus. For example, complete invariance to several common data augmentations (c.g. shifts in brightness or cropping) is undesirable since augmentations of inputs from different classes can collide. Furthermore, in many cases there are explicit ways to specify invariances (e.g. converting images to grayscale) that researchers avoid in favor of specifying them indirectly via augmentations (e.g. hue shifts). These observations suggest that specifying invariances is not the sole role of these views.
2 we observe
Instead, we suspect that augmentations serve as form of feature dropout--peevnttig any one feature from becoming a shortcut feature and suppressing the learning of other features. We study this idea empirically in Viewmaker Networks, recently proposed method that appears to learn drop out different features in the input via adversarial training. We apply viewmaker and expert views to datasets with two associated down- stream tasks, one involving classifying the main input (e.g., an image or audio recording) and one involving a simple overlaid element (e.g., digit, shape, letter, or speech snippet). We observe that the viewmaker augmentations selectively obscure these overlaid features. Despite this, the viewmaker representations still learn both downstream tasks well, while expert views often struggle on one or the other. This further suggests that being label-pree pressrving is not a necessary property of good views, as long as the label information is still sometimes accessible.
Finally, we formalize the intuition that feature dropout can aid learning with a theoretical analysis of a simple linear contrastive setting. In this setting, characterize how the noisiness of each feature directly determines how quickly features are learned, and uncover an interaction between features governing how fast they are learned. In particular, and show that adding noise to the "easiest" feature can increase the rate at which other features are learned. This further indicates that label-destroying augmentations may have direct role in ensuring that contrastive models learn a broad range of features for downstream tasks.
show how learning one feature quickly can suppress the learning of other features,
Overall, these findings suggest the need revisit common assumptions about the role of augmentations for contrastive learning in the foundation model setting, and move towards a better understanding of how to train generalist models that learn diverse features from unlabeled data.
2 COMMON PRACTICES ARE AT ODDS WITH THE "INVARIANCE" EXPLANATION
We begin by briefly exploring several common augmentations used in contrastive learning for natural images, and explore how they come into conflict with the common assumption described above. First, we observe that many common augmentations can affect the label of the input, depending on the downstream task. For example, many downstream image recognition tasks require color information (e.g. identifying bird species) or brightness (e.g. scene or time-of-day classification), implying that invariance to these characteristics would be undesirable. Yet hue shifts, greyscaling, and brightness shifts are common augmentations used in contrastive learning Chen et al. (2020b); He et al. (2020)
Second, repeated application of some augmentations causes challenges for all downstream tasks. For example, applying brightness shifts repeatedly results in any image turning completely black or completely white. Thus the class label cannot be truly invariant to this augmentation, since inputs from different classes
2

---

Under review conference paper at ICLR 2023
experience an "augmentation collision" at this black or white image (this is formalized in Appendix B This argument also applies to other augmentations, including shifts in contrast and random masking.
Third, some augmentations are commonly used despite ways of explicitly encoding invariance to them. For example, two image augmentations are hue shifts and greyscaling. Invariance to both of these augmentations can be explicitly encoded by always converting an image to greyscale. Yet doing so not common practice because color information is still desirable for many downstream tasks.
The contradictions between the invariance rationale for augmentations contrastive learning and these common practices suggest the need for additional explanations for the role of augmentations.
3 VIEWMAKER NETWORKS SUCCEED DESPITE DESTROYING LABEL. INFORMATION
As another point of evidence that good views need not be label-preserving, consider the behavior of viewmaker networks (Tamkin et al. 2021b), a generative model which produces augmentations for contrastive learning. Intuitively, viewmakers learn stochastic augmentation policy that makes the contrastive task hard as possible for the encoder. The stochastic augmentations are parameterized as additive perturbations bounded by L1 norm, meaning the viewmaker can alter but not completely destroy the original image.
Formally, given an input E N, a viewmaker network V₂ is trained jointly with an encoder Eg to optimize the minimax expression:
81)
Vele, 62)
max min max Eo x V 81)111
Egg x
V 0
[[V 82)
Here multiview loss function (e.g- (Chen al. 2020b; He et al.) 2020 ), a minibatch of inputs, II II is the L norm, is the distortion budger controlling the strength of the views, and é1, 82 N(0, 1) are random inputs that enable the viewmaker to learn stochastic augmentation policy. We clamp the output of the viewmaker for images to [0, 1] as in Tamkin et al. (0221b).
Viewmaker networks learn to stochastically alter different parts of the input, including task-relevant features, meaning that these augmentations not label-preserving. Nevertheless, networks enable strong performance on multiple downstream tasks, including often better performance than expert-designed augmentations. Moreover, this feature dropout capability of viewmaker networks may help them to learn many features well rather than focusing on the easiest ones.
will see shortly, viewmaker
3.1 DATASETS
We consider the behavior of viewmaker networks on four datasets, including three image and one audio dataset. Each dataset is constructed in such way as support two distinct downstream classification tasks, enabling us to examine how well each downstream task is learned. The presence two downstream tasks enables to analyze the foundation model setting where wish to learn features relevant for multiple downstream tasks, as opposed to one set or the other.
Image datasets The three image datasets are based on the canonical CIFAR-10 image-recognition dataset (Krizhevsky) 2009, (MIT-License). One task is always to predict the CIFAR-10 object label (e.g. airplane or bird). The other task is dependent on an additional feature overlaid on the image: C+Shapes: The CIFAR- 10 image is overlaid with one of three canloml shapes: square, triangle, circle.
Note that invariance is not to be confused with the related but distinct property of equivariance, often discussed as desirable property of network architectures (e.g. see Fukushima & Miyake (1982);(Cher et al. (2020a))
"Continuous reduction in contrast eventually produces single-color images, given finite precision images
3

---

Under review conference paper at ICLR 2023
The second task is to predict what shape was overlaid (N=3 classes). C+Digits: The CIFAR-10 images are overlaid with four copies of a randomly-sampled digit from the MNIST dataset. The second task is to predict the digit class (N=10 classes). C+Letters: The CIFAR- 10 images are overlaid with four copies of a randomly-colored English letter. The second task is to predict the class of the letter (N=26 classes).
Audio dataset The audio dataset is created by overlaying the audio of spoken digit (from the ANSI dataset (Becker et al., 2018), MIT License) with a random background sound (collected from one of three possible classes: cafe, machinery, and traffic) (Saki et al. 2016 Saki & Kehtarnavaz (016). The tasks are to predict the digit class (N=10 classes) and to predict the sound class (N=3 classes). Inputs are presented to the network as log mel spectrograms.
3.2 EXPERIMENTS
Pretraining We pretrain with the SimCLR algorithm for 200 epochs with a batch size of 256 and a temperature of 0.1. We use a ResNet-18 model with standard modifications for smaller inputs (including a smaller stride and no initial maxpool) as used in Tamkin et al. (202Tb). For the expert augmentations, we use the standard SimCLR augmentations for the image datasets (Chen et al., 2020b), and the SpecAug (Park et al., 2019) augmentations for the audio datasets, which randomly mask out different frequency and time bands, well as the WaveAug (Kharitonov et al. (202I) augmentations, which alter various properties of the waveform such as the pitch and speed. For the viewmaker augmentations, we use a budget of c = 0.05 for the image datasets, and 0.125 for the audio datasets, where P is the number of pixels in the input.
Linear Evaluation We evaluate the quality of the learned representations by training a linear softmax classifier on Top of the prepool representations. We train for 100 epochs, using the same parameters as Viewmaker (Tamkin al. 2021b), training separate linear classifiers using the same pretrained network for each downstream task (Chen et al. 2020b). Augmentations are applied during training but not evaluation.
3.3 RESULTS
Qualitative evidence of feature dropout Visually, the viewmaker augmentations seem to stochastically alter different aspects of the input, as shown in Figure [ In addition to modifying the background of each input, the viewmaker also selectively modifies the additional synthetic features added each domain: C+Digits: The viewmaker augmentations selectively add pixels to the MNIST digits, making it difficult to distinguish which number is present. C+Shapes: The viewmaker augmentations sometimes draw squares around the shape in the center, making it difficult to determine the shape class. C+Letters: The viewmaker draws letter-like markings top of the letters, obscuring the letter identity and color. Audio: The viewmaker identifies the narrow band corresponding to the speech and applies perturbations to it. As can be seen in Figure Il these label-destroying augmentations are quite common, occuring sizeable fraction of the sampled views.
Quantitative evidence of feature dropout We also measure this selectivity of features quantitatively in Appendix [C.2 and Figure 4 We augment images 1,200 times and observe the impact on the predictive probability of the correct object class. Two clear modes appear for the viewmaker augmentations, but not expert augmentations. This corresponds to the fraction of time the viewmaker destroys the overlaid feature information (low P(correct object class)) and preserves it (high P(correct object class)).
Viewmaker succeeds despite destroying label information As shown in Table and Table 2 viewmaker networks are able to achieve good accuracy on both tasks, while expert augmentations frequently achieve lower performance one both tasks. For example, on the image tasks, while expert views achieve slightly
4

---

Under review conference paper at ICLR 2023
BB
B
B
B
B
BB
Figure Comparison of viewmaker and expert augmentations on datasets with multiple features. The viewmaker augmentations adapt to the particular semantics of the input data, and make targeted perturbations which remove the class-relevant information of the synthetic features (e.g. occluding the digit, shape, letter, or speech). Despite this, the encoder network is still able to learn strong representations. Rows (from top): Digits, Shapes, Letters, Audio. Columns (from left): Expert augmentations, viewmaker augmentations, difference between original and viewmaker augmentation, rescaled to [0.1]. Center image in each grid is the original. Audio Expert views shown are Spectral views.
5

---

Under review conference paper at ICLR 2023
Viewmaker (CIFAR-I0) 10)
Expert (CIFAR-10)
Viewmaker (Object)
Expert (Object)
CIFAR- 10 Only
84.5
86.2
C+Shape
79.8
76.0
100.0
100.0
C+Digit
69.3
58.8
94.3
86.7
C+Letter
71.9
74.8
96.9
94.1
Table I: Transfer accuracy different features. Viewmaker networks are able to achieve good performance across multiple downstream tasks, while expert views sometimes falter. Networks are pretrained on the datasets on the left, and transfer accuracy is reported for the different conditions on the columns. Runs averages of three seeds (with the exception of CIFAR-10 Only, which taken from (Tamkin et al. 2021b)
Speech Accuracy
Background Sound Accuracy
Viewmaker
Spectral
Waveform
Viewmaker
Spectral
Waveform
Speech Only
92.4
97.0
76.7
Bkgd. Sound Only
100.0
32.64
100.0
Speech + Sound
60.8
10.1
53.6
97.0
47.2
43.3
Table 2: Audio transfer accuracies. Viewmaker networks achieve good performance across multiple tasks, while expert views sometimes suffer catastrophic drops as another feature added. Networks are pretrained the datasets the left, and transfer accuracy is reported for the different conditions the columns. Runs are averages of three seeds.
higher performance the image only, they experience large drop in accuracy when the synthetic feature is added. In two of these cases (Shape and Digit) the viewmaker models are able to achieve a higher accuracy on both the image and the synthetic feature, while the third (Letters) viewmakers achieve slightly lower accuracy on the images but achieve half the error on the synthetic object. For the audio experiments the picture is similar-the the viewmaker is able to avoid catastrophic drops in performance learning both features together, achieving the highest accuracy on both, while the expert views experience larger drops and worse overall performance. Note that the high performance of expert views for our control tasks (CIFAR- (CIFAR-1O/Speechl) 10/Specch/Sound Only) indicates that the viewmaker views are not merely better all-around views, but that they specifically help the model learn multiple features.
These results provide additional evidence that babee-pres views are not necessary for learning good representations--and that the ability to perform feature dropout may improve learning of multiple tasks.
4 THEORETICAL ANALYSIS OF FEATURE INTERACTIONS IN LINEAR CONTRASTIVE SETTING
this section, we theoretically analyze a simple linear model that captures the essence of how label-destroying augmentations can improve downstream accuracy. We study a setting where the data contains many underlying features that are relevant to downstream classification tasks, and where these features are preserved to varying degrees across augmentations. We will show that linear model trained with contrastive objective learns these features, and that adding noise to one feature can speed the learning of other features during gradient descent. One difference between the linear setting theoretically analyze and Section is that in this section we add stochastic Gaussian noise to destroy features across augmentations, as opposed to the more bimodal feature dropout behavior in Figure
6

---

Under review conference paper at ICLR 2023
PAR
Representation fa(w)
01
0 W22
12
θ1
O2
8x
Wi
W1S
P211
1128
W
WK'
WK3
Input
(a)
(b)
Figure 2: We show how label-destroying augmentations can aid learning of other features in linear contrastive setting: (a) The correlation of the kth feature of an augmentation pair, shown for d 2. Each pair ug (1)
(8)
and ukk have correlated projections onto the ground truth μı direction, representing the feature conserved across augmentations. (b) Feedforward linear network which computes the representation (w).. As each feature μą is learned (02 + p&) the representations of the two views fe (w((),, fe(u()) become more similar, decreasing the contrastive loss.
4.1 DATA MODEL AND SETTING
We study a model which consists of data with distinct features, each corresponding to some ground truth unit-vector directions f1,....,M MK Bdd We sample each data point R and its augmentation (a.k.a. its positive pair or its view) v E R K as follows. For k E I,...,,, K, the kth row of u, which we denote ug- is drawn from the Gaussian distribution (0, Id). The kth row of the augmentation, Uk. drawn from the same distribution, but is correlated with ug in the μ4 pa-direction (and is otherwise independent in the other directions). The strength of the correlation governed by parameter ag E [0, in the following sense: oI pu = agug μu + /1 S. where & (O, 1). Thus the larger dg, the stronger the correlation that feature across the two views. Figure a)) visualizes the correlation of (ux, U&) in augmented pair. Formally,
La we can write that (ux, Uk) ~ N/ ORPAPE
ausuaa Ta )) for vector a E [0, 1]..
We will learn model θ E R Rxxd, which represents collection of K feature extractors, as pictured Figure bb... The model Ô, with rows {0x [0=]ee[K) maps data point w E R. K'dd to representation fe(w) E x by computing a score w 0g for each element the representation. That is, (fe(w)) = w Ox.
Our goal is that the model θ will be useful for downstream classification task which depends on the ground truth features. A good representation will capture ground truth features that are correlated across augmentations, such that 6₁ is aligned with μ* -4A-
Training. We will study the the evolution of as we optimize a standard constrictive learning objective using gradient descent (Dosovitskiy et al. 2014 Chen al. 2020b). At each round of gradient descent, we sample fresh batch of m data points and their augmentations, (U, V) = {(((), 2(6) he(mm] For each
i E [m]). we compute a similarity score Zij := ())( fe(v0)) = Σ. (of u (6) product of their K K-dimensional representations. We then compute the logits Pıj = Σ, exp(2x)) using the
(0f
T
Uk (i) using the dot
exp(zr))
softmax function, and use the classwise cross entropy loss function L(O; U, V) - -og(pa))
7

---

Under review conference paper at ICLR 2023
4.2
MAIN RESULT
We will study gradient descent (GD) the cross entropy loss, and consider how adding noise feature affects the learning of the other features. As suggested earlier, we can measure how well we learn the kth feature by measuring the alignment of 0g with μ4 -Aa- A natural way to measure this alignment is the [ 0z1 (ln]0el)
acute angle between +/i and 0g. given by arccos
||0= Lemma E.1 in Appendix proves that this
quantity directly determines the test accuracy on a natural downstream lnnear classification task.
Formally, we say we add noise to some feature k' of a data point v, if for some 3 E (0,1), we let ön Bew + -1- where { ~ (O, Ta), and üg us for k + K. Thus if (u, v) were a pair generated with the correlation coefficients REK)) then the distribution of (u, 6) comes from the modified correlation coefficients {ā] with the single modification d&g = Bag. We now present our main theorem:
Theorem 4.1 (Noise improves feature learning). There exists a universal constant C, such that the following holds. Let ((++1) = O( - (VV V) + AO("),, and ((++)) = ((+)) - (VVL(U () + 10 where V is V with any amount of added noise in the k feature. This has the effect of changing agr to du
for any oy < ag. Then for any k + K. if 10% pxl S '-022 11xxll. ||0 11 = 10% and ||0x|1 2
cx(1-93)
[ 0 1102 112
then for a small enough step size n, Eu,y
arccos
[10 1la Eu,
arccos
We briefly comment on the three assumptions on θ in the theorem. The first assumption, 10% /4 S -- |0&| requires that 0% is not too aligned with μt that is, the result applies to all features l that aren't wen't already learned too well. The second two assumptions are satisfied if the k'th feature has been learned to some extent, and the norm of 0g and fg are small, which can be enforced throughout training with & regularization.
The theorem guarantees that at any point in training, we add noise to the k'th feature, the next step of GD learns all other features better than if we didn't It add noise. To validate the implication of this result for the complete trajectory of GD, we include simulations in Appendix [ Our experiments show that introducing noise part-way through training to dominant features can significantly speed the alignment of weak features, with only small cost to the alignment of the dominant features. We prove our result in Appendix including intuition and a technical overview of the steps in Section E.3
5 RELATED WORK
Understanding contrastive and multiview learning Many prior works have laid the foundations for current
contrastive and multiview learning algorithms (Becker & Hinton| 1992 Hadsell al. 2006 Dosovitskiy
let al.
2014 Wu et al. 2018, Bachman et al.) | Misra & van der Maaten 2020: He et al. 2020 Chen
ct al. 2020b Several works perform analysis studies of contrastive learning to identify important factors
(Cole et al. 2021 Zhao et al. 2021) or how contrastive models differ from supervised learning (Yang et al.
2020 Ericsson et al. 2021a] Karthik et al. 202T). HaoChen al. (202T) study contrastive learning using the concept of an augmentation graph. This model assumes the fraction of non-label preserving augmentations is "extremely small;" interestingly, we show in practice it can quite large and still yield good performance. [aang ett al. (202)) theoretically study contrastive learning under an assumption of label-preserving augmentations, though they show that such an assumption alone does not suffice to learn. Most relevant to our work, Tiin et al. (2020); Ericsson et al. (2021b) study how the information shared between different views impacts learning of downstream tasks. We complicate this picture by analyzing the foundation model setting where a single model must learn features that are useful for multiple tasks that are not known in advance. In this setting, we find that label-destooying perturbations, thought to be harmful by Tian et al. (2020). are useful for preventing one feature from suppressing others.
8

---

Under review conference paper at ICLR 2023
Feature suppression Our work is closely connected to the notion of feature suppression (Hermann & Lampinen 2020). where the presence of one feature can crowd out or suppress the learning of other features. Several works have explored the relevance of this concept in contrastive learnings. For example, the original SimCLR paper (Chen et al. 2020b) noted that color jitter augmentation was necessary to prevent the network for using only the color profile of the input to solve the contrastive task. Followup work (Chen et al. 2021) explores this phenomenon in more detail, characterizing how different hyperparameters and dataset features affect feature suppression. Other works have attempted to address feature suppression in contrastive learning, either via auxiliary losses (Li et al. 2020) or by modifying representations in the latent space (Robinson let al. 202T). Our work relates to these in two ways. First, we empirically and theoretically investigate feature suppression as alternate rationale for the role of augmentations, as opposed to invariance. Second, we show that an existing method, viewmaker networks (Tamkin et al. 2021b). can identify and potentially neutralize suppressing features in an interpretable way, resulting in better performance than expert augmentations.
Spurious correlations and shortcut features Outside the framing of feature suppression. several other works explore how classifiers can learn or make use of unwanted features. Shorteut features (Geirhos et al. 2020) describe often-simple features (e.g. the average color of an input) which are learned by networks the expense of more salient features (e.g. the object class). This notion is connected to spurious correlations
(Simon 1954) in deep learning which have been explored extensively (Sagawa et al. 2019 2020 Srivastava
et al. 2020 Tu et al. 2020 Xiao et al. 2021), including in the context of self-supervised learning (Minderer
al. 2020 Other works have also performed theoretical analysis of how related dynamics affect learning in
the supervised setting (Li et al., 2019 [Shah et al. 2020). Our work suggests that viewmaker networks may be a useful tool as well here-both an interpretability tool to visualize the different features network relies on, and as a way to reduce reliance on particular features without completely destroying the information.
6 DISCUSSION AND CONCLUSION
We have presented several different arguments against the commonly-articulan belief that the main role of augmentations is to specify invariances for contrastive learning model. First, common augmentations such as brightness shifts would result in useless representations if networks became truly invariant to them. Second, viewmaker networks succeed at contrastive learning despite learning lestr-dgg perturbations which drop out different features in the imput. Finally, we present an analysis of a linear contrastive setting where we prove that lesto-dg views actually have a positive effect on contrastive learning if the goal is to avoid learning one feature at the expense of others.
Our work has limitations. For example, our empirical analysis is limited to four synthetic datasets spanning vision and audio, whereas self-supervised learning may be applied to naturalistic data spanning much wider range of modalities (Tamkin et al.) 2021a 2022). In addition, our theoretical analysis considers linear contrastive setting, whereas current neural networks are highly nonlinear. Improving upon both of these fronts is an exciting area for future work.
On the other hand, understanding augmentations dropping out casy features suggests possible ways of improving the performance of self-supervised learning. For instance, viewmaker networks cap the extent to which views can differ from the underlying image. Our analysis here suggests the role of this cap indirectly sets the dropout rate of different features in the input; some way of directly encoding this objective may yield more flexible and performant viewmaker approaches.
The challenge of learning a broad range of useful features lies at heart of self-supervised learning. We hope our work sheds light on this challenge in contrastive learning, especially these objectives continue develop and are applied more broadly and at larger scale.
9

---

Under review conference paper at ICLR 2023
Ethics Statement Our work is centered on conceptual understanding, making it challenging confidently predict societal impacts. Better conceptual understanding of existing methods may help us understand the failure modes and successes of current models better, which may have positive impacts. However, if this understanding enables the development of more powerful methods, the work may indirectly accentuate whatever social impacts (positive or negative) those applications have.
Reproducibility Statement We include hyperparameters and experimental settings for our experiments in Section g complete statements of our theoretical results in Appendix E and will release our codebase to reproduce our results.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. ArXiv, abs/2204. 14198, 2022.
Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
Soren Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. ArXiv, htss/880.00318, 2018.
Suzanna Becker and Geoffrey E. Hinton. orgf-orggni neural network that discovers surfaces in random-do stereograms. Nature, 355:161-163, 1992.
Rishi Bommassni, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren F. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hssu,Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning. Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitrion, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, 0at./2200.02255 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
10

---

Under review conference paper at ICLR 2023
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski. and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-999 9999994422442..-- 2020.
Shuxiao Chen, E. Dobriban, and Jane Lee. A group-theoretic framework for data augmentation. arXiv: Machine Learning, 2020a.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020b.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neurul Information Processing Systems, 34, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Baindoor Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Mísra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brenman Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.0231 2022.
Kathleen S.
Elijah Cole, Xuan S. Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge J. Belongie. When does contrastive visual representation learning work? ArXiv, abs/2105.058877. 2021.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NIPS, 2014.
Linus Fricsson, Henry Gouk, and Timothy M. Hospedales. How well do self-supervised models transfer? 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5410-5419, 202,
Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. ArXiv, abs/211 11//2111.11111 113998 2021b.
Kunihiko Fukushima and Sei Miyake. Neocognitron: self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pp. 267-285. Springer, 1982.
Robert Geirhos, Jörn-Henri Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. ArXiv, abs/2004.07780, 2020.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Christopher Olah. Multimodal neurons in artificial neural networks. 2021.
Ccan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems, 33: 22224,,, 2020.
11

---

Under review conference paper at ICLR 2023
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR (06), 2: 1735-1742, 2006.
Jeff HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9733 2020.
Katherine L. Hermann and Andrew Kyle Lampinen. What shapes feature representations? exploring datasets, architectures, and training. ArXiv, abs/2006. 124403, 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L.. Sifre. Training compute-optimal large language models. ArXiv, abs/2203.1 15556, 2022.
A. Tarun Karthik, Mike Wu, Noah D. Goodman, and Alex Tamkin. Tradeoff's between contrastive and supervised learning: An empirical study. ArXiv, 0abs/2112.05344 2021.
Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Fierre-Emmann Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech representations in the time domain. In 2021 IEEE Spoken Language Technology Workshop (SLT), pp. 215-222, IEEE, 2021.
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
Tianhong Li, Lijie Fan, Yuan Yuan, Hao He. Yonglong Tian, Rogério Schmidt Feris, Piotr Indyk, and Dina
Katabi. Addressing feature suppression in unsupervised visual representations. 2020.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. ArXiv, abs/1907. abs/1907.04595, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692, 2019.
Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. In /CML, 2020.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6706-6716, 2020.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv: 08779, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763 PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. sero-shot text-to-imase generation. ArXiv, abs/2102.12092. 2021.
12

---

Under review conference paper at ICLR 2023
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas A generalist agent. ArXiv, abs/2205.061722 2022.
Joshua Robinson, Li Sun, Ke Yu, K. Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances neural information processing systems, 34:4974-4986, 2021.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, abs/1911.083 1.8731,, 2019.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameteri- zation exacerbates spurious correlations. ArXiv, abs/2005, (bs/2005.00435, 2020.
Fatemeh Saki and Nasser Kehtarnavaz. Automatic switching between noise classification and speech en- hancement for hearing aid devices. 2016 38th Anmual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), pp. 736-739, 2016. doi: 10.11090EBBC.2216.750888
Fatemeh Saki, Abhishek Sehgal, Issa Panahi, and Nasser Kehtarnavaz. Smartphone- based real-time clas- sification of noise signals using subband features and random forest classifier. In 2016 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 10.1 DD.100/ICASSSP201.7777222
2016. doi:
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. ArXiv, abs/2006.07710, 2020.
Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Association, 49(267):467-477 479, 1954.
Megha Srivastava, Tatsunoii B. Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In ICML, 2020.
Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah D. Goodman. Dabs: domain-agnostic benchmark for self-supervised learning. ArXiv, 1b22221 2021a.
Alex Tamkin, Mike Wu, and Noah D. Goodman. Viewmaker networks: Learning views for unsupervised representation learning. ArXiv, abs/2010.07432, 202Ib.
Alex Tamkin, Gaurab Banerjee, Mohamed Owda, Vincent Liu, Shashank Rammoorthy, and Noah Goodman. Dabs 2.0: Improved datasets and algorithms for universal self-supervision. 2022.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv, abs/2005. 10243, 2020.
Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633, 2020.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203. 2022.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. 2018 JEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 3742, 2018.
13

---

Under review conference paper at ICLR 2023
Kai Y. Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. ArXiv, abs/2006.09994, 2021.
Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang. Shanghang Zhang, and Pengtao Xie. Transfer learning or self-supervised learning? tale of two pretraining paradigms. ArXiv, abs/2007.044 ..00444 2020.
Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? ArXiv, sass22006.06006, 2021.
A CODE RELEASE
We are preparing our code and will release it on public GitHub repo
FORMALIZATION OF OBSERVATION IN SECTION
Definition B.1 (Invariance). A function f R + +R" is invariant to a set of transformations if and only if f fo g(z) f(x) for all x E Rh for all g G.
Definition B.2 (Augmentation collision). An augmentation collision occurs if. for two inputs Fos 2b and set of transformations G, there exist ga ,...,9u 99b ,...,
(na). (1) (ma) G for some nay Thy E N such that
ga o.... (na) (Ia) gb²² o... (mb) (Ix).
Observation B.3. there exists augmentation collision for imputs Fa 2h and transformation G, and f is invariant to G, then f(x,)= = f(x)).
Proof. By the definition of an augmentation collision, Ja o .... o (na) (xx) = (1)
9b ..... (ms) (zz). By the (1) (mo)
(1) (na) (xa)) f gb ... go (rx). Applying invariance,
definition of function, have f o go .... fu we obtain fxxa f(x)).
Applying this observation, we observe that the downstream labeling function f is invariant to a class of augmentations, then there cannot be augmentation collision for inputs with different labels. However, common augmentations such as brightness shifts can reduce any image to a black or white image, resulting in augmentation collapse between any two inputs.
C ADDITIONAL FEATURE DROPOUT EXPERIMENTS
C.I QUANTIFYING THE IMPORTANCE OF FEATURE DROPOUT
To assess the importance of label-destroying augmentations to the success of the viewmaker, we experiment with a setup where the viewmaker cannot destroy the information in the object class. To do this, we compute a mask around the object and zero out any perturbation from the viewmaker within that mask. We then perform pretraining and transfer usual.
As we report in Table the accuracy of the CIFAR-I0 class label drops precipitously, as expected. At the same time, the accuracy of two of the other objects remains mostly constant (shape and digits), while the accuracy for letters declines modestly (perhaps because the color of the letter is now able to suppress the learning of the letter class.
14

---

Under review conference paper at ICLR 2023
Viewmaker 10)
Mask Viewmaker (C-10)
Viewmaker (Object)
Mask- Viewmaker (Object)
C+Shape 79.8
26.0
100.0
95.8
C+Digit
69.3
50.7
94.3
95.0
C+Letter 71.9
23.2
96.9
71.8
Table Experiments with a masked viewmaker which is unable destroy the object class. Transfer accuracy on CIFAR-I0 (C-10) and the object task (Shape, Digit, or Letter). The Mask- Vaes-Vieemaker has its perturbation masked such that it cannot destroy the label of the object. This results in the features in the object suppressing the CIFAR-10 10 accuracy, while leaving the object accuracy relatively unscathed.
M
N
N
RR
M
M
R
G
R
GG
Figure 3: Non-label destroying Viewmaker perturbation examples.
C.2 QUANTIFYING THE DEGREE OF FEATURE DROPOUT
We perform an exploratory analysis to testing how well different views drop out the features in an input. We augment 1, 1,200 examples (CIFAR-10 image plus overlaid object) using a given augmentation policy (either the expert or viewmaker augmentations). We then encode the model with a classifier trained off of the other augmentation policy (i.e. expert for viewmaker augmentations the reverse) in order test how well the augmentations drop out the features. We use a different encoder to see the effects of the augmentations prior the encoder having chance to adapt to them.
We observe a bimodal behavior for the viewmaker views, shown in Figure 4 suggesting that the model is adapting to the semantics of the input and has learned to stochastically drop out the simple feature some fraction of the time. By contrast, the expert views display no such structure. Using the corresponding encoder and views leads models performing uniformly well, as shown in Figure
D END-TO-END SIMULATIONS OF LINEAR SETTING
We empirically test the performance of the full trajectory of gradient descent when we add noise to the data. We study a setting with one weak feature with correlations coefficient 01 I 0.5, and 50 dominant features with ag for k 2, ... 51. We compare two approaches run 150 iterations of GD without adding noise. the second, we run 50 iterations of GD without noise, and then add noise to the dominant features for the remaining 100 iterations.
the same data: in the first approach,
ttop) we compare the alignment of Feature (the weak feature) and Feature 2 (one of the dominant features) to the ground truth in the two approaches. We observe that adding noise consistently accelerates the
15

---

Under review conference paper at ICLR 2023
(a) Viewmaker Shapes
(b) Viewmaker Letters
(c) Viewmaker Digits
0.5
030
022
034
32
020
38
Expert Shapes
(c) Expert Letters
(f) Expert / Digits
Figure 4: Viewmaker augmentations stochastically drop out simple features added to the input. Prob- ability of correct answer for different augmentations (Viewmaker or Expert) and different examples from different datasets (Shapes, Letters, Digits). Each histogram shows single example from each dataset randomly augmented 1200 times, and the corresponding probabilities the correct answer. The viewmaker augmentations display a bimodal structure, indicating that the simple feature is selectively either destroyed preserved. The expert augmentations by contrast lack such structure, reflecting their lack of adaptation to the structure of each input.
16

---

Under review conference paper at ICLR 2023
Ploraed
(a) Viewmaker / Shapes
(b) Viewmaker Letters
(c) Viewmaker Digits
304
(d) Expert Shapes
(e) Expert Letters
(f) Expert Digits
Figure 5: Evaluating views with their respective encoder does not reveal bimodal structure for view- maker or expert views. Details are the same as in Figure with the exception that views are evaluated on their corresponding encoder.
17

---

Under review conference paper at ICLR 2023
Figure 6: Alignment of features with verses without added noise. From left right: 0i
0.125, 0.25, 0.375, 0.5. The top plots show the alignment of Features I (weak) and 2 (dominant) to the ground truth; the bottom plots shows the probability predicting the correct augmentation pair from the batch. Standard deviation bars are shown for the mean alignment over 200 runs. We used dimension 5, and a batch size of m 25.
learning of the weak feature (blue), with little cost to the dominant features (red). The affect is consistent among many choices for (r,, the correlation coefficient of the weak feature. We also plot in Figure((botttom) the probability of predicting the correct class (pair) of the view under both approaches. We observe that this probability drops sharply when we add noise, which we believe is the mechanism for faster learning with noise.
We remark that we chose to add noise to all the dominant features (instead of a single k' a in our theorem) to accentuate the effect of adding noise. We observed a similar effect, but smaller, when we added noise to fewer features, or when there were fewer than 50 dominant features.
FULL PROOFS OF PROPOSITIONS AND THEOREMS
We begin by stating and proving Lemma E. on the downstream classification accuracy.
Lemma E.1 (Downstream classification accuracy). Suppose we draw labeled data points (u, y) E RRKx x 1+1,1}, where as before, uk ~ (O La) for & E [K], and the label is given by sign(ui (ut pa). Then the best linear classifier R on the representations fe(u) KRK achieves an test error of 1 arccos That is
[ 0xl
T l
[
[ OL
arccos
T
min Pr(sign(a" fe())) + sign(M uz)] aERK
(1)
Thus if 0g and pa are orthogonal, then the test error is 50%. the angle between O2 and the t. is zero, then we achieve perfect classification accuracy.
Proof. It is easy to see that the best linear classifier will (up to scaling) be equal to the vector sign 0x )ek- Such a classifier predicts the correct sign whenever sign(a" fe())) = sign( 0x) sig ux) equals
sign ux), which occurs exactly a arrccoe
TPgl22
fraction of the time.
In the rest of this section, prove main theoretical result, Theorem which shows that T 9c decreases faster expectation during gradient descent we add noise to the k feature.
arccos 1
18

---

Under review conference paper at ICLR 2023
E. NOTATION.
We let dig denote the d-function which equals if = and 0 otherwise. For parameter è = ) [E[K]
we let 0g == u 0g be the projection of 0g in the μlù direction. We let 0 = 02. 0 be the projection of 0g orthogonal to the feature μlb.
Throughout this section, we consider the ground truth directions to be fixed, and we fix some initial correlation vector a. We let Pa denote the distribution from which the pair (u. v) is drawn from the Gaussian distribution described in Section 4 with correlation coefficients a. When unspecified, the variables U, V are drawn from the distribution Pa Since we study what happens when we vary og,, for [0, 1], we use the shorthand IP, to denote the distribution Pon ax()). where a<()) = and c((x):: or for all other k.
We denote (.O,U,V that we are considering L; for some fixed i, we omit the superscripts on the ith data point or its pair. That is, denote uk = u
(E((p)))en ei) log(p;)), which we abbreviate by Li. When it is clear
(6) and Uk:
ok
E.2 PRELIMINARIES
The following facts about of the derivative of the cross entropy loss are easy derived. Lemma F.2.
0C:
Ozst
Ozig
0zi
Σ (pij - 6ez)
ΣΣ
Pij
(2)
00
00
00
d0
iti
where
Ozij
(uk
(5) 1k
(0)
(5)T
(3) (1) +
1k
4k
)kk
(3)
00
We will also need the following facts on Gaussian random variables. The first, Stein's Lemma, well known. Lemma E.3 (Stein's Lemma).
Ex~Nio.0; (xf() = o
(4)
The next two lemmas are proved in Section
E.4
Lemma E.4. There exists some constant C such that following holds. Ifo S to and S S.5 S J. then for any c E {0, 3], and ~N(0, o²) we have
Ex [|X] (xxp((()))exx(X1 s Co..
(5)
If additionally d E (0, p &/ and Y (O p²). then
Ex [xx[9|] exp( (XY]] < Co
(6)
Lemma E.5. For some universal constant C. any a E [0, 11, 2 0, cE (0,1,2,3,4), we have
Ex~N(0.07) [(ex(()) 1) S Cla..
E.3 APPROACH AND LEMMAS
Intuition for proof of Theorem 4.1 Our proof involves comparing the gradient of the loss in the Og
direction, V = 30g 8 L in the setting with noise to the setting without noise. Loosely, our goal is to show that
for any k. the projection of the this gradient onto the ground truth direction, 146 V sign(u 0x), increases
19

---

Under review conference paper at ICLR 2023
when when increase the noise. The main intuition comes from an expansion of this gradient in Lemma E.7 which shows that E VA sign(μı 0x) approximately scales with Σ, Pa). Now observe that Psi. the probability of correctly matching the ith view to its pair, decreases when we add noise feature K'. Thus adding noise will increase 2l Vv sign(u Oz), thereby improving the alignment.
the remainder of this section, we outline our proof of Theorem this section. We prove all the lemmas below in Section R.4
InE (1 [o 1 1l2
To understand E
arccos
for small enough step size, we first claim that it suffices to
understand the expected projection of the gradient with respect to 0g in the μi direction and in the 0g direction. We use the notation V (C(,)
50,
Lemma E.6. Let 0% = 0k (V= + A0k). Then
1 BU. arccos
| O
[10x|12 ))
O (()
lim 4-5 17
|10
arccOs
.BBU.V
10
where N some negative value that depends only on Oz.
[nEo(++2) 16/
Inf (0))))))) ||0 ll
Now, since we care about the quantity Eu,v aarcoss positive, it suffices to show that derivative
E
arccos
being
d Eu.v-P -(KEO)(H of V
dx
|
is negative for all x E [āw, aw) Indeed, from Lemma
E.6 we have that
lim 1 70 r7
Bu.yweog arccos
(IuT of 10 (10
arccos
[ 0sl
))
(8)
|10x112
rak' d N
0 V (M )x
Bu..
(- VA)+
dx,
(9)
dx.
10
Jäy
so if the derivative is negative for the full range, then the difference in arccosines is positive.
the following lemma we compute the derivative of [v[ with respect to æ. Lemma E.7.
d BU. [v]] m da
d
aci
EU, SU.V-PP
dx
100g
(Ō(zs) 2n0)
}
1 x2 AT
uk
Ec, NP.
popplc (0%uw) (uT e00
00,
iti
We will analyze this quantity by explicitly taking the expectation with respect some set of random variables.
(i) Let = (Ux, Ve, Uv. Ver consist of the random variables u ug and uh uk for all i E [m]] Define
(6)
exp Eeni of 20 (2))
45 to be the logits when all variables in S are set to 0 (Thus explicitly, ds Σ, eND ERameer o0 " oT (T
We will use the notation j ~ o denote the distribution on [m] with mass diz j.
20

---

Under review conference paper at ICLR 2023
Let
(olez 2
h(S) (ff (1E (0)
u
(10)
80g
and
h1 (S) = (o) (1- 2
204
(of us (f]
(11)
which are the terms that appear in the right hand side of Lemma E. after PiiPij. Observe that Es(h(5) h1(ss] =
The following four lemmas serve to bound d Epapiy (S) ha (S)) "junk" terms, and our goal will be to show that these terms are small. We will control closely the terms of the form EpaPey (h1 (S)).
du Es |A] v and dar Es 10 We call the terms of the form
Lemma E.8 (Junk Terms for μal term.). If |10x1| I 1 and ||0x S 1, then for some universal constant C
Es [posPiptl (h(S) - h1)) S Cqaais (10 0 II + |10% 111 + on (100~1("100111))
Lemma E.9 (Good Term for term.). /f 1f || < and |10xr || S 1, then some universal constant C
Es [PaPssTT (s) 2 2ax (1 x²) )quas (110 (100)11110)) IIIo
C(110 113 [1=C(]0==8++10a
Plugging these two lemmas into Lemma E.7 yields the following corollary.
Corollary E.9.1 (Total μa term.). If for a sufficiently large constant C, 10 uul I 1-a2 1x 11xx.1 16 and [10x]12 < ax(1-03)) then
(nE 0x) d da
EE
[
VA
2 m
Euvvs
2n qua 2a 21 11 Σ
2
Lemma E.10 (Junk Terms for Og term.). If|10u11 S 1 and 110 I 1, then for some universal constant C
Es [puPig0[ (h(S) - fa (S))] & Cquas (10w 0x + |10 11 + au (10x 111 + llll [1
Lemma E.11 (Good Term for 0g term.). If |0&1| - 1 and 1|0x $ 1, then for some universal constant C
Es [PaPty hs))] s (1 - 113 2on (108) 110111 (1 + C(110= + [10&11))
Plugging these two lemmas into Lemma E.7 E.7 yields the following corollary.
Corollary E.11.1 (Total 0g term.). for sufficiently large constant C, ||o|||| S 1 1 110u.13 3| = 110% |10x112 < c=((1=²²)
then
(nT )2) |1 dx
d Er,
[0 vR]
S 2
Eu,yy
Σ qunn au 11
Combining Corollaries E.9. and E.11.1 we obtain the following lemma.
Lemma E.12. for a sufficiently large constant C, |!o!! < -يي" 11 ||0 11s = 110% 110+112 aall-> then
Eu,,~ (uuTon)u Tv)) ( 0.
(12)
10x 11
Theorem 4.1 now follows.
21

---

Under review conference paper at ICLR 2023
E.4 PROOFS LEMMAS
To prove the Lemmas E.4 and H.5 we will use the following well-known formula for the moment generating function (MGF) of the norm-n distribution.
Lemma E.13 (MGF of half-normal distribution). The MGF of the half-normal distribution is
Ox-Nvo. /2())
where (t) is the cumulative distribution of a normal random variable.
Proof of Lemmal E.4.
x²
Ex [|x] xxp(et
|>| exp(t|x|) exp(22 exp
dr
v 2m
20
v1
20
x²
1a12 exp(((=)))
dr
2
2T
2
v1 2
20 exp(t2)),
20
where ~1- To evaluate this, we use the MGF of the half-normal distribution in Lemma E. for some constant for all E 31,2.3,4).
E.13
Thus
de dtc e!|xi
< <Ex-N(o.I) Ex=N(o,1)x50 [[|x]<441x]]
{C( e//2
So for some constant C' (whose value changes throughout this equation), so long as o S 1 C+ V1 20 202 exp((Z)) = /1 - 2n 20 2x exp(111))
I / 20 (1 (())
S Co
This proves the first statement in the lemma. To prove the second, we first take the expectation over X, and using the half-Gaussiaa MGF before, obtain
Ey [[] exp(t) exp([XX)] S CEY [ + (t (T1) (++12))) (
Now applying the first statement to take the expectation over Y, we obtain
Ey (1 (t (+)))))++)))))) Co
Proof of Lemma E.5
We prove the lemma by induction on c. Suppose e 0. Then by plugging in the MGF
for the half-norma distribution from Lemma E.13 for some constant C, we have
E Ex~ o
((e XI 1)] 2/ /2
(13)
s 26 e/2 2/2
+
(14)
2
< ( - 1)
+ 12
(15)
< Ct,
(16)
22

---

Under review conference paper at ICLR 2023
thus
Ex- -)) -1]] 1)] Cto.
Now for 2 by Stein's Lemma, have (for new constant C),
Ex=N00.>)11X/((6 - 1)] = [xxxN10-)) c- sign( sign(X)(e"1x1
1)1
(17)
=0 d (Ix) sign(x) 11)
(18)
dx
=o²Bx~((000)) (c-2) (1x) - -1) + -1x (te!
}]
(19)
S Cta
(20)
where in the last step we used the inductive hypothesis and Lemma E.4
Proof of Lemma E.6
First observe that
1K4
[
110x112 ))
lim
BU.v
arccos
110 1l2
arccOs
1
1 (1 17/) 17 V )
I 0kl
lim n-+ 1)
Eu.v
arccos
110x(1 77) - v/l2ll2
arccos
110x112
1
1 (0x - I-クx VAJI )
[ 0k
lim n-0 7)
Eu,y
arccos
110x - V
arccos
||0/112
d
1 (Ox= v)
(0)
Eu.y
arccos
dn
10 2ll2
T1 since limy-+0 T-
0. Now
d drj
|K (0p v))
| 0& 110x112
d
[p (0p- V/ll (0)
arccos
(0) arccos'
100 V
dn
11 V
[uE ||0x|12
sign(u uh 0x P V 0xll +
1A
V
T
arccos'
|
o po C (L)
N
| 112
In 10
where N = arccos' observing derivative arccos( (z) is negative whenever x is positive.
[ile TT The lemma follows by taking the expectation over U, V, and
Proof of Lemma E.7 First observe that by symmetry, we have
d Bu. [va] m dx
EU.
de
00g
23

---

Under review conference paper at ICLR 2023
To make this expectation easier to analyze, we express the random variable (U)z), V(a)) Ps as an interpolation of Gaussians in the coordinate p Let & ~ N 1), and define (U, V) ~ Pi. such that
U ur. (8) (8) For x E [0, 1), define (U(x), (z:)) to have
V1-222. x²E. and otherwise be the same V). It is easy check that (U(x), V(z)) 1...
(8) Me U (z)
(8) up w
(21)
Now
d
(OU.V
(CC()))())
V())
BU.
BU,V
dæ
00n
dr
00,.
Taking the derivative of the cross-entropy loss, we have
d (L)
d de
Σ j71
[(zig - Zzi)
Pia
dx
00p
00g
dpsj
dpk (4)
Ue (1)) O(zij 2zd)
Σ
d (x)
dr
00k
iti
dzs
V1 V1-22 525
5)
(D(z)) 2zi)
PiyPa
duT (2)
(2)
00
JFL
where the variables Ziy and Pti are the similarity scores and the softmaxes from the data (U(x), V (xr).. Here
the first line by Lemma E.2 and the second line holds by chain rule since 0s., 0zu does not depend on UK
00, 86,
The third line uses the proof of Claim 1.14 to take the derivative Pıj. and Equation 21 to take the (5) derivative of uh UM (I)).
Now we reparameterize u Ug
(1)
-I as follows:
x
x 2h (x).
Me k
x
Plugging in this reparameterization and
dasi
0% Mi O. uw,,
obtain
du (x)
U,
(0lz) 230)
RU.V
PisPii (0 (0T (ue) (1 x1
dr
00,
x
00
iti
We now prove Lemmas E.8
10 and E.
Notation. Since is fixed throughout, drop the (1) superscripts and let uy u. and e2 = = We will introduce the following random variables, which are all independent, to simplify the exposition:
(i) Eg = 0 Ug for ii Thus & (0 [[@x11?).
Eg = TT for j # i. Thus E/ AO, ||0
24

---

Under review conference paper at ICLR 2023
& = (0 uk + (0))) - (xux). Thus & ~ (O |1 + - -
c ( uR- Thus E ~ NO, 10 112). C ( a. Thus Ç ~ (0,, (1 a2:)1102113))
(0.) Tun- Thus NN0, I1)
y (ou.)_uu Thus y/ ~ ~N, | 11))
7 = (0t)Tux: /Tu.- Thus NO, 1 11 T/. (0²))* uu.. Thus v/6 (0, 119)
For any such random variable X, we use xx to denote its variance. Observe that
exp (0% O u exp 0% uf....ff.of (()
PosPy exp (0% u Uk) exp (0% (f
qa4i Eyingg exp 0 1x o exp Of (2 (1) E Byrng exp 0% un o exp 0Z (12))
We will use the following two claims in the proofs of all four lemmas.
Claim E.14. For 3 E (Ei Ey, Exs E So Th Th x, x') }. let By == a (0% un Of o/ + off Then
OpaPt
S pipP (13,) [B:] + y By
03
If additionally E (5,, E St.Si, Cl n.,, ) and 1. {By E then
OprPis 0y 03
S Pupl ((13 + |8.1 + 2 13 (1y + 15vl + 2By'ng h5s + 21 1 11 2(E (BBrral1
Proof. By a straightforward quotient-rule computation of the derivative of 9: of S, obtain
recalling that Øsy is independent
Opu
03 Pij (B, - Bg By.Par)
By applying product to the expression above, we obtain
@prPis
03 PaPts (3, + B, 3y
Taking absolute values and using the fact that Pi S 1, obtain the first result.
Next take the derivative of Pij with respect to both B and 7. Using the expression above for obtain
Ops
"p we
Opr
Pij ((3g - EE 1Byr (5g - (1 - B3 +
dy 03
and
i @puPii By 03
PuPPis ((3, 3g 3B. (7g 7/ 23 N Pp)')) - 2 + 2( Be (By
The second result follows by taking absolute values and the fact that Pij' 1.
25

---

Under review conference paper at ICLR 2023
Claim E.15.
Pij exp (1 1) exp (10 Byrngg (exp (OE of exp (1 1))
Ki
Proof.
This follows directly from using Jenson's 'ss inequality on the distribution j' q to show that
1 Bynng exp o O " exp
S By [exp (-0% 1 o222) exp (-0R (2)
(8 u. oF
S Byiing exp (10% unOf uf u/ exp (f
Claim E.16
Pij
Zzj
where Zz exp (10f un ok 1l exp (10% un 04 (/21) Bninnn [exp (10% un uf exp (10 22)
Note that for any 2 O, we have - x/ S max (x - 1, 1). By Claim the desired value given in this claim.
Proof.
E.15
Ps. 96
1 is at most
Now
Egin exp (o OFOOf o exp (0% (FF
Pij
exp 00 1R v(2) exp (0 (0
S <exp 10% un 1221) exp (16 Byrnng (exp (10 ol exp (1 [))
This yields the claim.
Proof of Lemma
E28 Expanding h(S) - hS() we see that we need to control the following terms:
1. (a) Es [papis (v% (NT )x)) (N 44 [,] (b) Es [papls (v' (Kh xHb (M ue)]]
2. (a) Es [papas (1l (KT - uk)) (MT Ex]: (b) Es [papo ((v' (KT - xH((ww) (x]))
3. (a)
Bs [pePis (nl (KT xx( (MT uxy)] |. (b) JayEs [pupes (w'(-xE()) (KT uwy)]
Es PuPis (rk (HTWu ue - xpT (5a(V - 2z (6) ( p1s
4. (a)
(b)
Es (papis (v/ (yEmu - vx)) (5i(1x
Es PapPis (7% (HTrus uw - vx) (v(ex yTt)
5. (a)
(b)
Es ppapss (y/ (yE ug - m(FFvw)) (y((x - ar u - (1) yz p4:
26

---

Under review conference paper at ICLR 2023
We begin by bounding the terms where the expression after PuPPi has two independent mean-O terms, mainly (Ia), (2a), (4a). The first step is to apply Stein's 's Lemma (Lemma E.3) twice to these two terms, which we will call | and y. Let (S \ {8, 7)) be the terms after PuPii- Then we have
OpaPis
Es(pap.y87g(5 \ (B,7))]l S oz
Es
lo(S (B,
0
Next we apply the final result in Claim 1.14 to bound the absolute value of achieve
@pupu By 03
8
Once we do this, we
[s Bagp, (B,r))]l s 03 qug Es
Zlg(S> (8,x))
Σ
Cy c 1y
where Eg Cpte C for some constant C, and PuPP Resges
Finally, we use the bound on Z from
Claim E.15 and then Lemma E.4 to take the expectation over S, iteratively applying Lemma E.4 to cach variable in S. Thus we have, for some (different) constant C,
1. Es [papes (nt (n - vw)) (nE ux & Cauamll0 112111110000 (1110w11100011 S Caaasil 10
S
Cqadsy o 10 11
2. Ess [pupes (1/ ( vwf (n ui
S
20000
11x |110x|
Cquau 12 11 - Canqvil0x 10
3. Es (pupls (nt (KT - =xp(ew)) (El(0r uk "² 1x)
3 1110
Cqaqsl104 |1100==11|00011 11 < Cywas |0 1|a.
Now we consider the remaining 7 terms. Here we decompose the expression inside the expectation as PaPiy (,8,) where 3 S. We proceed before, but only apply Stein's Lemma once, to B. Applying Steins, the expression for ApaePss given in the first result of Claim E. we obtain 8,
Opapis 03
l \ |i]
Es(paPi; ()]1 < of
Es
< Lgg Es
Zlg(S Σ Gr133r
j'Emm]
(22)
where Eye(m) Cy' < C for some constant C, and Z = Claim E.15 an use Lemma E.4 take the expectation over S, again iteratively over each variable.
PiPp
Finally, we plug in bound for Z in
9.
Thus we have, for some (different) constant C,
1. Es [papis (v/ (Kkuu (pTTew) []]uR))]] < Couq, 11: 01 Cquq. [0.11910|| ll.
2. Es [papes (v/ (KT vx) (u]u+<))]] < Cqnnioo | 0 S
Caaa, 10..1
3. logEs [paprs (vl (nÉ-u - xmw (of uxf]]]
S Carqaq50",10000 100 |10!!
Cauqugy | 1 11 |110 S Ca | 11
4. JayEs (papls (V' (-x(1) (KFuwN)) - Ca [10
Cauguau 110 11
27

---

Under review conference paper at ICLR 2023
5. Es (pupes (V/ (n =xnf(w)) (Ei(12 - (Ei(u=-o&)) -z) (2())(())))) (ss) )
1100 110 11
Cauqvllo ||
6. agBS (aaPes (1k ( ve)) (olox - ox op us)
Cartaly 10 |110
Canquas 10 |1 || Carquass 11
7. Es puiPis (v' (NE - =x (x(Vrx oguE D px)) )]
< C
Cauqu 111001111
Combining the bounds on these 10 terms proves the lemma:
Es [PaPespII (h(S) - h1 hs)) - Cquais (10 11 + 110% + ax (109-11110111))
Proof of Lemma E.10 The proof of Lemma E. is nearly identical, besides some differences in the terms we need to bound. We list them below:
1. (a) Es [pupts (ng (KE xpf vw (0) uxb (b) |S [papis (v/ (pf vw (of ux
2. (a) [Es [papes (7k (nb vw) (0) [xE:)]]] (b) Es [papes (l// (pTrux- - -optuw-pppra vx (0 ux)i
3. (a) JagEs [pepts (n) (KT xpf (0% ua)]l (b)laREs [papos (v/ (KE zul (na))
4. lagBs [PaPty (w/ (-x()) (O |+,)]l
We use the same approach as before. For the terms (1a) and (2a) we apply Stein's Lemma to (1% E and (v%, G) respectively. For (lb), (2b), (3a) and (3b) and (4), we apply Stein's Lemma to Ej. E., 1/e yis and E E.4 as before, we obtain the following result:
respectively. Using Claim E.15 and then Lemma
1. Es [pupes (n/ (KE zpFF.w)) uw)) (0% [xE,]]]
Cquqnoo 10,]100 1101111011110011
Caranll0 111 < Cgans 10, |0
2. Ess [pupes (v/ (HE uu uw - =x (0) uxE:)]
Cquqioo 1/6 10110101111110
Cara, 11 11 |1 S Cyagcill 100 1 1xx11
3. EEs [puply (v/ (PE ux) (0f m
Cquanoo |
01 1110
11x1110001110000
4. EEs [pupls (3/ (KT upk (0T u&Ei)] 11xx
Cquqroo 18 1111@:
5. layEs [pupes (Tt (N ux - xbZ 10) uny)] Couqagi | 11100 0 |1|@!!!
Cauauaoo 110 1110% 1110
28

---

Under review conference paper at ICLR 2023
6. lgg [paPis (u' (nguy - x)) (nay] Cauquay 0 (|lllllll11l!!!
Carlah 11&l 11
7. aggS [paPes (0' ()) (o unv)]
Canquanjo 10 ||
Cou.gay | 1 11 Oz II
Combining the bounds on these terms, proves the lemma:
Ess [papesOk (h(S) hs)) Cqaqs (100~1/"10x)) + 11100= ar (100 100011011001110011 + 100 |1110x11 (1011))
We now prove the lemmas on the non-junk terms.
Proof of Lemma
E.9
(00)) uuuuuuuu (2 unon)(0)) usz)
Es puptij
Es guats (101))" "un "w (2 unax (o!!) u1s)
+ Es [(DuPL) - guaus) (109) Jun (2 (T un)]
20.qua pu ph Pk 2a
[
(101))) Tuuukuuu ) (uT (8()) uss)
q:i
Now by Claim E.16 we have
PuPl gictes
1
S ZiZy
(where the variable's Ze, Zj are defined in the
Claim
16 so
PuiPij
1)
uu (olu
$ Es (Z,Z, - 1) (e)))) u ug u 1u (n!l) um
Es
quits
SC (10x + 110 ||0| 1110
Here the second inequality follows from applying Lemma E.5 first, and then Lemma E. E.4 repeatedly for the remainder of the variables in S. This proves the lemma. Note that we need to apply Lemma E.5 several times single variable X E S. Indeed
write
(Z:Z, - 1) (0 uu
1 (0!)
(0!) "n
(E, exp([1xx11)) BJB B|X|=
(ExS((exp((tx)))
1)) B| + (E,S, 1)) x|
for some distribution on f, and for some terms Se, lez and B that are independent of X, and E {0. 2). Then to take the expectation of this term over X, we first apply Lemma E.5 on X to the first term, and iteratively apply Lemma to the random variables appearing in the next terms.
Proof of Lemma E.17
1
Es (PPPe, O (S) Es PuPss (1o))) u ug (2(oll) Tu (ol))
Ess quais (10 ( ((el) umn))
Es (priPie (391))) ( uorkke (2au((0!)) mu)))]
(
(PhiPiss
-1) (10%)) uu ms) ((ol)) uk )9]
20 of 1u 110 + 2a
gse
29

---

Under review conference paper at ICLR 2023
Now by Claim E.16
we have
PrePe
1 S - 10
PaPti
-1) 1) (co)) uu (101) us
$ Es (Z+7g -1) (0) u u (10!)) us )
S (1 + 1|0 113) ||0| ||e!!
Again the second inequality follows from applying Lemma E.5 first (several times as described in the previous lemma), and then Lemma E.4 repeatedly for the remainder of the variables in S. Taking absolute values proves the lemma.
30


Reviews:
[{"review_id": "uvm7TUeEr4", "reviewer": "Reviewer_AVvr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the role of data augmentation in contrastive learning is studied, especially in terms of whether label-preserving augmentation is useful or not. First the authors investigated the empirical evidence using a Viewmaker network (Tamkin et al., 2021b), which automatically finds effective augmentation. The results show that the augmentation that changes label information makes NNs more robust and generalizable to many downstream tasks. To explain this, a theoretical analysis is presented. ", "summary_of_the_review": "While the motivation is interesting, the novelty is not significant enough. Also, the theoretical analysis is counterintuitive and needs more careful assessment. ", "strength_and_weaknesses": "Strengths\n1. Overall the paper is well written.\n1. Motivation is clear and the problem this paper addresses is important.\n\nWeaknesses\n1. The novelty is somehow limited. I recognized that the main contributions are two-fold: (a) Empirically finding that label-disturbing augmentation is useful, contrary to common opinion. (b) Theoretical analysis of that. Although both are nice, this paper doesn't provide a concrete outcome (new method, new algorithm, etc). \n1. I have several concerns for the theoretical part. \n- The problem considered in 4.1, to estimate the correlation between u and v can be analytically solved by the eigenvalue decomposition of the covariance $cov[u, v]=E[uv^\\top]$. For simplicity, let K=1. The empirical covariance $\\Sigma = 1/n \\sum_i u_i v_i^\\top $ is then converges to $\\alpha \\mu\\mu^\\top$ for large n, and $\\mu$ is given as the largest eigenvector of $\\Sigma$ where $\\alpha$ is the eigenvalue. Similarly, the correlation btw u and $\\tilde{v}$ is given as the eigenvector of $cov[u, \\tilde{v}]$. In this case, the eigenvector is the same $\\mu$, but the eigenvalue is decreased from $\\alpha$ to $\\beta\\alpha$ ($\\beta \\le 1$). The scale of eigenvalue determines the strength of true signal (i.e. correlation) and larger eigenvalue makes the estimation problem easier. So the noisy problem ($cov[u, \\tilde{v}]$) is more difficult to obtain an accurate solution. This observation contradicts what Theorem 4.1 suggests. Why does this happen? Did I overlook something?\n  - Note that for $K>1$ we can have the same idea by thinking $K$ eigenvalue decomposition problems.\n- As mentioned above, the noisy problem should be vulnerable to data noise. However, the speech results (table 2) show the radical augmentation makes NNs robust to noise. Why this is possible?\n- Although the theoretical part is intriguing, it is written a bit dense. Also, any intuitions behind Theorem 4.1 are not explained. Since the formal proof is over 10 pages, I really want some proof sketch in the main paper. \n- It's better to contain experiments that verify the theorem. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RuFMaHQBDGj", "reviewer": "Reviewer_mMWc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper challenges the common belief in literature in self supervision that augmentations need to be label preserving, in order to perform well in downstream tasks. The authors first show that when using a Viewmaker network to automatically derive good views, the augmentations derived by the model may drop significant information about the label of the downstream task. Using this observation, the authors then demonstrate theoretically that in a simple problem, adding noise to the features of the inputs helps the optimization process to find a better representation.", "summary_of_the_review": "Overall, I think this is an interesting paper, due to the fact that it presents a novel interpretation on the necessity of invariance in the augmentations. I lean towards acceptance due to the interesting aspects of the above, however I also believe that there are a few issues in the paper that need to be addressed, with respect to how the quantitative evaluation of invariance is performed and how the theoretical result is linked to the rest of the paper.\n", "strength_and_weaknesses": "Strengths:\n- This is an interesting paper, in that it questions a commonly accepted paradigm for self supervised learning, in that the augmentations applied on the inputs need to be carefully selected, so as not to affect the downstream task. Since this field is overall novel, I believe that challenging commonly accepted views is important for the field as a whole.\n\n- The authors also make a convincing argument regarding the use of feature destroying augmentations, in the setting of multitask learning. According to the authors, while previous works only consider the setting of a single downstream task, they argue that in order for a model to perform well in a multitude of different downstream tasks, no single feature should be prioritized over the others. This is an interesting argument for the case of the use of a variety of augmentations, whether they are label preserving or not.\n\n- The experiments provided by the authors demonstrate that the use of Viewmaker networks to automatically extract (possibly label destroying) views show benefits over the use of expert augmentations, which is an argument for their case that invariance is not necessarily required.\n\nWeaknesses:\n- I believe that the motivating examples behind the arguments made by the authors in Section 2 can be improved. In particular, the authors argue that the brightness augmentation used in practice is not label preserving, while this may only be true in extreme cases. Furthermore, I also believe that their motivating argument would be cleaner if they described hue shifts and greyscaling the same way they describe brightness augmentations, in that they may not be label preserving despite their widespread use. I also think that the fact that an augmentation may be invariant for a given task and not for a different one should be stressed here.\n\n- The authors use Viewmaker networks to automatically extract augmentations, and make their central observation that augmentations need not be label preserving based on this. While this is a good argument, I am not sure if it is adequately supported. From the qualitative side, Figure 1 claims that Viewmaker views tend to destroy labels for one of the tasks. A similar argument can be made for the expert views in some cases (particularly in row 4, where the expert views corrupt the input greatly). From the quantitative side, the authors analyze how the predictive capabilities of a classifier are affected by the Viewmaker and the expert augmentations. However, this is done using a single sample of the data, and deriving several views from it. I believe that this should be done with a multitude of the samples of the dataset, in order to avoid any possible bias arising from this sample.\n\n- The authors provide a good theoretical argument on the use of noise on a given feature, and how doing so actually improves the learning of other features. However, in the setting that the authors describe, I find it slightly weird that neither of the two views of the samples that we consider contain by themselves any information about the directions $\\mu_k$ that we want to learn. This information is rather incorporated into their joint distribution, and thus it seems more natural to me to consider them both as a single sample, instead of two views of the same object. This makes the theoretical example somewhat disjoint from the rest of the work, if I understand correctly. Moreover, the theoretical result states that if a feature is corrupted, then we can learn the rest of the features well. I think it is also useful to know in the main theorem how much we need to corrupt that feature to achieve this (thus, what is the total effect on the alignment of our learned vectors with the ground truth).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JrfNjzCJvp", "reviewer": "Reviewer_Xou4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the question, what role does data augmentation play in (contrastive) self-supervised learning? As the authors write they complicate the simplistic view that data augmentation implements invariances: \"We complicate this picture by showing that label-destroying augmentations are often crucial in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks.\"\n\nRather the authors show that data augmentation can lead to a more balanced representation of features (essentially by feature level dropout), so that a broader set of down stream tasks is served.\n\nThe evidence presented is experimental, indicating that a combined (ensemble) approach of augmentation interventions lead to better general down stream generalizability. A theoretical analysis of SGD with data augmentation (added noise) complements the experimental evidence.\n", "summary_of_the_review": "Clear hypothesis with strong experimental evidence, supported by somewhat weaker theory.\n\nIn rebuttal please related the present hypothesis to the prior work of Ericsson et al.", "strength_and_weaknesses": "Strengths:\nRather convincing experiments are presented supporting the hypothesis that data augmentation leads to more balanced feature weights.\n\nWeaknesses: While carefully supported by creative designed experiments the idea is not entirely new. In fact work by Ericsson et al. [1] has pointed to a quite similar conclusion - this work is not cited in the present contribution.\n\n[1] Ericsson, L., Gouk, H. and Hospedales, T.M., 2021. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. arXiv preprint arXiv:2111.11398.\n\nThe paper's theorem is relatively weak evidence (many conditions, weak conclusion)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "52Sr8q091a", "reviewer": "Reviewer_WzK9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper questions the widespread argument/intuition for the success of contrastive learning that a good augmentation must be label preserving. The paper argues that label-destroying augmentations may be important for contrastive learning to learn diverse general purpose representations. It further hypothesizes that label destroying augmentations serve as some kind of “feature dropout” since they can prevent one “shortcut feature” from suppressing other features. The following arguments are provided to support the hypothesis:\n- Discussion (in Section 2) about why some standard augmentations are not label-preserving and why invariance to those augmentations is undesirable. E.g. while random gray scaling is an augmentation, we often do not want invariance w.r.t. it since color information can be useful for downstream tasks.\n- Experiments with ViewMaker networks (a recent method to automatically learn augmentations) on hybrid datasets (e.g. CIFAR with superimposed digits) where the learned augmentations are generally better than standard augmentations. The paper observes that despite performing better, ViewMaker has more label-destroying tendencies, as verified with some visualizations of the augmentations and other statistical evidence (questions about this later)\n- A theoretical result with a linear model that tries to capture that idea that adding noise to some features (to simulate label destruction) can speed up the learning of other features (for one gradient step).", "summary_of_the_review": "The message and experimental findings about ViewMaker networks are original, to the best of my knowledge, and overall I find them quite interesting. However I believe that the paper is lacking in execution of the ideas, more precisely in its justification of the *necessity* of label-destroying augmentations. I am thus inclined to assign a score of weak reject, but I would be happy to change the score if there is some more justification for the point about necessity, through experiments or theory.", "strength_and_weaknesses": "**Strengths**\n\n- This paper raises interesting points about how label-destroying augmentations may not be harmful for contrastive learning, questioning the conventional wisdom about contrastive learning and the role of augmentations.\n\n- The experimental findings on the hybrid datasets that ViewMaker networks learn more label-destroying augmentations than “standard” augmentations but perform slightly (for CIFAR) or much better (for audio), are also very interesting.\n\n- The paper is clearly written and easy to follow for most part\n\n**Weaknesses**\n\nMy main concern is the claim from the abstract that “label-destroying augmentations are often crucial in the foundation model setting” is not adequately justified. While the paper sufficiently argues why label-destroying augmentations *may not be harmful*, the results in the paper fail to convince me that they are *crucial*.\n\n(W1) The main evidence provided is the ViewMaker experiments, but in my view the findings are not sufficient to conclude that label-destroying augmentations from ViewMaker were “crucial” for its superior performance.\nThere could be many other reasons why those are better augmentations and it is also possible (in theory) that the label destroying augmentations are somehow even holding it back. I believe that some more experiments, where the only thing that changes is the level/amount of label-destroying augmentations, would be required to make such a conclusion.\n\nOne idea that comes to mind is the following: Run contrastive learning with only those ViewMaker augmentations that do not destroy labels a lot (e.g. pick augmentations that have P(correct) greater than some threshold in Figure 3). If filtering out such augmentations leads to a decline in performance, that would provides evidence that label destroying augmentations were indeed crucial.\nSome such experiment that isolates the effect of the presence or absence of label destroying augmentations, without other potential confounding factors, seems important for the conclusions in the paper.\n\n(W2) Similarly the theoretical result from Section 4 does not characterize the necessity or benefit of label destroying augmentations well enough.\nThe result shows that if noise is added to some feature (to “destroy” it), then one step of gradient descent will lead to better learning of other features.\nThis result fails to paint a more global picture of what would happen if contrastive learning is run with augmentations that do and do not destroy augmentations/features every now and then.\nWhile interesting, the one step gradient result that is presented is quite intuitive, since at the extreme if one feature is completely destroyed, then other features start becoming more important for the contrastive loss.\nHowever it is not clear to me that the result provides a “simple linear model that captures the essence of how label-destroying augmentations can improve downstream accuracy “, as claimed on page 6.\nSome more comments:\n- Is this result saying that with label-destroying augmentations, the useful features will be learned faster? Because it seems to me (although I do not have a proof) that with or without label-destroying augmentations, all the features should be learned eventually. I'm not sure if any suppression of features will happen without the occasional destruction of features.\n- Even if an end-to-end result cannot be shown, even some simulation experiments with contrastive learning on this example for two augmentation schemes, one that occasionally destroys features while another that doesn’t destroy, might help shed some more light and strengthen the claim that label destroying augmentations can help.\n- Will help to include a proof sketch or at least the key steps in the proof, even if for a special case like \\beta=0, for the reader to get some intuition.\n\n\nOther comments\n\n- In Section 2, I’m not sure if an augmentation “applied repeatedly” is the right way to think of this. A function $f$ might just be invariant on the support of the input distribution and one augmentation applied to those images, and not necessarily on augmentations of augmentations. This will not lead to the “catastrophic augmentation collisions” that is shown in Appendix B.\n\n- The bimodal distribution in probabilities in Figure 2 is quite interesting. However there might be some confounding factors since the histograms for ViewMaker and Expert augmentations use different network (based on the description in Appendix C).\nTo get rid of confounding factors it would help to plot histogram from both using both networks (so 4 plots instead of 2). Was this image randomly selected? Is the finding true for other images as well?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject