Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning
Authors: Alex Tamkin
Margalit Glasgow
Xiluo He
Noah Goodman
Venue: NeurIPS 2023 poster (2023)
Keywords: self-supervised learning
contrastive learning
TL;DR: We identify a "feature dropout" property of augmentations that influences how different features are learned in contrastive learning

Abstract: What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. for digits superimposed on photographs, predicting the class of one vs. the other). We find that Viewmaker Networks, a recently proposed model for learning augmentations for contrastive learning, produce label-destroying augmentations that stochastically destroy features needed for different downstream tasks. These augmentations are interpretable (e.g. altering shapes, digits, or letters added to images) and surprisingly often result in better performance compared to expert-designed augmentations, despite not preserving label information. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.

Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning
Alex Tamkin, Margalit Glasgow, Xiluo He, Noah Goodman Stanford University
Abstract
What role do augmentations play in contrastive learning? Recent work suggests that good augmentations labee-pres with respect to specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful the foundation model setting, where the goal is learn diverse, general-purpose representations for multiple downstream tasks. We perform con- trastive learning experiments on range of image and audio datasets with multiple downstream tasks (e.g. synthetic datasets combining two classes, such as images and digits, and naturalistic datasets labeled with dozens of attributes). In controlled experiments where we destroy features at different rates, we find that destroying one feature modest fraction of the time can improve learning of other features, while still enabling the dropped out feature to be learned well. Additionally, we show how this hypothesis can help explain the success of Viewmaker Networks, which generate augmentations that appear to target and destroy different features in the input examples, yet often result in better performance than standard augmen- tations across tasks. To support our empirical results, theoretically analyze a simple contrastive learning setting with a linear model. In this setting, we show that label-destroying augmentations are crucial for preventing suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.
set of features from
Introduction
In recent years, foundation models [ have exhibited remarkable progress on a range of Al tasks 1 38 crucial characteristic of foundation models is that they can be 36 6:
13 31 37
adapted for a range of downstream tasks. For example, a foundation model trained on ImageNet should ideally not only perform well at object classification, but should also have learned general features useful for localization, segmentation, and other visual tasks. Indeed, this is borne out by recent work showing the high accuracy of foundation models on a range of downstream tasks 19, as well as a range of analysis work showing that models learn high-level semantic features including texture, color, pose, and style [19].
One popular strategy for training foundation models involves training models to match transformed versions (known views augmentations) of the same input. For example, image views might include common data augmentations such as cropping or color jitter ( while views for speech might include pitch modulation spectrogram masking 35). This family of objectives includes contrastive approaches such as SimCLR and MoCo, as well as non-contrastive approaches such as
BYOL and SwAV 9 23 20 7
Given the central importance of these views for defining the self-supervised task, much work has focused on the question of what views lead to high-quality representations. The prevailing consensus, exemplified by Tian et al. [52]. holds that views should be label-preserving with respect
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

to a downstream task. In other words, because the contrastive loss will produce representations which are invariant to features that vary across views, any information we wish to preserve in the representations should not be altered by such views. As Tian et al. [52) write: "A good set of views are those that share the minimal information necessary to perform well at the downstream task."
Here, we question whether this assumption-in particular, with its focus on a single task-is enough explain why contrastive foundation models succeed on a range of downstream tasks, In Section 2 we observe that the actual choice and application of views in practice does not align with this prevailing consensus. For example, complete invariance to several common augmentations (e.g. shifts in brightness or cropping) is undesirable since augmentations of inputs from different classes can collide. Furthermore, in many cases there are explicit ways to specify invariances (e.g. converting images to grayscale) that researchers avoid, instead specifying them indirectly via augmentations (e.g. hue shifts). These observations suggest that specifying invariances is not the sole role of these views.
Instead, we suspect that augmentations serve as a form of feature feature from becoming a shortcut feature and suppressing the learning of other features. We study this idea empirically with a set of synthetic datsets constructed by overlaying a simple element (e.g. a digit, shape, letter, or speech snippet) on an image or audio recording. We find that adding such simple feature can dramatically decrease how well the other feature is learned, but that stochastically "dropping out" the simple feature can enable both features to be learned well. Next, we use this perspective to explain the success of Viewmaker Networks, a recently proposed method that generates augmentations for contrastive learning via adversarial training. We apply viewmaker and expert views to these same synthetic datasets, as well as naturalistic dataset of facial images annotated with 40 different attributes (e.g. "wearing lipstick" or "blond hair"). Across these settings, we find that viewmaker augmentations learn to selectively obscure various features in the input. Despite this, the viewmaker representations still learn the downstream tasks well, while expert views often struggle on one or more of the attributes. This further suggests that being label -abel-preserving is not necessary property of good views, as long as the label information is still sometimes accessible.
preventing any one
Finally, we formalize the intuition that feature dropout can aid learning with theoretical analysis of a simple linear contrastive setting. In this setting, we characterize how the noisiness of each feature directly determines how quickly features are learned, and uncover an interaction between features governing how fast they are learned. In particular, we show how learning one feature quickly can suppress the learning of other features, and show that adding noise to the "easiest" feature can increase the rate which other features are learned. This further indicates that label-destroying augmentations may have a direct role in ensuring that contrastive models learn a broad range of features for downstream tasks.
Overall, these findings suggest the need to revisit common assumptions about the role of augmenta- tions for contrastive learning in the foundation model setting, and move towards better understanding of how to train generalist models that learn diverse features from unlabeled data.
2 Common practices are at odds with the "invariance" explanation
We begin by briefly exploring several common augmentations used in contrastive learning for natural images, and explore how they come into conflict with the common assumption described above. First, observe that many common augmentations can affect the label of the input, depending on the downstream task. For example, many downstream image recognition tasks require color information (e.g. identifying bird species) or brightness (e.g. scene or time-of-day classification), implying that invariance to these characteristics would be undesirable. Yet hue shifts, greyscaling, and brightness shifts are common augmentations used in contrastive learning 9 23
Second, repeated application of some augmentations causes challenges for all downstream tasks. For example, applying brightness shifts repeatedly results in any image turning completely black or completely white. Thus the class label cannot be truly invariant to this augmentation, since inputs from different classes can experience 'augmentation collision" at this black or white image (this is

---

formalized in Appendix B This argument also applies to other augmentations, including shifts in and random masking.
Third, some augmentations are commonly used despite ways of explicitly encoding invariance to them. For example, two image augmentations are hue shifts and greyscaling. Invariance to both of these augmentations can be explicitly encoded by always converting an image to greyscale. Yet doing so is not common practice because color information still desirable for many downstream tasks.
The contradictions between the invariance rationale for augmentations in contrastive learning and these common practices suggest the need for additional explanations for the role of augmentations.
3 Controlled experiments demonstrate the benefits of feature dropout in settings with multiple features
In this section, we present controlled experiments on synthetic data demonstrating how label- destroying augmentations can balance the learning of mulúplee features during contrastive learning. Our core toolkit is to overlay images with set of synthetic features, As we will show, the presence of these synthetic features causes the network learn the synthetic features very well the expense of the image features, measured by downstream classification accuracy. However, "dropping out" these features some fraction of the time during contrastive learning enables us trade-off how well each feature is learned, while not resulting in complete invariance to either set of features.
3.1 Datasets
We consider the behavior of viewmaker networks on four synthetic datasets, including three image and one audio dataset. Each dataset constructed in such a way as to support two distinct downstream classification tasks, enabling us to examine precisely how well each downstream task is learned. The presence of two downstream tasks enables us to analyze the foundation model setting where we wish to learn features relevant for multiple downstream tasks, as opposed to set or the other.
Image datasets The three image datasets are based on the canonical CIFAR- 10 image- maggerrecognition dataset (28) (MIT-License). One task is always to predict the CIFAR- 10 object label (e.g. airplane or bird). The other task is dependent on an additional feature overlaid on the image: C+Shapes: The CIFAR-10 image is overlaid with one of three randomly-colore shapes: square, triangle, or a circle. The second task to predict what shape was overlaid (N=3 classes). C+Digits: The CIFAR-10 images are overlaid with four copies of a randomly-samples digit from the MNIST dataset. The second task is to predict the digit class (N=10 classes). C+Letters: The CIFAR-I0 images are overlaid with four copies of a randomly-colored English letter. The second task is to predict the class of the letter (N=26 classes).
Audio dataset The audio dataset is created by overlaying the audio of a spoken digit (from the AudioMNIST dataset (3), MIT License) with random background sound (collected from one of three possible classes: cafe, machinery, and traffic) [4:44] 42). The tasks are to predict the digit class (N=10 classes) and to predict the sound class (N=3 classes). Inputs are presented the network as log mel spectrograms.
3.2 Experiments
Pretraining We pretrain with the SimCLR algorithm for 200 epochs with batch size of 256 and a temperature of O. We use a ResNet- 18 model with standard modifications for smaller inputs (including a smaller stride and no initial maxpool) used in Tamkin et al. [49]. We use the standard SimCLR augmentations for the image datasets [9], with the exception of gaussian blurring, which the SpecAug [35] augmentations, which randomly mask out different frequency and time bands, as well the WaveAug [27] augmentations, which alter various properties of the waveform such the pitch and speed.
found to have impact downstream performance (Appendix C.4.. For audio,
Note that invariance is not be confused with the related but distinct property equivariance, often discussed as a desirable property network architectures (c-g. see Fukushima and Miyake [17: Chen at al. (8). "Continuous reduction contrast eventually produces single-color images, given finite precision images.
3

---

Linear Evaluation We evaluate the quality of the learned representations by training a linear softmax classifier on top of the prepool representations. We train for 100 epochs. using the same parameters as Tamkin et al. 49, using SGD with learning rate 0.01, momentum 0.9, weight decay O, and batch size 128. We train separate linear classifiers using the same pretrained network for each downstream task [9. Augmentations are applied during training but not evaluation.
Results As shown in Figure we see an interaction between the two features, where dropping out the synthetic feature improves learning of the main image or audio class. Across settings, we see regions where both features are still learned well, providing concrete example of how feature dropout can be useful when learning multiple features during contrastive learning.
Main
Atammra Labil
Mun Lobel
0.79
0.790
0.50
0.000
02000
0.00
76
103
0.000
23
30
75
100
Probabilty c'Dxpau
Probabily Cropout
(a) Shapes
(b) Digits
Atamata Labal
Main Label Alemale
1.0000
17000
0.7500
0.500
0.000
0.200
0.000
25
75
100 0.00
25
50
75
100
Probeaiity of Drepect
Frecablly Dropppu
(c) Letters
(d) Audio
Figure 1: Linear probing accuracy (y-axis) after contrastive learning with varying rates of dropout of the synthetic feature (X-axis). In all cases, we see clear tradeoff between features, where dropping out the synthetic feature improves learning of the object class.
4 Viewmaker Networks Succeed Despite Destroying Label Information
As another point of evidence that good views need not be label-preserving, we consider viewmaker networks [49]. generative model which produces augmentations for contrastive learning. Intuitively, viewmakers learn a stochastic augmentation policy that makes the contrastive task as hard as possible for the encoder. The stochastic augmentations are parameterized as additive perturbations bounded by an L1 norm, meaning the viewmaker can alter but not completely destroy the original image.
Formally, given an input æ E N, a viewmaker network Vo is trained jointly with an encoder Eg to optimize the minimax expression:
(Eo( x
Vas 82) (x, 52)
max min L
Eo
( Eg x +
Here L is multiview loss function (e.g. 9 (23)) is minibatch of inputs. ||-1|1 is the L1 norm, is the distortion budget controlling the strength of the views, and 8,, 82 ~ (O, are random inputs that enable the viewmaker to learn stochastic augmentation policy. We clamp the output of the viewmaker for images to [0, 1] as in Tamkin et al. 44).
Viewmaker networks learn to stochastically alter different parts of the input, including tesk-relevant features, meaning that these augmentations are not preservin.... Nevertheless, as we will see shortly, viewmaker networks enable strong performance on multiple downstream tasks, including often better performance than expert-designed augmentations. Moreover, this feature dropout capability of viewmaker networks may help them learn many features well rather than just the casiest.

---

Digits
Shapes
Letters
BB
Speech
Expert view
Viewmaker view
Viewmaker delta
Figure Comparison viewmaker and expert augmentations on datasets with multiple features. The viewmaker augmentations adapt to the particular semantics of the input data, and make targeted perturbations which remove the class-relevant information of the synthetic features (e.g. occluding the digit, shape, letter, or speech). Despite this, the encoder network is still able to learn strong representations. Rows (from top): Digits, Shapes, Letters, Audio. Columns (from left): Expert augmentations, viewmaker augmentations, difference between original and viewmaker augmentation. rescaled to [0,1]. Center image each grid the original. Audio Expert views shown are Spectral views.
4.1 Experiments and Results
Experimental Settings We use the same experimental settings as Section however without manual dropout of the synthetic features. one of experiments, we the standard augmentations from Chen et al. (9], which we henceforth refer to as the expert angmentations. For the experiments with viewmaker augmentations, we use budget of 0.05.P for the image datasets, and 0.125P for the audio datasets, where P is the number of pixels in the input.
Additional naturalistic dataset with 40 attributes To further validate the behavior of viewmaker on realistic multi-feature datasets, we consider the CelebA dataset, large database faces annotated with 40 different features. These features cover wide spectrum of facial attributes, such as "Has Bangs" "Wearing Lipstick" and "Smiling." and enable us further analyze whether viewmaker networks learn a broader range of features than commonly-usee augmentations.

---

VM (CIFAR-10)
Expert (CIFAR-I0)
VM (Object)
Expert (Object)
CIFAR-10 Only
84.5
86,2
C+Shape
79.8
76,0
100.0
100.0
C+Digit
69.3
58.8
94.3
86.7
C+Letter
71.9
74.8
94.1
Table 1: Transfer accuracy different features. Viewmaker (VM) networks are able to achieve good performance across multiple downstream tasks, while expert views sometimes falter. Networks are pretrained on the datasets on the left, and transfer accuracy is reported for the different conditions on the columns. Runs are averages of three seeds (with the exception of CIFAR- 10 Only, from 49
Speech Accuracy
Background Sound Accuracy
Viewmaker
Spectral
Waveform
Viewmaker
Spectral
Waveform
Speech Only
92.4
97.0
76.7
Bkgd. Sound Only
100.0
100.0
100.0
Speech + Sound
71.3
64.6
50.2
100.0
99.9
98.0
Table 2: Audio transfer accuracies. Viewmaker networks achieve good performance across multiple tasks, while expert views sometimes suffer catastrophic drops another feature is added. Networks are pretrained on the datasets on the left, and transfer accuracy is reported for the different conditions on the columns. Runs averages of three seeds.
4.2 Results on two-feature datasets
Qualitative evidence of feature dropout Visually, the viewmaker augmentations seem to stochasti- cally alter different aspects of the input, as shown in Figure In addition modifying the background of each input, the viewmaker also selectively modifies the additional synthetic features added to each domain: C+Digits: The viewmaker augmentations selectively add pixels to the MNIST digits, making it difficult to distinguish which number is present. C+Shapes: The viewmaker augmentations sometimes draw squares around the shape in the center, making it difficult to determine the shape class. C+Letters: The viewmaker draws letter-like markings top of the letters, obscuring the letter identity and color. Audio: The viewmaker identifies the narrow band corresponding the speech and applies perturbations to it. As can be seen in Figure 2 these label-destroying augmentations are quite common, occuring in a sizable fraction of the sampled views.
Quantitative evidence of feature dropout We also measure this selectivity of features quantita- tively in Appendix C.2 and Figure 6 We augment images 1,200 times and observe the resulting probability assigned the correct object class. Two clear modes appear for viewmaker, but not expert, augmentations. This corresponds to the fraction of time the viewmaker destroys the overlaid feature information (low P(correct object class)) and preserves (high P(correct object class)).
Viewmaker succeeds despite destroying label information As shown in Table I and Table 2 viewmaker networks achieve good accuracy on both tasks, while expert augmentations frequently achieve lower performance on one or both. On image tasks, for example, while expert views achieve slightly higher performance when classifying the image only, they see a large drop in accuracy when
Figure 3: Viewmaker networks capture broader range of features on a naturalistic dataset. Linear evaluation Fl score CelebA for viewmaker and expert views. Attributes are sorted from lowest to highest accuracy within each augmentation type.
03
00

---

Representation fe(w)
Q1
O2
Ox
2221
1223
PKA
PR
Input
(a)
(b)
Figure 4: We show how label-destroying augmentations can aid learning of other features in a linear contrastive setting: (a) The correlation of the kthh feature of augmentation pair, shown for Each pair (0 and ( have correlated projections onto the ground truth μtş direction, representing the feature conserved across augmentations.
(b) Feedforward linear network which computes the representation fe(w), As each feature /4 is learned (0g > p&) the representations of the two views
fe("))) fe(v(!) become more similar, decreasing the contrastive loss.
the synthetic feature is added. In two of these cases (Shape and Digit) the viewmaker models able to achieve a higher accuracy on both the image and the synthetic feature, while on the third (Letters) they achieve slightly lower accuracy on the images but achieve half the error on the synthetic object. For the audio experiments the picture is similar-veewmake avoids catastrophic drops in performance learning both features together, achieving the highest accuracy on both, while the expert views see larger drops and worse overall performance. Note that the high performance of expert views for our control tasks (CIFAR- (CIFAR-10/Speech) 10/Speech/Sound Only) indicates that the viewmaker views are not merely better all- all-around views, but that they specifically help the model learn multiple features.
These results provide additional evidence that aabel-preserving views good representations- that feature dropout may improve learning of multiple tasks.
necessary for learning
4.3 Results on naturalistic dataset
We observe similar qualitative and quantitative results for the CelebA dataset. We train models using the same settings in Section 3.2 using budget of 0.01, and indeed find that viewmakers capture a much broader range of features, achieving an average Fl Score of 0.509 over the 40 features, compared to 0.334 for the SimCLR augmentations. In addition, the viewmaker augmentations clearly capture wider range of features, can be seen in Figure 3 especially at the tail of the distribution. Furthermore, we see bimodal disruption patterns in over two-thirds of the CelebA features, as shown in Figure indicating significant feature dropout across in most attributes. We also show qualitative results Figure F demonstrating that the viewmaker alters attributes such as facial features, hair color, and background elements in the scene. These results further support the hypothesis that viewmaker networks exhibit feature dropout, yet capture a broader range of features than expert views.
Theoretical Analysis of Feature Interactions in Linear Contrastive Setting
In this section, we analyze simple linear model that captures the essence of how label-destroying augmentations can improve downstream accuracy. We study setting where the data contains many underlying features that are relevant downstream classification tasks, and where these features are preserved to varying degrees across augmentations. We will show that a linear model trained with contrastive objective learns these features, and that adding noise to one feature can speed learning of others during gradient descent. One difference between the linear setting we analyze and Section 4 is that here add stochastic Gaussian noise to destroy features across augmentations, as opposed to the bimodal feature dropout behavior of viewmaker networks seen in Figure 2
7

---

5.1 Data Model and Setting
We study a model which consists of data with K distinct features, each corresponding to some ground truth unit-vector directions Pi..... MK E R We sample each data point E R Kxxd and its augmentation (a.k.a. its positive pair or its view) v E IR xxx as follows. For E 1...., ..K,. K, the kth row of u, which we denote ug, is drawn from the Gaussian distribution, (0, Ta). The kth row of the augmentation, tk, is drawn from the same distribution, but is correlated with ug in the yu-direction (and is otherwise independent in the other directions). The strength of the correlation is governed by parameter og [0, in the following sense: of 14 ag Ms + v - a where & ~ (O, 1). Thus the larger org the stronger the correlation in that feature across the two views. Figure 4 visualizes the correlation of (ux, U&) in an augmented pair. Formally, we can write that aKHAPli
(ue, 04)) ~ (0. (o
Id
))
for vector a E 10
La
We will learn a model O E RKxd which represents collection of K feature extractors, as pictured Figure 4 b). The model θ, with rows [oK] maps a data point w E R& to representation fo ()) E R by computing a score w 0g for each element in the representation. That is, (fe(w)) = w 0x. Our goal is that the model e will be useful for a downstream classification task which depends the ground truth features. A good representation will capture ground truth features that are correlated across augmentations, such that 0g is aligned with j4g or -KK-
Training. We will study the the evolution of @ as we optimize a standard constrastive learning objective using gradient descent [14 p At each round of gradient descent, we sample fresh batch of m data points and their augmentations, (U, V) {(u(), e(() hie[m]. For each i, E [m]] we
compute a similarity score 2ij == (f)) (u(()) fee((()) - >. (0% (9) 10% o using the dot product exp(z:;) of their K -ddiennsional representations. We then compute the logits P.j == Σpp' exp(=ag)) using the softmax function, and use the classwise cross entropy loss function
== -ogg(p))
5.2
Main Result
We will study gradient descent (GD) on the cross entropy loss, and consider how adding noise to one feature affects the learning of the other features. As suggested earlier, learn the kth feature by measuring the alignment of O2 with μa or - A natural way to measure this alignment is the acute angle between +u and 0g. given by arccos Appendix proves that this quantity directly determines the test accuracy natural downstream linear classification task.
measure how well
[+(0=) 0z)
[ Lemma E. in
Formally, we say we add noise to some feature ke' of a data point v, if for some 3 E [o. 1), we define the new noisy data point e to have coordinates نы Bu, + / B2g, where ~ A0 Ia), and Tg = D2 for l # k.. Thus if (u, v) were a pair generated with the correlation coefficients { [e([]] then the distribution of (u, ū) comes from the modified correlation coefficients {ā}KE[K] with the single modification a Sag. We now present our main theorem:
Theorem 5.1 (Noise improves feature learning). There exists universal constant C, such that the following holds. Let ((++1) (+)) -n(V(U,VV V) + (Q(()), where V is V with any amount of added noise in the k fea- ture. This has the effect of changing axk to ây for any dg.. < Oo.. Then for any l + W, if
()()) m(VVLU, + XO(1), and ((t+1)
10% pol < ---22 110x|l 110x-13 < po and |0x]112 < oc(1-ag)) then for a small enough step size [4Toktty)
]
[ a 1 110
Euyy
arccos
110 1100-"1> 1|2
arccos
We briefly comment on the three assumptions on θ in the theorem. The first assumption, 10 pu S
1- ||@x|| requires that 0₂ is not too aligned with #4: that is, the result applies to all features & that are not already learned too well. The second two assumptions are satisfied if the k'th feature has been learned some extent, and the norm of 0g and 0gr are small, which can be enforced throughout training with & regularization.
8

---

The theorem guarantees that at any point in training, if we add noise to the k'th feature, the next step of GD learns all other features better than if we did not add noise. To validate the implication of this result for the complete trajectory of GD, we include simulations in Appendix D Our experiments show that introducing noise part-way through training to dominant features can significantly speed the alignment of weak features, with only a small cost to the alignment of the dominant features. We prove our result in Appendix B including intuition and an overview of the ideas in Section E.3
Related work
Understanding contrastive and multiview learning Many prior works have laid the foundations
Several works analyze contrastive learning to identify important factors 12 58 or how contrastive models
for current contrastive and multiview learning algorithms 4 21
14
55
2
34
23 9.
differ from supervised learning [57 15 26). HaoChen et al. [22] study contrastive learning using
the concept of an augmentation graph. This model assumes the fraction of non-label preserving augmentations is "extremely small"; interestingly, we show in practice this can be quite large and still yield good performance. Wang et al. [54] thcoretically study contrastive learning under an assumption of label-preserving augmentations, though they show that such an assumption alone does not suffice to learn. Most relevant to work, Tian al. [52]; Ericsson et al. [16] study how the information shared between different views impacts learning of downstream tasks. We complicate this picture by analyzing the foundation model setting where single model must learn features for multiple tasks that are not known in advance. In this setting, we find that label-destroying perturbations, thought to be harmful by Tian et al. [52]. are useful for preventing one feature from suppressing others.
Feature suppression Our work is closely connected to the notion of feature suppression [24, where the presence of one feature can suppress the learning of others. Several works explore this concept in contrastive learning. For example, the original SimCLR paper [ noted that color jitter augmentation was necessary prevent the network from using only the color profile of the input to solve the contrastive task. Followup work [10] characterizes how hyperparameters and dataset features affect feature suppression, including through range of synthetic experiments that vary the amount of information shared between views. Other works have attempted to address feature suppression in contrastive learning, either via auxiliary losses (29) by modifying representations in the latent space [39] Our builds upon these works by empirically and theoretically investigating feature suppression as alternate rationale for the role of augmentations, as opposed invariance. We also show that an existing method, viewmaker networks [49), can identify and potentially neutralize suppressing features in an interpretable way, resulting better performance than expert augmentations, These insights may also generalize to other self-supervised settings, such as language modeling, where multiple features may compete
Spurious correlations and shortcut features Outside the framing of feature suppression, several other works explore how classifiers can learn make use of unwanted features. Shortcut features [1] describe often-simple features (e.g. the average color of an input) which are learned by networks the expense of more salient features (c.g. the object class). This notion is connected to spurious
correlations [45] in deep learning which have been explored extensively 40 41
46 53 (56). including
in the context of self-supervised Icarning (33 [TT Other works have also performed theoretical analysis of how related dynamics affect learning in the supervised setting 30 4m In general, if such spurious correlations are known in advance, one can often design augmentations to remove such correlations and improve learning. However, our work suggests that viewmaker networks may be a useful tool in cases where such features are not known a priori-both as an interpretability tool to visualize the different features a network relies on, and as way to reduce reliance particular features without completely destroying the information.
Discussion and Conclusion
We explore the idea that augmentations in contrastive learning function as sort of "feature dropout." First, we show that in datasets with multiple features, dropping out one set of features improves learning of the other features. Second, feature dropout may help explain how viewmaker networks can learn a wide range of features well, despite producing augmentations which appear to destroy different features in the input. Finally, we theoretically analyze linear contrastive setting where

---

prove that label-destooying views have a positive effect on contrastive learning if the goal is to avoid learning one feature at the expense of others.
Our work has limitations: for example, while our experiments consider image and audio data, self- supervised learning may be applied to much wider range of modalities 48] 50]. In addition, our theoretical analysis considers a linear contrastive setting, whereas current neural networks are highly nonlinear. Improving upon both of these fronts an exciting area for future work. On the other hand, understanding augmentations as dropping out easy features suggests possible ways of improving the performance of superviedd learning. For example, guided version of viewmaker might enable prioritizing a subset of important features for learning, or might enable dropping out unwanted features such as watermarks, sensitive information, other image artifacts.
The challenge of learning a broad range of useful features lies at the heart of self-supervised learning. We hope our work sheds light this challenge in contrastive learning, especially as these objectives continue to develop and are applied more broadly and at larger scale.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14199 2022.
[2] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurlPS, 2019.
[3] Soren Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. ArXiv, 0tls/1807.0533 2018.
[4] Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355:161-163, 1992.
[5] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kath- leen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie. Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong. Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafisky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mir- chandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou,, Joon Sung Park, Chris Piech, Eva Portelance, Christo- pher Ports, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'c Dorsa Sadigh. Shiori Sagawa, Keshav Santhanam. Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang. Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2 10t.22008.7788 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:187771901,, 2020.
10

---

[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244 2020.
[8] Shuxiao Chen, E. Dobriban, and Jane Lee. A group group- theoretic framework for data augmentation. arXiv: Machine Learning. 2020.
[9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
[10] Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.
[11] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Baindoor Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchin- son, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim. Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz. Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, Abss2204.002311 2022.
[12] Elijah Cole, Xuan S. Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge J. Belongie. When does contrastive visual representation learning work? ArXiv, flbs/2155.0587,, 2021.
[13] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, 181004805, 2019.
[14] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Dis- criminative unsupervised feature learning with convolutional neural networks. NIPS, 2014.
[15] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales, How well do self-supervised models transfer? 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5410-5419, 2021.
[16] Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. ArXiv, abs/2111.11333 2021.
[17] Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets, pages 267-285. Springer, 1982.
[18] Robert Geirhos, Hörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. ArXiv, abs/2004.07780, 2020.
[19] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Christopher Olah. Multimodal neurons in artificial neural networks. 2021.
[20] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent-a new approach self-supervised learning. Advances in Neural Information Processing Systems, 33:21271-2128 2020.
[21] Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) 2:1735-1742, 2006.

---

[22] Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self- supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
[23] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 9726-9735, 2020.
[24] Katherine L. Hermann and Andrew Kyle Lampinen. What shapes feature representations? exploring datasets, architectures, and training. ArXiv, abs/2006. 12433, 2020.
[25] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203. 15556, 2022.
[26] A.
Tarun Karthik, Mike Wu, Noah D. Goodman, and Alex Tamkin. Tradeoffs between con- 2021.
trastive and supervised learning: An empirical study. ArXiv,
[27] Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Enn Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech representations in the time domain. In 2021 IEEE Spoken Language Technology Workshop (SL.T), pages 215-222 IEEE, 2021.
[28] Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, 2009.
[29] Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogério Schmidt Feris, Piotr Indyk, and Dina Katabi. Addressing feature suppression in unsupervised visual representations. 2020.
[30] Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. ArXiv, abs/1907.044 005995 2019.
[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. ArXiv, abs/1907. 1692, 2019.
[32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV, pages 3730-3734 IEEE Computer Society, 2015. ISBN 978- 1-4673-8391-2. URL http://bbbb uni- trier
[33] Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. In /CML, 2020.
[34] Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant repre- sentations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6706-6716, 16, 2020.
[35] Daniel S Park, William Chan, Yu Zhang, Chung-Chena Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech 2019.
recognition. arXiv preprint
[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 1748-8763. PMLR, 2021.
[37] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv, 12292 2021.
[38] Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent, ArXiv, abs/2205.06// 100.0000.61...100 2022.
12

---

[39] Joshua Robinson, Li Sun, Ke Yu, K. Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems, 34:4974-4986, 2021.
[40] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv, abs/1911.08733 2019.
[41] Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameteriizttoon exacerbates spurious correlations. ArXiv, (ass/2005.04345 2020.
[42] Fatemeh Saki and Nasser Kehtarnavaz. Automatic switching between noise classification and speech enhancement for hearing aid devices. In 2016 38th Annual International Conference of the IEEE Engineering Medicine and Biology Society (EMBC), pages 736-739, 2016. doi:
10. LLL11100/...221...55559
[43] Fatemeh Saki, Abhishek Sehgal, Issa Panahi, and Nasser Kehtarnavaz. Smartphone-based real-time classification of noise signals using subband features and random forest classifier. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
pages 2204-2208, 2016. doi: 10.1 10.110//AASP22211.77770000
[44] Harshay Shah, Kaustav Tamuly, Aditi Raghumathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. ArXiv, abs/2006.07710, 2020.
[45] Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Association, 49(267):467 479, 1954.
[46] Megha Srivastava, Tatsunori B. Hashimoto, and Percy Liang. Robustness spurious correlations via human annotations. In ICML, 2020.
[47] Alex Tamkin, Dan Jurafsky, and Noah Goodman. Language through a prism: A spectral approach for multiscale language representations. Advances in Neural Information Processing Systems, 33:5492-5504, 2020.
[48] Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel Fein, Colin Schultz, and Noah D. Goodman. Dabs: A dogaii-agnostic benchmark for self-supervised learning. ArXiv, abs/21 12.062,, 2021.
[49] Alex Tamkin, Mike Wu, and Noah D. Goodman. Viewmaker networks: Learning views for unsupervised representation learning. ArXiv, abs/2010.07432, 2021.
[50] Alex Tamkin, Gaurab Banerjee, Mohamed Owda, Vincent Liu, Shashank Rammoorthy, and Noah Goodman. Dabs 2.0: Improved datasets and algorithms for universal sulervuuervis.on. 2022.
[51] Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task. arXiv preprint arXiv:2204.0844 2022.
[52] Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv, abs/2005. 10243, 2020.
[53] Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics, 8:621-633, 2020.
[54] Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203. 34577, 2022.
[55] Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3733-3742, 2018.
(56) Kai Y. Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. ArXiv, abs/2006.09994, 2021.
13

---

[57] Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang. Shanghang Zhang, and Pengtao Xie. Transfer learning or supe-vuue learning? tale of two pretraining paradigms. ArXiv, his/2007.042344 2020.
[58] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? ArXiv, abs/2006.06606, 2021.
14

---

A Code release
Our code is open-sourced http:://stttub.cooo//llloofeuuuu
B Formalization of observation in Section 2
Definition B.1 (Invariance). A function f R - R invariant to a set of transformations if and only if f g(z) = f(x) for all z E R and for all G.
Definition B.2 (Augmentation collision). An augmentation collision occurs if. for two inputs Eas 3bb (1)
and set of transformations G, there exist Gu
(1)
.--..99 9u 96 1-..:96 99
(ma) (1)
(ma) E G for some Bay Thb E N
such that ga (1) o... 9i
(ma) (Ia) = 9b .... 90
(ma) (zb).
Observation B.3. If there exists an augmentation collision for inputs Ia Tb and transformation set G, and I is invariant G, then f(x = ((xa).
(1) Proof. By the definition of an augmentation collision, Ja .... ga (Ia) = 9b (m)) (m6). Applying
(ma)
(1) (ma) (xt).
ga
By the definition of a function, we have foga (1) (mo) (Ia) = f sogb o.... gu invariance, we obtain (xa) (x+)).
........ .....fe
Applying this observation, we observe that if the downstream labeling function f is invariant to a class of augmentations, then there cannot be augmentation collision for inputs with different labels. However, common augmentations such as brightness shifts can reduce any image to a black or white image, resulting in augmentation collapse between any two inputs.
C
Additional feature dropout experiments
C.1 Quantifying the importance of feature dropout
To assess the importance of label-destroying augmentations to the success of the viewmaker, we experiment with a setup where the viewmaker cannot destroy the information in the object class. To do this, we compute a mask around the object and zero out any perturbation from the viewmaker within that mask. We then perform pretraining and transfer usual.
As we report in T the accuracy of the CIFAR-10 class label drops precipitously. as expected. At the same time, the accuracy of two of the other objects remains mostly constant (shape and digits), while the accuracy for letters declines modestly (perhaps because the color of the letter is now able to suppress the learning of the letter class.
Viewmaker (C-10)
Mask Viewmaker (C-10)
Viewmaker (Object)
Mask-Viewnnnn Viewmaker (Object)
C+Shape 79.8
26.0
100.0
95.8
C+Digit 69.3
50.7
94.3
95.0
C+Letter 71.9
23.2
96.9
71.8
Table 3: Experiments with a masked viewmaker which is unable to destroy the object class. Transfer accuracy on CIFAR-10 (C-10) and the object task (Shape, Digit, or Letter). The Mask- Viewmaker has its perturbation masked such that it cannot destroy the label of the object. This results in the features in the object suppressing the CIFAR-10 accuracy, while leaving the object accuracy relatively unscathed.
C.2 Quantifying the degree of feature dropout
We perform an exploratory analysis to testing how well different views drop out the features in an input. We augment 1, 1,200 examples (CIFAR-10 image plus an overlaid object) using given augmentation policy (either the expert or viewmaker augmentations). We then encode the model with classifier trained off of the other augmentation policy (i.e. expert for viewmaker augmentations
15

---

M
M
RR
M
M
N
R
G
Figure Non-label destroying Viewmaker perturbation examples.
Pharent
Assrost)
(a) Viewmaker Shapes
(b) Viewmaker Letters
(c) Viewmaker Digits
0.8
224
034
92
(d) Expert / Shapes
(e) Expert/ / Letters
(1) Expert Digits
Figure 6: Viewmaker augmentations stochastically drop out simple features added to the input. Probability of the correct answer for different augmentations (Viewmaker or Expert) and different examples from different datasets (Shapes, Letters, Digits). Each histogram shows single example from each dataset randomly augmented 220 times, and the corresponding probabilities of the correct answer. The viewmaker augmentations display bimodal structure, indicating that the simple feature is selectively either destroyed or preserved. The expert augmentations by contrast lack such structure, reflecting their lack of adaptation the structure of each input.
the reverse) in order to test how well the augmentations drop out the features. We different encoder to see the effects the augmentations prior to the encoder having a chance to adapt to them.
We observe bimodal behavior for the viewmaker views, shown in Figure 6 suggesting that the model adapting to the semantics of the input and has learned to stochastically drop out the simple feature some fraction of the time. By contrast, the expert views display no such structure. Using the corresponding encoder and views leads to models performing uniformly well, shown in Figure 7
C.3 Additional visualizations for CelebA
We show feature dropout histograms for CelebA for each of the 40 features in Figure The prevalence bimodal distributions demonstrates high degree feature dropout across attributes. Histograms shown are viewmaker augmentations encoder trained with expert views.
16

---

Paaraidd
Pix Recend
(a) Viewmaker / Shapes
(b) Viewmaker Letters
(c) Viewmaker Digits
(d) Expert/ / Shapes
(e) Expert Letters
(l) Expert/ Digits
Figure 7: Evaluating views with their respective encoder does not reveal bimodal structure for viewmaker or expert views. Details are the same as in Figure 6 with the exception that views are evaluated on their corresponding encoder.
We also show views and diffs for CelebA in Figure 8 These views show high degree of sensitivity to the input semantics, and appear to modify characteristics such as the background, hair color, and facial features.
C.4 Importance of Gaussian Blur Transformation
We show experiments for CIFAR-10 (with no overlaid features) on SimCLR (expert views) with and without Gaussian blur augmentations. We find that the inclusion of the Gaussian blur augmentation does not significantly impact CIFAR-10 accuracy.
CIFAR-10 10 Accuracy
Blur
86.3%
No Blur
86.1%
Table 4: Experiments with Gaussian blur augmentation. Transfer accuracy CIFAR-10 using expert (SimCLR) views with and without Gaussian blur.
D End-to-end Simulations of Linear Setting
We empirically test the performance of the full trajectory of gradient descent when we add noise to the data. We study a setting with one weak feature with correlations cocfficient a1 S 0.5, and 50 the same
dominant features with ag 1 for & 2, 51. We compare two approaches data: in the first approach, we run 150 iterations GD without adding noise. In the second, we run 50 iterations of GD without noise, and then add noise to the dominant features for the remaining 100 iterations.
In Figure 10 top), we compare the alignment of Feature (the weak feature) and Feature 2 (one of the dominant features) the ground truth in the two approaches. We observe that adding noise consistently accelerates the learning of the weak feature (blue), with little cost to the dominant features (red). The affect consistent among many choices for a,, the correlation coefficient of the weak feature. We also plot in Figure 10 bottom) the probability of predicting the correct class (pair)
17

---

Expert view
Viewmaker view
Viewmaker delta
Figure 8: Comparison of viewmaker and expert augmentations CelebA. The viewmaker augmentations adapt to the particular semantics of the input data, and make targeted perturbations that alter features such as facial features, hair color, background, and skin tone. Columns (from left): Expert augmentations, viewmaker augmentations, difference between original and viewmaker augmentation, rescaled to [0. Center image in each grid is the original.
18

---

Most Celeb-A features are dropped out by viewmaker. Accuracy of a linear classifier trained on expert images and evaluated on 2000 different augmentations of different images. Over two-thirds of the features exhibit bimodal structure, indicating feature dropout by the viewmaker augmentations.
Figure 9:
of the view under both approaches. We observe that this probability drops sharply when we add noise, which we believe the mechanism for faster learning with noise.
We remark that we chose to add noise to all the dominant features (instead single k'' a in our theorem) to accentuate the effect of adding noise. We observed similar effect, but smaller, when added noise to fewer features, or when there were fewer than 50 dominant features.
E Full proofs of propositions and theorems
We begin by stating and proving Lemma on the downstream classification accuracy.
Lemma E.1 (Downstream classification accuracy). Suppose we draw labeled data points (u, y) E R Kxd x {+1,1), where as before, uy ~ NO, Ia) for [K], and the label is given by sign(ui uu (Ax) Then the best linear classifier a E I2 on the representations fe E R achieves an test error of
arecos [nTeul) That is
[ 0cl
arccos
0x
min Pr(sign(a" fe(u)) # sign(uς ux)] = aekk
(1)
19

---

13673
Piier
Figure 10: Alignment of features with verses without added noise. From left to right: a1 0. 0.125, 0.25, 0.375,, 0.5. The top plots show the alignment of Features (weak) and 2 (dominant) the ground truth; the bottom plots shows the probability of predicting the correct augmentation pair from the batch. Standard deviation bars shown for the mean alignment over 200 runs. We used dimension = 5, and a batch size of m 25.
Thus if 0g and pa are orthogonal, then the test error is 50%. If the ungle between Og and the t.p is zero, then achieve perfect classification accuracy.
Proof. It is easy to see that the best linear classifier a will (up to scaling) be equal to the vector sign(uf (K]6))))))) Jek. Such classifier predicts the correct sign whenever sign( (an fe (u))
[ oul 1
sign(u 0k) sign(0} ux) equals sign(uţ ux), which occurs exactly of the time.
fraction
In the rest of this section, we prove our main theoretical result, Theorem 5.1 which shows that
1 1C
arccos
decreases faster in expectation during gradient descent if we add noise the k
10.12
feature.
E.1 Notation.
We let 6iy denote the 5-function which equals 1 if j and 0 otherwise. For parameter θ (0x (f]]))])) we let 0!! == uu 0g be the projection of 0g in the μı direction. We let OF = 0g 0 be the projection 02 orthogonal the feature μag
Throughout this section, we consider the ground truth directions to be fixed, and we fix some initial correlation vector We let Pa denote the distribution from which the pair (u, v) is drawn from the Gaussian distribution described in Section with correlation coefficients a. When unspecified, the variables U, V drawn from the distribution Im... Since study what happens when we vary ag,, for x E [0, 1]. we use the shorthand P,, to denote the distribution a(a)* where axx) = x, and a(x)+ aę for all other kt.
We denote L: (,;VV CE((Ptj jejj in, e4)
log(pa), which abbreviate by L. When it is clear that we are considering C; for some fixed i, we omit the superscripts on the ith data point or its pair. That is, denote 24 us
(4) and 1g = tx (1)
E.2 Preliminaries
The following facts about of the derivative of the cross entropy loss are easy derived. Lemma E.2.
Oc,
Ozig
Ozig 00
Ozh
Σ (prj - diz)
ΣΣ Pii
(2)
00
00 S
00
where
0zij
(u ( T
TT
(1) (1) kk
4
)
(3)
00,
We will also need the following facts on Gaussian random variables. The first, Stein's I Lemma, well known.
20

---

Lemma E.3 (Stein's 's Lemma).
Ex~N(0,03) (X)0) x x
(4)
The next two lemmas are proved Section E.4
Lemma E.4. There exists some constant C such that following holds. If S j and 1 then for any (0,,, and X (2, we have
Ex [xp(1) (xxxp(()))) S Co.. If additionally d e (0,1,2,3), S t and Y NNO., p²). then
(5)
Ex [X exp([ (XX(( S Co
(6)
Lemma E.S. For some universal constant C, for any o E [0, 11. 2 0 30,1,,,3,4), we have Ex~N(0 [(exp(t|XX) - 1) X & Ct
E.3 Approach and Lemmas
Intuition for proof of Theorem 5.1 Our proof involves comparing the gradient of the loss in the 6g direction, V == 36
2 C in the setting with noise to the setting without noise. Loosely, our
goal is to show that for any k, the projection of the this gradient onto the ground truth direction, 14 V sign fz), increases when when increase the noise. The main intuition comes from an expansion of this gradient in Lemma E.7 which shows that Eyg Va sign(p? f2x) approximately scales with E, (1 Pa). Now observe that Pas the probability of correctly matching the ith view to its pair, decreases when we add noise to feature k'. Thus adding noise will increase ni V sign(u (k), thereby improving the alignment.
In the remainder this section, we outline our proof Theorem 5.1 in this section. We prove all the lemmas below in Section [.4]
+) I
To understand Egy arccos
for small enough step size, we first claim that it suffices
[ 112
to understand the expected projection of the gradient with respect to 0g in the μi direction and the 9g direction. We use the notation V aL().
00
Lemma E.6. Let Of = 0x - 1ha-7(Ve (V k + 20x). Then
1u (u 110 1
[k
110xll2 ))
NEEUV (( 0 (
lim 7-
arccos
arccos
10x
where is some negative value that depends only on Oz.
Now, since we care about the quantity Ey,y arccos being positive, it suffices to show that derivative
[n InTo(112)) 1e 1|2
Ing a (egace121 1 1a
arceos
d
du Eu.vrP
10
negative for all E awr) Indeed, from Lemma E.6 we have that
1
[IuTOL] |
)
[ On|
lim Bui 900 17
arccos
EU.V arccos
(8)
11 112
110x|12
aak' d N
0% V (
Bu.v~R
- (Pf
dx,
(9)
dx
10.3
so if the derivative is negative for the full range, then the difference in arccosines is positive. In the following lemma compute the derivative of [ with respect to
21

---

Lemma E.7.
d
aL; 008k
dr Ru,
[vk]
Bu. VN
dr
Bu.v.R. PisPie (fE (nt opt o (olay - znl )
)
x² 0 Pk
00k
We will analyze this quantity by explicitly taking the expectation with respect to some set of random variables. Let S {UR, Va, Un, ] consist of the random variables u uig and ugg (())
(8)
for all E [m]. Define dij to be the logits when all variables in S are set to 0 (Thus explicitly,
exp Expr,er of of
qui
We will use the notation j ~ q denote the distribution on En exp DErL " (
[m] with mass Gig onj. j.
Let
h(S)= = (w) (uE - -xpke
(Dley,
znl -
(10)
30k
and
hu (S) = ux)) (11=22)t 200h (1u (0
(11)
which are the terms that appear the right hand side of Lemma E.7 /atter PaPpi- Observe that Es(h(S) hs()
The following four lemmas serve to bound d Es [ V and de Es 10 V We call the terms of the form (PPPPi(h(()) h (S)) "junk" terms, and our goal will be show that these terms are small. We will control more closely the terms of the form RppsPis (h, (S)).
de
Lemma E.8 (Junk Terms for uou term.). !f |1000 < and [|0x/] S 1, then for some universal constant C
(M(() Es [popisnl (M(S) - hu (S)) S Cqaai (100s 100011110000 00 |10 111 + ar (100-11110(11))
Lemma E.9 (Good Term for μuu. term.). If f1&11 & and ||0 || S 1, then for some universal constant
C
Es [PaPls h1 (S)] 2 2ax(1 x² q²)g (Iee) (1 11 110x112))
Plugging these two lemmas into Lemma E.7 yields the following corollary.
Corollary E.9.1 (Total l4 term.). If for sufficientll large constant C, 10T mel I 0 ||0 @| < 1s], and |0 < a1 (1-a) then
1-a
110
( 0k) dr
d Er, [ V
2 m
quliy20 1001 11 Σ
Lemma E.10 (Junk Terms for O2 term.). If 00xl S and |10x-11 S 1, then for some universal constant C
Es [papss In (S)) S Cqaai (10 110xa1 |101 10 1001l+ + an (10 11 [110!! + !10l 111 (1100111100)))
Lemma E.11 (Good Term for 0g term.). If [10xll < and ||0 |l s 1, then for some universal constant C
Es [PaP, hz (S)] s (1 22 (101 [IIe!(]))) (1 C(11 ([&x12))
Plugging these two lemmas into Lemma E.7 yields the following corollary.
22

---

Corollary E.11.1 (Total 0g term.). ffrr sufficiently large constant C, 110 1|0x11 < 2x(1=x²) then
|10 - - 10x:1. 110
(1T 02
Bp.
[o
V
- 2
m
Eus
Σ
qua an
10 11
0
dx
2,3
Combining Corollaries E.9.1 and E.11.1
we obtain the following lemma.
Lemma E.12. for sufficiently large constant C, |10 - - 110x1 1|0x 1 z 10% 110x11
a1
then
(
Eu.v~ (uEO))(u
0. 10
(12)
Theorem now follows.
E,4
Proofs of Lemmas
To prove the Lemmas E.4 and E.5 we will use the following well-known formula for the moment generating function (MGF) of the half-normal distribution.
Lemma E.13 (MGF of half-normal distribution). The MGF of the half-normal distribution is
Ex-Noo.njxsaeee 2e (2
where (() is the cumulative distribution of a normal random variable.
Proof of Lemma E.4
x²
Ex exx))e
exp(tx²)
dr
0v2
20
vi
2
x.2
|o| exp(t|<=) exp
dx
V1 V1=
2T
2
V1-20-
20 2022|B22~w000))22222222 exp(1Z)),
where r VI-20² To evaluate this, we use the MGF the half-normal distribution in Lemma E.13 Thus for constant C, for all (1,2,3,4}, 4),
d' e dtc
Ex~WOOD)IX0
|[ < EEx~10.1)J330
cC(1+())
So for some constant C (whose value changes throughout this equation), so long as
c
V1 11-22 exp(tz)
V1 of expx(((tt
< VI - (t))))
< Co..
This proves the first statement in the lemma. To prove the second, we first take the expectation over X, and using the half-Gaussian MGF before, obtain
By [|]TУ] (X)ex((xxxx[)]) s CEy [ + (1+ (()))))
Now applying the first statement to take the expectation over Y. we obtain
Ey [[ +1 (t (Y (1) < Co
23

---

Proof of Lemmal E.S We prove the lemma by induction on Suppose = 0. Then by plugging in the MGF for the holf-normal distribution from Lemma E.13 for some constant C, we have
E [x.,vo,))xxooee XI
1)] =
(13)
S 1/ 11t
(14)
2
< 12 -1) + 11
(15)
S Ct2
(16)
thus
Ex-NOo. = - 1]] S Cto.
Now for 1, by Stein's Lemma, we have (for constant C),
3x=.w(0.02)11]]](( - 1)] Ex- e-1 sign sign(X)(e!Ix)) - 1)]
(17)
d x1xx
a Ex~N((0,0))
(18)
dx
o' Ex
(c-2) 2) (1x10 - -1)) + (
(19)
Cto
(20)
where in the last step we used the inductive hypothesis and Lemma E.4
Proof of Lemma E.6
First observe that
1nL. 0kl
))
7-20
lim
EU.v
arccos
110 11
arccOS
1102112
1
[ (0x 112) - v/ VVV
[ Okl
lim n n0 7)
arccos
arocos
110,.(1 77) V
110x|12
1
|KT (Ox- T- VWJI }
[ 0ml
lim n n 7)
-
EU.V
arccos
10 T Valla
arccos
110x112
d
|KT (O,, - v)) (0)) 10 17 1
EU.y
arccos
dn
since lim, +0 1-nx = 0. Now
[nE0s] Ozl 102112
d
|uL (Op -/ [/) (0)
arecos
(0) = arccos'
dn
[10 VAll2
dn
110 - V ll2
of V.
[u[0xl
sign( 0x 04 V 1x + 10
0z
T
arccos'
||0x1l2
O V 02)²
N
p Ou A V + (NT
10 12
where N arccos'
100 Ill 0.7 The lemma follows by taking the expectation over U, V. and observing derivative of arccos(2) (x)
[ 16
ezl
is negative whenever & is positive.
Proof of Lemma E.7 First observe that by symmetry, we have
d BU.VaR [v = m d EU.
ac;
dr
da
00p
24

---

To make this expectation easier to analyze, we express the random variable (U(x), (()) ~ P3 as an interpolation Gaussians in the coordinate ur og) Let & W(O, 1), and define (U, V) P1. such that 1
(8) 1 Ug/ For [0, 1), define (U),,, (z)) to have
(1)
H o (8) (z) u
( u + VI x²C,
(21)
and otherwise be the same (U, V). is easy to check that (U(z:) V(x)) P..
Now
d
aa, (O,VV,
d OL. V(a))
BU..or
dx
08k
dr
08,
Taking the derivative of the cross-entropy loss, we have
d (L(())
d dr
Σ Ptù
dx
00k
00
jt
dp
dp (4)
ke' (x) (izig 24s)
v (x)
dr
00k
iti
dza
(013)) 2sr)
Σ
PiyPas
duI u (x)
V1
00,
where the variables zıj and pij the similarity scores and the softmaxes from the data (U(x), Vzx)..
Here the first line is by Lemma E.2 and the second line holds by chain rule since
03sy
8sn does
a62
00g
not depend on Ugg The third line uses the proof of Claim E.14 to take the derivative of pij. and Equation 21 to take the derivative of 1E (x))
Now we reparameterizee p1 u
VIーzzh as follows:
T Pk Ug
T W Us!
x
u Ug (x).
x22
x2
Plugging in this reparameterization and
day du
th pi oa ux we obtain
(x)
d
OL,(O;U,V)
-1
PiyPei (f (1 u
(D(zs) zzz)
)
Bu.v~P
dr
00,
30
We now prove Lemmas E.8
10 and E.11
Since is fixed throughout, we drop the (i) superscripts and let ( and Uy We will introduce the following random variables, which are all independent, to simplify the exposition:
Notation.
Ey = T. Up for j # i. Thus Ey ~ NO,, 11
(i)
for j # i. Thus E) ~ No, ||0 112) E) = T (
* &. = ( uh + (o!)T(uz 04 Thus & ~ NO, 110 + (1-af)100 af)110/11).
u.. - E TTw. Thus E ~ (O. |10 11 112).
C (0 "uw awww). Thus C (0, (1 11-aa)))10 02.01011111
y (T "u. Thus y N1 \(00111!!11))
25

---

v' = ( Thus v' ~ (0, 10 11
7 (0 )z jmu.. Thus Y (0, 10 113) m/ (Tz)) Thus v/. NO, 113))
For any such random variable X, we use xx to denote its variance. Observe that
exp (0% ux Uz) exp (f (fF
exp
(8% uu u/k exp (0% ux 0
PuPii
Biingg exp 08% ( v exp (f (1)) B, Si exp 08% U [/²) exp (o (2))
We will use the following two claims in the proofs of all four lemmas.
Claim E.14. For B E {5,, EJn Ca, Eo So Thr The x'), let By
a 8s (0% Oh uf + Of
Then
Op:ii 03
S PuPPi (13,1 + 13,1 + 21E y 11
If addiiioonally E {Eg Sj. St. E, St Tk. ) and 7 1 {B3 [(m) then
0 OpaPis
S PuPi ((13)1 + |3:1 2 ~~ م~ B,,l)) (15sl + 15el + 21
07 03
+ PaPPy y 1 Fy. 1
Proof. By straightforward quotient-rule computation of the derivative of Pig diy
recalling that qi is
independent of S, we obtain
Pp:j
Pij (3, - Bypr))))P))))
0g
By applying product to the expression above, obtain
@paPis
03 PaPP (3, + B, - 3y
Taking absolute values and using the fact that Pı I 1, we obtain the first result.
Next we take the derivative of pay with respect to both 3 and 7. Using the expression above for dp: we obtain
0 Opri
Pij ((3g By (5g By
+ (y
oy 03
and
0 OpaPij
PiPPis ((3, + + 2 y (5g + + 2
0y 03
+ PaPli 1-2B Te Piy + 2(E ( Piy (Bzing Th
By
The second result follows by taking absolute values and the fact that Pij S 1.
Claim E.15.
Pij < exp (1T ok (221) exp (F Bynng [exp (10 ( (1))) exp OF
4ig
Proof:
This follows directly from using Jenson's inequality on the distribution jr ~ to show that
Eyinn [exp (- 1- uoF of exp (-0Fm))
(-0f
0 1n v (672) exp (ffum-of =ltr
Eyin
exp
Eyi eexp (10% (o exp (Z
26

---

Claim E.16.
Pij
1
S Zij 1,
Øij
where Zj := exp (10% uoT (1)) exp (10% ux 16 0 1) Byingg /exp (10% uxo u exp (10T ok 1)]
Proof. Note that for any x: 2 O, we have |1 - x I max (x - 1, - 1). By Claim E.15 Poj at most the desired value given in this claim. Now
is
eex (o OFuu ul) exp (f
Eying
Pij
exp (f unof vl2) exp O8 (6
S exp (10% un 122) exp (10 Byrnug (exp (10% (u exp
This yields the claim.
Proof of Lemma
E.8 Expanding 1(5) - hu (S), we see that we need to control the following terms:
papps (v% (n uw xFE )) (o (xو,)) )| (b) [Es [papl (v/ uu =xE[ew) (pt unEs)] 2. Bs [paPes (y) (KENE - m.f )) (MT u (b) Es [paPls ((v' (KZ - =uE (N ((])]
1. (a) Es
[a)
3. (a) Es [pepo (nk (pE - zh vw)) (M4 4x (l)) a [papes (=(()))() (ul uny)]l
4. (a) Es PuPPy (v% (Kg =ww (E((v=- - = psx)
(b)
Es pappi (y/ (nE ue =x((vw) (Si(ex -
- (()))T mmx
Es ppapis (7% uu - ap uy(u - -ux (1) pm )
5. (a)
(b)
Es pupls (y/ (Ny us vw)) (e(2 - aguz - o yT y p*)
We hegin by bounding the terms where the expression after PaPPi has independent mean-0 terms, mainly (1a), (2a), (4a). The first step is to apply Stein's Lemma (Lemma E.3) twice to these two terms, which will call / and y. Let ( (3,7)) be the terms after PaPaj- Then we have
0 @paPis Oy
Es(papi,B79(S (8, 1)11 S oz
Es
Lo(S \ (8,7))
03
Next we apply the final result in Claim E.14 to bound the absolute value of this, achieve
Opapi 0s
Once we do
Es 8rgp (B]+ S 230 Bu Zlg(S> (1 (8,1
Σ
cp B3 1ntll
3'
where Ey G S C for some constant C, and Z == PaPP Qae
Finally, we use the bound on Z
from Claim E.15 and then Lemma E.4 to take the expectation over S. iteratively applying Lemma E.4 to each variable in S. Thus we have, for some (different) constant C,
1. Es [purt (1t (nE =uf(vw)) (NT &n) Cauamll00 12100 11 < Cqaacill0 Ow | 10
10
2. Es [psiPij (1% (NEWE =)k(vw)) (1g unEs)] Cquqs,100 1100011110m11 Cquani 11100
Cqango? o2,10
110x [1100]]]
27

---

3. Ess PuPt (n% (p ux - xpZ (1(0 of y uzx )] Cquqso of o. Cququll0 | |110011 - Cquqyll Ox 19|00110.
1011111011 -
Now consider the remaining terms. Here we decompose the expression inside the expectation as PuPPiBB((S \ (), where 3 E S. We proceed as before, but we only apply Stein's 's Lemma once, to obtain
B. Applying Steins, the expression for SpeePa given in the first result of Claim E.14 ag
Es (s 8)] - o Es
OppPis 08
l0(S 81|
oug Es
2
Zlo(S \ B)
Σ
cy1881
J'E(m]
(22)
where Emmm Cy' C' for some constant C, and Z == PePP 9:
Finally, we plug in bound for Z in Claim E. an use Lemma E.4 to take the expectation over S, again iteratively over each variable.
Thus we have, for some (different) constant C,
1. Es [piePs (V' (pā - xxp[xx)) (0T uxEe)]
<
2 11
10xx)111xx191100
2. IEs [PuPls (3/ (KT -xpE(w)) (HT uxEi)]
Cquqsyo 10
(1a0 |
3. ls [paPls (v% (KÉ. uN =x) (1f uxy)] Cauauaii 01 11 I110@111 S Cauquiis | |1
Casqa4yo |0 111111100111
4. lagEs [PaPes (=(()x()) (Mf uwy)
S
22 10
Cou.aaa 110 11 1110
(()) 5. Es puiPs (w' (KÉ - =x) (Ei(vh- ( ( s 00 S
Clannll0u 1 11100%
6. Lu (papis (7% (NE vxf( (I(UVs - o² pm
W
Coumuuyeo 21000 110000010000 Cangaaii 11 S Cangeess1000 11110
7. Es ppipis (v' (NÉ - vwr) (o((r= - axuu - ( u pm
W
Cquq 2110x11100 = Cquqs 1110
Combining the bounds on these 10 terms proves the lemma:
Ess [papesnll (b(S) (b(()) - ha (S))] < Cqugi (100 (10w11"10021 + 11l 111001 + an (10x-11(10111))
Proof of Lemma E.10 The proof of Lemma E.10 is nearly identical, besides some differences in the terms we need to bound. We list them below:
1. (a) Es [papls (vl (NEmur =ub (0% [x] (b) Es [papts (v' (vE - mn (oz ux$;
2. Es [papy (vt (NEMU - mmE (o7 [i)] (b) Es [paps (tv/ (KE - TTE (0% [x]]
3. (a) JalEs [pupl (vt (K us xp( (0% u (b) (b)| [pupys (v/ (KT 2p((.w)) (na)
4. aayE [puPl (v/ (-x(1)) (0% uuy)
28

---

We use the same approach as before. For the terms (la) and (2a) we apply Stein's 's Lemma to (7% E3) and (1/- E.) respectively. For (1b). (2b), (3a) and (3b) and (4), we apply Stein's 's Lemma to Ej- Ei. 1/6- Mi and E respectively. Using Claim E.15 and then Lemma E.4 as before, we obtain the following result:
1. Es [psiis (1b (p - vx) (0 [x]))]]
S Cquqnoo |10 [+11110
Cququll0 191 s Cquaull |
2. IEs [PuPls (vb ux - x (0% uxE:)]
I Cquqyo
o/ o0 |00 |II| |||
Cquqiyl18 12 - Cqaqi 022 [1xx=11
3. EEs [PuPu (v/ ( - =x (0% ux,,
Cquano 10 |II
|
4. Es |PSPes (V' (Pt xx (0% uuLii
Cquq,o [10a|] 0
5. la.E [paites (n/ (KŽNu =xkeww) (0% uuy)] Cauquaiy ( |1 1110
Co 1100 |1le
6. layEs [papls (v' (H ug zx)) (nwy)
Canqaaio |1
!lll!
Coununy 0 |
7. aaıE [puPls (v' (-x()) (Of uay)]l
S
Couqaguo 10 111
Couduai [1 || IlI !l
Combining the bounds on these 7 terms, proves the lemma:
Es [PiPt5 (h(S) h2 (s)) S Cq (10kr (10%) 11 + |l 1110x11 + as (1ew (10x~1111ee 1111111111 Iell 1|
We now prove the lemmas on the non-junk terms.
Proof of Lemma
E.9
(ol:)"uwuuf (2ui (oa) us))
Es PuPii
Es (aans (1 uu (2 (e!)) ux)] Es (priDes (agiss) ((e)) P Fuunugur (u mxx)
20 pe 20
(
PiiPis
-1)
(1e))) uu (ut us (Bll) ux
Pis
Now by Claim E. we have
PepPe
1/ s Z 1 (where the variable's Z;, By are defined in the
Claim
1.1 so
PuPPi 9:a4i
1) (10)) (umuttt)))) (uf
< Es (Z,Z, - 1) (ell) ug ui por
pi un (e!))
Es
C (1 110x-11²) |10] |11|@!!1
Here the second inequality follows from applying Lemma for the remainder of the variables in S. This proves the lemma. Note that we need to apply Lemma B.5 several times to a single variable X E S. Indeed we can write
E.5 first, and then Lemma E.4 repeatedly
(Z,Z, 1) (el)) )T uu uff pe/ [u (o.))) u (E, cxp((te 1) X|
(ExS((exp((txx))) 1)) B| + (E,S: 1)) B|X
29

---

for some distribution on é, and for some terms St, tg, and B that are independent of X, and E {0, 1. Then to take the expectation of this term over X, we first apply Lemma E.5 to on X the first term, and iteratively apply Lemma E.5 to the random variables appearing in the next terms. I
Proof of Lemma E.11
1 x²
Es (n [n)] Ess
(2(0!!)) una (ol)) )]
PiPPij
Es 4ag ((0))) uu (E (20x((0)))) un))))))))))) un)²)]
uum (20m((0l))) ((oo)((TT))))) mm)))]]
+ Es
(poPis qu9is)
( qietes
(Pillss -1) (10 u (10 99.
2anqu0 0h Me 110 + 2an
Now by Claim E.16
we have
PicPe
1 S - s,
PiiPij
1)
(uu (109)) ux) )
(10))
$ Es (Z,Z, -1) T (10!) )
Es
qu9iz
I I (1 2 + 11 |10| |10!11?
Again the second inequality follows from applying Lemma E.5 first (several times as described the previous lemma), and then Lemma E.4 repeatecyy for the remainder of the variables in S. Taking absolute values proves the lemma.
30


Reviews:
[{"review_id": "W680313Kxt", "reviewer": "Reviewer_1FJs", "summary": "The authors of this work challenge the belief that label-preserving augmentations are necessary for contrastive learning by demonstrating that label-destroying augmentations can be useful for learning representations useful for a variety of downstream tasks.  The authors treat augmentations as feature dropout, and demonstrate how straightforward contrastive learning techniques struggle to learn multiple features that are present in a visual scene.  This was demonstrated through synthetically created datasets.  Furthermore, the authors discover that feature dropout helps learn representations that encode multiple competing features - the authors believe that this is what explains why Viewmaker networks can help learn a wide range of features well.", "questions": "It is encouraging that Viewmaker networks seem to overcome it due to feature-level dropout.  Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations? \n\nWhat are some ways this idea can be demonstrated for more natural-looking images?  Currently, the synthetic datasets created indeed highlight the feature suppression affect inherent in contrastive learning techniques.  However, are there any practical cases of natural images where we would like multiple features to be learned from a given image that could showcase the approach outlined in the paper?\n", "limitations": "No negative societal impact.", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "The strengths of this work lies in its thorough experimentation and its theoretical justification regarding the benefits of learning representations of multiple features when noise is added to corrupt a particular feature.  The work was clearly and well-presented, and convincing arguments were made when challenging previously-held beliefs regarding augmentations for contrastive learning.  Furthermore, linking the idea of stochastic feature dropout with the success of Viewmaker networks is an interesting discovery.", "weaknesses": "In the introduction, the authors suspect that augmentations serve as a form of feature dropout that prevents any one feature from becoming a shortcut feature and suppressing the learning of other features.  This idea has been noted before in prior work, as also mentioned in the Related Work section (in “Feature Suppression”).  In the original SimCLR paper [Ting et. al, 2020], it was discovered that certain combinations of augmentations improved the representations learned.  In particular, color jittering was found to consistently improve performance; it has been speculated that the addition of color jittering as a data augmentation helps to destroy the color histogram values for pixels from the same patch, which can easily shortcut the recognition of two positives.  Therefore, it is already generally accepted that augmentations in contrastive losses serve the effective purpose of feature dropout.  On the other hand, the authors utilize a set of synthetic datasets to determine if certain features suppress the learning of other features; this line of work was also previously done in prior work [Ting, et. al, 2022].  It must be noted that the original authors of [Ting et. al, 2022] did not attempt stochastic dropout of the secondary features.\n\nThe paper can be made better with improvements to the formatting - for example, the entirety of page 5 is devoted to a singular figure.\n\nI believe most of the entire work is predicated on the feature suppression effect, which go through great lengths to show even though it is a well-known property of contrastive losses.  I agree that the authors explore this idea thoroughly in their work, extending it to multiple modalities, adding theoretical justification, and explaining the potential reason VIewmaker networks work well, but fundamentally I believe the work lacks substantial core novelty beyond what was already established in prior work.\n\nTing Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.\n\nTing Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful review! We are glad that they found our experiments and theoretical results to be thorough. From our understanding, the reviewer’s main point of concern is clarifying the paper's additional contribution on top of the two papers cited.\n\nWe are happy to state this more clearly, and include the below discussion in the paper:\n\nAs the reviewer notes, our paper focuses on “feature dropout”—how stochastic ablation of features in an input  can broaden the range of features learned by a network. This investigation complements the two works cited: [1] conjectures that stochastically destroying color information is useful by qualitatively comparing color histograms (Figure 6), while [2] shows that varying the size of MNIST digits (MultiDigits), number of MNIST digits (DigitOnImageNet), and number of shared bits (RandBit) can affect whether one feature is learned more than others. Our study adds to the existing literature in four ways:\n1. Studying how augmentations can reduce the strength of features. In particular, this enables us to study the phenomenon conjectured by [1] using similar tools as those considered by [2].\n2. This choice of studying feature dropout is especially important, as viewmaker networks seem to prominently display this dropout phenomenon. In other words, a neural network **emergently discovered** the same phenomenon that human experts have speculated is important for learning good representations! This contribution has both scientific interest as well as empirical implications, given the promise of generative views in settings where good augmentations are not known (e.g. varieties of scientific or sensor data).\n3. We go beyond synthetic data to conduct an in-depth study on the naturalistic CelebA dataset, which has over 40 annotated attributes (Section 4, Figure 2, Appendix D3, and Figure 8).\n4. We have a novel theoretical contribution that is not present in recent work, demonstrating theoretically how feature dropout can aid learning.\n\nIn sum, there are multiple distinct contributions in our work that complement [1] and [2]. We will include this discussion in the next draft, including highlighting papers [1] and [2] more prominently in the introduction and related work, and would be more than happy to incorporate any additional suggestions from the reviewer for framing these contributions in the discussion period.\n\n\n**Q1: Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations?**\n\nWe believe viewmaker networks are likely a very good starting point for researchers in domains where good augmentations are less well-studied (e.g. audio, many types of scientific data, wearable sensors, etc). However, they are likely to be less helpful in cases where we have well-developed augmentations that encode strong priors over what features are helpful to humans (e.g. in natural images).\n\n**Q2: Natural looking images***\n\nWe believe our study of the naturalistic CelebA dataset, which has over 40 annotated features, is a good example of this direction. We show that viewmaker augmentations enable the encoder to capture a much broader range of features in this setting.\n\nWe thank the reviewer again for their thoughtful comments and review!"}, {"review_id": "moIGMmeT7n", "reviewer": "Reviewer_eJFh", "summary": "**Summary**. The authors question the common assumption in contrastive learning that good augmentations should be label-preserving, and suggest that dropping features out (for example, with data augmentation) actually is helpful in practice because it helps learning of \"weaker\" features: features that may be helpful on downstream tasks, but are not necessary for the contrastive task. To do this, they create synthetic datasets, use view-maker augmentations, and consider a linear toy example.\n\n**My position.** I think this work is interesting: I find it interesting to read, but there are some limitations in the work that make me question it's central thesis. Overall, I marginally would like to see it at NeurIPS because I found it interesting and think it would lead to an interesting discussion, but some technical concerns limit my enthusiasm about the paper.", "questions": "1. Do you think feature dropout changes the optimal solution, or just makes the optimisation process easier?\n\n2. L84: \"repeated application of some augmentations causes challenges for all downstream tasks\". Are people using repeated augmentation in practice? I'm not sure this is a real problem.\n\n3. Are the expert views really expert for CelebA?\n\n4. For Fig. 4, how are the features dropped out? They just aren't added to the image?", "limitations": "I would have liked to see further discussion of the limitations of the work that I mentioned above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. I enjoyed reading and thinking about this work, I think the question is interesting, and in general, I like the approach of questioning common assumptions in the field. The choice of data augmentation is clearly very important for contrastive learning, so that's great.\n\n2. The paper is reasonably well written.", "weaknesses": "# Major Weaknesses / Concerns\n\n1. The paper emphasises \"multiple downstream tasks\", but what this means precisely is unclear, and **I am not sure whether the experimental setup reflects the problems we actually care about.** In the experiments, the features for the added digit/shape, etc are basically completely independent of the rest of the image features. I'm not sure whether this holds in real life settings of multiple downstream tasks e.g., if classifying both digit rotation and digit identity, the features of the \"object\" are shared; does this count as multiple downstream tasks here? The paper emphasises \"the need for analyzing the interaction between multiple downstream tasks\" (L19), but I'm not sure, precisely, what this might mean. \n\n1. Instead of using synthetic experiments where another feature like a shape is overlaid on top of the base image, it would be better to use for example, a latent variable generative model, and downstream classification where different sets of features only are important for the class label. I think this would be closer to real world scenarios.\n\n2. The paper quotes Tian et al. \"a good set of views are those that share the minimal information necessary to perform well at the downstream task\". I think this is a slight strawman (i.e., simplification of the prevailing view), because it is well known that \"heavy\" data augmentation is necessary for good performance (e.g., experiments from the original SimCLR paper). I suggest making the nuance more clear, and make clear the aim of the work is to help clarify this confusion.\n\n3. There are no error bars on the experiment results, though multiple seeds are run.\n\n4. I'm not convinced that the toy example with a linear model captures the important aspects of the problem. I don't think that learning the last linear layer after a feature is what is challenging about feature learning, but rather, it is learning the complex, non-linear representation. I am more interested in how dropping out features would affect learning of the representation, rather than learning of the feature layer. (See also Suggestion 2).\n\n5. I am concerned about sensitivity to the viewmaker pertubation budget, $\\epsilon$. As far as I can tell, the viewmaker objective basically always tries to make the contrastive task as hard as possible, which is to remove as much information about the base image identity as possible (i.e., remove information about the shape as well as the base image), because the contrastive task is to identify the base image identity from augmented pairs of images. If the budget is too large, all the images would become noise. The objective directly incentivises feature dropout! Therefore, the choice of $\\epsilon$ is really important. I want to understand how it is that you chose this, and how sensitive your results and findings are to it. \n\n6. I don't think the expert augmentations are really \"expert\" augmentations in the sense that they don't preserve label information for the synthetic datasets. Some of the augmentations will destroy information e.g., by flipping a 9, it no longer becomes a digit. I would like to see augmentations where you only flip e.g., the digits that are invariant to flipping. Then, the comparison between viewmaker and expert is more fair.\n\n7. The results in Table 1 I find hard to interpret. Is the claim that the view-maker does feature dropout, and it performs better, suggesting that feature dropout is good? Because the difference in performance for shape are small, the performance for both the object and image class is poor for digit, and for letter, the expert augmentations outperform viewmaker. Also, should it not be possible that we can add the synthetic data without losing performance on the CIFAR-10 task?\n\n8. \"Two clear modes appear for viewmaker, but not for expert augmentations\" (L181/182). Some of the expert augmentations seem to have very small probability of the correct label, suggesting that they perhaps always dropout the label? I was surprised by this. (This is ~Fig. 6 in the appendix). Can you explain this please\n\n9. It seems many of the experimental results assume that viewmaker is doing feature dropout, and the expert augmentations are not i.e., the expert augmentations are good. I'm not sure this is actually the case.\n\n10. WRT the theoretical analysis, I have some concerns. (i) the setup is not like contrastive learning. Here, we have $K$ features in a higher dimension space, but really the feature space is low dimensional. But contrastive representations live on a hypersphere, with the contrastive loss encouraging points to spread out across the hypersphere. (ii) \"A good representation will capture ground true features ... such that $\\theta_k$ is aligned with $\\mu_k$ ...\" I am confused by this; isn't there permutational and rotational invariance here? i.e., $\\theta_1$ could be aligned with $\\mu_5$. This, as a result, undermines your theoretical results.\n\n11. In figure 4 in the appendix, the synthetic feature dropping out only improves the accuracy of the main image class quite a small amount, and dropping the feature out also drastically reduces the accuracy of the synthetic task. What one should do in this scenario is quite unclear. Perhaps we would do better by just constructing new, strong data augmentation that preserves label information. That seems to be the fairest comparison.\n\n\n## Other suggestions\n1. L22: pitching the paper as foundation models seems like overselling because the work focuses on contrastive learning.\n\n2. If the hypothesis is feature dropout, one could also try dropping out features that are learnt during contrastive pre-training e.g., adding dropout regularisation before or after the projection head. This would (approximately) correspond directly to feature dropout, and would be somewhat similar to the linear contrastive example you use. Do you think this would work? Have you considered this?\n\n3. Figure 4 and Figure 6 should not be in the appendix if they are mentioned in the main paper as prominently as they are.\n\n4. I suggest streamlining Figure 1 e.g., including only one row, making it more visually clear which image is the base image and which images are augmentations.\n\n5. I would like to see the accuracy of the contrastive task (i.e., how accurate is the linear layer when the class templates are the mean representations of augmente images) for the different viewmaker augmentations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review! We are glad they found the results interesting, and appreciated our “questioning common assumptions in the field.”\n\n**[Q1 and Q2] Do our synthetic experiments generalize to the the real world?**\n\nThis is an important question, a big reason we conducted experiments on CelebA, a naturalistic dataset with over 40 annotated features per image (with complex interactions, as the reviewer suggests).\n\n**Q3: Adding nuance to discussion of prevailing view.**\n\nWe agree. We can add something like the follow line (but are open to feedback): “However, the strong augmentations used in real-world settings [Chen et al, 2020], complicate this picture, lending the need for further analysis”\n\n**Q4: Runs have multiple seeds but no error bars.**\n\nWe were not able to run enough trials for error bars to be useful, but believe the three seeds we ran go help reduce any variance in the results.\n\n**Q5: Validity of linear model**\n\nWe agree it would be interesting to examine non-linear feature learning as well, however there are few theoretical tools that can do this at the moment. We do believe that this linear model captures the “important” aspects of the problem (the impact of feature dropout), as the linear model can be viewed as learning a representation capturing K features from a high dimensional dataset.  \n\n\n**Q6: Choice of budget**\n\nWe agree the choice of the budget is important. The original Viewmaker paper found [Tamkin et al, 2021], as well as a recent preprint [Bayrooti et al, 2023] show that good performance is possible at a range of budgets. We also found similar results (below).\n\n|Budget|CIFAR Acc.|Letter Acc| \n|-|-|-|\n|0.1|0.719|0.969|\n|0.125|0.717|0.949|  \n|0.15|0.721|0.957|\n\n\n**Q7: New experiments without flipping.** \n\nThe performance decreases for both the CIFAR-10 label and the digit label without horizontal flipping, suggesting that it aids (rather than injures) MNIST accuracy.\n\n||CIFAR Acc|Letter Acc|\n|-|-|-|  \n|No Flip|0.693|0.943|\n|Flip|0.572|0.923|\n\n\n**Q8: Interpreting results in Table 1.**\n\nYes, automatic feature dropout performed by Viewmaker enables the model to better capture features across tasks. Accuracies are generally lower when synthetic features are overlaid, but this makes the task harder overall so the decrease is expected.\n\n**Q9: Why expert views have small probability in Figure 6 (Appendix)**\nThis could be due: 1) no feature dropout, or 2) the models not being robust to the other model’s augmentations. Thus, while we can see that viewmakers perform feature dropout, it is not conclusive evidence of how much expert views are doing feature dropout.\n\n**Q10: Are expert views also doing dropout?**\nExpert views are likely doing dropout to some degree (e.g. see Figure 6 in Chen et al)! However, we believe viewmaker networks may perform feature dropout in an automatic and more thorough way.\n\n**Q11: Choices made in theoretical setup.**\n \n(i) We are not certain we understand the reviewer’s question, so we would be happy to respond more with further clarification. The K-dimensional representations are encouraged to be spread out by the contrastive loss, though other work (e.g. [Tian et al, 2021]) has found that the hypersphere constraint is not a crucial component of contrastive learning and can be removed. \n\n(ii) In our setting,  the total dimension of the data point is d*K, not d, and each of the K features is learned by a different set of parameters. Figure 3b may also help clarify the reviewer’s confusion.  We do not believe this undermines our theoretical result, but would be happy to clarify further given additional context in the discussion period.\n\n**Q12: Augmentations if label-information is known**\n\nWe agree (and will mention in our revision) that if label information is known, design augmentations targeting this information. However, in the general case cases this information will not be known in advance, nor will all the possible features that we want to learn be known.\n\n**S1. Foundation models framing**\n\nWe believe we study one kind of foundation model, at least as defined in [Bommasani et al, 2021] where SimCLR was provided as an example (but acknowledge the term is sometimes used to refer to LLMs).\n\n**S2. Dropping out features in projection head.**\n\nThis is an interesting idea and we have had similar thoughts. Our initial investigations suggested that the network would learn to make redundant sets of features to counteract the effects of dropout, however further experimentation could be interesting.\n\n**S3. Figures in appendix.**\n\n We agree it would be best to have these figures in the main text, and will use the extra page towards that end.\n\n**S4. Suggestion for Figure 1. **\n\nWe will experiment with streamlining Figure 1. We agree that only having one row may decrease the amount of space needed for this figure and thank the reviewer for the suggestion.\n\n**Q1: Does feature dropout change the optimal solution?**\n\n Yes, we believe so: for example, consider a feature perfectly predictive of the label (say a watermark). The model might only rely on that feature, ignoring the rest of the image. Dropping out the watermark would force the network to rely on other parts of the input.\n\n**Q2: Repeated augmentations.**\n\nYes, people do not do this explicitly. However, consider e.g. amateur photographers, who often mis-expose or bracket images, resulting in a broad range of brightness shifts. A brightness shift applied to a very lightest image could result in a pure white image, destroying the class information.\n\n**Q3: Are the expert views really expert for CelebA?**\n\n We believe they are a reasonable choice, as the expert views were designed for use on a broad range of natural image datasets. However, we certainly agree that better face-specific augmentations could be developed.\n\n**Q4: How are features dropped out?**\n\nCorrect, these features are simply not added to the image.\n\nWe thank the reviewer again for their thoughtful comments and review!\n"}, {"review_id": "d3nJ3OZqdL", "reviewer": "Reviewer_jcen", "summary": "This paper questions a common assumption on data augmentations in contrastive unsupervised representation learning, specifically label-preserving, where data augmentations do not change the label of the original sample. Indeed, this assumption does not always holds in data augmentations used in contrastive unsupervised representation learning in practice as explained in Sec. 2. The authors claim that such a label-destroying data augmentation can be seen as feature-dropout, and this would be useful for universal representation learning, where the learnt feature extractor can give good discriminative representations for multiple downstream tasks.\nThe authors perform such label-destroying data augmentations are useful on synthetic datasets and Viewmaker networks, which are previously proposed generative models to improve downstream tasks' performance of contrastive unsupervised representation learning.\nTo give a theoretical understanding of this label-destroying data augmentation, the authors provide a theoretical justification of the usefulness of noise under a linear contrastive model with Gaussian-based data distribution in Sec. 5.\n", "questions": "Q1: Could you give the details of experiments as pointed out by W1?\nQ2: L84: Is there any concrete example paper using \"brightness shifts repeatedly\"? In my understanding, SimCLR (Appendix A. in Chen et al. [2020]) applies it to a sample at most once.\nQ3: Clarification on data augmentation: \n\nL129 says\n\n> data augmentations for the image datasets [9]\n\nCould you clarify which data augmentation was used in SimCLR v1 paper? SimCLR paper used slightly different data augmentation depending on ImageNet or CIFAR-10. Concretely, B.9. in the SimCLR paper for CIFAR-10 says\n\n> \"For data augmentation ... leaving out Gaussian blur\"\n\nIn my understanding, the implementation of this paper uses the ImageNet version: not excluding Gaussian blur according to `load_default_transforms` and `load_default_unnorm_transforms` in `feature-dropout-code/src/datasets/datasets.py`.\n\nQ4: Can Figures 4, 6, 8, and 9 be in the main body? Their explanations are in the main body to discuss the proposed analyses but I needed to see them in the Appendix.\n\n\n### Typos and notation errors:\n\n- L83: `\\citep` for `Chen et al. [9] and He et al. [23]`?\n- Fig. 4: Fig 4(b)'s caption is \"Letters\", but a label in the legend is \"Digits accuracy\". Similar typos are in Fig. 4(c): caption is \"Digits\", but the label is \"Letters accuracy\".\n- L261: aren't -> are not\n- L265: didn't -> did not \n", "limitations": "As mentioned in Weaknesses, the analysed linear model is a more restricted model class than widely used linear models in contrastive learning paper.\n", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Originality: Questioning the commonly used assumption on data augmentation sounds novel problem.\nQuality: The empirical and theoretical results give a better view of the problem (commonly used assumptions should be revisited)\nClarity: The paper is well-written and clearly explains the motivation of the problem with concrete examples.\nSignificance: Unsupervised representation learning for many downstream tasks is a desired algorithm in practice; this result gives another research direction to create a better \"view\" for such representation learning.\n", "weaknesses": "## W.1 Details of experiments and results are missing\n\n- I could not find how to perform the transfer accuracy reported in Tables 1 and 2. The details should be given in the manuscript or Appendix.\n- The audio results in Sec. 3 are in neither Figure 4 nor others.\n\n## W.2 Assumptions on the theoretical analysis\n\nThe empirical and theoretical results have a gap in data, model, and data augmentation. For example, the analysed linear model has a sparse connection and is independent of the other weights as in Fig. 3(b). Still, empirically used models are non-linear and the weights are non-independent via a fully connected or pooling layer.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review! We are glad they found our research question novel and significant, and to give a better view of the problem.\n\n\n**W1a: Details of transfer experiments.** \n\nTo perform the transfer experiments in Tables 1 and 2 we use the same setup as in the Viewmaker Paper [Tamkin et al, 2021] (see L132-135). We will replicate the details in the paper for the next revision to make the paper more self-contained.\n\n**W1b: Audio results in Figure 4.**\n\nWe thank the reviewer for this observation. We will update Figure 4 to include the audio results. As with the image datasets, we see a stark tradeoff between features as the dropout rate is increased (the gap in accuracy changes by over 50%)\n\n\n**W2. Limitations of theoretical results.**\n\nThese are all true limitations of our theoretical setup! While we mention several limitations in the paper, we will add the limitations mentioned here (e.g. sparse connectivity). Unfortunately it is currently challenging to get strong theoretical results for features learned in deep nonlinear networks. However we find it encouraging that despite such a simplified setting, we are able to see a clear feature dropout effect in our theoretical setting. White not conclusive, this is an encouraging sign as to the generality of the feature dropout phenomenon.\n\n**Q1: Experimental settings**\n\nWe use the settings from Tamkin et al 2021, and will include them in the paper (see response to W1a above)\n\n**Q2 Repeated brightness shifts.** \n\nCorrect, as far as we know people do not do repeated augmentations explicitly. Our point here is just that if T(x) has support under the data distribution D (our assumption), then if we were to sample T(x) from D, we would then produce T(T(x)), which in effect is a repeated augmentation. This causes an augmentation collision in the sense we discuss in the paper. One case where this assumption is reasonable is brightness shifts: amateur photographers often mis-expose images, resulting in a broad range of brightness shifts. A brightness shift applied to a very light image could result in a pure white image, destroying the class information.\n\n**Q3: We use gaussian blur?** \n\nThis is a good catch; indeed we do use gaussian blur in our experiments unlike the CIFAR-10 experiments in the paper. However, we notice no impact on the CIFAR accuracy in our results for including this information. We will update the paper to note this.\n\n**CIFAR-10 (no overlaid features) with SimCLR (expert views):**\nNo blur: 0.8614 \nBlur: 0.8632\n\n\n**Q4: Moving figures to main text.** \n\nYes, we will consolidate Figure 1 and move up the important figures from the appendix given the extra page provided if our paper is accepted.\n\n**Typos**\n\nWe are grateful for these suggestions and have fixed them on our copy of the manuscript.\n\nWe thank the reviewer again for their thoughtful comments and review."}]

Author rebuttal:
We thank the reviewers for their reviews! We briefly highlight two points mentioned across more than one reviewer.

**Results on naturalistic datasets**

Our experiments on the naturalistic CelebA dataset are a significant contribution of our work, but we realize from the reviewers’ feedback that we should highlight them more prominently in the next revision. We will do so by using the additional space provided by reducing the size of Figure 1 and the additional page provided after the review process.

**Additional experiments**

We include 4 sets of additional experiments, covering the effects of gaussian blurs, horizontal flipping, varying the distortion budget, and varying the dropout rate for audio data. We find that these additional results further reinforce the main claims of the paper, and appreciate the reviewers’ suggestions to run them.


Meta-review: This paper's scores were mixed.

- The authors argue in this work that label-destroying augmentations can be helpful for contrastive representation learning, by a dropout-like robustifying mechanism, complicating the simplistic viewpoint that good augmentations are those that preserve the label. Reviewers agreed that the hypothesis is crisp, and the empirical study is thorough and high-quality.
- One reviewer was concerned with the overlap between this work and the *feature suppression* phenomenon. The authors clarified contributions not present in prior work: (1) unifying prior accounts; (2) benefits of feature dropout for Viewmaker networks; (3) novel experiments (synthetic, audio, CelebA); (4) theoretical analysis.

Overall, after discussions, the paper surpasses the publication threshold, due to the former point.
Decision: Accept (poster)