Title: Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: self-supervised learning

contrastive learning
Abstract: What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. for digits superimposed on photographs, predicting the class of one vs. the other). We find that Viewmaker Networks, a recently proposed model for learning augmentations for contrastive learning, produce label-destroying augmentations that stochastically destroy features needed for different downstream tasks. These augmentations are interpretable (e.g. altering shapes, digits, or letters added to images) and surprisingly often result in better performance compared to expert-designed augmentations, despite not preserving label information. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.

Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning

Abstract

What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. synthetic datasets combining two classes, such as images and digits, and naturalistic datasets labeled with dozens of attributes). In controlled experiments where we destroy features at different rates, we find that destroying one feature a modest fraction of the time can improve learning of other features, while still enabling the dropped out feature to be learned well. Additionally, we show how this hypothesis can help explain the success of Viewmaker Networks, which generate augmentations that appear to target and destroy different features in the input examples, yet often result in better performance than standard augmentations across tasks. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, we show that label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.

1 Introduction

In recent years, foundation models [5] have exhibited remarkable progress on a range of AI tasks [13; 31; 37; 36; 6; 11; 25; 1; 38]. A crucial characteristic of foundation models is that they can be adapted for a range of downstream tasks. For example, a foundation model trained on ImageNet should ideally not only perform well at object classification, but should also have learned general features useful for localization, segmentation, and other visual tasks. Indeed, this is borne out by recent work showing the high accuracy of foundation models on a range of downstream tasks [9], as well as a range of analysis work showing that models learn high-level semantic features including texture, color, pose, and style [19].

One popular strategy for training foundation models involves training models to match transformed versions (known as views or augmentations ) of the same input. For example, image views might include common data augmentations such as cropping or color jitter [9], while views for speech might include pitch modulation or spectrogram masking [27; 35]. This family of objectives includes contrastive approaches such as SimCLR and MoCo, as well as non-contrastive approaches such as BYOL and SwAV [9; 23; 20; 7].

Given the central importance of these views for defining the self-supervised task, much work has focused on the question of what views lead to high-quality representations. The prevailing consensus, exemplified by Tian et al. [52], holds that views should be label-preserving with respect

to a downstream task. In other words, because the contrastive loss will produce representations which are invariant to features that vary across views, any information we wish to preserve in the representations should not be altered by such views. As Tian et al. [52] write: ' A good set of views are those that share the minimal information necessary to perform well at the downstream task .'

Here, we question whether this assumption-in particular, with its focus on a single task-is enough to explain why contrastive foundation models succeed on a range of downstream tasks. In Section 2, we observe that the actual choice and application of views in practice does not align with this prevailing consensus. For example, complete invariance to several common augmentations (e.g. shifts in brightness or cropping) is undesirable since augmentations of inputs from different classes can collide. Furthermore, in many cases there are explicit ways to specify invariances (e.g. converting images to grayscale) that researchers avoid, instead specifying them indirectly via augmentations (e.g. hue shifts). These observations suggest that specifying invariances is not the sole role of these views.

Instead, we suspect that augmentations serve as a form of feature dropout -preventing any one feature from becoming a shortcut feature and suppressing the learning of other features. We study this idea empirically with a set of synthetic datsets constructed by overlaying a simple element (e.g. a digit, shape, letter, or speech snippet) on an image or audio recording. We find that adding such a simple feature can dramatically decrease how well the other feature is learned, but that stochastically 'dropping out' the simple feature can enable both features to be learned well. Next, we use this perspective to explain the success of Viewmaker Networks, a recently proposed method that generates augmentations for contrastive learning via adversarial training. We apply viewmaker and expert views to these same synthetic datasets, as well as a naturalistic dataset of facial images annotated with 40 different attributes (e.g. 'wearing lipstick' or 'blond hair'). Across these settings, we find that viewmaker augmentations learn to selectively obscure various features in the input. Despite this, the viewmaker representations still learn the downstream tasks well, while expert views often struggle on one or more of the attributes. This further suggests that being label-preserving is not a necessary property of good views, as long as the label information is still sometimes accessible.

Finally, we formalize the intuition that feature dropout can aid learning with a theoretical analysis of a simple linear contrastive setting. In this setting, we characterize how the noisiness of each feature directly determines how quickly features are learned, and uncover an interaction between features governing how fast they are learned. In particular, we show how learning one feature quickly can suppress the learning of other features, and show that adding noise to the 'easiest' feature can increase the rate at which other features are learned. This further indicates that label-destroying augmentations may have a direct role in ensuring that contrastive models learn a broad range of features for downstream tasks.

Overall, these findings suggest the need to revisit common assumptions about the role of augmentations for contrastive learning in the foundation model setting, and move towards a better understanding of how to train generalist models that learn diverse features from unlabeled data.

2 Common practices are at odds with the 'invariance' explanation

We begin by briefly exploring several common augmentations used in contrastive learning for natural images, and explore how they come into conflict with the common assumption described above. First, we observe that many common augmentations can affect the label of the input, depending on the downstream task. For example, many downstream image recognition tasks require color information (e.g. identifying bird species) or brightness (e.g. scene or time-of-day classification), implying that invariance to these characteristics would be undesirable. Yet hue shifts, greyscaling, and brightness shifts are common augmentations used in contrastive learning [9; 23].

Second, repeated application of some augmentations causes challenges for all downstream tasks. For example, applying brightness shifts repeatedly results in any image turning completely black or completely white. Thus the class label cannot be truly invariant to this augmentation, since inputs from different classes can experience an 'augmentation collision' at this black or white image (this is

formalized in Appendix B). 1 This argument also applies to other augmentations, including shifts in contrast 2 and random masking.

Third, some augmentations are commonly used despite ways of explicitly encoding invariance to them. For example, two image augmentations are hue shifts and greyscaling . Invariance to both of these augmentations can be explicitly encoded by always converting an image to greyscale. Yet doing so is not common practice because color information is still desirable for many downstream tasks.

The contradictions between the invariance rationale for augmentations in contrastive learning and these common practices suggest the need for additional explanations for the role of augmentations.

3 Controlled experiments demonstrate the benefits of feature dropout in settings with multiple features

In this section, we present controlled experiments on synthetic data demonstrating how labeldestroying augmentations can balance the learning of multiple features during contrastive learning. Our core toolkit is to overlay images with a set of synthetic features. As we will show, the presence of these synthetic features causes the network to learn the synthetic features very well at the expense of the image features, as measured by downstream classification accuracy. However, 'dropping out' these features some fraction of the time during contrastive learning enables us to trade-off how well each feature is learned, while not resulting in complete invariance to either set of features.

3.1 Datasets

We consider the behavior of viewmaker networks on four synthetic datasets, including three image and one audio dataset. Each dataset is constructed in such a way as to support two distinct downstream classification tasks, enabling us to examine precisely how well each downstream task is learned. The presence of two downstream tasks enables us to analyze the foundation model setting where we wish to learn features relevant for multiple downstream tasks, as opposed to one set or the other.

Image datasets The three image datasets are based on the canonical CIFAR-10 image-recognition dataset [28] (MIT-License). One task is always to predict the CIFAR-10 object label (e.g. airplane or bird ). The other task is dependent on an additional feature overlaid on the image: C+Shapes: The CIFAR-10 image is overlaid with one of three randomly-colored shapes: a square, a triangle, or a circle. The second task is to predict what shape was overlaid (N=3 classes). C+Digits: The CIFAR-10 images are overlaid with four copies of a randomly-sampled digit from the MNIST dataset. The second task is to predict the digit class (N=10 classes). C+Letters: The CIFAR-10 images are overlaid with four copies of a randomly-colored English letter. The second task is to predict the class of the letter (N=26 classes).

Audio dataset The audio dataset is created by overlaying the audio of a spoken digit (from the AudioMNIST dataset [3], MIT License) with a random background sound (collected from one of three possible classes: cafe, machinery, and traffic) [43; 42]. The tasks are to predict the digit class (N=10 classes) and to predict the sound class (N=3 classes). Inputs are presented to the network as log mel spectrograms.

3.2 Experiments

Pretraining We pretrain with the SimCLR algorithm for 200 epochs with a batch size of 256 and a temperature of 0.1. We use a ResNet-18 model with standard modifications for smaller inputs (including a smaller stride and no initial maxpool) as used in Tamkin et al. [49]. We use the standard SimCLR augmentations for the image datasets [9], with the exception of gaussian blurring, which we found to have no impact on downstream performance (Appendix C.4). For audio, we use the SpecAug [35] augmentations, which randomly mask out different frequency and time bands, as well as the WaveAug [27] augmentations, which alter various properties of the waveform such as the pitch and speed.

Linear Evaluation We evaluate the quality of the learned representations by training a linear softmax classifier on top of the prepool representations. We train for 100 epochs, using the same parameters as Tamkin et al. [49], using SGD with learning rate 0.01, momentum 0.9, weight decay 0, and batch size 128. We train separate linear classifiers using the same pretrained network for each downstream task [9]. Augmentations are applied during training but not evaluation.

Results As shown in Figure 1, we see an interaction between the two features, where dropping out the synthetic feature improves learning of the main image or audio class. Across settings, we see regions where both features are still learned well, providing a concrete example of how feature dropout can be useful when learning multiple features during contrastive learning.

(a) Shapes

(b) Digits

(c) Letters

(d) Audio

Main Label

AlternateLabel

1.00

0.75

0.50

0.25

0.00

0

25

50

75

100

Probability of Dropout

Main LabelAlternate Label

1.0000

0.7500

0.5000

0.2500

0.0000

0

25

50

75

100

Probability of Dropout

Main LabelAlternate Label

1.0000

0.7500

0.5000

0.2500

0.0000

0

25

50

75

100

Probability of Dropout

Main Label

Alternate Label

1.0000

0.7500

0.5000

0.2500

0.0000

0

25

50

75

100

Probability of Dropout

4 Viewmaker Networks Succeed Despite Destroying Label Information

As another point of evidence that good views need not be label-preserving, we consider viewmaker networks [49], a generative model which produces augmentations for contrastive learning. Intuitively, viewmakers learn a stochastic augmentation policy that makes the contrastive task as hard as possible for the encoder. The stochastic augmentations are parameterized as additive perturbations bounded by an L 1 norm, meaning the viewmaker can alter but not completely destroy the original image.

Formally, given an input x ∈ N , a viewmaker network V ψ is trained jointly with an encoder E θ to optimize the minimax expression:

Here L is a multiview loss function (e.g. [9; 23]), x is a minibatch of inputs, || · || 1 is the L 1 norm, ϵ is the distortion budget controlling the strength of the views, and δ 1 , δ 2 ∼ N (0 , I ) are random inputs that enable the viewmaker to learn a stochastic augmentation policy. We clamp the output of the viewmaker for images to [0 , 1] as in Tamkin et al. [49].

Viewmaker networks learn to stochastically alter different parts of the input, including task-relevant features, meaning that these augmentations are not label-preserving. Nevertheless, as we will see shortly, viewmaker networks enable strong performance on multiple downstream tasks, including often better performance than expert-designed augmentations. Moreover, this feature dropout capability of viewmaker networks may help them learn many features well rather than just the easiest.

Speech

Letters

Shapes

Digits

Expert view

Viewmaker view

Viewmaker delta

6

M

BB

营

200

4.1 Experiments and Results

Experimental Settings We use the same experimental settings as Section 3, however without manual dropout of the synthetic features. In one set of experiments, we use the standard augmentations from Chen et al. [9], which we henceforth refer to as the expert augmentations . For the experiments with viewmaker augmentations , we use a budget of ϵ = 0 . 05 P for the image datasets, and ϵ = 0 . 125 P for the audio datasets, where P is the number of pixels in the input.

Additional naturalistic dataset with 40 attributes To further validate the behavior of viewmaker on realistic multi-feature datasets, we consider the CelebA [32] dataset, a large database of faces annotated with 40 different features. These features cover a wide spectrum of facial attributes, such as 'Has Bangs" 'Wearing Lipstick" and 'Smiling," and enable us to further analyze whether viewmaker networks learn a broader range of features than commonly-used augmentations.

4.2 Results on two-feature datasets

Qualitative evidence of feature dropout Visually, the viewmaker augmentations seem to stochastically alter different aspects of the input, as shown in Figure 2. In addition to modifying the background of each input, the viewmaker also selectively modifies the additional synthetic features added to each domain: C+Digits: The viewmaker augmentations selectively add pixels to the MNIST digits, making it difficult to distinguish which number is present. C+Shapes: The viewmaker augmentations sometimes draw squares around the shape in the center, making it difficult to determine the shape class. C+Letters: The viewmaker draws letter-like markings on top of the letters, obscuring the letter identity and color. Audio: The viewmaker identifies the narrow band corresponding to the speech and applies perturbations to it. As can be seen in Figure 2, these label-destroying augmentations are quite common, occuring in a sizable fraction of the sampled views.

Quantitative evidence of feature dropout We also measure this selectivity of features quantitatively in Appendix C.2 and Figure 6. We augment images 1,200 times and observe the resulting probability assigned to the correct object class. Two clear modes appear for viewmaker, but not expert, augmentations. This corresponds to the fraction of time the viewmaker destroys the overlaid feature information (low P(correct object class)) and preserves it (high P(correct object class)).

Viewmaker succeeds despite destroying label information As shown in Table 1 and Table 2, viewmaker networks achieve good accuracy on both tasks, while expert augmentations frequently achieve lower performance on one or both. On image tasks, for example, while expert views achieve slightly higher performance when classifying the image only, they see a large drop in accuracy when

Expert

Viewmaker

0.8

0.6

F1-Score

0.4

0.2

0.0

Features

u

(4)

k

v

(4)

k

v

(3)

k

u

(3)

k

u

(2)

k

v

(2)

k

u

(1)

k

v

(1)

k

-

µ

k

µ

k

w

11

w

12

w

13

w

21

w

21

w

23

w

K

1

w

K

2

w

K

3

. . .

θ

1

θ

2

θ

K

θ

T

1

w

1

θ

T

2

w

2

θ

T

K

w

K

. . .

Representation

f

Θ

(

w

)

Input

w

(a)

(b)

the synthetic feature is added. In two of these cases (Shape and Digit) the viewmaker models are able to achieve a higher accuracy on both the image and the synthetic feature, while on the third (Letters) they achieve slightly lower accuracy on the images but achieve half the error on the synthetic object. For the audio experiments the picture is similar-viewmaker avoids catastrophic drops in performance learning both features together, achieving the highest accuracy on both, while the expert views see larger drops and worse overall performance. Note that the high performance of expert views for our control tasks (CIFAR-10/Speech/Sound Only) indicates that the viewmaker views are not merely better all-around views, but that they specifically help the model learn multiple features.

These results provide additional evidence that label-preserving views are not necessary for learning good representations-and that feature dropout may improve learning of multiple tasks.

4.3 Results on naturalistic dataset

We observe similar qualitative and quantitative results for the CelebA dataset. We train models using the same settings in Section 3.2, using a budget of 0.01, and indeed find that viewmakers capture a much broader range of features, achieving an average F1 Score of 0.509 over the 40 features, compared to 0.334 for the SimCLR augmentations. In addition, the viewmaker augmentations clearly capture a wider range of features, as can be seen in Figure 3, especially at the tail of the distribution. Furthermore, we see bimodal disruption patterns in over two-thirds of the CelebA features, as shown in Figure 9, indicating significant feature dropout across in most attributes. We also show qualitative results in Figure 8 demonstrating that the viewmaker alters attributes such as facial features, hair color, and background elements in the scene. These results further support the hypothesis that viewmaker networks exhibit feature dropout, yet capture a broader range of features than expert views.

5 Theoretical Analysis of Feature Interactions in Linear Contrastive Setting

In this section, we analyze a simple linear model that captures the essence of how label-destroying augmentations can improve downstream accuracy. We study a setting where the data contains many underlying features that are relevant to downstream classification tasks, and where these features are preserved to varying degrees across augmentations. We will show that a linear model trained with a contrastive objective learns these features, and that adding noise to one feature can speed learning of others during gradient descent. One difference between the linear setting we analyze and Section 4 is that here add stochastic Gaussian noise to destroy features across augmentations, as opposed to the bimodal feature dropout behavior of viewmaker networks seen in Figure 2.

5.1 Data Model and Setting

We study a model which consists of data with K distinct features, each corresponding to some ground truth unit-vector directions µ 1 , . . . , µ K ∈ R d . We sample each data point u ∈ R K × d and its augmentation (a.k.a. its positive pair or its view ) v ∈ R K × d as follows. For k ∈ 1 , . . . , K , the k th row of u , which we denote u k , is drawn from the Gaussian distribution N (0 , I d ) . The k th row of the augmentation, v k , is drawn from the same distribution, but is correlated with u k in the µ k -direction (and is otherwise independent in the other directions). The strength of the correlation is governed by parameter α k ∈ [0 , 1] in the following sense: v T k µ k = α k u T k µ k + √ 1 -α 2 k ξ , where ξ ∼ N (0 , 1) . Thus the larger α k , the stronger the correlation in that feature across the two views. Figure 4(a) visualizes the correlation of ( u k , v k ) in an augmented pair. Formally, we can write that

We will learn a model Θ ∈ R K × d , which represents a collection of K feature extractors, as pictured in Figure 4(b). The model Θ , with rows { θ k } k ∈ [ K ] , maps a data point w ∈ R K × d to a representation f Θ ( w ) ∈ R K by computing a score w T k θ k for each element in the representation. That is, ( f Θ ( w )) k = w T k θ k . Our goal is that the model Θ will be useful for a downstream classification task which depends on the ground truth features. A good representation will capture ground truth features that are correlated across augmentations, such that θ k is aligned with µ k or -µ k .

Training. We will study the the evolution of Θ as we optimize a standard constrastive learning objective using gradient descent [14; 9]. At each round of gradient descent, we sample a fresh batch of m data points and their augmentations, ( U, V ) := { ( u ( i ) , v ( i ) } i ∈ [ m ] . For each i, j ∈ [ m ] , we compute a similarity score z ij := ⟨ f Θ ( u ( i ) ) , f Θ ( v ( j ) ) ⟩ = ∑ k ( θ T k u ( i ) k )( θ T k v ( j ) k ) using the dot product of their K -dimensional representations. We then compute the logits p ij := exp( z ij ) ∑ j ′ exp( z ij ′ ) using the softmax function, and use the classwise cross entropy loss function L (Θ; U, V ) := -log( p ii ) .

5.2 Main Result

We will study gradient descent (GD) on the cross entropy loss, and consider how adding noise to one feature affects the learning of the other features. As suggested earlier, we can measure how well we learn the k th feature by measuring the alignment of θ k with µ k or -µ k . A natural way to measure this alignment is the acute angle between ± µ k and θ k , given by arccos ( | µ T k θ k | ∥ θ k ∥ 2 ) . Lemma E.1 in Appendix E proves that this quantity directly determines the test accuracy on a natural downstream linear classification task.

̸

Formally, we say we add noise to some feature k ′ of a data point v , if for some β ∈ [0 , 1) , we define the new noisy data point ˜ v to have coordinates ˜ v k ′ = βv k ′ + √ 1 -β 2 ξ , where ξ ∼ N (0 , I d ) , and ˜ v k = v k for k = k ′ . Thus if ( u, v ) were a pair generated with the correlation coefficients { α k } k ∈ [ K ] , then the distribution of ( u, ˜ v ) comes from the modified correlation coefficients { ˜ α } k ∈ [ K ] with the single modification ˜ α k ′ = βα k . We now present our main theorem:

̸

Theorem 5.1 (Noise improves feature learning) . There exists a universal constant C , such that the following holds. Let Θ ( t +1) = Θ ( t ) -η ( ∇L ( U, V ; Θ) + λ Θ ( t ) ) , and ˜ Θ ( t +1) = Θ ( t ) -η ( ∇L ( U, ˜ V ; Θ) + λ Θ ( t ) ) , where ˜ V is V with any amount of added noise in the k ′ feature. This has the effect of changing α k ′ to ˜ α k ′ for any ˜ α k ′ < α k ′ . Then for any k = k ′ , if | θ T k µ k | ≤ 1 -α 2 k ′ C ∥ θ k ∥ , ∥ θ k ′ ∥ 3 ≤ | θ T k ′ µ k | , and ∥ θ k ∥ 2 ≤ α k (1 -α 2 k ′ ) C , then for a small enough step size η , E U,V [ arccos ( | µ T k θ ( t +1) k | ∥ θ ( t +1) k ∥ 2 )] > E U, ˜ V [ arccos ( | µ T k ˜ θ ( t +1) k | ∥ ˜ θ ( t +1) k ∥ 2 )] .

We briefly comment on the three assumptions on Θ in the theorem. The first assumption, | θ T k µ k | ≤ 1 -α 2 k ′ C ∥ θ k ∥ requires that θ k is not too aligned with µ k - that is, the result applies to all features k that are not already learned too well. The second two assumptions are satisfied if the k ′ th feature has been learned to some extent, and the norm of θ k and θ k ′ are small, which can be enforced throughout training with ℓ 2 regularization.

The theorem guarantees that at any point in training, if we add noise to the k ′ th feature, the next step of GD learns all other features better than if we did not add noise. To validate the implication of this result for the complete trajectory of GD, we include simulations in Appendix D. Our experiments show that introducing noise part-way through training to dominant features can significantly speed the alignment of weak features, with only a small cost to the alignment of the dominant features. We prove our result in Appendix E, including intuition and an overview of the ideas in Section E.3.

6 Related work

Understanding contrastive and multiview learning Many prior works have laid the foundations for current contrastive and multiview learning algorithms [4; 21; 14; 55; 2; 34; 23; 9]. Several works analyze contrastive learning to identify important factors [12; 58] or how contrastive models differ from supervised learning [57; 15; 26]. HaoChen et al. [22] study contrastive learning using the concept of an augmentation graph. This model assumes the fraction of non-label preserving augmentations is 'extremely small'; interestingly, we show in practice this can be quite large and still yield good performance. Wang et al. [54] theoretically study contrastive learning under an assumption of label-preserving augmentations, though they show that such an assumption alone does not suffice to learn. Most relevant to our work, Tian et al. [52]; Ericsson et al. [16] study how the information shared between different views impacts learning of downstream tasks. We complicate this picture by analyzing the foundation model setting where a single model must learn features for multiple tasks that are not known in advance. In this setting, we find that label-destroying perturbations, thought to be harmful by Tian et al. [52], are useful for preventing one feature from suppressing others.

Feature suppression Our work is closely connected to the notion of feature suppression [24], where the presence of one feature can suppress the learning of others. Several works explore this concept in contrastive learning. For example, the original SimCLR paper [9] noted that color jitter augmentation was necessary to prevent the network from using only the color profile of the input to solve the contrastive task. Followup work [10] characterizes how hyperparameters and dataset features affect feature suppression, including through a range of synthetic experiments that vary the amount of information shared between views. Other works have attempted to address feature suppression in contrastive learning, either via auxiliary losses [29] or by modifying representations in the latent space [39]. Our builds upon these works by empirically and theoretically investigating feature suppression as an alternate rationale for the role of augmentations, as opposed to invariance. We also show that an existing method, viewmaker networks [49], can identify and potentially neutralize suppressing features in an interpretable way, resulting in better performance than expert augmentations. These insights may also generalize to other self-supervised settings, such as language modeling, where multiple features may compete [47].

Spurious correlations and shortcut features Outside the framing of feature suppression, several other works explore how classifiers can learn or make use of unwanted features. Shortcut features [18] describe often-simple features (e.g. the average color of an input) which are learned by networks at the expense of more salient features (e.g. the object class). This notion is connected to spurious correlations [45] in deep learning which have been explored extensively [40; 41; 46; 53; 56], including in the context of self-supervised learning [33; 51]. Other works have also performed theoretical analysis of how related dynamics affect learning in the supervised setting [30; 44]. In general, if such spurious correlations are known in advance, one can often design augmentations to remove such correlations and improve learning. However, our work suggests that viewmaker networks may be a useful tool in cases where such features are not known a priori-both as an interpretability tool to visualize the different features a network relies on, and as a way to reduce reliance on particular features without completely destroying the information.

7 Discussion and Conclusion

We explore the idea that augmentations in contrastive learning function as a sort of 'feature dropout.' First, we show that in datasets with multiple features, dropping out one set of features improves learning of the other features. Second, feature dropout may help explain how viewmaker networks can learn a wide range of features well, despite producing augmentations which appear to destroy different features in the input. Finally, we theoretically analyze a linear contrastive setting where we

prove that label-destroying views have a positive effect on contrastive learning if the goal is to avoid learning one feature at the expense of others.

Our work has limitations: for example, while our experiments consider image and audio data, selfsupervised learning may be applied to a much wider range of modalities [48; 50]. In addition, our theoretical analysis considers a linear contrastive setting, whereas current neural networks are highly nonlinear. Improving upon both of these fronts is an exciting area for future work. On the other hand, understanding augmentations as dropping out easy features suggests possible ways of improving the performance of self-supervised learning. For example, a guided version of viewmaker might enable prioritizing a subset of important features for learning, or might enable dropping out unwanted features such as watermarks, sensitive information, other image artifacts.

The challenge of learning a broad range of useful features lies at the heart of self-supervised learning. We hope our work sheds light on this challenge in contrastive learning, especially as these objectives continue to develop and are applied more broadly and at larger scale.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv , abs/2204.14198, 2022.

Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS , 2019.

Soren Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals. ArXiv , abs/1807.03418, 2018.

Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature , 355:161-163, 1992.

Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv , abs/2108.07258, 2021.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems , 33:9912-9924, 2020.

Shuxiao Chen, E. Dobriban, and Jane Lee. A group-theoretic framework for data augmentation. arXiv: Machine Learning , 2020.

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv , abs/2002.05709, 2020.

Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems , 34, 2021.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Baindoor Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Oliveira Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv , abs/2204.02311, 2022.

Elijah Cole, Xuan S. Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge J. Belongie. When does contrastive visual representation learning work? ArXiv , abs/2105.05837, 2021.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv , abs/1810.04805, 2019.

Alexey Dosovitskiy, Jost Tobias Springenberg, Martin A. Riedmiller, and Thomas Brox. Discriminative unsupervised feature learning with convolutional neural networks. In NIPS , 2014.

Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. How well do self-supervised models transfer? 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 5410-5419, 2021.

Linus Ericsson, Henry Gouk, and Timothy M. Hospedales. Why do self-supervised models transfer? investigating the impact of invariance on downstream tasks. ArXiv , abs/2111.11398, 2021.

Kunihiko Fukushima and Sei Miyake. Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition. In Competition and cooperation in neural nets , pages 267-285. Springer, 1982.

Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard S. Zemel, Wieland Brendel, Matthias Bethge, and Felix Wichmann. Shortcut learning in deep neural networks. ArXiv , abs/2004.07780, 2020.

Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Christopher Olah. Multimodal neurons in artificial neural networks. 2021.

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in Neural Information Processing Systems , 33:21271-21284, 2020.

Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06) , 2:1735-1742, 2006.

Jeff Z. HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for selfsupervised deep learning with spectral contrastive loss. In NeurIPS , 2021.

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9726-9735, 2020.

Katherine L. Hermann and Andrew Kyle Lampinen. What shapes feature representations? exploring datasets, architectures, and training. ArXiv , abs/2006.12433, 2020.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv , abs/2203.15556, 2022.

A. Tarun Karthik, Mike Wu, Noah D. Goodman, and Alex Tamkin. Tradeoffs between contrastive and supervised learning: An empirical study. ArXiv , abs/2112.05340, 2021.

Eugene Kharitonov, Morgane Rivière, Gabriel Synnaeve, Lior Wolf, Pierre-Emmanuel Mazaré, Matthijs Douze, and Emmanuel Dupoux. Data augmenting contrastive learning of speech representations in the time domain. In 2021 IEEE Spoken Language Technology Workshop (SLT) , pages 215-222. IEEE, 2021.

Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.

Tianhong Li, Lijie Fan, Yuan Yuan, Hao He, Yonglong Tian, Rogério Schmidt Feris, Piotr Indyk, and Dina Katabi. Addressing feature suppression in unsupervised visual representations. 2020.

Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. ArXiv , abs/1907.04595, 2019.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv , abs/1907.11692, 2019.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In ICCV , pages 3730-3738. IEEE Computer Society, 2015. ISBN 978-1-4673-8391-2. URL http://dblp.uni-trier.de/db/conf/iccv/iccv2015.html#LiuLWT15 .

Matthias Minderer, Olivier Bachem, Neil Houlsby, and Michael Tschannen. Automatic shortcut removal for self-supervised representation learning. In ICML , 2020.

Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representations. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 6706-6716, 2020.

Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08779 , 2019.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. ArXiv , abs/2102.12092, 2021.

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Manfred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. ArXiv , abs/2205.06175, 2022.

Joshua Robinson, Li Sun, Ke Yu, K. Batmanghelich, Stefanie Jegelka, and Suvrit Sra. Can contrastive learning avoid shortcut solutions? Advances in neural information processing systems , 34:4974-4986, 2021.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. ArXiv , abs/1911.08731, 2019.

Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameterization exacerbates spurious correlations. ArXiv , abs/2005.04345, 2020.

Fatemeh Saki and Nasser Kehtarnavaz. Automatic switching between noise classification and speech enhancement for hearing aid devices. In 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) , pages 736-739, 2016. doi: 10.1109/EMBC.2016.7590807.

Fatemeh Saki, Abhishek Sehgal, Issa Panahi, and Nasser Kehtarnavaz. Smartphone-based real-time classification of noise signals using subband features and random forest classifier. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 2204-2208, 2016. doi: 10.1109/ICASSP.2016.7472068.

Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. ArXiv , abs/2006.07710, 2020.

Herbert A Simon. Spurious correlation: A causal interpretation. Journal of the American statistical Association , 49(267):467-479, 1954.

Megha Srivastava, Tatsunori B. Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In ICML , 2020.

Alex Tamkin, Dan Jurafsky, and Noah Goodman. Language through a prism: A spectral approach for multiscale language representations. Advances in Neural Information Processing Systems , 33:5492-5504, 2020.

Alex Tamkin, Vincent Liu, Rongfei Lu, Daniel E Fein, Colin Schultz, and Noah D. Goodman. Dabs: A domain-agnostic benchmark for self-supervised learning. ArXiv , abs/2111.12062, 2021.

Alex Tamkin, Mike Wu, and Noah D. Goodman. Viewmaker networks: Learning views for unsupervised representation learning. ArXiv , abs/2010.07432, 2021.

Alex Tamkin, Gaurab Banerjee, Mohamed Owda, Vincent Liu, Shashank Rammoorthy, and Noah Goodman. Dabs 2.0: Improved datasets and algorithms for universal self-supervision. 2022.

Alex Tamkin, Dat Nguyen, Salil Deshpande, Jesse Mu, and Noah Goodman. Active learning helps pretrained models learn the intended task. arXiv preprint arXiv:2204.08491 , 2022.

Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning. ArXiv , abs/2005.10243, 2020.

Lifu Tu, Garima Lalwani, Spandana Gella, and He He. An empirical study on robustness to spurious correlations using pre-trained language models. Transactions of the Association for Computational Linguistics , 8:621-633, 2020.

Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Chaos is a ladder: A new theoretical understanding of contrastive learning via augmentation overlap. arXiv preprint arXiv:2203.13457 , 2022.

Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3733-3742, 2018.

Kai Y. Xiao, Logan Engstrom, Andrew Ilyas, and Aleksander Madry. Noise or signal: The role of image backgrounds in object recognition. ArXiv , abs/2006.09994, 2021.

Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang, Shanghang Zhang, and Pengtao Xie. Transfer learning or self-supervised learning? a tale of two pretraining paradigms. ArXiv , abs/2007.04234, 2020.

Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? ArXiv , abs/2006.06606, 2021.

A Code release

Our code is open-sourced at https://github.com/xiluohe/feature-dropout.

B Formalization of observation in Section 2

Definition B.1 (Invariance) . A function f : R m → R n is invariant to a set of transformations G if and only if f ◦ g ( x ) = f ( x ) for all x ∈ R m and for all g ∈ G .

Definition B.2 (Augmentation collision) . An augmentation collision occurs if, for two inputs x a , x b and set of transformations G , there exist g (1) a , . . . , g ( n a ) a , g (1) b , . . . , g ( n b ) b ∈ G for some n a , n b ∈ N such that g (1) a ◦ . . . ◦ g ( n a ) a ( x a ) = g (1) b ◦ . . . ◦ g ( n b ) a ( x b ) .

Observation B.3. If there exists an augmentation collision for inputs x a , x b and transformation set G , and f is invariant to G , then f ( x a ) = f ( x b ) .

Applying this observation, we observe that if the downstream labeling function f is invariant to a class of augmentations, then there cannot be an augmentation collision for inputs with different labels. However, common augmentations such as brightness shifts can reduce any image to a black or white image, resulting in an augmentation collapse between any two inputs.

C Additional feature dropout experiments

C.1 Quantifying the importance of feature dropout

To assess the importance of label-destroying augmentations to the success of the viewmaker, we experiment with a setup where the viewmaker cannot destroy the information in the object class. To do this, we compute a mask around the object and zero out any perturbation from the viewmaker within that mask. We then perform pretraining and transfer as usual.

As we report in Table 3, the accuracy of the CIFAR-10 class label drops precipitously, as expected. At the same time, the accuracy of two of the other objects remains mostly constant (shape and digits), while the accuracy for letters declines modestly (perhaps because the color of the letter is now able to suppress the learning of the letter class.

C.2 Quantifying the degree of feature dropout

We perform an exploratory analysis to testing how well different views drop out the features in an input. We augment a 1,200 examples (CIFAR-10 image plus an overlaid object) using a given augmentation policy (either the expert or viewmaker augmentations). We then encode the model with a classifier trained off of the other augmentation policy (i.e. expert for viewmaker augmentations

(b) Viewmaker / Letters

(c) Viewmaker / Digits

(d) Expert / Shapes

(e) Expert / Letters

(f) Expert / Digits

RR

99

86

66

6-6

KK

R

品

KK

RIR

0.8

0.7

0.4

0.5

0.3

PTO

0.3

0.2

0.2

0.2

0.1

0.1

I'0

0.0

0.0

0.2

0.4

0.6

0.8

0.0

0.2

0.4

0.6

1.0

0'0

0.0

0.2

0.4

0.6

0.8

P(correct)

P(correct)

P(correct)

0.16

0.14

0.8

8°0

0.7

0.12

0.6

0.10

0.6

0.08

PTOb

0.4

0.4

0.06

E'0

0.04

0.2

0.2

0.02

0.1

0.00

0.0

0.2

0.4

0.6

8'0

1.0

0.0

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.0

0'0

0.2

0.4

0.6

0.8

1.0

P(correct)

P(correct)

P(correct)

or the reverse) in order to test how well the augmentations drop out the features. We use a different encoder to see the effects of the augmentations prior to the encoder having a chance to adapt to them.

We observe a bimodal behavior for the viewmaker views, shown in Figure 6, suggesting that the model is adapting to the semantics of the input and has learned to stochastically drop out the simple feature some fraction of the time. By contrast, the expert views display no such structure. Using the corresponding encoder and views leads to models performing uniformly well, as shown in Figure 7.

C.3 Additional visualizations for CelebA

We show feature dropout histograms for CelebA for each of the 40 features in Figure 9. The prevalence of bimodal distributions demonstrates a high degree of feature dropout across attributes. Histograms shown are viewmaker augmentations on an encoder trained with expert views.

(b) Viewmaker / Letters

(c) Viewmaker / Digits

(d) Expert / Shapes

(e) Expert / Letters

(f) Expert / Digits

1.0

1.0

0.8

8°0

0.8

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.4

P(correct)

0.6

0.8

1.0

.4

0.5

0.6

P(correct)

0.7

0.8

0.9

0°0

0.65

0.70

0.75

0.80

P(correct)

0.85

0.90

0.95

1.00

1.0

0.12

0.5

0.8

0.10

0.08

0.6

amq

0.4

0.04

0.2

0.2

0.02

0.1

0.0

0.0

0.2

0.4

0.6

8'0

1.0

00'0

0.0

0.2

0.4

0.6

0.8

1.0

0°0

0.0

0.2

0.4

9'0

0.8

1.0

P(correct)

P(correct)

P(correct)

We also show views and diffs for CelebA in Figure 8. These views show a high degree of sensitivity to the input semantics, and appear to modify characteristics such as the background, hair color, and facial features.

C.4 Importance of Gaussian Blur Transformation

We show experiments for CIFAR-10 (with no overlaid features) on SimCLR (expert views) with and without Gaussian blur augmentations. We find that the inclusion of the Gaussian blur augmentation does not significantly impact CIFAR-10 accuracy.

D End-to-end Simulations of Linear Setting

We empirically test the performance of the full trajectory of gradient descent when we add noise to the data. We study a setting with one weak feature with correlations coefficient α 1 ≤ 0 . 5 , and 50 dominant features with α k = 1 for k = 2 , · · · , 51 . We compare two approaches run on the same data: in the first approach, we run 150 iterations of GD without adding noise. In the second, we run 50 iterations of GD without noise, and then add noise to the dominant features for the remaining 100 iterations.

In Figure 10(top), we compare the alignment of Feature 1 (the weak feature) and Feature 2 (one of the dominant features) to the ground truth in the two approaches. We observe that adding noise consistently accelerates the learning of the weak feature (blue), with little cost to the dominant features (red). The affect is consistent among many choices for α 1 , the correlation coefficient of the weak feature. We also plot in Figure 10(bottom) the probability of predicting the correct class (pair)

Expert view

Viewmaker view

Viewmaker delta

LEN

L

of the view under both approaches. We observe that this probability drops sharply when we add noise, which we believe is the mechanism for faster learning with noise.

We remark that we chose to add noise to all the dominant features (instead of a single k ′ a in our theorem) to accentuate the effect of adding noise. We observed a similar effect, but smaller, when we added noise to fewer features, or when there were fewer than 50 dominant features.

E Full proofs of propositions and theorems

We begin by stating and proving Lemma E.1 on the downstream classification accuracy.

Lemma E.1 (Downstream classification accuracy) . Suppose we draw labeled data points ( u, y ) ∈ R K × d ×{ +1 , 1 } , where as before, u k ∼ N (0 , I d ) for k ∈ [ K ] , and the label is given by sign( u T k µ k ) . Then the best linear classifier a ∈ R K on the representations f Θ ( u ) ∈ R K achieves an test error of 1 π arccos ( | µ T k θ k | ∥ θ k ∥ 2 ) . That is

̸

Alignment of Features: [μ[9]/9xl2

Alignment of Features: [μu9x|/]6x12

Alignment of Features: [μ/6x|/]6vl2

Alignment of Features: [μ{9]/]9xl2

1.0

1.0

1.0

1.0

0.8

0.8

0.8

0.8

0.6

0.4

0.2

十 No Noise: feature 2

十No Noise: feature 1

Noise: feature 2

Noise: feature 1

0.6

0.4

2'0

No Noise: feature 1

+

+

+

No Noise: feature 2

Noise: feature 1

Noise: feature 2

0.6

0.4

0.2

+

+

十

No Noise: feature 1

Noise: feature 1

Noise: feature 2

No Noise: feature 2

0.6

0.4

0.2

No Noise: feature 1

+

+

+

No Noise: feature 2

Noise: feature 1

Noise: feature 2

0

20

40

60

80

100

120

140

0

20

40

60

80

100

120

140

0

40

60

80

100

120

140

0

20

40

60

80

100

120

140

1.0

P(correct)

1.0

P(correct)

1.0

P(correct)

1.0 -

P(correct)

0.8

0.8

0.8

0.8

0.6

0.4

No Noise

Noise

0.6

0.4

No Noise

Noise

0.6

0.4

No Noise

Noise

0.6

0.4

No Noise

Noise

0.2

0.2

0.2

0.2

0.0

0

20

40

60

80

100

120

140

0.0

0

20

40

60

80

100

120

140

0.0

0

20

40

60

80

100

120

140

0.0

0

20

40

60

80

100

120

140

Thus if θ k and µ k are orthogonal, then the test error is 50% . If the angle between θ k and the ± µ k is zero, then we achieve perfect classification accuracy.

Proof. It is easy to see that the best linear classifier a will (up to scaling) be equal to the vector sign( µ T k θ k ) e k . Such a classifier predicts the correct sign whenever sign( a T f Θ ( u )) =

In the rest of this section, we prove our main theoretical result, Theorem 5.1, which shows that arccos ( | µ T k θ k | ∥ θ k ∥ 2 ) decreases faster in expectation during gradient descent if we add noise to the k ′ feature.

E.1 Notation.

We let δ ij denote the δ -function which equals 1 if i = j and 0 otherwise. For a parameter Θ = { θ k } k ∈ [ K ] , we let θ ∥ k := µ k µ T k θ k be the projection of θ k in the µ k direction. We let θ ⊥ k = θ k -θ ∥ k be the projection of θ k orthogonal to the feature µ k .

Throughout this section, we consider the ground truth directions to be fixed, and we fix some initial correlation vector α . We let P α denote the distribution from which the pair ( u, v ) is drawn from the Gaussian distribution described in Section 5 with correlation coefficients α . When unspecified, the variables U, V are drawn from the distribution P m α . Since we study what happens when we vary α k ′ , for x ∈ [0 , 1] , we use the shorthand P x to denote the distribution P m α ( x ) , where α ( x ) k ′ = x , and α ( x ) k = α k for all other k .

We denote L i (Θ; U, V ) = CE( { p ij } j ∈ [ m ] , e i ) = -log( p ii ) , which we abbreviate by L i . When it is clear that we are considering L i for some fixed i , we omit the superscripts on the i th data point or its pair. That is, we denote u k := u ( i ) k and v k := v ( i ) k .

E.2 Preliminaries

The following facts about of the derivative of the cross entropy loss are easy derived.

Lemma E.2.

where

̸

We will also need the following facts on Gaussian random variables. The first, Stein's Lemma, is well known.

Lemma E.3 (Stein's Lemma) .

The next two lemmas are proved in Section E.4.

Lemma E.4. There exists some constant C such that following holds. If σ ≤ 1 C , and 0 ≤ t ≤ 1 σ , then for any c ∈ { 0 , 1 , 2 , 3 } , and X ∼ N (0 , σ 2 ) we have

If additionally d ∈ { 0 , 1 , 2 , 3 } , ρ ≤ 1 C and Y ∼ N (0 , ρ 2 ) , then

E.3 Approach and Lemmas

Intuition for proof of Theorem 5.1. Our proof involves comparing the gradient of the loss in the θ k direction, ∇ k := ∂ ∂θ k L in the setting with noise to the setting without noise. Loosely, our goal is to show that for any k , the projection of the this gradient onto the ground truth direction, µ T k ∇ k sign( µ T k θ k ) , increases when when increase the noise. The main intuition comes from an expansion of this gradient in Lemma E.7, which shows that E µ T k ∇ k sign( µ T k θ k ) approximately scales with ∑ i (1 -p ii ) . Now observe that p ii , the probability of correctly matching the i th view to its pair, decreases when we add noise to feature k ′ . Thus adding noise will increase µ T k ∇ k sign( µ T k θ k ) , thereby improving the alignment.

In the remainder of this section, we outline our proof of Theorem 5.1 in this section. We prove all the lemmas below in Section E.4.

To understand E U,V [ arccos ( | µ T k θ ( t +1) k | ∥ θ ( t +1) k ∥ 2 )] for a small enough step size, we first claim that it suffices to understand the expected projection of the gradient with respect to θ k in the µ k direction and in the θ k direction. We use the notation ∇ k = ∂ L (Θ; U,V ) ∂θ k .

where N is some negative value that depends only on θ k .

Now, since we care about the quantity E U,V [ arccos ( | µ T k θ ( t +1) k | ∥ θ ( t +1) k ∥ 2 )] -E U, ˜ V [ arccos ( | µ T k ˜ θ ( t +1) k | ∥ ˜ θ ( t +1) k ∥ 2 )] being positive, it suffices to show that derivative

is negative for all x ∈ [ ˜ α k ′ , α k ′ ] . Indeed, from Lemma E.6, we have that

so if the derivative is negative for the full range, then the difference in arccosines is positive.

In the following lemma we compute the derivative of E [ ∇ k ] with respect to x .

Lemma E.7.

̸

We will analyze this quantity by explicitly taking the expectation with respect to some set of random variables. Let S = { U k , V k , U k ′ , V k ′ } consist of the random variables u ( i ) k ′ , u ( i ) k , and v ( i ) k ′ , v ( i ) k for all i ∈ [ m ] . Define q ij to be the logits when all variables in S are set to 0 (Thus explicitly,

̸

q ij = exp ( ∑ ˜ k = k,k ′ θ T ˜ k u ( i ) ˜ k θ T ˜ k v ( j ) ˜ k ) ∑ j ′ exp ( ∑ ˜ k = k,k ′ θ T ˜ k u ( i ) ˜ k θ T ˜ k v ( j ′ ) ˜ k ) ). We will use the notation j ∼ q to denote the distribution on [ m ] with mass q ij on j .

Let

̸

and

which are the terms that appear in the right hand side of Lemma E.7 after p ii p ij . Observe that

The following four lemmas serve to bound d dx E S [ µ T k ∇ k ] and d dx E S [ θ T k ∇ k ] . We call the terms of the form E p ii p ij ( h ( S ) -h 1 ( S )) 'junk' terms, and our goal will be to show that these terms are small. We will control more closely the terms of the form E p ii p ij ( h 1 ( S )) .

Lemma E.8 (Junk Terms for µ k term.) . If ∥ θ k ∥ ≤ 1 and ∥ θ k ′ ∥ ≤ 1 , then for some universal constant C

Lemma E.9 (Good Term for µ k term.) . If ∥ θ k ∥ ≤ 1 and ∥ θ k ′ ∥ ≤ 1 , then for some universal constant C

Plugging these two lemmas into Lemma E.7 yields the following corollary.

Corollary E.9.1 (Total µ k term.) . If for a sufficiently large constant C , | θ T k µ k | ≤ 1 -α 2 k ′ C ∥ θ k ∥ ,

Lemma E.10 (Junk Terms for θ k term.) . If ∥ θ k ∥ ≤ 1 and ∥ θ k ′ ∥ ≤ 1 , then for some universal constant C

Lemma E.11 (Good Term for θ k term.) . If ∥ θ k ∥ ≤ 1 and ∥ θ k ′ ∥ ≤ 1 , then for some universal constant C

Plugging these two lemmas into Lemma E.7 yields the following corollary.

Corollary E.11.1 (Total θ k term.) . If for a sufficiently large constant C , ∥ θ ∥ k ∥ ≤ 1 -x 2 C ∥ θ k ∥ , ∥ θ k ′ ∥ 3 ≤ ∥ θ ∥ k ′ ∥ , ∥ θ k ∥ 2 ≤ α k (1 -x 2 ) C , then

Combining Corollaries E.9.1 and E.11.1, we obtain the following lemma.

Lemma E.12. If for a sufficiently large constant C , ∥ θ ∥ k ∥ ≤ 1 -x 2 C ∥ θ k ∥ , ∥ θ k ′ ∥ 3 ≤ ∥ θ ∥ k ′ ∥ , ∥ θ k ∥ 2 ≤ α k (1 -x 2 ) C , then

Theorem 5.1 now follows.

E.4 Proofs of Lemmas

To prove the Lemmas E.4 and E.5, we will use the following well-known formula for the moment generating function (MGF) of the half-normal distribution.

Lemma E.13 (MGF of half-normal distribution) . The MGF of the half-normal distribution is

where Φ( t ) is the cumulative distribution of a normal random variable.

Proof of Lemma E.4.

where r = σ √ 1 -2 σ 2 t . To evaluate this, we use the MGF of the half-normal distribution in Lemma E.13. Thus for some constant C , for all c ∈ { 1 , 2 , 3 , 4 } ,

So for some constant C (whose value changes throughout this equation), so long as σ ≤ 1 C ,

This proves the first statement in the lemma. To prove the second, we first take the expectation over X , and using the half-Gaussian MGF as before, we obtain

Now applying the first statement to take the expectation over Y , we obtain

Proof of Lemma E.5. We prove the lemma by induction on c . Suppose c = 0 . Then by plugging in the MGF for the half-normal distribution from Lemma E.13, for some constant C , we have

thus

Now for c ≥ 1 , by Stein's Lemma, we have (for a new constant C ),

where in the last step we used the inductive hypothesis and Lemma E.4.

Proof of Lemma E.6. First observe that

since lim η → 0 η 1 -ηλ = 0 . Now

where N = arccos ′ ( | µ T k θ k | ∥ θ k ∥ 2 ) 1 ∥ θ k ∥| µ T k θ k | . The lemma follows by taking the expectation over U, V , and observing derivative of arccos( x ) is negative whenever x is positive.

Proof of Lemma E.7. First observe that by symmetry, we have

To make this expectation easier to analyze, we express the random variable ( U ( x ) , V ( x )) ∼ P x as an interpolation of Gaussians in the coordinate µ T k ′ v ( i ) k ′ . Let ξ ∼ N (0 , 1) , and define ( U, V ) ∼ P 1 , such that µ T k ′ v ( i ) k ′ = µ T k ′ u ( i ) k ′ . For x ∈ [0 , 1) , define ( U ( x ) , V ( x )) to have

and otherwise be the same as ( U, V ) . It is easy to check that ( U ( x ) , V ( x )) ∼ P x .

Now

Taking the derivative of the cross-entropy loss, we have

̸

̸

where the variables z ij and p ij are the similarity scores and the softmaxes from the data ( U ( x ) , V ( x )) . Here the first line is by Lemma E.2, and the second line holds by chain rule since ∂z ij ∂θ k -∂z ii ∂θ k does not depend on v ( i ) k ′ . The third line uses the proof of Claim E.14 to take the derivative of p ij , and Equation 21 to take the derivative of µ T k ′ v ( i ) k ′ ( x ) .

Now we reparameterize µ T k ′ u ( i ) k ′ -x √ 1 -x 2 ξ as follows:

Plugging in this reparameterization and dz ii dµ T k ′ v ( i ) k ′ ( x ) = θ T k ′ µ k ′ θ T k ′ u k ′ , we obtain

̸

We now prove Lemmas E.8, E.9, E.10, and E.11.

Notation. Since i is fixed throughout, we drop the ( i ) superscripts and let u k = u ( i ) k and v k = v ( i ) k . We will introduce the following random variables, which are all independent, to simplify the exposition:

̸

̸

̸

For any such random variable X , we use σ 2 X to denote its variance. Observe that

We will use the following two claims in the proofs of all four lemmas.

Claim E.14. For β ∈ { ξ j , ξ ′ j , ξ i , ξ ′ i , ζ ′ i , η i , η ′ i , x, x ′ } , let ¯ β j ′ := ∂ ∂β ( θ T k u k θ T k v ( j ′ ) k + θ T k ′ u k ′ θ T k ′ v ( j ′ ) k ′ ) . Then

If additionally γ ∈ { ξ j , ξ ′ j , ξ i , ξ ′ i , ζ ′ i , η i , η ′ i } and γ ⊥ { ¯ β j ′ } j ′ ∈ [ m ] , then

Proof. By a straightforward quotient-rule computation of the derivative of p ij q ij , recalling that q ij is independent of S , we obtain

By applying product to the expression above, we obtain

Taking absolute values and using the fact that p ij ′ ≤ 1 , we obtain the first result.

Next we take the derivative of p ij with respect to both β and γ . Using the expression above for ∂p ij ∂β , we obtain

and

The second result follows by taking absolute values and the fact that p ij ′ ≤ 1 .

Proof. This follows directly from using Jenson's inequality on the distribution j ′ ∼ q to show that

Claim E.16.

Proof. Note that for any x ≥ 0 , we have | 1 -x | ≤ max ( x -1 , 1 x -1 ) . By Claim E.15, p ij q ij -1 is at most the desired value given in this claim.

Now

This yields the claim.

Proof of Lemma E.8. Expanding h ( S ) -h 1 ( S ) , we see that we need to control the following terms:

We begin by bounding the terms where the expression after p ii p ij has two independent mean-0 terms, mainly (1a), (2a), (4a). The first step is to apply Stein's Lemma (Lemma E.3) twice to these two terms, which we will call β and γ . Let βγg ( S \ { β, γ } ) be the terms after p ii p ij . Then we have

Next we apply the final result in Claim E.14 to bound the absolute value of ∣ ∣ ∣ ∂ ∂γ ∂p ii p ij ∂β ∣ ∣ ∣ . Once we do this, we achieve

where ∑ j ′ ,ℓ ∈ [ m ] c j ′ ,ℓ ≤ C for some constant C , and Z := p ii p ij q ii q ij . Finally, we use the bound on Z from Claim E.15, and then Lemma E.4 to take the expectation over S , iteratively applying Lemma E.4 to each variable in S . Thus we have, for some (different) constant C ,

Now we consider the remaining 7 terms. Here we decompose the expression inside the expectation as p ii p ij βg ( S \ β ) , where β ∈ S . We proceed as before, but we only apply Stein's Lemma once, to β . Applying Steins, the expression for ∂p ii p ij ∂β given in the first result of Claim E.14, we obtain

where ∑ j ′ ∈ [ m ] c j ′ ≤ C for some constant C , and Z := p ii p ij q ii q ij . Finally, we plug in a bound for Z in Claim E.15, an use Lemma E.4 to take the expectation over S , again iteratively over each variable. Thus we have, for some (different) constant C ,

Combining the bounds on these 10 terms proves the lemma:

Proof of Lemma E.10. The proof of Lemma E.10 is nearly identical, besides some differences in the terms we need to bound. We list them below:

We use the same approach as before. For the terms (1a) and (2a) we apply Stein's Lemma to ( η ′ i , ξ j ) and ( η ′ i , ξ i ) respectively. For (1b), (2b), (3a) and (3b) and (4), we apply Stein's Lemma to ξ j , ξ i , η ′ i , η i , and ξ ′ i respectively. Using Claim E.15 and then Lemma E.4 as before, we obtain the following result:

∣ ∣ E S [ p ii p ij ( y ′ ( µ T k ′ u k ′ -xµ T k ′ v k ′ )) ( θ T k u k ξ i )]∣ ∣ ≤ Cq ii q ij σ 2 ξ i ∥ θ k ∥∥ θ k ∥∥ θ ∥ k ′ ∥ ≤ Cq ii q ij ∥ θ k ∥ 4 ∥ θ ∥ k ′ ∥

∣ ∣ α k E S [ p ii p ij ( η ′ i ( µ T k ′ u k ′ -xµ T k ′ v k ′ )) ( θ T k u k y )]∣ ∣ ≤ Cα k q ii q ij σ 2 η ′ i ∥ θ k ′ ∥∥ θ k ∥∥ θ ∥ k ∥ = Cα k q ii q ij ∥ θ ⊥ k ′ ∥ 2 ∥ θ k ′ ∥∥ θ k ∥∥ θ ∥ k ∥

∣ ∣ α k E S [ p ii p ij ( y ′ ( µ T k ′ u k ′ -xµ T k ′ v k ′ )) ( η i y ) ]∣ ∣ ≤ Cα k q ii q ij σ 2 η i ∥ θ k ∥∥ θ ∥ k ′ ∥∥ θ ∥ k ∥ = Cα k q ii q ij ∥ θ ⊥ k ∥ 2 ∥ θ k ∥∥ θ ∥ k ′ ∥∥ θ ∥ k ∥ .

Combining the bounds on these 7 terms, proves the lemma:

We now prove the lemmas on the non-junk terms.

Proof of Lemma E.9.

Now by Claim E.16, we have ∣ ∣ ∣ p ii p ij q ii q ij -1 ∣ ∣ ∣ ≤ Z i Z j -1 (where the variable's Z i , Z j are defined in the Claim E.16) so

Here the second inequality follows from applying Lemma E.5 first, and then Lemma E.4 repeatedly for the remainder of the variables in S . This proves the lemma. Note that we need to apply Lemma E.5 several times to a single variable X ∈ S . Indeed we can write

for some distribution on ℓ , and for some terms S ℓ , t ℓ , and B that are independent of X , and c ∈ { 0 , 1 , 2 } . Then to take the expectation of this term over X , we first apply Lemma E.5 to on X to the first term, and iteratively apply Lemma E.5 to the random variables appearing in the next terms.

Proof of Lemma E.11.

Now by Claim E.16, we have ∣ ∣ ∣ p ii p ij q ii q ij -1 ∣ ∣ ∣ ≤ Z i Z j -1 , so

Again the second inequality follows from applying Lemma E.5 first (several times as described in the previous lemma), and then Lemma E.4 repeatedly for the remainder of the variables in S . Taking absolute values proves the lemma.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper questions a common assumption on data augmentations in contrastive unsupervised representation learning, specifically label-preserving, where data augmentations do not change the label of the original sample. Indeed, this assumption does not always holds in data augmentations used in contrastive unsupervised representation learning in practice as explained in Sec. 2. The authors claim that such a label-destroying data augmentation can be seen as feature-dropout, and this would be useful for universal representation learning, where the learnt feature extractor can give good discriminative representations for multiple downstream tasks.\nThe authors perform such label-destroying data augmentations are useful on synthetic datasets and Viewmaker networks, which are previously proposed generative models to improve downstream tasks' performance of contrastive unsupervised representation learning.\nTo give a theoretical understanding of this label-destroying data augmentation, the authors provide a theoretical justification of the usefulness of noise under a linear contrastive model with Gaussian-based data distribution in Sec. 5.\n\nSTRENGTHS:\nOriginality: Questioning the commonly used assumption on data augmentation sounds novel problem.\nQuality: The empirical and theoretical results give a better view of the problem (commonly used assumptions should be revisited)\nClarity: The paper is well-written and clearly explains the motivation of the problem with concrete examples.\nSignificance: Unsupervised representation learning for many downstream tasks is a desired algorithm in practice; this result gives another research direction to create a better \"view\" for such representation learning.\n\nWEAKNESSES:\n## W.1 Details of experiments and results are missing\n\n- I could not find how to perform the transfer accuracy reported in Tables 1 and 2. The details should be given in the manuscript or Appendix.\n- The audio results in Sec. 3 are in neither Figure 4 nor others.\n\n## W.2 Assumptions on the theoretical analysis\n\nThe empirical and theoretical results have a gap in data, model, and data augmentation. For example, the analysed linear model has a sparse connection and is independent of the other weights as in Fig. 3(b). Still, empirically used models are non-linear and the weights are non-independent via a fully connected or pooling layer.\n\nQUESTIONS:\nQ1: Could you give the details of experiments as pointed out by W1?\nQ2: L84: Is there any concrete example paper using \"brightness shifts repeatedly\"? In my understanding, SimCLR (Appendix A. in Chen et al. [2020]) applies it to a sample at most once.\nQ3: Clarification on data augmentation: \n\nL129 says\n\n> data augmentations for the image datasets [9]\n\nCould you clarify which data augmentation was used in SimCLR v1 paper? SimCLR paper used slightly different data augmentation depending on ImageNet or CIFAR-10. Concretely, B.9. in the SimCLR paper for CIFAR-10 says\n\n> \"For data augmentation ... leaving out Gaussian blur\"\n\nIn my understanding, the implementation of this paper uses the ImageNet version: not excluding Gaussian blur according to `load_default_transforms` and `load_default_unnorm_transforms` in `feature-dropout-code/src/datasets/datasets.py`.\n\nQ4: Can Figures 4, 6, 8, and 9 be in the main body? Their explanations are in the main body to discuss the proposed analyses but I needed to see them in the Appendix.\n\n\n### Typos and notation errors:\n\n- L83: `\\citep` for `Chen et al. [9] and He et al. [23]`?\n- Fig. 4: Fig 4(b)'s caption is \"Letters\", but a label in the legend is \"Digits accuracy\". Similar typos are in Fig. 4(c): caption is \"Digits\", but the label is \"Letters accuracy\".\n- L261: aren't -> are not\n- L265: didn't -> did not\n\nLIMITATIONS:\nAs mentioned in Weaknesses, the analysed linear model is a more restricted model class than widely used linear models in contrastive learning paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n**Summary**. The authors question the common assumption in contrastive learning that good augmentations should be label-preserving, and suggest that dropping features out (for example, with data augmentation) actually is helpful in practice because it helps learning of \"weaker\" features: features that may be helpful on downstream tasks, but are not necessary for the contrastive task. To do this, they create synthetic datasets, use view-maker augmentations, and consider a linear toy example.\n\n**My position.** I think this work is interesting: I find it interesting to read, but there are some limitations in the work that make me question it's central thesis. Overall, I marginally would like to see it at NeurIPS because I found it interesting and think it would lead to an interesting discussion, but some technical concerns limit my enthusiasm about the paper.\n\nSTRENGTHS:\n1. I enjoyed reading and thinking about this work, I think the question is interesting, and in general, I like the approach of questioning common assumptions in the field. The choice of data augmentation is clearly very important for contrastive learning, so that's great.\n\n2. The paper is reasonably well written.\n\nWEAKNESSES:\n# Major Weaknesses / Concerns\n\n1. The paper emphasises \"multiple downstream tasks\", but what this means precisely is unclear, and **I am not sure whether the experimental setup reflects the problems we actually care about.** In the experiments, the features for the added digit/shape, etc are basically completely independent of the rest of the image features. I'm not sure whether this holds in real life settings of multiple downstream tasks e.g., if classifying both digit rotation and digit identity, the features of the \"object\" are shared; does this count as multiple downstream tasks here? The paper emphasises \"the need for analyzing the interaction between multiple downstream tasks\" (L19), but I'm not sure, precisely, what this might mean. \n\n1. Instead of using synthetic experiments where another feature like a shape is overlaid on top of the base image, it would be better to use for example, a latent variable generative model, and downstream classification where different sets of features only are important for the class label. I think this would be closer to real world scenarios.\n\n2. The paper quotes Tian et al. \"a good set of views are those that share the minimal information necessary to perform well at the downstream task\". I think this is a slight strawman (i.e., simplification of the prevailing view), because it is well known that \"heavy\" data augmentation is necessary for good performance (e.g., experiments from the original SimCLR paper). I suggest making the nuance more clear, and make clear the aim of the work is to help clarify this confusion.\n\n3. There are no error bars on the experiment results, though multiple seeds are run.\n\n4. I'm not convinced that the toy example with a linear model captures the important aspects of the problem. I don't think that learning the last linear layer after a feature is what is challenging about feature learning, but rather, it is learning the complex, non-linear representation. I am more interested in how dropping out features would affect learning of the representation, rather than learning of the feature layer. (See also Suggestion 2).\n\n5. I am concerned about sensitivity to the viewmaker pertubation budget, $\\epsilon$. As far as I can tell, the viewmaker objective basically always tries to make the contrastive task as hard as possible, which is to remove as much information about the base image identity as possible (i.e., remove information about the shape as well as the base image), because the contrastive task is to identify the base image identity from augmented pairs of images. If the budget is too large, all the images would become noise. The objective directly incentivises feature dropout! Therefore, the choice of $\\epsilon$ is really important. I want to understand how it is that you chose this, and how sensitive your results and findings are to it. \n\n6. I don't think the expert augmentations are really \"expert\" augmentations in the sense that they don't preserve label information for the synthetic datasets. Some of the augmentations will destroy information e.g., by flipping a 9, it no longer becomes a digit. I would like to see augmentations where you only flip e.g., the digits that are invariant to flipping. Then, the comparison between viewmaker and expert is more fair.\n\n7. The results in Table 1 I find hard to interpret. Is the claim that the view-maker does feature dropout, and it performs better, suggesting that feature dropout is good? Because the difference in performance for shape are small, the performance for both the object and image class is poor for digit, and for letter, the expert augmentations outperform viewmaker. Also, should it not be possible that we can add the synthetic data without losing performance on the CIFAR-10 task?\n\n8. \"Two clear modes appear for viewmaker, but not for expert augmentations\" (L181/182). Some of the expert augmentations seem to have very small probability of the correct label, suggesting that they perhaps always dropout the label? I was surprised by this. (This is ~Fig. 6 in the appendix). Can you explain this please\n\n9. It seems many of the experimental results assume that viewmaker is doing feature dropout, and the expert augmentations are not i.e., the expert augmentations are good. I'm not sure this is actually the case.\n\n10. WRT the theoretical analysis, I have some concerns. (i) the setup is not like contrastive learning. Here, we have $K$ features in a higher dimension space, but really the feature space is low dimensional. But contrastive representations live on a hypersphere, with the contrastive loss encouraging points to spread out across the hypersphere. (ii) \"A good representation will capture ground true features ... such that $\\theta_k$ is aligned with $\\mu_k$ ...\" I am confused by this; isn't there permutational and rotational invariance here? i.e., $\\theta_1$ could be aligned with $\\mu_5$. This, as a result, undermines your theoretical results.\n\n11. In figure 4 in the appendix, the synthetic feature dropping out only improves the accuracy of the main image class quite a small amount, and dropping the feature out also drastically reduces the accuracy of the synthetic task. What one should do in this scenario is quite unclear. Perhaps we would do better by just constructing new, strong data augmentation that preserves label information. That seems to be the fairest comparison.\n\n\n## Other suggestions\n1. L22: pitching the paper as foundation models seems like overselling because the work focuses on contrastive learning.\n\n2. If the hypothesis is feature dropout, one could also try dropping out features that are learnt during contrastive pre-training e.g., adding dropout regularisation before or after the projection head. This would (approximately) correspond directly to feature dropout, and would be somewhat similar to the linear contrastive example you use. Do you think this would work? Have you considered this?\n\n3. Figure 4 and Figure 6 should not be in the appendix if they are mentioned in the main paper as prominently as they are.\n\n4. I suggest streamlining Figure 1 e.g., including only one row, making it more visually clear which image is the base image and which images are augmentations.\n\n5. I would like to see the accuracy of the contrastive task (i.e., how accurate is the linear layer when the class templates are the mean representations of augmente images) for the different viewmaker augmentations.\n\nQUESTIONS:\n1. Do you think feature dropout changes the optimal solution, or just makes the optimisation process easier?\n\n2. L84: \"repeated application of some augmentations causes challenges for all downstream tasks\". Are people using repeated augmentation in practice? I'm not sure this is a real problem.\n\n3. Are the expert views really expert for CelebA?\n\n4. For Fig. 4, how are the features dropped out? They just aren't added to the image?\n\nLIMITATIONS:\nI would have liked to see further discussion of the limitations of the work that I mentioned above.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors of this work challenge the belief that label-preserving augmentations are necessary for contrastive learning by demonstrating that label-destroying augmentations can be useful for learning representations useful for a variety of downstream tasks.  The authors treat augmentations as feature dropout, and demonstrate how straightforward contrastive learning techniques struggle to learn multiple features that are present in a visual scene.  This was demonstrated through synthetically created datasets.  Furthermore, the authors discover that feature dropout helps learn representations that encode multiple competing features - the authors believe that this is what explains why Viewmaker networks can help learn a wide range of features well.\n\nSTRENGTHS:\nThe strengths of this work lies in its thorough experimentation and its theoretical justification regarding the benefits of learning representations of multiple features when noise is added to corrupt a particular feature.  The work was clearly and well-presented, and convincing arguments were made when challenging previously-held beliefs regarding augmentations for contrastive learning.  Furthermore, linking the idea of stochastic feature dropout with the success of Viewmaker networks is an interesting discovery.\n\nWEAKNESSES:\nIn the introduction, the authors suspect that augmentations serve as a form of feature dropout that prevents any one feature from becoming a shortcut feature and suppressing the learning of other features.  This idea has been noted before in prior work, as also mentioned in the Related Work section (in “Feature Suppression”).  In the original SimCLR paper [Ting et. al, 2020], it was discovered that certain combinations of augmentations improved the representations learned.  In particular, color jittering was found to consistently improve performance; it has been speculated that the addition of color jittering as a data augmentation helps to destroy the color histogram values for pixels from the same patch, which can easily shortcut the recognition of two positives.  Therefore, it is already generally accepted that augmentations in contrastive losses serve the effective purpose of feature dropout.  On the other hand, the authors utilize a set of synthetic datasets to determine if certain features suppress the learning of other features; this line of work was also previously done in prior work [Ting, et. al, 2022].  It must be noted that the original authors of [Ting et. al, 2022] did not attempt stochastic dropout of the secondary features.\n\nThe paper can be made better with improvements to the formatting - for example, the entirety of page 5 is devoted to a singular figure.\n\nI believe most of the entire work is predicated on the feature suppression effect, which go through great lengths to show even though it is a well-known property of contrastive losses.  I agree that the authors explore this idea thoroughly in their work, extending it to multiple modalities, adding theoretical justification, and explaining the potential reason VIewmaker networks work well, but fundamentally I believe the work lacks substantial core novelty beyond what was already established in prior work.\n\nTing Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.\n\nTing Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.\n\nQUESTIONS:\nIt is encouraging that Viewmaker networks seem to overcome it due to feature-level dropout.  Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations? \n\nWhat are some ways this idea can be demonstrated for more natural-looking images?  Currently, the synthetic datasets created indeed highlight the feature suppression affect inherent in contrastive learning techniques.  However, are there any practical cases of natural images where we would like multiple features to be learned from a given image that could showcase the approach outlined in the paper?\n\nLIMITATIONS:\nNo negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-12"}