SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Alex Tamkin, Margalit Glasgow, Xiluo He, Noah Goodman  
Stanford University
What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. synthetic datasets combining two classes, such as images and digits, and naturalistic datasets labeled with dozens of attributes). In controlled experiments where we destroy features at different rates, we find that destroying one feature a modest fraction of the time can improve learning of other features, while still enabling the dropped out feature to be learned well. Additionally, we show how this hypothesis can help explain the success of Viewmaker Networks, which generate augmentations that appear to target and destroy different features in the input examples, yet often result in better performance than standard augmentations across tasks. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, we show that label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.
In recent years, foundation models [5] have exhibited remarkable progress on a range of AI tasks [13; 31; 37; 36; 6; 11; 25; 1; 38]. A crucial characteristic of foundation models is that they can be adapted for a range of downstream tasks. For example, a foundation model trained on ImageNet should ideally not only perform well at object classification, but should also have learned general features useful for localization, segmentation, and other visual tasks. Indeed, this is borne out by recent work showing the high accuracy of foundation models on a range of downstream tasks [9], as well as a range of analysis work showing that models learn high-level semantic features including texture, color, pose, and style [19].
One popular strategy for training foundation models involves training models to match transformed versions (known as views or augmentations) of the same input. For example, image views might include common data augmentations such as cropping or color jitter [9], while views for speech might include pitch modulation or spectrogram masking [27; 35]. This family of objectives includes contrastive approaches such as SimCLR and MoCo, as well as non-contrastive approaches such as BYOL and SwAV [9; 23; 20; 7].
Given the central importance of these views for defining the self-supervised task, much work has focused on the question of what views lead to high-quality representations. The prevailing consensus, exemplified by Tian et al. [52], holds that views should be label-preserving with respect
to a downstream task. In other words, because the contrastive loss will produce representations which are invariant to features that vary across views, any information we wish to preserve in the representations should not be altered by such views. As Tian et al. [52] write: "A good set of views are those that share the minimal information necessary to perform well at the downstream task."
Here, we question whether this assumption—in particular, with its focus on a single task—is enough to explain why contrastive foundation models succeed on a range of downstream tasks. In Section 2, we observe that the actual choice and application of views in practice does not align with this prevailing consensus. For example, complete invariance to several common augmentations (e.g. shifts in brightness or cropping) is undesirable since augmentations of inputs from different classes can collide. Furthermore, in many cases there are explicit ways to specify invariances (e.g. converting images to grayscale) that researchers avoid, instead specifying them indirectly via augmentations (e.g. hue shifts). These observations suggest that specifying invariances is not the sole role of these views.
Instead, we suspect that augmentations serve as a form of feature dropout—preventing any one feature from becoming a shortcut feature and suppressing the learning of other features. We study this idea empirically with a set of synthetic datasets constructed by overlaying a simple element (e.g. a digit, shape, letter, or speech snippet) on an image or audio recording. We find that adding such a simple feature can dramatically decrease how well the other feature is learned, but that stochastically "dropping out" the simple feature can enable both features to be learned well. Next, we use this perspective to explain the success of Viewmaker Networks, a recently proposed method that generates augmentations for contrastive learning via adversarial training. We apply viewmaker and expert views to these same synthetic datasets, as well as a naturalistic dataset of facial images annotated with 40 different attributes (e.g. "wearing lipstick" or "blond hair"). Across these settings, we find that viewmaker augmentations learn to selectively obscure various features in the input. Despite this, the viewmaker representations still learn the downstream tasks well, while expert views often struggle on one or more of the attributes. This further suggests that being label-preserving is not a necessary property of good views, as long as the label information is still sometimes accessible.
Finally, we formalize the intuition that feature dropout can aid learning with a theoretical analysis of a simple linear contrastive setting. In this setting, we characterize how the noisiness of each feature directly determines how quickly features are learned, and uncover an interaction between features governing how fast they are learned. In particular, we show how learning one feature quickly can suppress the learning of other features, and show that adding noise to the "easiest" feature can increase the rate at which other features are learned. This further indicates that label-destroying augmentations may have a direct role in ensuring that contrastive models learn a broad range of features for downstream tasks.
Overall, these findings suggest the need to revisit common assumptions about the role of augmentations for contrastive learning in the foundation model setting, and move towards a better understanding of how to train generalist models that learn diverse features from unlabeled data.
2 Common practices are at odds with the "invariance" explanation
We begin by briefly exploring several common augmentations used in contrastive learning for natural images, and explore how they come into conflict with the common assumption described above. First, we observe that many common augmentations can affect the label of the input, depending on the downstream task. For example, many downstream image recognition tasks require color information (e.g. identifying bird species) or brightness (e.g. scene or time-of-day classification), implying that invariance to these characteristics would be undesirable. Yet hue shifts, greyscale, and brightness shifts are common augmentations used in contrastive learning [9; 23].
Second, repeated application of some augmentations causes challenges for all downstream tasks. For example, applying brightness shifts repeatedly results in any image turning completely black or completely white. Thus the class label cannot be truly invariant to this augmentation, since inputs from different classes can experience an "augmentation collision" at this black or white image (this is
formalized in Appendix B). This argument also applies to other augmentations, including shifts in contrast and random masking.
Third, some augmentations are commonly used despite ways of explicitly encoding invariance to them. For example, two image augmentations are hue shifts and greyscale. Invariance to both of these augmentations can be explicitly encoded by always converting an image to greyscale. Yet doing so is not common practice because color information is still desirable for many downstream tasks.
The contradictions between the invariance rationale for augmentations in contrastive learning and these common practices suggest the need for additional explanations for the role of augmentations.
3 Controlled experiments demonstrate the benefits of feature dropout in settings with multiple features
In this section, we present controlled experiments on synthetic data demonstrating how label-destroying augmentations can balance the learning of multiple features during contrastive learning. Our core toolkit is to overlay images with a set of synthetic features. As we will show, the presence of these synthetic features causes the network to learn the synthetic features very well at the expense of the image features, as measured by downstream classification accuracy. However, "dropping out" these features some fraction of the time during contrastive learning enables us to trade-off how well each feature is learned, while not resulting in complete invariance to either set of features.
We consider the behavior of viewmaker networks on four synthetic datasets, including three image and one audio dataset. Each dataset is constructed in such a way as to support two distinct downstream classification tasks, enabling us to examine precisely how well each downstream task is learned. The presence of two downstream tasks enables us to analyze the foundation model setting where we wish to learn features relevant for multiple downstream tasks, as opposed to one set or the other.
Image datasets The three image datasets are based on the canonical CIFAR-10 image-recognition dataset [28] (MIT-License). One task is always to predict the CIFAR-10 object label (e.g. airplane or bird). The other task is dependent on an additional feature overlaid on the image: C+Shapes: The CIFAR-10 image is overlaid with one of three randomly-colored shapes: a square, a triangle, or a circle. The second task is to predict what shape was overlaid  $(N = 3$  classes). C+Digits: The CIFAR-10 images are overlaid with four copies of a randomly-sampled digit from the MNIST dataset. The second task is to predict the digit class  $(N = 10$  classes). C+Letters: The CIFAR-10 images are overlaid with four copies of a randomly-colored English letter. The second task is to predict the class of the letter  $(N = 26$  classes).
Audio dataset The audio dataset is created by overlaying the audio of a spoken digit (from the AudioMNIST dataset [3], MIT License) with a random background sound (collected from one of three possible classes: cafe, machinery, and traffic) [43; 42]. The tasks are to predict the digit class  $(N = 10$  classes) and to predict the sound class  $(N = 3$  classes). Inputs are presented to the network as log mel spectrograms.
Pretraining We pretrain with the SimCLR algorithm for 200 epochs with a batch size of 256 and a temperature of 0.1. We use a ResNet-18 model with standard modifications for smaller inputs (including a smaller stride and no initial maxpool) as used in Tamkin et al. [49]. We use the standard SimCLR augmentations for the image datasets [9], with the exception of gaussian blurring, which we found to have no impact on downstream performance (Appendix C.4). For audio, we use the SpecAug [35] augmentations, which randomly mask out different frequency and time bands, as well as the WaveAug [27] augmentations, which alter various properties of the waveform such as the pitch and speed.
<sup>1</sup>Note that invariance is not to be confused with the related but distinct property of equivariance, often discussed as a desirable property of network architectures (e.g. see Fukushima and Miyake [17]; Chen et al. [8]).
Linear Evaluation We evaluate the quality of the learned representations by training a linear softmax classifier on top of the prepool representations. We train for 100 epochs, using the same parameters as Tamkin et al. [49], using SGD with learning rate 0.01, momentum 0.9, weight decay 0, and batch size 128. We train separate linear classifiers using the same pretrained network for each downstream task [9]. Augmentations are applied during training but not evaluation.
Results As shown in Figure 1, we see an interaction between the two features, where dropping out the synthetic feature improves learning of the main image or audio class. Across settings, we see regions where both features are still learned well, providing a concrete example of how feature dropout can be useful when learning multiple features during contrastive learning.
(a) Shapes
(c) Letters
Figure 1: Linear probing accuracy (y-axis) after contrastive learning with varying rates of dropout of the synthetic feature (x-axis). In all cases, we see a clear tradeoff between features, where dropping out the synthetic feature improves learning of the object class.
(b)Digits
(d) Audio
4 Viewmaker Networks Succeed Despite Destroying Label Information
As another point of evidence that good views need not be label-preserving, we consider viewmaker networks [49], a generative model which produces augmentations for contrastive learning. Intuitively, viewmakers learn a stochastic augmentation policy that makes the contrastive task as hard as possible for the encoder. The stochastic augmentations are parameterized as additive perturbations bounded by an  $L_{1}$  norm, meaning the viewmaker can alter but not completely destroy the original image.
Formally, given an input  $x \in \mathbb{N}$ , a viewmaker network  $V_{\psi}$  is trained jointly with an encoder  $E_{\theta}$  to optimize the minimax expression:
$$
\max  _ {\psi} \min  _ {\theta} \mathcal {L} \left(E _ {\theta} \left(x + \epsilon \frac {V _ {\psi} (x , \delta_ {1})}{| | V _ {\psi} (x , \delta_ {1}) | | _ {1}}\right), E _ {\theta} \left(x + \epsilon \frac {V _ {\psi} (x , \delta_ {2})}{| | V _ {\psi} (x , \delta_ {2}) | | _ {1}}\right)\right)
$$
Here  $\mathcal{L}$  is a multiview loss function (e.g. [9; 23]),  $x$  is a minibatch of inputs,  $||\cdot ||_1$  is the  $L_{1}$  norm,  $\epsilon$  is the distortion budget controlling the strength of the views, and  $\delta_1,\delta_2\sim N(0,I)$  are random inputs that enable the viewmaker to learn a stochastic augmentation policy. We clamp the output of the viewmaker for images to [0, 1] as in Tamkin et al. [49].
Viewmaker networks learn to stochastically alter different parts of the input, including task-relevant features, meaning that these augmentations are not label-preserving. Nevertheless, as we will see shortly, viewmaker networks enable strong performance on multiple downstream tasks, including often better performance than expert-designed augmentations. Moreover, this feature dropout capability of viewmaker networks may help them learn many features well rather than just the easiest.
Figure 2: Comparison of viewmaker and expert augmentations on datasets with multiple features. The viewmaker augmentations adapt to the particular semantics of the input data, and make targeted perturbations which remove the class-relevant information of the synthetic features (e.g. occluding the digit, shape, letter, or speech). Despite this, the encoder network is still able to learn strong representations. Rows (from top): Digits, Shapes, Letters, Audio. Columns (from left): Expert augmentations, viewmaker augmentations, difference between original and viewmaker augmentation, rescaled to [0,1]. Center image in each grid is the original. Audio Expert views shown are Spectral views.
4.1 Experiments and Results
Experimental Settings We use the same experimental settings as Section 3, however without manual dropout of the synthetic features. In one set of experiments, we use the standard augmentations from Chen et al. [9], which we henceforth refer to as the expert augmentations. For the experiments with viewmaker augmentations, we use a budget of  $\epsilon = 0.05P$  for the image datasets, and  $\epsilon = 0.125P$  for the audio datasets, where  $P$  is the number of pixels in the input.
Additional naturalistic dataset with 40 attributes To further validate the behavior of viewmaker on realistic multi-feature datasets, we consider the CelebA [32] dataset, a large database of faces annotated with 40 different features. These features cover a wide spectrum of facial attributes, such as "Has Bangs" "Wearing Lipstick" and "Smiling," and enable us to further analyze whether viewmaker networks learn a broader range of features than commonly-used augmentations.
VM (CIFAR-10)	Expert (CIFAR-10)	VM (Object)	Expert (Object)
CIFAR-10 Only	84.5	86.2	-	-
C+Shape	79.8	76.0	100.0	100.0
C+Digit	69.3	58.8	94.3	86.7
C+Letter	71.9	74.8	96.9	94.1
Table 1: Transfer accuracy on different features. Viewmaker (VM) networks are able to achieve good performance across multiple downstream tasks, while expert views sometimes falter. Networks are pretrained on the datasets on the left, and transfer accuracy is reported for the different conditions on the columns. Runs are averages of three seeds (with the exception of CIFAR-10 Only, from [49]).
Speech Accuracy	Background Sound Accuracy
Viewmaker	Spectral	Waveform	Viewmaker	Spectral	Waveform
Speech Only	92.4	97.0	76.7	-	-	-
Bkgd. Sound Only	-	-	-	100.0	100.0	100.0
Speech + Sound	71.3	64.6	50.2	100.0	99.9	98.0
Table 2: Audio transfer accuracies. Viewmaker networks achieve good performance across multiple tasks, while expert views sometimes suffer catastrophic drops as another feature is added. Networks are pretrained on the datasets on the left, and transfer accuracy is reported for the different conditions on the columns. Runs are averages of three seeds.
4.2 Results on two-feature datasets
Qualitative evidence of feature dropout Visually, the viewmaker augmentations seem to stochastically alter different aspects of the input, as shown in Figure 2. In addition to modifying the background of each input, the viewmaker also selectively modifies the additional synthetic features added to each domain: C+Digits: The viewmaker augmentations selectively add pixels to the MNIST digits, making it difficult to distinguish which number is present. C+Shapes: The viewmaker augmentations sometimes draw squares around the shape in the center, making it difficult to determine the shape class. C+Letters: The viewmaker draws letter-like markings on top of the letters, obscuring the letter identity and color. Audio: The viewmaker identifies the narrow band corresponding to the speech and applies perturbations to it. As can be seen in Figure 2, these label-destroying augmentations are quite common, occurring in a sizable fraction of the sampled views.
Quantitative evidence of feature dropout We also measure this selectivity of features quantitatively in Appendix C.2 and Figure 6. We augment images 1,200 times and observe the resulting probability assigned to the correct object class. Two clear modes appear for viewmaker, but not expert, augmentations. This corresponds to the fraction of time the viewmaker destroys the overlaid feature information (low P.correct object class)) and preserves it (high P(correct object class)).
Viewmaker succeeds despite destroying label information As shown in Table 1 and Table 2, viewmaker networks achieve good accuracy on both tasks, while expert augmentations frequently achieve lower performance on one or both. On image tasks, for example, while expert views achieve slightly higher performance when classifying the image only, they see a large drop in accuracy when
Figure 3: Viewmaker networks capture a broader range of features on a naturalistic dataset. Linear evaluation F1 score on CelebA for viewmaker and expert views. Attributes are sorted from lowest to highest accuracy within each augmentation type.
(b)
Figure 4: We show how label-destroying augmentations can aid learning of other features in a linear contrastive setting: (a) The correlation of the  $k$ th feature of an augmentation pair, shown for  $d = 2$ . Each pair  $u_{k}^{(i)}$  and  $v_{k}^{(i)}$  have correlated projections onto the ground truth  $\mu_{k}$  direction, representing the feature conserved across augmentations. (b) Feedforward linear network which computes the representation  $f_{\Theta}(w)$ . As each feature  $\mu_{k}$  is learned ( $\theta_{k} \rightarrow \mu_{k}$ ) the representations of the two views  $f_{\Theta}(u^{(i)})$ ,  $f_{\Theta}(v^{(i)})$  become more similar, decreasing the contrastive loss.
the synthetic feature is added. In two of these cases (Shape and Digit) the viewmaker models are able to achieve a higher accuracy on both the image and the synthetic feature, while on the third (Letters) they achieve slightly lower accuracy on the images but achieve half the error on the synthetic object. For the audio experiments the picture is similar—viewmaker avoids catastrophic drops in performance learning both features together, achieving the highest accuracy on both, while the expert views see larger drops and worse overall performance. Note that the high performance of expert views for our control tasks (CIFAR-10/Speech/Sound Only) indicates that the viewmaker views are not merely better all-around views, but that they specifically help the model learn multiple features.
These results provide additional evidence that label-preserving views are not necessary for learning good representations—and that feature dropout may improve learning of multiple tasks.
4.3 Results on naturalistic dataset
We observe similar qualitative and quantitative results for the CelebA dataset. We train models using the same settings in Section 3.2, using a budget of 0.01, and indeed find that viewmakers capture a much broader range of features, achieving an average F1 Score of 0.509 over the 40 features, compared to 0.334 for the SimCLR augmentations. In addition, the viewmaker augmentations clearly capture a wider range of features, as can be seen in Figure 3, especially at the tail of the distribution. Furthermore, we see bimodal disruption patterns in over two-thirds of the CelebA features, as shown in Figure 9, indicating significant feature dropout across in most attributes. We also show qualitative results in Figure 8 demonstrating that the viewmaker alters attributes such as facial features, hair color, and background elements in the scene. These results further support the hypothesis that viewmaker networks exhibit feature dropout, yet capture a broader range of features than expert views.
5 Theoretical Analysis of Feature Interactions in Linear Contrastive Setting
In this section, we analyze a simple linear model that captures the essence of how label-destroying augmentations can improve downstream accuracy. We study a setting where the data contains many underlying features that are relevant to downstream classification tasks, and where these features are preserved to varying degrees across augmentations. We will show that a linear model trained with a contrastive objective learns these features, and that adding noise to one feature can speed learning of others during gradient descent. One difference between the linear setting we analyze and Section 4 is that here add stochastic Gaussian noise to destroy features across augmentations, as opposed to the bimodal feature dropout behavior of viewmaker networks seen in Figure 2.
5.1 Data Model and Setting