SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We study a model which consists of data with  $K$  distinct features, each corresponding to some ground truth unit-vector directions  $\mu_1,\ldots ,\mu_K\in \mathbb{R}^d$ . We sample each data point  $u\in \mathbb{R}^{K\times d}$  and its augmentation (a.k.a. its positive pair or its view)  $v\in \mathbb{R}^{K\times d}$  as follows. For  $k\in 1,\dots ,K$ , the  $k$ th row of  $u$ , which we denote  $u_{k}$ , is drawn from the Gaussian distribution  $\mathcal{N}(0,I_d)$ . The  $k$ th row of the augmentation,  $v_{k}$ , is drawn from the same distribution, but is correlated with  $u_{k}$  in the  $\mu_k$ -direction (and is otherwise independent in the other directions). The strength of the correlation is governed by parameter  $\alpha_{k}\in [0,1]$  in the following sense:  $v_{k}^{T}\mu_{k} = \alpha_{k}u_{k}^{T}\mu_{k} + \sqrt{1 - \alpha_{k}^{2}}\xi$ , where  $\xi \sim \mathcal{N}(0,1)$ . Thus the larger  $\alpha_{k}$ , the stronger the correlation in that feature across the two views. Figure 4(a) visualizes the correlation of  $(u_{k},v_{k})$  in an augmented pair. Formally, we can write that  $(u_{k},v_{k})\sim \mathcal{N}\left(0,\left( \begin{array}{cc}I_{d} & \alpha_{k}\mu_{k}\mu_{k}^{T}\\ \alpha_{k}\mu_{k}\mu_{k}^{T} & I_{d} \end{array} \right)\right)$ , for a vector  $\alpha \in [0,1]^k$ .
We will learn a model  $\Theta \in \mathbb{R}^{K\times d}$ , which represents a collection of  $K$  feature extractors, as pictured in Figure 4(b). The model  $\Theta$ , with rows  $\{\theta_k\}_{k\in [K]}$ , maps a data point  $w\in \mathbb{R}^{K\times d}$  to a representation  $f_{\Theta}(w)\in \mathbb{R}^{K}$  by computing a score  $w_{k}^{T}\theta_{k}$  for each element in the representation. That is,  $(f_{\Theta}(w))_k = w_k^T\theta_k$ . Our goal is that the model  $\Theta$  will be useful for a downstream classification task which depends on the ground truth features. A good representation will capture ground truth features that are correlated across augmentations, such that  $\theta_{k}$  is aligned with  $\mu_{k}$  or  $-\mu_{k}$ .
**Training.** We will study the evolution of  $\Theta$  as we optimize a standard constructive learning objective using gradient descent [14; 9]. At each round of gradient descent, we sample a fresh batch of  $m$  data points and their augmentations,  $(U,V) \coloneqq \{(u^{(i)},v^{(i)})_{i\in [m]}\}$ . For each  $i,j \in [m]$ , we compute a similarity score  $z_{ij} \coloneqq \langle f_{\Theta}(u^{(i)}),f_{\Theta}(v^{(j)})\rangle = \sum_{k}(\theta_{k}^{T}u_{k}^{(i)})(\theta_{k}^{T}v_{k}^{(j)})$  using the dot product of their  $K$ -dimensional representations. We then compute the logits  $p_{ij} \coloneqq \frac{\exp(z_{ij})}{\sum_{j'}\exp(z_{ij'})}$  using the softmax function, and use the classwise cross entropy loss function  $\mathcal{L}(\Theta;U,V) \coloneqq -\log(p_{ii})$ .
5.2 Main Result
We will study gradient descent (GD) on the cross entropy loss, and consider how adding noise to one feature affects the learning of the other features. As suggested earlier, we can measure how well we learn the  $k$ th feature by measuring the alignment of  $\theta_{k}$  with  $\mu_{k}$  or  $-\mu_{k}$ . A natural way to measure this alignment is the acute angle between  $\pm \mu_{k}$  and  $\theta_{k}$ , given by  $\arccos \left(\frac{|\boldsymbol{\mu}_{k}^{T}\boldsymbol{\theta}_{k}|}{\|\boldsymbol{\theta}_{k}\|_{2}}\right)$ . Lemma E.1 in Appendix E proves that this quantity directly determines the test accuracy on a natural downstream linear classification task.
Formally, we say we add noise to some feature  $k'$  of a data point  $v$ , if for some  $\beta \in [0,1)$ , we define the new noisy data point  $\tilde{v}$  to have coordinates  $\tilde{v}_{k'} = \beta v_{k'} + \sqrt{1 - \beta^2}\xi$ , where  $\xi \sim \mathcal{N}(0,I_d)$ , and  $\tilde{v}_k = v_k$  for  $k\neq k'$ . Thus if  $(u,v)$  were a pair generated with the correlation coefficients  $\{\alpha_k\}_{k\in [K]}$  then the distribution of  $(u,\tilde{v})$  comes from the modified correlation coefficients  $\{\tilde{\alpha}\}_{k\in [K]}$  with the single modification  $\tilde{\alpha}_{k'} = \beta \alpha_k$ . We now present our main theorem:
Theorem 5.1 (Noise improves feature learning). There exists a universal constant  $C$ , such that the following holds. Let  $\Theta^{(t + 1)} = \Theta^{(t)} - \eta (\nabla \mathcal{L}(U,V;\Theta) + \lambda \Theta^{(t)})$ , and  $\tilde{\Theta}^{(t + 1)} = \Theta^{(t)} - \eta (\nabla \mathcal{L}(U,\tilde{V};\Theta) + \lambda \Theta^{(t)})$ , where  $\tilde{V}$  is  $V$  with any amount of added noise in the  $k^{\prime}$  feature. This has the effect of changing  $\alpha_{k'}$  to  $\tilde{\alpha}_{k'}$  for any  $\tilde{\alpha}_{k'} < \alpha_{k'}$ . Then for any  $k \neq k'$ , if  $|\theta_k^T\mu_k| \leq \frac{1 - \alpha_{k'}^2}{C} \| \theta_k\|$ ,  $||\theta_{k'}||^3 \leq |\theta_{k'}^T\mu_k|$ , and  $||\theta_k||^2 \leq \frac{\alpha_k(1 - \alpha_{k'}^2)}{C}$ , then for a small enough step size  $\eta$ ,  $\mathbb{E}_{U,V}\left[\arccos \left(\frac{|\mu_k^T\theta_k^{(t + 1)}|}{\|\theta_k^{(t + 1)}\|_2}\right)\right] > \mathbb{E}_{U,\tilde{V}}\left[\arccos \left(\frac{|\mu_k^T\tilde{\theta}_k^{(t + 1)}|}{\|\tilde{\theta}_k^{(t + 1)}\|_2}\right)\right]$ .
We briefly comment on the three assumptions on  $\Theta$  in the theorem. The first assumption,  $|\theta_k^T\mu_k|\leq \frac{1 - \alpha_{k'}^2}{C}\| \theta_k\|$  requires that  $\theta_{k}$  is not too aligned with  $\mu_{k}$  - that is, the result applies to all features  $k$  that are not already learned too well. The second two assumptions are satisfied if the  $k^{\prime}$  th feature has been learned to some extent, and the norm of  $\theta_{k}$  and  $\theta_{k^{\prime}}$  are small, which can be enforced throughout training with  $\ell_2$  regularization.
The theorem guarantees that at any point in training, if we add noise to the  $k^{\prime}$ th feature, the next step of GD learns all other features better than if we did not add noise. To validate the implication of this result for the complete trajectory of GD, we include simulations in Appendix D. Our experiments show that introducing noise part-way through training to dominant features can significantly speed the alignment of weak features, with only a small cost to the alignment of the dominant features. We prove our result in Appendix E, including intuition and an overview of the ideas in Section E.3.
Understanding contrastive and multiview learning Many prior works have laid the foundations for current contrastive and multiview learning algorithms [4; 21; 14; 55; 2; 34; 23; 9]. Several works analyze contrastive learning to identify important factors [12; 58] or how contrastive models differ from supervised learning [57; 15; 26]. HaoChen et al. [22] study contrastive learning using the concept of an augmentation graph. This model assumes the fraction of non-label preserving augmentations is "extremely small"; interestingly, we show in practice this can be quite large and still yield good performance. Wang et al. [54] theoretically study contrastive learning under an assumption of label-preserving augmentations, though they show that such an assumption alone does not suffice to learn. Most relevant to our work, Tian et al. [52]; Ericsson et al. [16] study how the information shared between different views impacts learning of downstream tasks. We complicate this picture by analyzing the foundation model setting where a single model must learn features for multiple tasks that are not known in advance. In this setting, we find that label-destroying perturbations, thought to be harmful by Tian et al. [52], are useful for preventing one feature from suppressing others.
Feature suppression Our work is closely connected to the notion of feature suppression [24], where the presence of one feature can suppress the learning of others. Several works explore this concept in contrastive learning. For example, the original SimCLR paper [9] noted that color jitter augmentation was necessary to prevent the network from using only the color profile of the input to solve the contrastive task. Followup work [10] characterizes how hyperparameters and dataset features affect feature suppression, including through a range of synthetic experiments that vary the amount of information shared between views. Other works have attempted to address feature suppression in contrastive learning, either via auxiliary losses [29] or by modifying representations in the latent space [39]. Our builds upon these works by empirically and theoretically investigating feature suppression as an alternate rationale for the role of augmentations, as opposed to invariance. We also show that an existing method, viewmaker networks [49], can identify and potentially neutralize suppressing features in an interpretable way, resulting in better performance than expert augmentations. These insights may also generalize to other self-supervised settings, such as language modeling, where multiple features may compete [47].
Spurious correlations and shortcut features Outside the framing of feature suppression, several other works explore how classifiers can learn or make use of unwanted features. Shortcut features [18] describe often-simple features (e.g. the average color of an input) which are learned by networks at the expense of more salient features (e.g. the object class). This notion is connected to spurious correlations [45] in deep learning which have been explored extensively [40; 41; 46; 53; 56], including in the context of self-supervised learning [33; 51]. Other works have also performed theoretical analysis of how related dynamics affect learning in the supervised setting [30; 44]. In general, if such spurious correlations are known in advance, one can often design augmentations to remove such correlations and improve learning. However, our work suggests that viewmaker networks may be a useful tool in cases where such features are not known a priori—both as an interpretability tool to visualize the different features a network relies on, and as a way to reduce reliance on particular features without completely destroying the information.
We explore the idea that augmentations in contrastive learning function as a sort of "feature dropout." First, we show that in datasets with multiple features, dropping out one set of features improves learning of the other features. Second, feature dropout may help explain how viewmaker networks can learn a wide range of features well, despite producing augmentations which appear to destroy different features in the input. Finally, we theoretically analyze a linear contrastive setting where we
prove that label-destroying views have a positive effect on contrastive learning if the goal is to avoid learning one feature at the expense of others.
Our work has limitations: for example, while our experiments consider image and audio data, self-supervised learning may be applied to a much wider range of modalities [48; 50]. In addition, our theoretical analysis considers a linear contrastive setting, whereas current neural networks are highly nonlinear. Improving upon both of these fronts is an exciting area for future work. On the other hand, understanding augmentations as dropping out easy features suggests possible ways of improving the performance of self-supervised learning. For example, a guided version of viewmaker might enable prioritizing a subset of important features for learning, or might enable dropping out unwanted features such as watermarks, sensitive information, other image artifacts.
The challenge of learning a broad range of useful features lies at the heart of self-supervised learning. We hope our work sheds light on this challenge in contrastive learning, especially as these objectives continue to develop and are applied more broadly and at larger scale.
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangoooi, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. ArXiv, abs/2204.14198, 2022.
[2] Philip Bachman, R. Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. In NeurIPS, 2019.
[3] Soren Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals. ArXiv, abs/1807.03418, 2018.
[4] Suzanne Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 355:161–163, 1992.
[5] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R'e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramér, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. On the opportunities and risks of foundation models. ArXiv, abs/2108.07258, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[57] Xingyi Yang, Xuehai He, Yuxiao Liang, Yue Yang, Shanghang Zhang, and Pengtao Xie. Transfer learning or self-supervised learning? a tale of two pretraining paradigms. ArXiv, abs/2007.04234, 2020.
[58] Nanxuan Zhao, Zhirong Wu, Rynson W. H. Lau, and Stephen Lin. What makes instance discrimination good for transfer learning? ArXiv, abs/2006.06606, 2021.
A Code release
Our code is open-sourced at https://github.com/xiluohe/feature-dropout.
B Formalization of observation in Section 2
Definition B.1 (Invariance). A function  $f: \mathbb{R}^m \to \mathbb{R}^n$  is invariant to a set of transformations  $G$  if and only if  $f \circ g(x) = f(x)$  for all  $x \in \mathbb{R}^m$  and for all  $g \in G$ .
Definition B.2 (Augmentation collision). An augmentation collision occurs if, for two inputs  $x_{a}, x_{b}$  and set of transformations  $G$ , there exist  $g_{a}^{(1)}, \ldots, g_{a}^{(n_{a})}, g_{b}^{(1)}, \ldots, g_{b}^{(n_{b})} \in G$  for some  $n_{a}, n_{b} \in \mathbb{N}$  such that  $g_{a}^{(1)} \circ \ldots \circ g_{a}^{(n_{a})}(x_{a}) = g_{b}^{(1)} \circ \ldots \circ g_{a}^{(n_{b})}(x_{b})$ .
Observation B.3. If there exists an augmentation collision for inputs  $x_{a}, x_{b}$  and transformation set  $G$ , and  $f$  is invariant to  $G$ , then  $f(x_{a}) = f(x_{b})$ .
Proof. By the definition of an augmentation collision,  $g_{a}^{(1)} \circ \ldots \circ g_{a}^{(n_{a})}(x_{a}) = g_{b}^{(1)} \circ \ldots \circ g_{a}^{(n_{b})}(x_{b})$ . By the definition of a function, we have  $f \circ g_{a}^{(1)} \circ \ldots \circ g_{a}^{(n_{a})}(x_{a}) = f \circ g_{b}^{(1)} \circ \ldots \circ g_{a}^{(n_{b})}(x_{b})$ . Applying invariance, we obtain  $f(x_{a}) = f(x_{b})$ .
Applying this observation, we observe that if the downstream labeling function  $f$  is invariant to a class of augmentations, then there cannot be an augmentation collision for inputs with different labels. However, common augmentations such as brightness shifts can reduce any image to a black or white image, resulting in an augmentation collapse between any two inputs.
C Additional feature dropout experiments
C.1 Quantifying the importance of feature dropout
To assess the importance of label-destroying augmentations to the success of the viewmaker, we experiment with a setup where the viewmaker cannot destroy the information in the object class. To do this, we compute a mask around the object and zero out any perturbation from the viewmaker within that mask. We then perform pretraining and transfer as usual.
As we report in Table 3, the accuracy of the CIFAR-10 class label drops precipitously, as expected. At the same time, the accuracy of two of the other objects remains mostly constant (shape and digits), while the accuracy for letters declines modestly (perhaps because the color of the letter is now able to suppress the learning of the letter class.
Viewmaker (C-10)	Mask-Viewmaker (C-10)	Viewmaker (Object)	Mask-Viewmaker (Object)
C+Shape	79.8	26.0	100.0	95.8
C+Digit	69.3	50.7	94.3	95.0
C+Letter	71.9	23.2	96.9	71.8
Table 3: Experiments with a masked viewmaker which is unable to destroy the object class. Transfer accuracy on CIFAR-10 (C-10) and the object task (Shape, Digit, or Letter). The Mask-Viewmaker has its perturbation masked such that it cannot destroy the label of the object. This results in the features in the object suppressing the CIFAR-10 accuracy, while leaving the object accuracy relatively unscathed.
C.2 Quantifying the degree of feature dropout
We perform an exploratory analysis to testing how well different views drop out the features in an input. We augment a 1,200 examples (CIFAR-10 image plus an overlaid object) using a given augmentation policy (either the expert or viewmaker augmentations). We then encode the model with a classifier trained off of the other augmentation policy (i.e. expert for viewmaker augmentations
Figure 5: Non-label destroying Viewmaker perturbation examples.
(a) Viewmaker / Shapes
(d) Expert / Shapes
Figure 6: Viewmaker augmentations stochastically drop out simple features added to the input. Probability of the correct answer for different augmentations (Viewmaker or Expert) and different examples from different datasets (Shapes, Letters, Digits). Each histogram shows a single example from each dataset randomly augmented 1200 times, and the corresponding probabilities of the correct answer. The viewmaker augmentations display a bimodal structure, indicating that the simple feature is selectively either destroyed or preserved. The expert augmentations by contrast lack such structure, reflecting their lack of adaptation to the structure of each input.
(e) Expert / Letters
(c) Viewmaker / Digits
(f) Expert / Digits
or the reverse) in order to test how well the augmentations drop out the features. We use a different encoder to see the effects of the augmentations prior to the encoder having a chance to adapt to them.
We observe a bimodal behavior for the viewmaker views, shown in Figure 6, suggesting that the model is adapting to the semantics of the input and has learned to stochastically drop out the simple feature some fraction of the time. By contrast, the expert views display no such structure. Using the corresponding encoder and views leads to models performing uniformly well, as shown in Figure 7.
C.3 Additional visualizations for CelebA
We show feature dropout histograms for CelebA for each of the 40 features in Figure 9. The prevalence of bimodal distributions demonstrates a high degree of feature dropout across attributes. Histograms shown are viewmaker augmentations on an encoder trained with expert views.
(a) Viewmaker / Shapes
(d) Expert / Shapes
Figure 7: Evaluating views with their respective encoder does not reveal bimodal structure for viewmaker or expert views. Details are the same as in Figure 6, with the exception that views are evaluated on their corresponding encoder.
(b) Viewmaker / Letters
(e) Expert / Letters
We also show views and diffs for CelebA in Figure 8. These views show a high degree of sensitivity to the input semantics, and appear to modify characteristics such as the background, hair color, and facial features.
C.4 Importance of Gaussian Blur Transformation
We show experiments for CIFAR-10 (with no overlaid features) on SimCLR (expert views) with and without Gaussian blur augmentations. We find that the inclusion of the Gaussian blur augmentation does not significantly impact CIFAR-10 accuracy.
CIFAR-10 Accuracy
Blur	86.3%
No Blur	86.1%
Table 4: Experiments with Gaussian blur augmentation. Transfer accuracy on CIFAR-10 using expert (SimCLR) views with and without Gaussian blur.
D End-to-end Simulations of Linear Setting
We empirically test the performance of the full trajectory of gradient descent when we add noise to the data. We study a setting with one weak feature with correlations coefficient  $\alpha_{1} \leq 0.5$ , and 50 dominant features with  $\alpha_{k} = 1$  for  $k = 2, \dots, 51$ . We compare two approaches run on the same data: in the first approach, we run 150 iterations of GD without adding noise. In the second, we run 50 iterations of GD without noise, and then add noise to the dominant features for the remaining 100 iterations.
In Figure 10(top), we compare the alignment of Feature 1 (the weak feature) and Feature 2 (one of the dominant features) to the ground truth in the two approaches. We observe that adding noise consistently accelerates the learning of the weak feature (blue), with little cost to the dominant features (red). The affect is consistent among many choices for  $\alpha_{1}$ , the correlation coefficient of the weak feature. We also plot in Figure 10(bottom) the probability of predicting the correct class (pair)
Viewmaker view
Figure 8: Comparison of viewmaker and expert augmentations on CelebA. The viewmaker augmentations adapt to the particular semantics of the input data, and make targeted perturbations that alter features such as facial features, hair color, background, and skin tone. Columns (from left): Expert augmentations, viewmaker augmentations, difference between original and viewmaker augmentation, rescaled to [0,1]. Center image in each grid is the original.
Viewmaker delta
Figure 9: Most Celeb-A features are dropped out by viewmaker. Accuracy of a linear classifier trained on expert images and evaluated on 2000 different augmentations of different images. Over two-thirds of the features exhibit a bimodal structure, indicating feature dropout by the viewmaker augmentations.
of the view under both approaches. We observe that this probability drops sharply when we add noise, which we believe is the mechanism for faster learning with noise.
We remark that we chose to add noise to all the dominant features (instead of a single  $k'$  a in our theorem) to accentuate the effect of adding noise. We observed a similar effect, but smaller, when we added noise to fewer features, or when there were fewer than 50 dominant features.
E Full proofs of propositions and theorems
We begin by stating and proving Lemma E.1 on the downstream classification accuracy.