Title: value: Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning
Authors: value: Alex Tamkin
Margalit Glasgow
Xiluo He
Noah Goodman
Venue: NeurIPS 2023
Keywords: value: self-supervised learning
contrastive learning

Abstract: value: What role do augmentations play in contrastive learning? Recent work suggests that good augmentations are label-preserving with respect to a specific downstream task. We complicate this picture by showing that label-destroying augmentations can be useful in the foundation model setting, where the goal is to learn diverse, general-purpose representations for multiple downstream tasks. We perform contrastive learning experiments on a range of image and audio datasets with multiple downstream tasks (e.g. for digits superimposed on photographs, predicting the class of one vs. the other). We find that Viewmaker Networks, a recently proposed model for learning augmentations for contrastive learning, produce label-destroying augmentations that stochastically destroy features needed for different downstream tasks. These augmentations are interpretable (e.g. altering shapes, digits, or letters added to images) and surprisingly often result in better performance compared to expert-designed augmentations, despite not preserving label information. To support our empirical results, we theoretically analyze a simple contrastive learning setting with a linear model. In this setting, label-destroying augmentations are crucial for preventing one set of features from suppressing the learning of features useful for another downstream task. Our results highlight the need for analyzing the interaction between multiple downstream tasks when trying to explain the success of foundation models.


Review discussion:
Originality: Questioning the commonly used assumption on data augmentation sounds novel problem.
Quality: The empirical and theoretical results give a better view of the problem (commonly used assumptions should be revisited)
Clarity: The paper is well-written and clearly explains the motivation of the problem with concrete examples.
Significance: Unsupervised representation learning for many downstream tasks is a desired algorithm in practice; this result gives another research direction to create a better "view" for such representation learning.
## W.1 Details of experiments and results are missing
- I could not find how to perform the transfer accuracy reported in Tables 1 and 2. The details should be given in the manuscript or Appendix.
- The audio results in Sec. 3 are in neither Figure 4 nor others.
## W.2 Assumptions on the theoretical analysis
The empirical and theoretical results have a gap in data, model, and data augmentation.
For example, the analysed linear model has a sparse connection and is independent of the other weights as in Fig. 3(b).
Still, empirically used models are non-linear and the weights are non-independent via a fully connected or pooling layer.
Q1: Could you give the details of experiments as pointed out by W1?
Q2: L84: Is there any concrete example paper using "brightness shifts repeatedly"? In my understanding, SimCLR (Appendix A. in Chen et al. [2020]) applies it to a sample at most once.
Q3: Clarification on data augmentation:
L129 says
> data augmentations for the image datasets [9]
Could you clarify which data augmentation was used in SimCLR v1 paper? SimCLR paper used slightly different data augmentation depending on ImageNet or CIFAR-10. Concretely, B.9. in the SimCLR paper for CIFAR-10 says
> "For data augmentation ... leaving out Gaussian blur"
In my understanding, the implementation of this paper uses the ImageNet version: not excluding Gaussian blur according to `load_default_transforms` and `load_default_unnorm_transforms` in `feature-dropout-code/src/datasets/datasets.py`.
Q4: Can Figures 4, 6, 8, and 9 be in the main body? Their explanations are in the main body to discuss the proposed analyses but I needed to see them in the Appendix.
### Typos and notation errors:
- L83: `\citep` for `Chen et al. [9] and He et al. [23]`?
- Fig. 4: Fig 4(b)'s caption is "Letters", but a label in the legend is "Digits accuracy". Similar typos are in Fig. 4(c): caption is "Digits", but the label is "Letters accuracy".
- L261: aren't -> are not
- L265: didn't -> did not
As mentioned in Weaknesses, the analysed linear model is a more restricted model class than widely used linear models in contrastive learning paper.
No ethics review needed.
We thank the reviewer for their review! We are glad they found our research question novel and significant, and to give a better view of the problem.
**W1a: Details of transfer experiments.**
To perform the transfer experiments in Tables 1 and 2 we use the same setup as in the Viewmaker Paper [Tamkin et al, 2021] (see L132-135). We will replicate the details in the paper for the next revision to make the paper more self-contained.
**W1b: Audio results in Figure 4.**
We thank the reviewer for this observation. We will update Figure 4 to include the audio results. As with the image datasets, we see a stark tradeoff between features as the dropout rate is increased (the gap in accuracy changes by over 50%)
**W2. Limitations of theoretical results.**
These are all true limitations of our theoretical setup!
While we mention several limitations in the paper, we will add the limitations mentioned here (e.g. sparse connectivity). Unfortunately it is currently challenging to get strong theoretical results for features learned in deep nonlinear networks.
However we find it encouraging that despite such a simplified setting, we are able to see a clear feature dropout effect in our theoretical setting. White not conclusive, this is an encouraging sign as to the generality of the feature dropout phenomenon.
**Q1: Experimental settings**
We use the settings from Tamkin et al 2021, and will include them in the paper (see response to W1a above)
**Q2 Repeated brightness shifts.**
Correct, as far as we know people do not do repeated augmentations explicitly. Our point here is just that if T(x) has support under the data distribution D (our assumption), then if we were to sample T(x) from D, we would then produce T(T(x)), which in effect is a repeated augmentation. This causes an augmentation collision in the sense we discuss in the paper. One case where this assumption is reasonable is brightness shifts: amateur photographers often mis-expose images, resulting in a broad range of brightness shifts. A brightness shift applied to a very light image could result in a pure white image, destroying the class information.
**Q3: We use gaussian blur?**
This is a good catch;
indeed we do use gaussian blur in our experiments unlike the CIFAR-10 experiments in the paper.
However, we notice no impact on the CIFAR accuracy in our results for including this information. We will update the paper to note this.
**CIFAR-10 (no overlaid features) with SimCLR (expert views):**
No blur: 0.8614
Blur: 0.8632
**Q4: Moving figures to main text.**
Yes, we will consolidate Figure 1 and move up the important figures from the appendix given the extra page provided if our paper is accepted.
**Typos**
We are grateful for these suggestions and have fixed them on our copy of the manuscript.
We thank the reviewer again for their thoughtful comments and review.
Thank you for your response, especially for providing additional experimental results for Q3, which is really helpful.
The first weakness would be addressed in a self-contained way: explaining the details of the experiment and adding the audio's result to Fig. 4.
I'd be happy to raise my rating to 6 from 5.
Thank you for raising your score!
1. I enjoyed reading and thinking about this work, I think the question is interesting, and in general, I like the approach of questioning common assumptions in the field. The choice of data augmentation is clearly very important for contrastive learning, so that's great.
2. The paper is reasonably well written.
# Major Weaknesses / Concerns
1. The paper emphasises "multiple downstream tasks", but what this means precisely is unclear, and **I am not sure whether the experimental setup reflects the problems we actually care about.** In the experiments, the features for the added digit/shape, etc are basically completely independent of the rest of the image features. I'm not sure whether this holds in real life settings of multiple downstream tasks e.g., if classifying both digit rotation and digit identity, the features of the "object" are shared; does this count as multiple downstream tasks here? The paper emphasises "the need for analyzing the interaction between multiple downstream tasks" (L19), but I'm not sure, precisely, what this might mean.
1.
Instead of using synthetic experiments where another feature like a shape is overlaid on top of the base image, it would be better to use for example, a latent variable generative model, and downstream classification where different sets of features only are important for the class label. I think this would be closer to real world scenarios.
2. The paper quotes Tian et al. "a good set of views are those that share the minimal information necessary to perform well at the downstream task". I think this is a slight strawman (i.e., simplification of the prevailing view), because it is well known that "heavy" data augmentation is necessary for good performance (e.g., experiments from the original SimCLR paper). I suggest making the nuance more clear, and make clear the aim of the work is to help clarify this confusion.
3. There are no error bars on the experiment results, though multiple seeds are run.
4. I'm not convinced that the toy example with a linear model captures the important aspects of the problem. I don't think that learning the last linear layer after a feature is what is challenging about feature learning, but rather, it is learning the complex, non-linear representation. I am more interested in how dropping out features would affect learning of the representation, rather than learning of the feature layer. (See also Suggestion 2).
5. I am concerned about sensitivity to the viewmaker pertubation budget, $\epsilon$. As far as I can tell, the viewmaker objective basically always tries to make the contrastive task as hard as possible, which is to remove as much information about the base image identity as possible (i.e., remove information about the shape as well as the base image), because the contrastive task is to identify the base image identity from augmented pairs of images. If the budget is too large, all the images would become noise. The objective directly incentivises feature dropout!
Therefore, the choice of $\epsilon$ is really important. I want to understand how it is that you chose this, and how sensitive your results and findings are to it.
6. I don't think the expert augmentations are really "expert" augmentations in the sense that they don't preserve label information for the synthetic datasets. Some of the augmentations will destroy information e.g., by flipping a 9, it no longer becomes a digit. I would like to see augmentations where you only flip e.g., the digits that are invariant to flipping.
Then, the comparison between viewmaker and expert is more fair.
7. The results in Table 1 I find hard to interpret. Is the claim that the view-maker does feature dropout, and it performs better, suggesting that feature dropout is good? Because the difference in performance for shape are small, the performance for both the object and image class is poor for digit, and for letter, the expert augmentations outperform viewmaker.
Also, should it not be possible that we can add the synthetic data without losing performance on the CIFAR-10 task?
8. "Two clear modes appear for viewmaker, but not for expert augmentations" (L181/182). Some of the expert augmentations seem to have very small probability of the correct label, suggesting that they perhaps always dropout the label? I was surprised by this. (This is ~Fig. 6 in the appendix). Can you explain this please
9. It seems many of the experimental results assume that viewmaker is doing feature dropout, and the expert augmentations are not i.e., the expert augmentations are good. I'm not sure this is actually the case.
10. WRT the theoretical analysis, I have some concerns. (i) the setup is not like contrastive learning. Here, we have $K$ features in a higher dimension space, but really the feature space is low dimensional.
But contrastive representations live on a hypersphere, with the contrastive loss encouraging points to spread out across the hypersphere. (ii) "A good representation will capture ground true features ... such that $\theta_k$ is aligned with $\mu_k$ ..." I am confused by this; isn't there permutational and rotational invariance here? i.e., $\theta_1$ could be aligned with $\mu_5$. This, as a result, undermines your theoretical results.
11. In figure 4 in the appendix, the synthetic feature dropping out only improves the accuracy of the main image class quite a small amount, and dropping the feature out also drastically reduces the accuracy of the synthetic task. What one should do in this scenario is quite unclear. Perhaps we would do better by just constructing new, strong data augmentation that preserves label information. That seems to be the fairest comparison.
## Other suggestions
1. L22: pitching the paper as foundation models seems like overselling because the work focuses on contrastive learning.
2. If the hypothesis is feature dropout, one could also try dropping out features that are learnt during contrastive pre-training e.g., adding dropout regularisation before or after the projection head. This would (approximately) correspond directly to feature dropout, and would be somewhat similar to the linear contrastive example you use. Do you think this would work? Have you considered this?
3. Figure 4 and Figure 6 should not be in the appendix if they are mentioned in the main paper as prominently as they are.
4. I suggest streamlining Figure 1 e.g., including only one row, making it more visually clear which image is the base image and which images are augmentations.
5. I would like to see the accuracy of the contrastive task (i.e., how accurate is the linear layer when the class templates are the mean representations of augmente images) for the different viewmaker augmentations.
1. Do you think feature dropout changes the optimal solution, or just makes the optimisation process easier?
2. L84: "repeated application of some augmentations causes challenges for all downstream tasks". Are people using repeated augmentation in practice? I'm not sure this is a real problem.
3. Are the expert views really expert for CelebA?
4. For Fig. 4, how are the features dropped out? They just aren't added to the image?
I would have liked to see further discussion of the limitations of the work that I mentioned above.
No ethics review needed.
We thank the reviewer for their review! We are glad they found the results interesting, and appreciated our “questioning common assumptions in the field.”
**[Q1 and Q2] Do our synthetic experiments generalize to the the real world?**
This is an important question, a big reason we conducted experiments on CelebA, a naturalistic dataset with over 40 annotated features per image (with complex interactions, as the reviewer suggests).
**Q3: Adding nuance to discussion of prevailing view.**
We agree. We can add something like the follow line (but are open to feedback): “However, the strong augmentations used in real-world settings [Chen et al, 2020], complicate this picture, lending the need for further analysis”
**Q4: Runs have multiple seeds but no error bars.**
We were not able to run enough trials for error bars to be useful, but believe the three seeds we ran go help reduce any variance in the results.
**Q5: Validity of linear model**
We agree it would be interesting to examine non-linear feature learning as well, however there are few theoretical tools that can do this at the moment. We do believe that this linear model captures the “important” aspects of the problem (the impact of feature dropout), as the linear model can be viewed as learning a representation capturing K features from a high dimensional dataset.
**Q6: Choice of budget**
We agree the choice of the budget is important. The original Viewmaker paper found [Tamkin et al, 2021], as well as a recent preprint [Bayrooti et al, 2023] show that good performance is possible at a range of budgets. We also found similar results (below).
|Budget|CIFAR Acc.|Letter Acc|
**Q7: New experiments without flipping.**
The performance decreases for both the CIFAR-10 label and the digit label without horizontal flipping, suggesting that it aids (rather than injures) MNIST accuracy.
||CIFAR Acc|Letter Acc|
|No Flip|0.693|0.943|
|Flip|0.572|0.923|
**Q8: Interpreting results in Table 1.**
Yes, automatic feature dropout performed by Viewmaker enables the model to better capture features across tasks. Accuracies are generally lower when synthetic features are overlaid, but this makes the task harder overall so the decrease is expected.
**Q9: Why expert views have small probability in Figure 6 (Appendix)**
This could be due: 1) no feature dropout, or 2) the models not being robust to the other model’s augmentations.
Thus, while we can see that viewmakers perform feature dropout, it is not conclusive evidence of how much expert views are doing feature dropout.
**Q10: Are expert views also doing dropout?**
Expert views are likely doing dropout to some degree (e.g. see Figure 6 in Chen et al)!
However, we believe viewmaker networks may perform feature dropout in an automatic and more thorough way.
**Q11: Choices made in theoretical setup.**
(i) We are not certain we understand the reviewer’s question, so we would be happy to respond more with further clarification. The K-dimensional representations are encouraged to be spread out by the contrastive loss, though other work (e.g. [Tian et al, 2021]) has found that the hypersphere constraint is not a crucial component of contrastive learning and can be removed.
(ii) In our setting,  the total dimension of the data point is d*K, not d, and each of the K features is learned by a different set of parameters. Figure 3b may also help clarify the reviewer’s confusion.  We do not believe this undermines our theoretical result, but would be happy to clarify further given additional context in the discussion period.
**Q12: Augmentations if label-information is known**
We agree (and will mention in our revision) that if label information is known, design augmentations targeting this information.
However, in the general case cases this information will not be known in advance, nor will all the possible features that we want to learn be known.
**S1. Foundation models framing**
We believe we study one kind of foundation model, at least as defined in [Bommasani et al, 2021] where SimCLR was provided as an example (but acknowledge the term is sometimes used to refer to LLMs).
**S2. Dropping out features in projection head.**
This is an interesting idea and we have had similar thoughts. Our initial investigations suggested that the network would learn to make redundant sets of features to counteract the effects of dropout, however further experimentation could be interesting.
**S3. Figures in appendix.**
We agree it would be best to have these figures in the main text, and will use the extra page towards that end.
**S4. Suggestion for Figure 1. **
We will experiment with streamlining Figure 1. We agree that only having one row may decrease the amount of space needed for this figure and thank the reviewer for the suggestion.
**Q1: Does feature dropout change the optimal solution?**
Yes, we believe so: for example, consider a feature perfectly predictive of the label (say a watermark). The model might only rely on that feature, ignoring the rest of the image. Dropping out the watermark would force the network to rely on other parts of the input.
**Q2: Repeated augmentations.**
Yes, people do not do this explicitly.
However, consider e.g. amateur photographers, who often mis-expose or bracket images, resulting in a broad range of brightness shifts. A brightness shift applied to a very lightest image could result in a pure white image, destroying the class information.
**Q3: Are the expert views really expert for CelebA?**
We believe they are a reasonable choice, as the expert views were designed for use on a broad range of natural image datasets.
However, we certainly agree that better face-specific augmentations could be developed.
**Q4: How are features dropped out?**
Correct, these features are simply not added to the image.
We thank the reviewer again for their thoughtful comments and review!
Thanks. Thanks for the response.
Q1/Q2: This is good, but the experiments here seem to be not mentioned that much (only once or twice in the paper), and most of the experiments are on other more toy settings.
Q3: Thanks!
Q4: I am concerned that the error bars are not useful. What does this mean?
Q5: I disagree. I'm happy with toy model settings, but in general, I'm not sure learning a faster linear model means we'll learn the representations better.
Q6: I'm not convinced by this experiment, the range of episilon is tiny.
Q7: Thanks.
Q8-10: Thanks.
Q11: Thanks, this is more clear. I suggest making this clearer in the manuscript.
Other points: thanks also.
overall, I raise score to weak accept.
Thank you for raising your score!
Q1/Q2: This is a good point, we will mention these results more prominently in the next revision.
Q4: We only meant that the number of samples (3) is small for plotting error bars
Q11: Thank you, we will do so!
The strengths of this work lies in its thorough experimentation and its theoretical justification regarding the benefits of learning representations of multiple features when noise is added to corrupt a particular feature.  The work was clearly and well-presented, and convincing arguments were made when challenging previously-held beliefs regarding augmentations for contrastive learning.
Furthermore, linking the idea of stochastic feature dropout with the success of Viewmaker networks is an interesting discovery.
In the introduction, the authors suspect that augmentations serve as a form of feature dropout that prevents any one feature from becoming a shortcut feature and suppressing the learning of other features.  This idea has been noted before in prior work, as also mentioned in the Related Work section (in “Feature Suppression”).  In the original SimCLR paper [Ting et. al, 2020], it was discovered that certain combinations of augmentations improved the representations learned.
In particular, color jittering was found to consistently improve performance; it has been speculated that the addition of color jittering as a data augmentation helps to destroy the color histogram values for pixels from the same patch, which can easily shortcut the recognition of two positives.
Therefore, it is already generally accepted that augmentations in contrastive losses serve the effective purpose of feature dropout.
On the other hand, the authors utilize a set of synthetic datasets to determine if certain features suppress the learning of other features; this line of work was also previously done in prior work [Ting, et. al, 2022].  It must be noted that the original authors of [Ting et. al, 2022] did not attempt stochastic dropout of the secondary features.
The paper can be made better with improvements to the formatting - for example, the entirety of page 5 is devoted to a singular figure.
I believe most of the entire work is predicated on the feature suppression effect, which go through great lengths to show even though it is a well-known property of contrastive losses.  I agree that the authors explore this idea thoroughly in their work, extending it to multiple modalities, adding theoretical justification, and explaining the potential reason VIewmaker networks work well, but fundamentally I believe the work lacks substantial core novelty beyond what was already established in prior work.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. ArXiv, abs/2002.05709, 2020.
Ting Chen, Calvin Luo, and Lala Li. Intriguing properties of contrastive losses. Advances in Neural Information Processing Systems, 34, 2021.
It is encouraging that Viewmaker networks seem to overcome it due to feature-level dropout.
Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations?
What are some ways this idea can be demonstrated for more natural-looking images?  Currently, the synthetic datasets created indeed highlight the feature suppression affect inherent in contrastive learning techniques.
However, are there any practical cases of natural images where we would like multiple features to be learned from a given image that could showcase the approach outlined in the paper?
No negative societal impact.
No ethics review needed.
We thank the reviewer for their thoughtful review! We are glad that they found our experiments and theoretical results to be thorough. From our understanding, the reviewer’s main point of concern is clarifying the paper's additional contribution on top of the two papers cited.
We are happy to state this more clearly, and include the below discussion in the paper:
As the reviewer notes, our paper focuses on “feature dropout”—how stochastic ablation of features in an input  can broaden the range of features learned by a network. This investigation complements the two works cited: [1] conjectures that stochastically destroying color information is useful by qualitatively comparing color histograms (Figure 6), while [2] shows that varying the size of MNIST digits (MultiDigits), number of MNIST digits (DigitOnImageNet), and number of shared bits (RandBit) can affect whether one feature is learned more than others. Our study adds to the existing literature in four ways:
1. Studying how augmentations can reduce the strength of features.
In particular, this enables us to study the phenomenon conjectured by [1] using similar tools as those considered by [2].
2. This choice of studying feature dropout is especially important, as viewmaker networks seem to prominently display this dropout phenomenon.
In other words, a neural network **emergently discovered** the same phenomenon that human experts have speculated is important for learning good representations! This contribution has both scientific interest as well as empirical implications, given the promise of generative views in settings where good augmentations are not known (e.g. varieties of scientific or sensor data).
3. We go beyond synthetic data to conduct an in-depth study on the naturalistic CelebA dataset, which has over 40 annotated attributes (Section 4, Figure 2, Appendix D3, and Figure 8).
4. We have a novel theoretical contribution that is not present in recent work, demonstrating theoretically how feature dropout can aid learning.
In sum, there are multiple distinct contributions in our work that complement [1] and [2]. We will include this discussion in the next draft, including highlighting papers [1] and [2] more prominently in the introduction and related work, and would be more than happy to incorporate any additional suggestions from the reviewer for framing these contributions in the discussion period.
**Q1: Then, do the authors believe that utilizing Viewmaker networks with contrastive losses are the best way to learn multi-feature aware representations?**
We believe viewmaker networks are likely a very good starting point for researchers in domains where good augmentations are less well-studied (e.g. audio, many types of scientific data, wearable sensors, etc).
However, they are likely to be less helpful in cases where we have well-developed augmentations that encode strong priors over what features are helpful to humans (e.g. in natural images).
**Q2: Natural looking images***
We believe our study of the naturalistic CelebA dataset, which has over 40 annotated features, is a good example of this direction. We show that viewmaker augmentations enable the encoder to capture a much broader range of features in this setting.
We thank the reviewer again for their thoughtful comments and review!
We thank the reviewers for their reviews! We briefly highlight two points mentioned across more than one reviewer.
**Results on naturalistic datasets**
Our experiments on the naturalistic CelebA dataset are a significant contribution of our work, but we realize from the reviewers’ feedback that we should highlight them more prominently in the next revision. We will do so by using the additional space provided by reducing the size of Figure 1 and the additional page provided after the review process.
**Additional experiments**
We include 4 sets of additional experiments, covering the effects of gaussian blurs, horizontal flipping, varying the distortion budget, and varying the dropout rate for audio data. We find that these additional results further reinforce the main claims of the paper, and appreciate the reviewers’ suggestions to run them.

Meta-review:
role: PC
data: comment: value: This paper's scores were mixed.

- The authors argue in this work that label-destroying augmentations can be helpful for contrastive representation learning, by a dropout-like robustifying mechanism, complicating the simplistic viewpoint that good augmentations are those that preserve the label. Reviewers agreed that the hypothesis is crisp, and the empirical study is thorough and high-quality.
- One reviewer was concerned with the overlap between this work and the *feature suppression* phenomenon. The authors clarified contributions not present in prior work: (1) unifying prior accounts; (2) benefits of feature dropout for Viewmaker networks; (3) novel experiments (synthetic, audio, CelebA); (4) theoretical analysis.

Overall, after discussions, the paper surpasses the publication threshold, due to the former point.

Decision: Accept (poster)