Deep Augmentation: Dropout as Augmentation for Self-Supervised Learning
Authors: Rickard Brüel Gabrielsson
Tongzhou Wang
Manel Baradad
Justin Solomon
Venue: Accepted by TMLR ()

Abstract: Despite dropout’s ubiquity in machine learning, its effectiveness as a form of data augmentation remains under-explored. We address two key questions: (i) When is dropout effective as an augmentation strategy? (ii) Is dropout uniquely effective under these conditions? To explore these questions, we propose Deep Augmentation, a network- and modality-agnostic method that applies dropout or PCA transformations to targeted layers in neural networks. Through extensive experiments on contrastive learning tasks in NLP, computer vision, and graph learning, we find that uniformly applying dropout across layers does not consistently improve performance. Instead, dropout proves most beneficial in deeper layers and can be matched by alternative augmentations (e.g., PCA). We also show that a stop-gradient operation is critical for ensuring dropout functions effectively as an augmentation, and that performance trends invert when moving from contrastive tasks to supervised tasks. Our analysis suggests that Deep Augmentation helps mitigate inter-layer co-adaptation---a notable issue in self-supervised learning due to the absence of labeled data. Drawing on these insights, we outline a procedure for selecting the optimal augmentation layer and demonstrate that Deep Augmentation can outperform traditional input-level augmentations. This simple yet powerful approach can be seamlessly integrated into a wide range of architectures and modalities, yielding notable gains in both performance and generalization.

Published in Transactions on Machine Learning Research (05/2025)
Deep Augmentation: Dropout as Augmentation for Self-Supervised Learning
Rickard -abre--abrielsson Tongzhou Wang
Manel Baradad
Justin Solomon
Massachusetts Institute of Technology
(brg, tongzhou, mane maneabaradad, jsolomon| @mit. edu
Reviewed on http://ww...ttp.e//////////////t
Abstract
Despite dropout's ubiquity in machine learning, its effectiveness form of data augmentation remains under-explored. We address two key questions: (i) When is dropout effective as augmenta- tion strategy? (ii) Is dropout uniquely effective under these conditions? To explore these questions, we propose Deep Augmentation, a network- and modality-agnostic method that applies dropout or PCA transformations to targeted layers in neural networks. Through extensive experiments on contrastive learning tasks in NL., computer vision, and graph learning, we find that uniformly applying dropout across layers does not consistently improve performance. Instead, dropout proves most beneficial in deeper layers and can be matched by alternative augmentations (e.g., PCA). We also show that stop-gre coopggradeett operation is critical for ensuring dropout functions effectively as augmentation, and that performance trends invert when moving from contrastive tasks to supervised tasks. Our analysis suggests that Deep Augmentation helps mitigate inter-layer co-adaptation-a notable issue in self-supervised learning due to the absence of labeled data. Drawing on these insights, we outline a procedure for selecting the optimal augmentation layer and demonstrate that Deep Augmentation can outperform traditional input-level augmentations. This simple yet powerful approach can be seamlessly integrated into wide range of architectures and modalities, yielding notable gains in both performance and generalization.
1 Introduction
Self-super learning has emerged as powerful paradigm in machine learning, enabling the creation of representations and pre-trained models without reliance on aumman-annoatted labels. It has propelled breakthroughs in computer vision (Chen et al., 2020), natural language processing (Devlin et al., 2019), graph learning (7hu et al., 2021), speech processing (Oord et al., 2016), and genomics (Zaheer et al., 2020). Within this landscape, contrastive learning (Oord et al., 2018; Chen et al., 2020) has gained particular prominence by leveraging augmen- tations that generate complementary pairs of samples, thereby preserving semantic structure (Shorten & Khoshgoftaar, 2019) and effectively expanding the training set.
Ourput
NN
-laeers
NN
NN
L layers
Augment
NN
/layers
Augment
Data
Despite recent progress, effective augmentation strategies in contrastive learning often hinge on domain-specific knowledge- for instance, cropping and blurring in im- ages (Chen et al., 2020), and token masking in NLP (Gao al.,
Figure 1: Left: Traditional augmentation. Right:
Deep Augmentation at layer L,
2021). Designing such augmentations can be consuminn and may not generalize well across diverse modalities. This motivates the exploration of more universally applicable techniques.

---

Published in Transactions on Machine Learning Research (05/2025)
A simple yet under-explored direction is to treat dropout (Labach et al., 2019; Salchin & Kang, 2023) not solely regularization tool, but also as form of data augmentation (Srivastava et al., 2014). Dropout randomly zeros out activations, and its potential as an augmentation has been noted in self-supervised settings (Gao et al., 2021); however, the precise conditions under which it proves most effective remain unclear.
To address this gap, we introduce Deep Augmentation (Figure 1), a network- and modality-agnostic technique for augmenting high-dimensional activations in neural networks. Deep Augmentation applies dropout (Srivastava al., 2014) or principal component analysis (PCA) (F.R.S.. 1901) to specific layers, optionally combined with a stop- gradient operation. This design enables to pose and investigate two central questions:
1. When is dropout effective as an augmentation strategy?
2. Is dropout uniquely effective under these conditions?
Through extensive experiments, we find that applying dropout uniformly across all layers does not consistently improve contrastive learning performance; instead, targeting deeper layers yields substantial gains. Moreover, we show that stop-gradient operation is sometimes critical for harnessing the full potential of dropout-based augmentation, as it prevents detrimental gradient flow through the augmented representations. Notably, similar effect can be achieved using PCA- Abased augmentations, suggesting that dropout is not uniquely suited to these conditions.
Interestingly, the performance trends reverse under supervised learning, indicating that Deep Augmentation mitigates inter-layer co-adaptation-a challenge that arises in the absence of labeled data. Our approach is straightforward to integrate into standard architectures such as ResNets (He et al., 2016). Transformers (Vaswani et al., 2017), and Graph Neural Networks (Kipf Welling, 2017). does not depend on manually designed augmentations or labeled datasets, making it an appealing option for wide range of tasks.
Contributions. In summary, our main contributions are as follows:
We show that dropout, when selectively applied to deeper layers (with or without stop-gradient), can substantially improve performance in contrastive settings, while offering little benefit (or even proving detrimental) in supervised tasks.
We demonstrate that dropout is not uniquely effective: a simple PCA-based augmentation can yield comparable gains. We highlight the significance of a stop-gradient operation for stabilizing and enhancing the effects of layer-targeted dropout, ensuring consistent performance boosts.
We propose a practical procedure for selecting which layer to augment, showing that this choice is key to mitigating inter-layer co-adaptation.
We thoroughly validate our approach on Transformers, ResNets, and Graph Neural Networks across multiple data modalities, underscoring the broad applicability of Deep Augmentation.
2
Related Work
Self-Supervised Learning. Self-supervised learning (Chen et al., 2020; Grill al., 2020; Caron et al., 2020; Chen & He, 2021; Rani et al., 2023) leverages abundant unlabeled data to train transferable representations. By replacing human- annotated labels with automatically generated lseudo-labels, it enables training larger models on extensive datasets while mitigating overfitting. This paradigm has become increasingly popular for learning high-quality representations across diverse downstream tasks.
Data Augmentation. Data augmentation traditionally alleviates the scarcity of labeled data in supervised tasks, expand- ing the training set with semantically faithful transformations (Lecun et al., 1998). While contrastive and supervised learning often employ similar augmentations (Chen et al., 2020), label-dependent methods such as Mixup (Zhang al., 2018a) are unique supervised settings. In most self-supervised pipelines, augmentations focus the input space, which, although intuitive, may not exploit the most semantically rich features available in deeper layers.
Hierarchical and Higher-Layer Features. Neural networks inherently learn hierarchical representations, with early layers capturing low-level patterns (c.g., edges) and deeper layers capturing higher-level semantics (c.g., textures, objects) (Bilal al., 2018; Brüel Gabrielsson & Carlsson, 2019). Distances between higher-layer features can align with human judgments of semantic similarity (Zhang et al., 2018b; Ulyanov et al., 2018), underscoring their practical importance. Recent work has emphasized that capturing semantic granularity in deeper layers can notably enhance
2

---

Published in Transactions on Machine Learning Research (05/2025)
downstream performance (Zhou et al., 2022; Hinton, 2023). Consequently, augmentations at these deeper layers can promote invariance to more abstract transformations than those found at the input level.
Higher-Layer Augmentation. Several approaches directly modify hidden or latent spaces. Manifold Mixup (Verma et al., 2018) applies mixup to hidden-layer outputs, and other work interpolates features for image classification (DeVries & Taylor, 2017). MODALS (Cheung & Yeung, 2021) unifies these ideas within cunfforeement-eerrnng framework. Together, these studies highlight the potential of deeper-layer augmentation as complement-or even alternative-to input-level transformations.
Dropout as Augmentation. Dropout (Labach et al., 2019; Salehin & Kang, 2023) is typically viewed as regularizer that randomly zeroes out activations (Bouthillier et al., 2015), yet it can also serve as a general-purpose data aug- mentation (Srivastava et al., 2014). Recent work has explored this perspective in self-supervised contexts (Gao et al., 2021), but often by uniformly applying dropout across all layers. In contrast, we show that selectively targeting deeper layers can yield significant gains in contrastive learning, thereby challenging the one-size-fits-al assumption underlying uniform dropout usage. Our findings extend prior efforts that examine dropout's 's role in specialized settings (Wu & Gu, 2015), demonstrating its nuanced effects.
Stop-Gradient & Information Collapse. goopggradient mechanisms have been explored in Siamese networks (Chen & He, 2021), where they help avoid trivial collapses (Jing et al., 2022) and lessen the reliance on large batches or negative samples. In SimCLR (Chen et al., 2020), applying stop-gradient to one side of a positive pair slightly degraded performance (Chen & Hc, 2021), indicating that the effect may be context-specific. We broaden this investigation by introducing stop-gratt gradient at different layers in conjunction with dropout- and PCA-based augmentations across diverse domains (including supervised tasks). From an information-theoneti perspective (Tian al., 2020; Shwartz-Ziv & LeCun, 2023; Tishby & Zaslavsky, 2015), our layer-targeter augmentation appears to reduce unwanted uniformity co-adaptation in latent features- an effect that can markedly improve contrastive learning performance while sometimes inhibiting performance on supervised tasks.
Analysis of Representation Learning. Prior theoretical works provide benchmarks for evaluating representation quality. For instance, Wang & Isola (2020) propose two criteria for contrastive-leaming representations: alignment of positive pairs and uniformity on the hypersphere. Kornblith et al. (2019) introduce a similarity index (equivalent to centered kernel alignment, CKA) for comparing representational similarity across neural networks. We draw on these and similar methods to analyze how our Deep Augmentation strategy shapes the internal representations.
3 Method
3.1
Preliminaries
Contrastive learning seeks to learn representations by drawing semantically similar pairs closer while pushing dissimilar pairs apart. Given dataset X {II,... .NN J. it forms pairs D {=)) where 2 and 2 are distinct views of the same underlying sample z; E X, yet semantically similar. Constructing such pairs is pivotal, as it defines the invariances captured by the learned representations. Commonly, pairs are generated by applying random transformations (e.g., cropping, flipping, distortion) to the same sample.
Formally, let Z ~ y be random variable, where yı is a probability distribution over some space Ω. (For instance, Ω could be discrete-c.g. cropping size-or continuous-c-2 blurring variance.) Let A Rd Ω - Ed be an augmentation function, and let B X be randomly drawn batch. For each sample n, B, draw 2 2 p. The features of the augmented pairs are defined as
I = fo(A(z,)))) +())
for E {1,2),
where fo is a neural network with learnable parameters A. The InfoNCE loss (Chen et al., 2020) for batch B is then:
B
cosine-sim (h:. Att/TT
1(0: B) = B log
(1)
B 2
cosine-sim(h sim) (13)/
where is a temperature parameter. This loss encourages fo to be invariant to the augmentation A and promotes uniform distribution of normalized features (Wang& & Isola, 2020).
3

---

Published in Transactions on Machine Learning Research (05/2025)
3.2 Deep Augmentation
A neural network So processes data by successively applying L layers. For -1 S S b < L, let ra denote the operation from layer a to layer b, where -1 represents the input itself, and fø is the identity. In particular, So 1, for any - -1 L. For example, applying an input augmentation corresponds to:
So --
(((A(1)))))
(A (A(f)-t-(x))), 2))
In this work, we investigate:
(z 2)) = f 1 (xz))
(2)
where 1 S 1 < L, as illustrated in Figure 1. Throughout, our primary augmentation A is dropout, but we also explore alternative PCA-inspired augmentation (Section 5). For dropout, z corresponds to random mask that zeros specific percentage of activations in Iú, meaning that of is the source stochasticity in the augmentation. Ideally, A should satisfy three properties:
1. Layer-agnostic-the same A applies at any layer / without modification.
2. Network-agnosticc-A does not depend specific architecture for fe.
3. aoodalit-agnostiic- A does not depend on the input domain.
Both dropout and our PCA-based approach satisfy these criteria.
We aim to identify which value(s) of yield the best representation dg. as measured by downstream performance. When combining Deep Augmentation with standard input-level transformations, we simply compose the two augmentations (i.e., first apply the input-space augmentation, then apply the in-network augmentation).
PCA Augmentation. To test whether dropout is uniquely effective in Deep Augmentation, we compare it with penucipa--cmppoeed removal variant. Given a bati-batchhaa Io- = {l,..... .......,,, K}. define the mean R EkEt, 2ke the centered samples 22 = 2g μl, and the stacked matrix X = [ø1, Øx] E Rd K and compute the singular- value decomposition X = UEV Denote by Vp E Rdxx the first right singular vectors (the top PCs). Each sample is augmented as
Ap(xa)
I (I-V
(a=
1D) + μ.
project away from PCs
Thus we subtract the components of z; that lie in the subspace spanned by the batch's 's top p principal directions. using the batch itself (not an external noise vector) to generate the perturbation. We refer to this augmentation as PCA in subsequent sections.
Stop-Gradient. equation 2, once > 1,, learnable layers exist before the augmentation. We optionally apply a stop-gradient operation at layer I, which prevents gradients from flowing into these earlier layers (Chen & He, 2021). This design enables us distinguish two regimes: one where the network learns invariance only the already applied augmentations (when using stop-gradient), and one where it additionally learns invariance to upcoming augmentations (when not using stop-gradient). In effect, by cutting off the gradient at the augmentation point, we prevent the upstream layers from "learning to undo" the perturbation, thereby preserving the desired invariances.
Partial Batch Sampling. By default, we apply Deep Augmentation to random 50% of each mini-batch. This enhances variation in the training process and ensures that some samples remain unaugmented, maintaining alignment with evaluation conditions. Moreover, this approach preserves learning in all layers even if some samples experience stop-gradient (it only applies to half of the batch). See Appendix A.5 for ablation study.
Co-Adaptation. We define co-adaptation between layers as scenario where two layers capture essentially the same information. To quantify this, we measure activation similarity using the centered kernel alignment (CKA) index (Korbblihh et al., 2019). A high CKA value indicates that the two layers are effectively redundant. Since deterministic transformations cannot create new information (Shwartz-Ziv & LeCun, 2023), strong co-adaptation implies that minimal additional processing occurs in the deeper layer. Conversely, weaker so-adaptation suggests more effective information filtering, which can foster better generalization.
4

---

Published in Transactions on Machine Learning Research (05/2025)
4 Main Results
We show that Deep Augmentation consistently and substantially improves contrastive learning performance across vision (Table 2), natural language processing (Table 1), and graph-based learning (Table 3). These gains primarily stem from the targeted use of dropout and sop-ggaadient they need not reflect the absolute best performance achievable under every possible Deep Augmentation hyperparameter.
Sentence Embeddings. We follow the approach of Gao et al. (2021), pre-training a BERT transformer (Devlin et al., 2019) on 10 randomly sampled sentences from English Wikipedia. Hyperparameters are tuned on the STS-B development set (Cer al., 2017), and final evaluations are conducted on seven standard semantic textual similarity (STS) tasks (Agirre et al., 2012; Cer al., 2017; Marelli al., 2014).
Vision. For images, we employ ResNet (He al., 2016) and follow the SimCLR framework (Chen et al., 2020), testing on CIFARIO, CIFARI00, and 100-class subset of ImageNet (Deng et al., 2009). Again, we target deeper layers for dropout and stop-gradient, leading to measurable improvements over standard SimCLR.
Graph Contrastive Learning. In graph-based tasks, we adopt the GCL framework (Zhu et al., 2021) with GCN backbone (Kipf Welling, 2017). We evaluate on COLLAB and IMBD-Multi (Yanardag Vishwanathan, 2015), well as NCI1 (Wale & Karypis, 2006) and PROTEINS (Borgwardt et al., 2005). Hyperparameters tuned on validation split, with results reported on separate test set. Table shows that applying Deep Augmentation in deeper layers benefits performance across most datasets.
Compute & Memory Savings. Stop-gradient can reduce both training time and memory usage. In our setup, the portion of the network cut off by stop-gradient no longer computes gradients, saving roughly 4x the compute time and 3 the memory for the affected layers. Most these savings are realized on the GPU. By selectively applying ttop-gradient deeper layers and to only half of each batch, overall training time drops to about 62.5% of the baseline, while memory usage is trimmed about 66%. Tables 1, 2, and provide approximate "Compute" metrics reflecting these savings.
Table 1: Contrastive Learning on Sentence Embeddings with Transformer. Performance on STS tasks (Spearman" 's correlation, where higher is better). SimCSE versus SimCSE with Deep Augmentation, specifically sayer-targeted dropout and stop-gradient at layer 8. Compute refers to the estimated use of compute time and memory, as compared to SiMCSE
Model
STS12
STS13
STS14
STS15
STS16
STS-B
SICK-R
Avg. 75.61
Compute 100%
SimCSE
81.18 66..7110.505
18.1321.818 188
80.82 585
78.47:2005
77-5420.00
71.49
SinCSSEDeepAug
69.00=1.111
81.82+0.127
81.84:0.429 44.48=
78.41=0.146
78.63-0.114
71.75
76.56=0.16
~79
Table 2: Contrastive Learning in Vision with ResNets. SimCLR versus SimCLR with Deep Augmentation, specifically layer-targeted dropout and stop-gradient layer 4, across all datasets. Values represent classification accuracies (higher is better). Compute refers to the estimated use of compute time and memory, compared to SimCLR.
Model
CIFARIO
CIFARI00
ImageNet100
Compute
SimCLR
90.37
61.64
79.38
100%
SinCLR+DeepAug
91.04
64.01
79.56
~66%
Table 3: Contrastive Learning on Graphs with GNNs, GCL (Graph Contrastive Learning) versus GCL with Deep Augmentation, specifically layer-targeted dropout and stop- noo-ggradeent at layer 6, across all datasets. Values represent classification accuracies (higher is better) measured flmi (Micro-averaged F1 Score). Compute refers the estimated use of compute time and memory, compared GCL.
Model
COLLAB
IMDB-Multi
NCII
PROTEINS
Compute 100%
GCL
72.40+0.6
53.33-1.1
73.97
72.32+1.5
GCL+DeepAug
73.80
52.89
75.83
73.21
~66%
5

---

Published in Transactions on Machine Learning Research (05/2025)
5 Ablations
Deep Augmentation introduces additional hyperparameters (c.g., the targeted layer, augmentation type, stop-gradient usage). In this section, we present ablation studies demonstrating its consistent performance across different datasets. We analyze the impact of (i) contrastive supervised settings, (ii) layer depth, (iii) stop- hop-gradient usage, (iv) dropout vs. PCA augmentations, and (v) pre-trained initialization.
5.1 Sentence Embeddings
SimCSE (Gao al., 2021) first showed that using only dropout can improve contrastive learning for sentence embeddings (built on pre-trained MLM model). We extend SimCSE's setup by applying layer-specific dropout PCA, with without stop- stop-gradicns gradient, both with and without additional MLM augmentations.
Augmentation, Layer, and Stop-Gradient. Figures 2a and 2b illustrate our results using (a) dropout and (b) PCA-based transformations on the STS-B development set. For PCA, we tested removing both the largest and sixth-largest principal component; Figure 2b reports the largest component, which worked best. The ixthh-componet results appear in the Appendix.
Overall, Deep Augmentation surpasses plain SimCSE across range of layer depths. Adding stop-gradient further boosts performance with dropout, though the gains are smaller for PCA.
0.84
SimCSE
Spearman on STS- B Development Set
0.84
SimCSE
Spearman on STS-B Development
Stop
Stup w/o Stop
w/o Stop
0.83
0.83
0.82
0.82
0.81
0.81
0
1
2
3
5
10
12
0
2
Layer
Layer
Dropout
(b) PCA
Figure 2: SimCSE vs. Deep Augmentation with (a) Dropout or (b) PCA, with and without stop-gradient. "Stop": stop-gradient. Deep Augmentation outperforms SimCSE.
Deep Augmentation and Masked Language Modeling (MLM). Figure 3 compares Deep Augmentation to MLL's original data augmentations in SimCSE. Although SimCSE already tunes dropout rates (0%, 1%, 5%, 10%. 15%, 20%), our fixed 50% dropout rate at a chosen layer still yields higher results, underscoring the robustness of layer- targeted augmentation alongside MLMM-his reduces dependece on pre-trained models and development sets, by supporting simultaneous Deep Augmentation and MLM training. Across standard STS tasks, the best Deep Augmentation setup exceeds SimCSE's strongest Spearman correlation (e.g.. 74.32 vs. 69.31). Moreover, even in purely MLM setting (i.e., contrastive objective), Deep Augmentation significantly improves performance (Appendix, Figure 51). and there the effect of stop-gradient is less pronounced.
Supervised Learning. To contrast Deep Augmentation's impact in supervised vs. contrastive learning, we train on STS-B directly as a supervised task. Figures 4a and 4b show results with dropout and PCA, respectively, with/without stop-gradient. Contrary to our contrastive findings, Deep Augmentation reduces performance in the supervised setting-especially in higher layers. This stark difference supports our hypothesis that leeper-layer augmentations help most when explicit supervision is provided, forcing the model learn robust invariances its own.

---

Published in Transactions on Machine Learning Research (05/2025)
0.8
Spearan on STS-B Develogment
MLM SimCSE
0.55
Stop
0.5
0.45
SimCSE
0
2
91 10 11 12
Layer
Figure 3: SimCSE Deep Augmentation with and without stop-gradient, both with MLM. "Stop": stop-gradient. *; includes hyperparameter search over dropout rates.
0.65
0.65
Spcarmee STS-B Developueus
0.8
Spearman on Development
0.6
0.55
0.55
0.5
0.5
0.45
0.45
0.4
0.4
0.35
0.35
0.3
Regular
0.25
Regular
Stop
Stop
0.25
wio Stop
0.15
w/o Stop
0.2
10 11
10 11 12
Layer
Layer
(a) Dropout
(b) PCA
Figure 4: Supervised only. Deep Augmentation with (a) Dropout (best performing across dropout rates .5, 22.5,,1125
and (b) PCA, with and witout stop-gradient, on STS-B.
5.2 Vision
For vision, our primary experiments CIFAR-100 (Appendix reports CIFAR-10). ImageNet1 00 results follow the same layer-targeted dropout and ttop-gradient configuration determined optimal for CIFAR.
Architecture. We adopt ResNet18 (Appendix A.1, Table 4). Compared to Transformers and GNNs, ResNet has less regularity across depths: ResNet contains convolutional layers upfront, followed by fully connected layer, with average pooling interspersed. This, in addition to the varying dimensionality across layers, makes vision-specific hyperparameters more challenging.
Dropout is Not Sufficient Alone. Unlike for sentence embeddings, using only dropout as the augmentation did not yield competitive results in vision. Consequently, complement deep augmentations with standard image augmentations. Future work could investigate adapting pre-trained vision models to new domains via dropout alone (similar to SimCSE in NLP).
Data Augmentation and Targeted Dropout. Figure 14, in Appendix, show that uniform dropout across all layers degrades contrastive learning, but layer-specific dropout can mitigate these losses. Notably, 50% dropout uniformly applied is detrimental, whereas targeting the same 50% rate certain layers has much less impact on performance.
Augmentation, Layer, and Stop-Gradient. Now apply Deep Augmentation with dropout (Figure 5a), with and without stop-gradient. Deep Augmentation with dropout and stop-gradient demonstrate significant performance improvements, particularly for layers 4 and 6; however, not using the stop-gradient did not achieve performance comparable to using stop-gradient. A small tuning of dropout rate yielded the results in Table 2 (Appendix A). We also evaluate Deep Augmentation with PCA augmentation, removing the largest and the sixth largest principal component. Removing the sixth largest yields superior performance (results with and without stop- coodggradient in Figure 5b). For the largest, see the Appendix. Similar dropout, stop-gree gradient consistently enhances performance, especially higher layers.
7

---

Published in Transactions on Machine Learning Research (05/2025)
65
65
Downstream Validation Accuracy
64
Downstream Validation Accuracy
64
63
63
62
62
61
61
60
SimCLR
SimCLR*
59
Stop
59
SimCLR
Stop*
Stop
58
w/o Stop
w/o Stop
57
0
2
4
5
2
5
6
Layer
Layer
(a) Dropout
(b) PCA
Figure 5: Contrastive learning. Deep Augmentation with (a) dropout or (b) PCA, with and without stop-gradient. *:
initialized with pre-trained SimCLR model. "Stop" short for sop-ggaddeet. Note: Layer is average pooling.
73
Downstrean Validation Accuracy
Downireum Validationn Accuracy
72
71
70
70
69
No Dropout
Stop
69
No Dropout
Stop
Stop
who Stop
2
68
Layer
Layer
(a) Dropout
(b) PCA
Figure 6: Supervised only. Deep Augmentation with (a) dropout or (b) PCA, with and without ttop-gradient. *: initialized with pre-trained SimCLR model. "Stop" short for soo-ggrddiett.
Initialization Freezing Weights. Augmentations may have more significant impact on higher layers that already possess useful, discriminative features. In addition, the concurrent objectives of learning features and maintaining invariance to their alterations could conflict, slowing down destabilizing training. In Figure 5a, 'SimCLR** and 'Stop** apply Deep Augmentation atop SimCLR-pretrainee, ResNet. While using pre-trained initialization outperforms not using pre-trained initialization, the gains are marginal, indicating that extensive pre-tres pre-training is not essential for Deep Augmentation. Similarly, freezing specific layers before or after the augmentation point does not yield notable benefits (Appendix A.7).
Supervised Learning. Figures 6a and 6b compare Deep Augmentation's 's effect in supervised vision tasks. Here, dropout PCA does not provide benefits. Notably, omitting stop-gradient actually performs better-opposite our contrastive results. Retaining basic data augmentations remains crucial; removing them lowers accuracy to around 59.02%.
5.3 Graphs
Finally, we evaluate Deep Augmentation on graph contrastive learning, extending our insights from vision and NLP to GNNs. We use standard Graph Contrastive Learning (GCL) augmentations (Zhu al., 2021) on COLLAB. IMDB- Multi, NCI1, and PROTEINS. In GCL, graph is augmented-une using operations such as node and edge deletion-
8

---

Published in Transactions on Machine Learning Research (05/2025)
create two distinct views, and the model is trained identify pairs of views originating from the graph. We use the mi (Micro-averaged F1 Score) metric; further details can be found in Appendix C.
Augmentation, Layer, and Stop-Gradient. Figures 7 and 8 show results for dropout and PCA (removing the sixth- largest principal component). Since graph inputs vary in embedding size per number nodes, we adapt PCA to operate over across node embeddings in the batch. While trends vary somewhat by dataset, Deep Augmentation with dropout and stop-gradient significantly boosts performance in most cases (especially higher layers).
0.73
6.75
Dovittran Ciee
11:00
Deomttran
0.79
0.54
0.33
OCL Stop
0.72
OCL Stop
Iivolo Suyp
0.7
0.32
Tager
Lager
Layer
Layer
(a) COLLAB
(b) IMDB-Mult
(c) NCTI
(d) PROTEINS
Figure 7: Graphs: Deep Augmentation with Dropout
0.74
Divttroue Euea
0.75
Doverren
0.7
Stvp Stop
wib wibaiep
0.72
0.00
6.5
0:73
Layer
Layos
(a) COLLAB
(b) IMDB-Multi
(c) NCII
(d) PROTEINS
Figure 8: Graphs: Deep Augmentation with PCA
5.4 Findings
In summary, our ablation studies indicate:
Contrastive vs. Supervised Learning: Deep Augmentation generally has opposite effects on self-supervised versus supervised learning. See Figure 9. is surprising that Deep Augmentation reduces performance in the supervised setting, particularly in higher layers, given that dropout is commonly effective in supervised learning. likely explanation is that our Deep Augmentation setting departs from conventional dropout: confine the mask to single layer, and adjust other hyperparameters. Accordingly, we emphasize the direction of the effect rather than the absolute numbers.
larger dropout rates,
Layer Depth: Applying augmentation to higher layers typically yields the largest gains in contrastive learning. Stop-Graud Gradient:: Improves contrastive performance across diverse data but often reduces accuracy in supervised tasks. Augmentation Type: Both dropout and PCA are effective, though they exhibit different behaviors and trade-offs.
Pre-trained Weights: Starting from pre-trained models is not essential for successful dropout- tropout-based augmentation.
6 Analysis
Deep Augmentation demonstrates strikingly different effects in contrastive and supervised learning. This section explores why Deep Augmentation benefits contrastive learning while offering little or no gains in supervised contexts, and how we can determine which layers to target for best results. We integrate alignment-uninn uniformitty metrics (Wang & Isola, 2020) and CKA similarity (Korbblit al., 2019) to show how these transformations reshape latent representations and mitigate co-adaptation among layers.

---

Published in Transactions on Machine Learning Research (05/2025)
Stop, Self-Supervised
Acjusted Performance Difference
(performance mance baseline) baselllsn
W/o Stop, Self-Supervised
Stop, Supervised
Stop, Supervised
-03
30
04
366
1.0 1.0
Layer Depth
Figure 9: Regression lines are fitted experimental results been 2-score normalized (with the original mean preserved). In the self-supervised setting, deep augmentation improves performance in higher layers when using stopggradeen, while the opposite trend is observed without stop- noo-ggradeett. In contrast, for supervised learning, deep augmentation generally does not yield benefits, and the trends across layers are reversed.
NLP, vision, and graph-based tasks. All data have
Contrastive Learning. Our findings indicate that Deep Augmentation helps reduce overfitting and eliminate spurious alignment in contrastive tasks, while also maintaining or enhancing uniformity across learned features. This is evident from alignment-unifon
analyses, which show improved invariance properties, and from CKA measures, which confirm that stronger feature transformations in targeted layers reduce inter-layer similarity (co-adaptation). Importantly, these techniques also identify which layers are most susceptible to co-adaptation, guiding the strategic selection of layers where Deep Augmentation is most beneficial.
Supervised Learning. In stark contrast, supervised learning sees little benefit-and can even suffer-when Deep Augmentation is applied. Because labeled tasks already specify ground-truth invariances (the intra-class differences), they do not rely broad augmentations to combat spurious alignments. In effect, the network is already "regularized" by the labels themselves, leading to naturally lower co-adaptation in deeper layers. Consequently, additional perturbations introduced by Deep Augmentation can degrade performance. This aligns with studies on information bottlenecks (Tishby & Zaslavsky, 2015; Shwartz-Zii Ziv & LeCun, 2023; Jing et al., 2022), which highlight how supervised training inherently enforces more constrained representation space compared to contrastive methods optimizing mutual information across potentially infinite augmentations.
Taken together, these findings illuminate how Deep Augmentation simultaneously combats overfitting and enforces useful invariances in contrastive tasks, exerts little positive influence-ant sometimes a negative one one-when the target invariances are already determined by label supervision.
6.1 Co-Adaptation Between Layers
Sentence Embeddings and Transformers. Figure 10 shows CKA similarity for a Transformer model under different conditions:
BERT: The pre- pre-trained baseline for SimCSE and Deep Augmentation.
SimCSE: Contrastive learning applied to BERT, notably reducing co-adaptation across its higher layers (i.e., 10-12). Layer 10 (w/o Stop): Deep Augmentation applied at Layer 10 (highlighted by red cross) without stop-gradient.
Layer (w/ Stop): Deep Augmentation with stop-gradient at Layer 8 (highlighted by red cross).
In BERT, layers 8-11 show a stretch of co-adaptation (between the black crosses). Deep Augmentation at or near these points reduces similarity in subsequent layers, overcoming the co-adaptation that persists even after SimCSE. Interestingly, choosing the earlier "cross" for stop- -ogggrraient and the later "cross" for without stop-gradien perform the best. This suggests potential extension: targeting multiple layers could further boost performance. Overall, CKA
10

---

Published in Transactions on Machine Learning Research (05/2025)
similarity index highlights co-adaptation issue between layers and determines what layer Deep Augmentation should be applied.
Appendix Figure 59 shows that, in a supervised setting, co-adaptation is already weaker even without Deep Augmen- tation, emphasizing that ground-truth labels inherently curb excessive vo-adaptation. Further details are provided in Appendix
BERT
SimCSE
Layer 10 without Stop
Layer 8 with Stop
14
Figure 10: CKA similarity index for "BERT." "SimCSE." "Layer 10 without Stop" (Deep Augmentation applied without stop-gradient Layer 10), and "Layer with Stop" (Deep Augmentation applied with stop-gradient Layer 8) the STS-B dataset. Black crosses mark the beginning and end of co-adaptation region in BERT, while red crosses "Layer 10 without Stop" and "Layer 8 with Stop" highlight the targets of Deep Augmentation. Optimal performance of Deep Augmentation is observed near the black crosses, indicating its effectiveness and guiding the selection of layers for targeting.
Images and ResNet. Figure 11 compares CKA similarity for:
Random: A randomly initialized ResNet18.
SimCLR: ResNet18 trained with SimCLR,
Layer without Stop: ResNet18 with Deep Augmentation at Layer (w/o Stop).
We find that co-adaptation between Layers and emerges after SimCLR training and is even stronger in the poorer- performing "Layer (w/o Stop)." Conversely, the pessforrrnnnnn models (Layer or Layer with stop-gradient) avoid this excessive similarity. This corroborates the Transformer and sentence embeddings findings: certain layers (e.g., Layer 4 in ResNet18) are especially prone to co-adaptation, pinpointing where Deep Augmentation can be most beneficial. See Appendix E for more details.
Appendix Figure 56 shows a similar pattern to Transformer and sentence embeddings in supervised training, with consistently lower co-adaptation than in self-supervised settings (particularly the last layer) further evidence that labeled tasks intrinsically limit over-adaptation between layers.
Random
SimCLR
Layer 4 without Stop
04
Figure 11: Indications of why Layer 4 is special in Figure 5a, the major divide between co-adaptation across layers. Layers 0)4 are convolutional. All nigh-performing NNs have the pattern of SimCLR, and failure cases have stronger co-adaptation between Layers 4 and 5. "Layer 4 without Stop" corresponds to the failure case of Deep Augmentation without stop-gradient at Layer
Sentence Embeddings and Transformer. Alignment and Uniformity measures for sentence embedding methods are in Figure 12, computed as SimCSE (Gao et al., 2021) w.r.t. ground truth (STS-B development set), during training, and with methods converging to higher density regions.
11

---

Published in Transactions on Machine Learning Research (05/2025)
Following Gao et al. (2021), we measure alignment and uniformity the STS-B development set (Figure 12), during training, and with methods converging to higher density regions. Without ML.,, Deep Augmentation trends toward better uniformity (lower is better) than SimCSE alone. Adding stop-gradient further enhances alignment at some expense of uniformity ("S" vs. "w/o S")) However, introducing MLM reverses the optimization direction, boosting alignment at some cost to uniformity. Again, the Deep Augmentation variants (with or without stop-gradient) tend to outperform the SimCSE baselines.
In Appendix Table E, compare alignment and uniformity in supervised learning. Deep Augmentation slightly improves alignment but not uniformity, consistent with the notion that supervised models already have high uniformity ground-truth labels, leaving less room for improvement.
Note, since these measures are computed on ground truth validation classes rather than augmentations, they offer a different perspective from the original introduction these measures for contrastive learning Wang & Isola (2020).
0.55
SimCSE S w/o SiMCSE+MLM S+MLM
0.5
Alignment
O
0.45
w/o S+MLM
D0
0.4
0.35
224
223
-22
221
119
Uniformity
Figure 12: Alignment and Uniformity (lower is better) for sentence embeddings on STS-B: SimCSE vs. Deep Augmentation (with and without stop nop-grradeent.. We also include these methods combined with the pre-training method of BERT, i.e., Masked Language Modeling (MLM). Arrows indicate the direction during training, which reverses when MLM is introduced. "S" is short for stop-gradin nop-gradient.
Images and ResNet. For vision, we standardize on SimCLR augmentations to measure alignment and uniformity (Figure 13). Checking multiple training checkpoints (e.g., epochs 300, 600, 900, 1200, 1500), observe that:
Higher training epochs improve uniformity test data and alignment on training data.
Layers 4 and 6 (w/ Stop) consistently outperform SimCLR on test alignment/uniforiity. avoiding the overfitting to aligmment that SimCLR shows on training data.
Layer 4 (w/o Stop) matches SimCLR on these metrics but underperforms on downstream tasks, highlighting these metrics' limitations in capturing latent space quality. Hence, our preference for the CKA similarity index for comprehensive assessment.
Appendix Figure 54 shows that, in supervised training, the differences are less pronounced. However, Layer 4 (w/ Stop) perform better on both metrics but this does not translate better classification accuracy. This highlights that the invariances in self supervuper learning and supervised learning remain fundamentally different, and superior performance on the self-supervised task does not necessarily translate to improved performance on the supervised downstream task. Avoiding overfitting to alignment crucial when that alignment differs from the ground truth, but less critical when alignment matches the ground truth. Again, tasks with ground-truth labels do not see the same overfitting challenges as contrastive setups, reducing the need for strong regularization from Deep Augmentation.
7 Discussion
Our work affirms that dropout, widely known as a regularizer, can indeed serve as an effective in-network augmentation method particularry within self-supervised pipelines where the risk of inter-layer co-adaptation runs high. The success of PCA-inspired augmentations further underscores that dropout is but avenue for improving contrastive training in
12

---

Published in Transactions on Machine Learning Research (05/2025)
training
0.25
training
Aligmment
0.28
CA
Alignment
0.2
2
0.15
O SimCLR
SimCLR
0.24
L4
O
L4 w/
à
L4 w/o
0.1
1,5
w/S
0.22
L6 w/
L6w/ w/S
-2.77 -2.76 22.7 -274
2.773
-2.2
-2.77
-2.76
-2.75
-2.74
-2.73
-2.72
Uniformity
Uniformity
Figure 13: Alignment and Uniformity (lower is better) of SimCLR augmentations CIFAR. Left: Test data. Right: Training data. Deep Augmentation outperforms SimCLR when measuring alignment and uniformity using SimCLR's s augmentations on the test set, and SimCLR overfits at Alignment on the training set. "L" is short for Layer and "S" is short for gradggradent..
deep networks. By providing a procedure for layer selection and underscoring the importance of soop-grddeet, we hope this work encourages more granular thinking about dropout-likk augmentations in deep learning, inspiring new ways to exploit activation-space transformations for both performance and generalization gains.
8 Acknowledgements
The MIT Geometric Data Processing Group acknowledges generous support of Army Research Office grants W91 IN2010016 and W91 INF2110295, National Science Foundation grant IIS2335492, from the CSAIL Future of Data program, from the MIT-IBM Watson AI Laboratory, from the Wistron Corporation, and from the ToyALa-CSAIL Joint Research Center.
References
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. SemEval-2012 task 6: pilot on semantic textual similarity. In *SEM 2012: The First Joint Conference Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https::tp//.... aac antaanttohoooooooo org/ 10112-115111
Alsallakh Bilal, Amin Jourabloo, Mao Ye, Xiaoming Liu, and Liu Ren. Do convolutional neural networks learn class hierarchy? IEEE Transactions on Visualization and Computer Graphics, 24(1):152-1622 jan 2018. doi: 10.1 1,9/ttcc (t199//vc.201..774444444 2017.2744683. URL htthttss: http:://ooiooooo 10. w...11982Ftteee 2017. 2017.2744 77777744668.
Karsten M. Borgwardt, Cheng Soon Ong, Stefan Schönauer, S. V. N. Vishwanathan, Alex J. Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, (suppl 1):447-i56, 06 2005. ISSN 1367-4803. doi: 10. 1001000/hoonffrmaatishiiiiiiiiii
URL https: /tw.///oii.//
Xavier Bouthillier, Kishore Konda, Pascal Vincent, and Roland Memisevic. Dropout data augmentation, 2015. URL
https: /ttp::///xxxvvvvv org www.orgggaaas///ooo 08700.
Rickard Brüel Gabrielsson and Gunnar Carlsson. Exposition and interpretation of the topology of neural networks. In 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pp. 1069-1076, 2019. doi: 10. 1 10.1109/CMLA.2019.000
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan,
13

---

Published in Transactions on Machine Learning Research (05/2025)
and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9912-9924. Curran As- sociates, Inc., 2020. URL https: //poceee///////eeeeeeeeee ngs neurips cttpppccpppeeeeea1111 lateeer_fllssseee wttpfffeeeeeeee//000000 fflle/ 704e60226916602288711aa2222222 wtt.:eaabbeeeeeeeeape..e..e Paper.
Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the 11th International Workshop on Semantic Evaluation (SemFxvll-2017), pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. 18855//1/SS7--20011
URL https: htttt:t://caaaatttoooo httpa//aaaannooyaaooeaa/.. wwg/orog//////// werg//17-2000 702001.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference Machine Learning. ICML'20.
JMLR.org, 2020.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In 202/ IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021. doi: 10. 00.1000/CCPPR444372222...055.3.0
Tsz-Him Cheung and Dit-Yan Yeung. (MODALS): Modality-agnostic automated data augmentation in the latent space. In International Conference on Learning Representations, 2021. URL https: http:///ppeeeeeeeeeeeeeeeeeeeeee net/ forum?
Ld=XjYgR6gbCES.
Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pullern Recognition, pp. 248-255, 2009. doi:
Jacob Devlin, Ming-Wor Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19- 1423. URL https:
https:///ccaattti http://aaaaaahtltaeeoee org/N1 14233.
Terrance DeVries and Graham W. Taylor. Dataset augmentation feature space, 2017. URL https::ttaaaata
org/ abs 11000/1002000 05538.
Karl Pearson F.R.S. Liii on lines and planes of closest fit to systems of points space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559-577 572, 1901. doi: 10.1080114884440094667777
Tianyu Gao, Xingeheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18663/v1/2021.. http:///2221eeemmmmeee
URL ht /ac///clantttt http:////aaaaatthogoyy/ h
org/ 2021. emnlp-main. 552.
nan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukeuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach self-supervised learning. In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284 Curran Associates, Inc., 2020. URL https: http:::::///ccpeppeeeeeee paper /2apr/2020// file/ htte:/eeeee/aaaee55eeeeeeee 42tee44eeee044111111 2e111144117111911
www.iiinnnnnnnninnieeeeeeeeeeeee /c/cc///ppe/eeeeeee files/ .ddf.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016. doi: 10.1109/CVPR.2016..0
Geoffrey Hinton. How to Represent Part-Whole Hierarchies in a Neural Network. Neural Computation, 35(3):413- 452, 02 2023. ISSN 0899-7667, doi: 10.1 162/neco a 01557. URL bttpp::///o.. www.:/o..........oooo..o.. 1ttp//...1.///ee...... 01557.
Li Jing, Pascal Vincent, Yann LeCun, and Yuandong Tian. supervised learning, 2022.
Understanding dimensional collapse in contrastive self-
14

---

Published in Transactions on Machine Learning Research (05/2025)
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning, 2020. URL
wttp://ww.w.ra///////////// 113622.
Thomas N. Kipf and Max Welling. Semi-super classification with graph convolutional networks. In International Conference on Learning Representations, 2017. URL https::////. /owpereveeeeee..ee///ee/.e..e..e net. fttt/forum?? /tt////mmmmmmss/4aargga
d=JJU44yyg1.
Simon Korbllihh, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019. URL httppps//aaxxii.oee wwtpaawwwww....///aa//a.aaaa///e /ttp://abb/115.00011111
Alex Labach, Hojjat Salchinejad, and Shahrokh Valaee. Survey of dropout methods for deep neural networks, 2019.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11)::278-232 1998. doi: 10.1199/5.72671.
Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'I4), pp. 216-223, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). URL http: http://wwwwww www. -rec- oreee..rg//pp.. pttp//eccceeen/////// 1rec2014/ // pdff 363 Bare__aaper.__ .dd..
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio, 2016. URL http:
/ //aaavvvorg//abaaee
609 03499. cite 109.v916...03499.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2018.
wttp://..rraaab//////aa//// /
URL httpt:////aaae
03748.
Learning: Veenu Rani, Syed Tufael Nabi, Munish Kumar, Ajay Mittal, and Krishan Kumar. Self-supervised Learpin A Succinet
Review. Arch. Comput. Methods Eng. pp. 1-15, January 2023. ISSN 1886-1784. doi:
Imrus Salchin and Dae-Ki Kang. review on dropout regularization approaches for deep neural networks within the scholarly domain. Electronics, 12(14), 2023. ISSN 2079-9292. doi: 10.3390eeeettrmnice11444400 URL
https: //t:://wwwww
Connor Shorten and Taghi M. Khoshgoftaar. survey on Image Data Augmentation for Deep Learning. J. Big Data, (1):1-48, December 2019. ISSN 2196-1115. doi: 1 /6////0533700199019777
Ravid Shwartz-Ziv and Yann LeCun. To compress or not to compress- self-supervised learning and information theory: A review, 2023.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-1999 1958, 2014. URL http: jm1 wwtpg//apprs/vvee wttpppeer/////ssseaaeeeeeeeeeeee
4 html.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning?, 2020.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle, 2015.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, vol- ume 30. Curran Associates, Inc., 2017. URL https: http://eocceeeeeeeeeeeee / 3f5ee24354 4eee2244444dee999
neurips. httpr/pss.cc/pppp///////.....
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najañi, Ioannis Mitliagkas, Aaron Courville, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states, 2018. URL https:// bttpps:/aawxw...rr////oo wttpaa///././0.0/ 052366526.....
15

---

Published in Transactions on Machine Learning Research (05/2025)
Nikil Wale and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. In Sixth International Conference on Data Mining (ICDM '06), pp. 678-689, 2006. doi: 10. 1099//0...0000....
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In International Conference Machine Learning, pp. 992999939. PMLR, 2020.
Haibing Wu and Xiaodong Gu. Towards dropout training for convolutional neural networks. Newral Networks, 71:1-10, November 2015. ISSN 0893-6080. doi: n0..0166..eumme22055.77... URL http:: http::////..ii..o//1100000000000 ] .enneeeet..... 2015. 07 007.
Pinar Yanardag and S. V. N. Vishwanathan. Deep graph kernels. In Longbing Cao, Chengqi Zhang, Thorsten Joachims, Geoffrey Webb, Dragos D. Margineantu, and Graham Williams (eds.), Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, August 10-13, 2015, pp. 1365-1374. ACM, 2015. doi: 10.1145/783355.2778447 URL https: //oi /0..//1..00 0rg/10.1145//7888100 2783417.
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS* Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829541
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018a. URL htttps:://ppeeeeeeteeee net/ forum?
id=riDdp1
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018b. URL https: /tapp:://axxivv wrtp/borggaaaaaoeeeeeeoe aas///ass/1111 03924.
Pan Zhou, Yichen Zhou, Chenyang Si, Weihao Yu, Teck Khim Ng, and Shuicheng Yan. Mugs: A multi-granular self-supervised learning framework. ArXiv, abs/2203.14415, 2022.
Yanqiao Zhu, Yichen Xu, Qiang Liu, and Shu Wu. empirical study of graph contrastive learning. Thirty-fifth
Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL
16

---

Published in Transactions on Machine Learning Research (05/2025)
A CIFAR
In this section, we outline the training specifics for our experiments on the CIFAR datasets, complemented by supplementary results and comparative analyses.
A.1 ResNet Architecture
The specifications for 8 are detailed in Table 4.
Table 4: Configuration of ResNet18 on CIFAR
ResNet18 on CIFAR
Layer
Type
#Neurons
-1
Input Data
32 x 3 = 3072
0
Conv(k=3, s=
322 x 64 = 65536
Conv(k=3, s=2)
322 x 64 = 65536
Conv(k=3, s=2)
162 x 128 32768
Conv(k=3, s=2)
82 x 256 16384
Conv(k=3, s=2)
42 512 8192
5
Avgpool
512
6
MLP
128
A.2 Dropout at All Layers Versus 50% Layer Targeted Dropout
In Figure 14, we compare dropout rates at all layers versus 50% dropout rate targeted specific layer.
Downstream Validaaion Accuracy
0%
1%
5%
10%
20%
50%
50
Layer
Figure 14: Comparing dropout rates all layers versus 50% dropout rate targeted specific layer. For ratio of dropped to total nodes when targeting a layer, see Appendix A.3; there is no trend.
A.3 Proportion of Dropped Nodes
When setting a dropout rate for a specific layer, exclusively affects that layer. Consequently. a 50% dropout rate at one layer results fewer neurons being dropped compared to a 50% dropout applied uniformly across all layers. Additionally, the same dropout rate can impact different numbers of neurons in various layers, reflecting the varying neuron counts in each layer.
In Table 5 we include the number of nodes in each layer, the total nodes across all layers. Thus, for 0.5 dropout, we show the proportion of dropped nodes when a layer is targeted. There is not trend between the proportion and performance.
A.4 Training Details
For implementation, we utilized the code provided by (Khosla al., 2020), available at this link. Our experiments were conducted with batch size of 1024, training each method for 1500 epochs.
17

---

Published in Transactions on Machine Learning Research (05/2025)
Table 5: Proportion of Dropped Nodes per Layer at 50% dropout
Layer
Dropped Nodes
Total Nodes
Proportion
0
0.5 65536
192640
0.17
1
0.5 65536
192640
0.17
2
0.5 x 32768
192640
0.085
3
0.5 x 16384
192640
0.043
4
0.5 x 8192
192640
0.021
0.5 x 512
192640
0.001
0.5 128
192640
0.0003
A.5 Naive Deep Augmentation with stop-gradient on CIFAR100
In Figure 15, include results of 50% dropout with stop-gradient individual layers 100% of the batch. Such naive augmentations generally give poor performances. All layers besides the input data layer lead to downstream accuracy of 1% (equivalent with random guess). The input data layer arrives at downstream accuracy of 61.38%.
Downstream: Validation Accuracy
60
30
LD
10
400
600
800
1,000
1,200
1,400
Num. Epochs Pre-training
Figure 15: CIFARI00. 50% dropout with stop-gradient applied at individual layers on 100% of the batch. Lee freezing earlier layers to random weights.
A.6 Including Deep Augmentation w/o stop gradient initialized with SimCLR
For completion, we also include Deep Augmentation without stop gradient, initialized with pre-trained SimCLR model, together with the other variants see Figure 16.
66
Downstreum Validiatio Accuracy
63
SimCLR
80
SimCLR
Stop
Stop*
Stop
58
w/o Stap"
Layer
Figure 16: Comparing sampling 50% and applying 50% dropout, with or without stop-gradient. initialized with pre-trained SimCLR model. Stop: short for stop-gradient
18

---

Published in Transactions on Machine Learning Research (05/2025)
A.7 Freezing
We repeat the experiment with pre mre-trained initialization but freeze all the layers up to and including the layer at which the targeted transformation occurs; see "Freeze before" in Figure 17. Compared to not freezing, this strategy gives very different results. In particular, the downstream performance of Layers 3 and 4 is critically reduced.
Deep Augmentation after frozen SimCLR layers may not work well due to co-adaptation between neurons, leading to overfitting, Suppose layer of NN exhibits strong co-adaptation within several subsets neurons, i.c., each subset encodes a single data feature. Randomly dropping neurons unlikely to remove a complete co-adapted subset of neurons. Ideally, features learned per neuron dropping any of them provides complementary view. Alternatively, features might be represented continuously among neurons in a layer such that dropout corresponds to something akin to blurring the feature continuously.
Because early layers have fewer parameters to distort the input data, such layers may have less co-adaptation. This might explain why earlier layers, rather than later layers, perform better when frozen during Deep Augmentation. Similarly, higher layers may benefit from higher dropout rates because they are more susceptible to co-adaptation, explaining why in Figure Sa, Deep Augmentation in higher layers yields the best downstream performance.
Reversely, may freeze the layers following the targeted layer; results labeled "Freeze after" in Figure 17. Compared to "Freeze before", Layer improves, Layer worsens, while Layer 4 performs similarly. This asserts that later layers, some more than others, benefit from learning to be invariant to Deep Augmentation.
We see that Deep Augmentation with freezing layers and initialized to SimCLR-model, works better for earlier layers than for later layers. Especially in Figure 39a and 43, we see that earlier layers outperform SimCLR earlier in the training. This suggests that incrementally freezing layers, and adding Deep Augmentation the next layer, might help improve performance and speed up training.
Downsttram Validation Accuracy
63
62
61
60
69
68
SimCLR
Freeze before
57
Freeze
0
3
Layer
Figure 17: Freezing layers before and after Deep Augmentation with stop-gradient, initialized with pre-trained SimCLR
model. For "Freeze before." Layer freezes nothing, and for "Freeze after" Layer 6 freezes nothing.
A.8 PCA Augmentation
In Figure 21, results demonstrate that removing the largest principal component from a batch sample is less effective than subtracting the sixth largest (Figure 28).
Figure 20 presents the six largest principal values from the layers of randomly initialized ResNet18 versus trained with SimCLR CIFARI00 Post-SimCLR training, the distribution of values becomes more uniform, and there is notable shift in the rank of layers before and after the training process.
A.9 Supervised Learning
For our supervised learning experiments, training was conducted for 100 epochs but otherwise using the same hyperparameters as those in the fine-tuning phase post pre-training, which lasted 28 epochs.
Figure presents results from supervised learning on CIFAR100, comparing the effects of uniform dropout across all layers with 50% dropout applied to targeted layer.
19

---

Published in Transactions on Machine Learning Research (05/2025)
Layer
Leparil
Leparo
Layer
Layer
0.2
083
015
Figure 18: Random init.
Figure 19: After SimCLR
Figure 20: The six largest principal value from the layers of random initialized ResNet18 and one trained with SimCLR on CIFAR100.
65
Dewnssircam Validation Accuracy
63
62
60
59
SimCLR
Stop
58
w/o Stop
57
Layer
Figure 21: PCA: Comparing sampling 50% of batch and subtracting the largest principal component from that sample, with and without stop-grade -rgadenae.n..
Figure 22 presents the results of supervised training but also includes standard deviations.
A.10 Domain Transfer: CIFAR100 to CIFAR10
We perform basic domain transfer experiments by taking networks pretrained CIFARI00 and finetuning them CIFAR10. In Figure 24 we include results comparing SimCLR with Deep Augmentation with and without stop-gradient, across layers. We also include performance for different checkpoints across training, see Figure 25a and Figure 25b for Deep Augmentation with and without stop gradient, respectively. Note the overfitting tendencies.
73
Downsrraa Validdation Accuracy
Downsteem Validaiion Accuracy
72
71
70
63
Dropout
No Drupout Stop
Stop
wio Stop
Stop
68
1
5
2
3
Layer
Layer
(a) Dropout
(b) PCA
Figure 22: Supervised only. Deep Augmentation with (a) dropout or (b) PCA, with and without stop-gradient. *; initialized with pre-trained SimCLR model. "Stop" short for stop- goo-ggraieett
20

---

Published in Transactions on Machine Learning Research (05/2025)
Dowentrraam Validation Accuracy
70
0%
1%
65
10%
20%
50%
50% Layer
Layer
Figure 23: Supervised on CIFAR100: Comparing dropout rates at all layers versus 50% dropout rate targeted at specific layer.
Dow wwwnsttaan Validation Accuracy
76
71
SimCLR
Stop
Layer
Figure 24:
Finetuning on CIFARIO of networks pre-trained on CIFARI00. Comparing SimCLR with Deep Augmenta-
tion with and without stop-gratd gradient.. Stop: short for stop-gradient.
A.11 Different dropout rates
In Figure 26, we tune over dropout rates 0.5, 0.25, and 0.125 and find that 0.125 at Layer 4 performs the best.
A.12 CIFAR10
We include results most of the experiments that were CIFARIO0, also CIFARIO. In general, results show the same trends as for CIFAR 100. In Figure 30, we include results comparing dropout rates across all layers to 50%
80
80
Downstrraa Validalion Accuracy
Downstream Validation Accureyy
79
79
78
78
LG
75
1.6
1,400
1,400
Num. Epochs
Num. Epochs
(a) With stop-gradicnt.
(b) Without stop-gradient
Figure 25: SimCLR and Deep Augmenation with and without stop-gradient pre-trained CIFAR100 and finetuned CIFARIO, for different checkpoints during training. Observe the overfitting behavior.
21

---

Published in Transactions on Machine Learning Research (05/2025)
65
Downstream Validation Accunicy
64
63
61
60
SimCLR
69
.5
.25
58
0
3
Layer
Figure 26: CIFARI00: Comparing sampling 50% of batch and applying dropout to that sample, with and without stop-gradient, for different dropout rates. "Stop" is short for stop-gradient.
dropout at single layers. Again, we see targeted dropout at some layers showing much better performance than dropout across all layers.
In Figure 31 we include results sampling 50% of batch and performing 50% dropout with and without stop-gradient, called "Stop" and "w/o Stop" respectively. We also include benchmark of SimCLR. Here **" refers the networks being initialized with pre-trained SimCLR model. Again, we see Layer 4 (with stop-gradient) and Layer 6 (with and without slop-gradient) stand out. It is also interesting to note that when initializing with a pre-trained SimCLR model, performance differs significantly more for Deep Augmentation with stop-gradient than without.
In Figures 27 and 28, we subtract the largest and sixth largest principal component from half the samples of the batch.
Down-treum Validarion Accuracy
91
90
SinCLR
Stop Stop
3
Layer
Figure 27: PCA CIFAR10: Comparing sampling 50% of batch and subtracting the largest principal component from that sample, with and without soopggradeent.
Dow-ntreum Validation Accuracy
91
90
SimCLR Stop
89
wio Stop
Layer
Figure 28: PCA CIFAR10: Comparing sampling 50% of batch and subtracting the sixth principal component from that sample, with and without stop-gradient.
In Figure 32, we include results of Deep Augmentation with stop- gradient but freezing layers up the targeted layer versus freezing after the targeted layer. Again, see the performance change, especially Layer 3 and 4 degrading. while Layer 2 improves.
22

---

Published in Transactions on Machine Learning Research (05/2025)
In Figure 29, we include results of supervised learning CIFARIO, with dropout across all layers well as 50% dropout at largeted-layer.
94
Dowmmtramm Validation Accuracy
963
92
91
0%
18
900
5%
89
10%
20%
88
50%
50%
87
Layer
Figure 29: Supervised CIFAR10: Comparing dropout rates at all layers versus 50% dropout rate targeted specific layer.
We perform basic domain transfer experiments by taking networks pretrained on CIFARIO and finetuning them on CIFARI00. In Figure 33 we include results comparing SimCLR with Deep Augmentation with and without stop- gradient, across layers. We also include performance for different checkpoints across training, see Figure 45a and Figure 45b for Deep Augmentation with and without stop gradient, respectively. Note the overfitting tendencies.
In Figure 34, we tune over dropout rates 0.5, 0.25, and 0.125 and fine that 0. 125 at Layer 4 performs the best.
Dowasircam Validation Accuracy
0%
15
10%
20%
50% at Layer
Layer
Figure 30: CIFARIO: Comparing dropout rates at all layers versus 50% dropout targeted specific layer.
Dowastream Validation Ascuracy
90
SimCLR
SimCLR*
3
Stop
Stop*
89
w/o Stop
w/o Stap"
2
Layer
Figure 31: CIFAR10: Comparing SimCLR with Deep Augmentation with and without stop-gradient. #: Initialized with pre-trained SimCLR model. Stop: short for stop-gradient.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Dowastream Validation Accuracy
SimCIR
Freeze before
Freeze after
Layer
Figure 32: CIFARIO: Comparing freezing layers before after Deep Augmentation with stop-gradient, initialized with pre-trained SimCLR model. Note that for "Freeze before" Layer -1 freezes nothing, and for "Freeze after" Layer 6 freezes nothing.
Downstream Validation Accurccy
46
SimCLR
Slop Stop
2
6
Layer
Figure 33:
Finetuning on CIFAR100 of networks pre-trained on CIFAR10. Comparing SimCLR with Deep Augmenta- tion with and without stop-gradient.
A.13 CIFAR100 across epochs
We include results where finetuned and tested checkpoints different epochs for various experiments. In Figure 35, we include results for dropout everywhere at different rates and 50% dropout at single layers.
In Figure 36, we include results for sampling 50% of each batch and performing 50% dropout on that sample, with and without stop-grade gradient.
In Figures 37 and 38, include results for sampling 50% of each batch and subtracting the largest and sixth largest (respectively) principal component from that sample, with and without stop-gradient.
In Figure 39, compare freezing layers before or after Deep Augmentation with goo-ggra initialized with pre-trained SimCLR model.
In Figure 40, we inlcude results for 50% sampling, 50% dropout, with and without stop-gradient, and initialized with pre-trained SimCLR model.
A.14 CIFAR10 across epochs
We include results where we finetuned and tested checkpoints at different epochs for various experiments. In Figure 41, we include results for dropout everywhere at different rates and 50% dropout at single layers.
In Figure 42, we include results for sampling 50% of each batch and performing 50% dropout on that sample, with and without gradien.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Dowassrcamm Validation Accuracy
SimCLR
Stop
.25
.125
Layer
Figure 34: CIFARIO: Comparing sampling 50% of batch and applying dropout to that sample, with and without stop-gradient, for different dropout rates. "Stop" is short for stop-gradient.
65
65
Downstroam Validition Accurecy
62
63
Downsream Validation
69
62
58
L0
57
66
60
L6
1,000
1,200
1,400
1,000 1,200
1,400
Num. Epochs Pre-trining
Num. Epochs Pre-training
Dropout layers
(b) 50% dropout at single layer
Figure 35: CIFAR100. Comparing dropout rates at all layers versus 50% dropout targeted at specific layer. Note difference in y-axis.
In Figure 43, we compare freezing layers before or after Deep Augmentation with stop- groddrra initialized with pre-trained SimCLR model.
In Figure 44, we inlcude results for 50% sampling, 50% dropout, with and without stop-gradient, and initialized with pre-trained SimCLR model.
65
65
Downsream Validation Accuracy
64
Downstream Validarioo Accuracy
64
63
62
62
1.0
60
60
1.3
:9
58
1.6
1.6
57
57
1,000
1,400)
Num. Epochs Pre-training
Nam. Epochs Pre-training
With gradieta
(b) Without stop-gradient
Figure 36: CIFARI00, Comparing sampling 50% and applying 50% dropout, with without stop-gradient.
25

---

Published in Transactions on Machine Learning Research (05/2025)
Downstream Validation Accuracy
64
Downstream Valieiation Accurccy
64
62
62
61
61
59
50
1.5
1.6
400
600
800
1,000
1,200
1,400
400
600
800
1,000
1,200
1,400
Epochs training
Epochs training
(a) With stop-gradient
(b) Without stop-gradient
Figure 37: PCA CIFARIO0 Comparing sampling 50% of batch and subtracting the largest principal component from that sample, with and without noppgrrdient..
Downxrcam Validarion Accuracy
64
Downereem Validariie Accuracy
64
63
63
62
L-1
L-1
61
61
:0
59
58
58
430
600
800
1,000
1,200
1,400
400
600
800
1,000
1,200
1,4001
Num. Epochs Pre-training
Nom. Epochs Pre-training
(a) With goo-ggradient
(b) Without stop cop-gradieett
Figure 38: PCA CIFARI00: Comparing sampling 50% of batch and subtracting the sixth largest principal component from that sample, with and without hoo-graaieet.
65
Downrreum Validation Accuracy
Downstream Validmtio Accuracy
63
63
62
62
L-1
L.D
60
60
59
57
57
600
1,200
400
1,400
Num. Epochs Pre-training
Num. Epochs Pre-training
(a) Freeze layers before Deep Augmentation
(b) Freeze layers after Deep Augmentation
Figure 39: 100.R100. Comparing freezing layers before or after Deep Augmentation with stop-gradient initialized with pre-trained SimCLR model.
26

---

Published in Transactions on Machine Learning Research (05/2025)
64
Downstreum Validation Accuracy
Downstream Validition Accuracy
63
63
62
62
61
L.0
60
60
59
58
L66
L6
57
600
1,400
Num. Epochs Pre Pre-training
Num. Eppchh
(a) With stop-gradient.
(b) Without stop-gradient.
Figure 40: CIFAR100. 50% sampling, 50% dropout, with and without sop-gradient, and initialized with pre-trained
SimCLR model.
92
92
Dow Validasion Accuracy
Downstrea Validation Accurwey
91
L.
88
.05
88
1.3
87
87
1.5
L66
86
86
1,200 1,400
600
1,000
1,400
Num. Epochs
Num. Epochs
(a) Dropout at all layers
(b) 50% dropout at single layer
Figure 41: CIFARIO. Comparing dropout rates at all layers versus 50% dropout targeted at specific layer. Note difference in y-axis.
92
92
Downtrream Validation Accuracy
Downstream Validatioo Accurney
91
91
90
LO
88
87
87
1.6
86
86
1,400
Num. Epochs
Num. Epochs
With graddet
(b) Without stap-gradient
Figure 42: CIFARIO. Comparing sampling 50% and applying 50% dropout, with without hop-gradient.
27

---

Published in Transactions on Machine Learning Research (05/2025)
Downstean Valitation Accurioy
Downstream Validation: Accurney
68
88
57
1.0
87
86
85
400
600
800
1,000
1,200
1,400
400
600
800
1,000
1,200
1,400
Num. Epochs
Num. Epochs
(a) Freeze layers before Deep Augmentation
(b) Freeze layers after Deep Augmentation
Figure 43: CIFARIO. Comparing freezing layers before or after Deep Augmentation with stop-grt nop-gradient initialized with pre-trained SimCLR model.
92
Downstreum Valilatoo Accuracy
Downstram Validation Accurrc
91
91
90
LO
1.3
88
1.5
87
L6
L6
86
86
600
1,000
1,200
1,400
400
800
1,000
1,200
1,400
Num. Epochs
Num. Epochs
(a) With stop-gradient.
(b) Without stop-gradient.
Figure 44: CIFARIO. 50% sampling, 50% dropout, with and without stop-grt -raddrna,,nt,, and initialized with pre-trained
SimCLR model.
Downstream Validation Accuracy
52
Downsrream Validdaiion Accuracy
52
50
50
19
49
47
47
L.
1.5
45
45
4000
600
800
1,000
1,200
1,400
400
600
800
1,000
1,206
1,400
Epochs
Epochs
(a) With stop- grogggra
(b) Without stop gradient
Figure 45: SimCLR and Deep Augmenation with and without stop-gradient pre-trained on CIFARIO and finetuned on CIFAR100, for different checkpoints during training. Observe the overfitting behavior.
28

---

Published in Transactions on Machine Learning Research (05/2025)
63
Validation Accurccy
64
Downstream Validurion Accuncc
64
63
63
61
G1
60
60
60
1.4
88
58
400
600
800
1,000
1,200
1,400
400
600
800
1,000
1,200
1,400
Num. Epochs
Num. Epochs
(a) Random initialization.
(b) SimCLR initialization
Figure 46: Deep Augmentation with stop- nop-gradient, only lower-to-hee to-hhher augmentation pairs.
Downstrem Validation Accuracy
64
62
61
60
59
L6
57
600
1,000
1,200
1,400
Num. Epochs
(a) Random initialization.
Figure 47: Deep Augmentation without stop-grrdion only lower-to-higher augmentation pairs.
A.15 CIFAR100 Miscellaneous Experiments
We include some preliminary results on different aspects of Deep Augmentation that deserve further investigation.
In Figure 46, we include results of Deep Augmentation with stop-gradient where each pair consists of one sample that has only input-data augmentation and another sample that has intut-data and higher-layer augmentation. Lee we remove all the higher-to-higher and lower-to-lower pairs. We see that for Layer and the performance does not change substantially, but for Layer 3 performance degrades substantially.
In Figure 47, include results of Deep Augmentation without stop-gradien where each pair consists of one sample that has only input-data augmentation and another sample that has input-data and higher-layer augmentation. T.e. we remove all the higher-to-higher and lower-to-lower pairs. We see that for the layers involved performance does not change substantially.
This suggests that lower-to-higher pairs are sufficient to make Deep Augmentation successful, but that certain layers are greatly helped by also including other lower-to-lower higher-to-higher pairs.
In Figure 48, include results of Deep Augmentation with goaddrndient& and freezing layers before, but initialized with random weights instead of initialized with a pre-trained SimCLR model. We note that several layers are severely hurt by this compared to the SimCLR pre-trained model initialization.
In Figure 49, we include results of Deep Augmentation with sto-ggradient and freezing layers before, but initialized with a model pre-trained with SimCLR and 20% dropout across all layers. We wanted high dropout everywhere was more helpful as a starting point for Deep Augmentation. Future work may investigate ways optimally train NN that dropout serves useful higher transformation.
model trained with
29

---

Published in Transactions on Machine Learning Research (05/2025)
Dow nownsscaam Validation Accuracy
L-1
10
800
1,000
1,200
1,400
Num. Epochs
Figure 48: Deep Augmentation with stop-gradient and random initialization, freeze layers before.
Downstean Validation Accuracy
641
62
60
158
57
36
100
600
800
1,000
1,200
1,100
Epochs Pre-training
Figure 49: Deep Augmentation with stop-gradient and SinnCLA--aaine---i--------opppoo initialization, freeze layers before.
30

---

Published in Transactions on Machine Learning Research (05/2025)
B Sentence Embeddings
B.1 Training Details
We used the training protocol of (Gao et al., 2021) with code released at link. Deep Augmentation Layer correspond to just after the first token-embeddings. Deep Augmentation the subsequent layers was applied after each transformer layer in the code, with the last Layer 13 corresponding to the output latent vector.
B.2 PCA Augmentation
In Figure 50, we include results with Deep Augmentation and subtracting the sixth largest principal component.
0.84
SimCSE
Spearman STS-B Developmentt
Stup
wio Step
0.82
0.81
Layer
Figure 50: PCA: SimCSE vs. Deep Augmentation to 50% of the batch and subtracting the sixth largest principal component from that sample, with and without stop-gradient. "Stop": stop-gradient. Deep Augmentation outperforms SimCSE..
B.3 Deep Augmentation and MLM only
In Figure 51, we include results with Deep Augmentation and MLM only, without contrastive learning, Deep Augmentation boosts performance substantially. This demonstrates that Deep Augmentation can boost performance in self-supervised learning settings beyond contrastive learning.
0.65
Spearmnn on STS-S- Develoment
0.53
-3
MLM Stop
0.4
wio Stop
0.35
0
91 10 11
12 13
Layer
Figure 51: MLL vs. MLM with Deep Augmentation with and without stop-gradient, both without contrastive learning. "Stop": nop-ggrdieent.
B.4 Additional Results
In Figure 52, we include results of different dropout rates and hyper- parem-tr settings for using Deep Augmentation with SimCSR.
31

---

Published in Transactions on Machine Learning Research (05/2025)
0.8
STS-B Spearman
SimCSE
S w/o S
0.75
w/o
S. w/o D
w/o S, .25
S,5
0.7
S. S,5
w/o .25
1
3
4
6
8
9 10
11 12
13
Layer
Figure 52: S: short for stop-gradient. D: short for default-dropout, referring to the 10% dropout (including attention- dropout) utilized by BERT and SimCSE. The decimal numbers refer to the Deep Augmentation drop out rate, and is .5 when unspecified.
C Graph Contrastive Learning
We follow the protocol and code of Zhu et al. (2021) that can be found at htt.g/uu...ccccy/y///e..... We pre-train for 1000 epochs and use the following data augmentations in GCL:
A. RandomChoice (
A. RWSamplin walk__ength==10),
NodeDrropppng
A. FeatureMasking (pf-0..
A. EdgeRemoving (pe=0.1)
1)
We use flmi -Miero-averaged F1 Score) as the evaluation metrie. The fimi metric computes the overall F1 score by aggregating the true positives (TP), false positives (FP), and false negatives (FN) across all classes. It is defined as:
Σ.
Precisionmiem
E, + FP.)
T.
Recallmicro
Σ,TP, + FN,)
F1 micro = 2 Precisionmicro Recallmicro
Precisionmicn Recall
This metric is particularly useful for evaluating performance in imbalanced multi-class multi-label classification tasks.
For ablation study with standard deviations, see Table 6.
C.1 Dropout Rate Ablation
In Figure 53, we present an ablation study on various dropout rates. Note that the results reported are evaluated on the test sets. For Table 6, we report test results corresponding to the hyperparameters achieving the highest validation accuracy.
32

---

Published in Transactions on Machine Learning Research (05/2025)
Table 6: Contrastive Learning on Graphs with GNNs, GCL (Graph Contrastive Learning) versus GCL with Deep Augmentation, dropout, PCA, and with and without stop-gret nop-gradient.
Model
COLLAB
IMDB-Mull BuuBii
NCII
PROTEINS
GCL
72.40::0.6
53.33
73.97=1.6
1..
GCL+De Drop L0
70.93.+2.3
56.44
73.07=0.7
2.992+2.9
GCL+DeepAug Drop L2 w/5
10.33
54.00
72.34-0.6
71.73+2.3
GCL+DeepAug Drop L4 w/ S
71.00
52.44
73.32=1.5
1..62+1.1
GCI.+DeepAug Drop 1.6 w/ S
73.80+1..3
52.89+4422
75.83=1.00
73.21+1.5
GCL+DeepAug Drop LO w/o S
71.87+2.7
56.89
1..51=1.7
73.51:2.6
GCL+DeepAAug Drop 1.2 w/o S
70.40+22.0
52.44-13.9
73.32:1225
73.81-22.1
GCL+DeepAug Drop L4 w/o S
70.93#1.6
53.5613..0
75.67=3.3
1..32+1.9
GCL++DeppAug Drop L6 w/o $
70.87
52.00+2..7
74.61
73.81+2.33
GCL+DeepAAug PCA LO w/ S
71.2+11.
54,44.1.88
73.411.9
74.4229
GCL+DecpAug PCA L2 w/ S
70.5343.0
54.44:1..7
73.48=2.4
13.21
GCL+DeepAug PCA L4 w/5 S 70.73
51.11#5.1
74.37=0.8
72.92+1.1
GCL+DeepAAug PCA L6 w/ S GCL+DeepAug PCA LO w/o S
72.0=0044
50.22:2.2
0.1
73.51
68. 68.2342.4
52.8913.8
74.78
744117
GCL+DeepAuug PCA L,2 w/o S
1..53+1.3
54.0++2.0
74.45=005
72.62+2.3
GCL+DeepAug PCA L4 w/o S GCI.+DeepAAug PCA 1.6 w/o
71.93+0.8
54.22+4.99
74.21=0.9
72.92+2.3
70.87+1.55
53.11+0.3
75.18+1.0
72.02+0.4
COLLAB
IMDB MULII
- wio Stop
- wel Stop (dm0 0.1)
wio Stop (dr-0.25)
wio Stop (d-0.5))
Stup (dr=0.5)
0.56
woo
Dorram Dimi
Step (d--0.1)
Stop (dr-8.1) 1)
Stop (dr=0 25)
(dr=0.25)
Step (dm=0.5)
istroam
Stop (dr=0.5)
0.7
0.68
Layer
Layer
(a) COLLAB
(b) MADB-MULTI
NCI!
PROTEINS
wio Stop (dr-0.1)
wio Stop (dr-L.)
(lm=0.25)
0.71
(1em0.25)
0 wi Stop (dr-0.5)
wio Stop (de-0.5)
Downitrra fimi
0.76
Stop (dr=0.1) I)
Downstrcam flmii
Stop (dr=0.1)
Stop
Stop (dr-0225)
Stop (dr=0.5)
0.72
Stup (dr=0.5)
0.74
0.7
0.72
Layer
Layer
(c) NCT1
(d) PROTEINS
Figure 53: Ablation plots for dropout rates across datasets. Accuracy on test set.
33

---

Published in Transactions on Machine Learning Research (05/2025)
D Alignment and Uniformity
First, we reiterate the fundamental drawbacks of Alignment and Uniformity for our work. Alignment is defined either with respect to set of augmentations (the original intent Wang & Isola (2020)) or with respect to embeddings from different datapoints within the same class (as in SimCSE Gao et al. (2021)).
This poses the following issues for our work: (i) We must be very carefully to compare the Alignment and Uniformity results of sentences with those of images. (ii) Since Deep Augmentation introduces new augmentations, for images, select only the default data augmentations for the Alignment and Uniformity metries enable comparison between Deep Augmentation, baselines, and other settings.
In Table 7, we present the Alignment and Uniformity measures for supervised models on sentence embeddings.
Table 7: Comparison of aligmment and uniformity metrics across different models of Supervised Setting on Sentence Embeddings. * Example of collapse
Model
Alignment
Uniformity
Random Init
0.032
-0.514
Regular
0.790
-2.415
Deep Augmentation L8 w/ stop
0.734
-2.318
Deep Augmentation L10 w/o stop
0.671
-2.153
Deep Augmentation L1 w/o stop (Fail)"
0.002
-0.019
1.3
Regular 1.4 w/ S
1.3
+ Regular
n 14 S
1.25
truining
L4 w/o S
G LA w/o
1.25
training
Alignment
Aligument
1.2
0
1.2
1.15
0
n
0
O
1.1
1.1
-2.22
-2.7 -2.68-222 2.64-2.62 -266
-2.72
22.08-2.66222 22..2-4-2666
226 2258
Uniformity
Uniformity
Figure 54: Alignment and Uniformity (lower is better) of Supervised model on SimCLR augmentations on CIFAR. Left: Test data. Right: Training data. Best performance on Alignment and Uniformity does not translate to best performance in supervised setting where the ground truth labels are known. "L" short for Layer and "S" is short for stop-gradient.
E CKA Similarity Index Analysis
We include more complete results using CKA similarity index.
In Figure 55, we include results for several configurations for ResNet18 and CIFAR100. "Layer without Stop" and "Layer 5 with Stop" do not perform well their downstream performance and share the same increased co-adaptation between layers and 5.
In Figure 57, we include results for several configurations for ResNet18 and CIFAR1O. The same trends that were observed on CIFARION 100 is also observed on CIFAR10.
Figure 58 displays results for various configurations on sentence embeddings and the STS-B development set. Deep Augmentation achieves optimal performance in the later co-adaptation region, with stop-gradient at its onset and without stop-gradient towards its conclusion.
In Figure 59, we also present the CKA similarity for supervised models. Additionally, Figure 60 shows the CKA similarity for randomly initialized model. The supervised setting exhibits significantly less co-adaptation between
34

---

Published in Transactions on Machine Learning Research (05/2025)
layers, particularly in the later layers. Although Deep Augmentation slightly decreases co-adaptation, this does not correlate with improved performance on the supervised task, suggesting that co-adaptation is less problematic for supervised learning compared to self-supervised learning. We include the "Deep Aug (Fail)" example to illustrate that training collapses, resulting in extremely low accuracy, are associated with very low co-adaptation, indicating that a nuanced level of co-adaptation is necessary to retain information from the data.
It is also worth noting that, since information cannot be created by deterministic function (i.e., I(X;Y) I(X; (Y))), the reduction in co ad-aapttaiion through transformations likely corresponds to removal of some in- formation from the input data distribution. This suggests that reduced co-adaptation and less overfitting may be linked
through the reduction of spurious information in the later layers of the neural network.
Random
SimCLR
Layer 4 without Stop
Layer 4 with Stop
Layer with Stop
Layer with Stop
Layer with Stop
Figure 55: CKA similarity index of ResNet18 for different pre- pre-training methods CIFAR100.
Self-Supervised
Random
SimCLR
Layer 4 without Stop
Layer with Stop
Supervised
Regular
Layer without Stop
Layer with Stop
Figure 56: CKA similarity index of ResNet18 for and Deep Augmentation in contrastive learning (self-supervised)
versus supervised learning settings on CIFAR100.
35

---

Published in Transactions on Machine Learning Research (05/2025)
Random
SimCLR
Layer 4 without Stop
Layer with Stop
Layer 2 with Stop
Layer with Stop
Figure 57: CKA similarity index of ResNet18 for different pre-training methods CIFARIO.
MIM & BERT
MLM & SimCSE
ML.M & Layer 10 without Stop
MLM & Layer with Stop
BERT
SünCSE
Layer Stop
Layer with Stop
Figure 58: CKA similarity index for different methods trained to produce sentence embeddings. Black crosses indicate the start and end co-adaptations stretch of layers in BERT, and red crosses indicate where the Deep Augmentation was applied. The layers which Deep Augmentation performs the best are around the black crosses the initialization "BERT".
36

---

Published in Transactions on Machine Learning Research (05/2025)
Self-Supervised
BERT
SinCCI
Layer without Stop
Layer Stop
Supervised
Deep Aug (Fail)
SimCSE
Layer 10 without Stop
Layer 8 with Stop
Figure 59: CKA similarity index for different methods trained to produce sentence embeddings. Black crosses indicate the start and end of co-adaptations stretch of layers in BERT, and red crosses indicate where the Deep Augmentation was applied. The layers which Deep Augmentation performs the best around the black crosses the initialization "BERT". Upper row is with contrastive learning -self-supe)vised) setting and lower row is in supervised setting.
Random Init
Figure 60: CKA similarity index for random initialized model.
37


Reviews:
[{"review_id": "B3isvyK5sY", "reviewer": "Reviewer_zJQJ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article studies a very interesting question which could result in \nimproved empirical performance of existing contrast learning models. \nThe study is quite complete in terms of the types of datasets such as text, image, and graphs.\nThe study is complementary to some existing results in the literature of data augmentation,\nwhich is based on changing input images.\n\nThe writing of the methodology could be made clearer, to help me to understand the main contributions. \nThe results could be strengthened by including error bars, as well as positioning with known results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article studies how to use dropout as a data augmentation strategy \nin the framework of contrast learning in self-supervised learning. \nFocusing on the questions on when this idea will work, \nthe effect of the choice of the layer in a neural network is studied.\nBased on extensive experiments, it is found that when using a deep layer. \nthe performance of classification problems could be improved. In particular, \nwhen dropout is combined with stop-gradient. \nThe effect of using PCA instead of dropout is also studied, reaching a similar conclusion.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Clarify the PCA augmentation method. What is the variable p in the definition of A_P? \nWhy there are no eigenvalues involved in the subtraction of each sample x_i? \nDoes the computation relies on the mini-batch I_b and how sensitive is the choice of batch size (if small)?\n\n- It is unclear what the dropout augmentation means, during the training and test. How should one interpret the variable z_i^j in the input?\n\n- The idea of stop gradient is not so clear. Do you use random initial weights for the layers before l if stop gradient is applied? It is a bit surprising that if these initial weights were random, one could still achieve good classification performance, as well as desired invariances. \n\n- Add std in Table 2 and Table 3 as in Table 1.\n\n- Clarify how the Compute is obtained from the cpu and memory usage, in which ratio?\n\n- The supervised learning results seem to be a bit of surprise. You mentioned that \"Deep Augmentation reduces performance in\nthe supervised setting - especially in higher layers\". This is contradictory to the original dropout article of Hinton\nwhere it is found that dropout can avoid significant overfitting. \n\n- Clarify which dropout rate is used in Figure 4. \n\n- ImageNet-100 dataset is not so meaningful to compare with existing results. Is it possible to include results on the standard ImageNet (1000 classes) dataset?\n\n- It is not so clear what are the ground-truth invariances mean in Section 6.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FSgKT8aSes", "reviewer": "Reviewer_eAmY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n**S1.**  I like the idea of looking at dropout as a data augmentation strategy.\n\n**S2.**  The paper shows that on a wide range of scenarios the proposed method is applicable.\n\n\n**Weaknesses**\n\n**W1.**  I understand that the authors want to show wide applicability of their method in different ANN architectures and data modalities.  However, in my opinion this makes it a paper that is hard to read, in two different ways, for the majority of the people that do machine learning in one way or another.  \n\nOn one hand, the flow seems a little bit awkward because the authors need to argue in different sections sequentially for the different ANN architectures and modalities that they have chosen to investigate.  This is really the nature of the paper, but somehow it feels a little bit odd.  On the other hand, personally, I cannot appreciate the diversity of experiments that are being conducted, and I am not sure about the importance of the results in two out of three cases that are studied.  In any case, for computer vision where I do have some familiarity the narrative makes sense.  So, I trust that the same is true for text and for graph neural networks.\n\nIn other words, something seems a little bit off with the flow of the paper, but honestly I do not have a good suggestion for the authors.  So, please take this with a grain of salt.\n\n**W2.**  I think the paper would benefit if the authors would spend a few paragraphs describing the methods that they use for the different modalities.  Here I am following up on the fact that personally I am not familiar with graph neural networks or with NLP-related tasks and therefore all the important models are completely unknown to me (e.g., GCL).  Adding a short paragraph explaining the importance of each method selected and perhaps comparing it against other similar methods in their respective field, I believe would be a good thing that all the readers would benefit from, and those who know can safely skip reading.  Also, I know that some description is given in Sections 5.x but I feel this is too little and also a little bit too late.  Something like related work or related methods background section would be nice to have (perhaps even in the appendix).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigates the idea of dropout as an augmentation strategy for self-supervised learning.  While this idea may seem not intuitive at first sight (at least it was not intuitive to me in the beginning), it is quite natural the way it is formulated and they way it is explored by the authors.  In particular, the authors consider multilayer artificial neural networks (ANNs) and consider the application of dropout at a particular layer of the ANN.  Thus, dropout can naturally be interpreted as an \"augmentation\" at the particular latent representation that would otherwise be communicated to the next layer in the architecture.  The authors call this approach \"Deep Augmentation\".  Among the findings of their work is that a similar result can be observed by performing a PCA-inspired \"augmentation\" at the same layer of the ANN.  Furthermore, the authors show experimentally that Deep Augmentation can outperform input-level augmentations.  Notably, the experiments cover a wide range of scenarios, allowing different ANN architectures and data modalities on which the ANNs are applied to.  These experiments include applications to sentence embeddings, standard computer vision datasets, and graph contrastive learning on COLLAB, IMDB-Multi, and PROTEINS.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see above the listed weaknesses as suggestions for improvement of the paper.  Below I have some clarification questions.\n\n**Questions for the authors**\n\n**Q1**  Can you explain the notation that you use in the equation that is on display for PCA Augmentation (page 4)?  In general, can you say a few more words and explain how this augmentation method works?\n\n**Q2**  In the tables of page 5 (e.g., Table 1, Table 2, etc.), what do the different numbers represent?  Shouldn't something be stated, at least in the caption?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MxPHocmN0X", "reviewer": "Reviewer_UJt2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n* Experiments and results are demonstrated across a wide variety of tasks with different model architectures. Ablations show differences in performance when dropout is applied to specific layers in the network.\n\n## Weaknesses\n\n* Ablations regarding dropout ratio should be brought into the main text. To me, this seems like a crucial and important hyperparameter for this method. In fact, the amount of masking seems quite important for MAE-style approaches, so analyzing various different dropout rates should be more strongly highlighted in the paper. \n* It seems that only Figure 34 contains an ablation on dropout rate for the proposed method (i.e., dropouts at an individual, specific layer). This should be expanded upon for other tasks (NLP or graph tasks).\n* I feel that the drop in performance for supervised objectives could be explored and analyzed more in depth.\n* I believe that the paper could benefit strongly from comparisons to similar data aumgnetation techniques, such as quantization [1] or masking (which is essentially dropout on the pixels for vision tasks) [2].\n\n[1] Wu, et. al. Randomized Quantization: A Generic Augmentation for Data Agnostic Self-supervised Learning.\n[2] Assran, et. al. Masked siamese networks for label-efficient learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the role of dropout as a potential source of data augmentation in standard constrastive learning objectives used in self-supervised learning. Their work studies two main problems, (1) when is dropout useful as a source of data augmentation and (2) is dropout uniquely effective, when compared to alternative data augmentation approaches.\n\nRegarding (1), the authors find that dropout (+ stop-gradient) helps performance in contrastive learning objectives but is generally harmful in supervised learning tasks. Furthermore, the stop-gradient operation seems crucial in SSL objectives, but is even more harmful for supervised learning. Both of these results are surprising and fairly interesting, although no explanation or further experimentation is provided.\n\nThe authors find that dropout hurts performance when applied uniformly across all layers, and is most beneficial when applied to target deeper layers in the network (when only applying to a single layer). The authors find that using PCA (to remove the component with the largest or 6th largest eigenvalue) as a data augmentation performs comparably to using dropout.\n\nExperiments are performed on vision (with ResNets), NLP (with BERT-style models), and graph tasks (with graph convolutional networks).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness above — I believe that moving ablations on dropout ratio into the main text is important and that adding ablations for the proposed method (i.e., dropout on individual targeted layers) on the other NLP and graph domains is quite important and are critical for my recommendation.\n\nI also highly suggest adding the comparisons to other related data augmentation techniques (masking in the input layer or quantization) -- this would significantly strengthen the work in my opinion.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a question, "Can dropout serve as data augmentation to enhance self-supervised learning?" Although naively adding dropout does not help much, the authors provide a remedy to explore how to enhance the performance. After several rounds of the revisions, the experiments have been much more complete, thus we accept this paper. Please incorporate the reviewers' feedback appropriately in the final version.
Decision: Accept (poster)