MoReDrop: Dropout without Dropping

Authors: duo  li
li  jiang
yichuan  ding
xue  liu
victor wai kin chan
Venue: ICML_cc_2024_Workshop_WANT
Published: 2024-05-24 15:34:33

Abstract: Dropout is a widely adopted technique that significantly improves the generalization of deep neural networks in various domains. However, the discrepancy in model configurations between the training and evaluation phases introduces a significant challenge: the model distributional shift. In this study, we introduce an innovative approach termed Model Regularization for Dropout (MoReDrop). MoReDrop actively updates solely the dense model during training, targeting its loss function optimization and thus eliminating the primary source of distributional shift. To further leverage the benefits of dropout, we introduce a regularizer derived from the output divergence of the dense and its dropout models. Importantly, sub-models receive passive updates owing to their shared attributes with the dense model. To reduce computational demands, we introduce a streamlined variant of MoReDrop, referred to as MoReDropL, which utilizes dropout exclusively in the final layer. Our experiments, conducted on several benchmarks across multiple domains, consistently demonstrate the scalability, efficiency, and robustness of our proposed algorithms.

Main paper: 
 introduction: 
 In recent years, deep Neural Networks (DNNs) (Salakhutdinov, 2014; Schmidhuber, 2015) have made significant advancements across a wide range of areas such as computer vision, reinforcement learning, and natural language processing (Deng et al., 2009; Mnih et al., 2015; He et al., 2016; Vaswani et al., 2017; Ho et al., 2020; Jumper et al., 2021; Kang et al., 2024) . While DNNs hold great promise with deeper networks (He et al., 2016; Wang et al., 2022) , the model complexity correspondingly escalates rapidly. This rapid escalation underscores the need for effective regularization techniques to mitigate overfitting and enhance the generalization capabilities of these deep models. Numerous strategies have been developed to tackle these challenges, with dropout gaining prominence due to its simplicity and efficacy extensively utilized in many recent AI breakthroughs (Hinton et al., 2012; Srivastava et al., 2014; Dosovitskiy et al., 2021; Jumper et al., 2021; Ramesh et al., 2022; Wang et al., 2024) .Dropout generally uses a Bernoulli-distributed mask applied to each layer before each training step, which also implies independently and randomly deactivating each neuron with probability p but activating all neurons during inference. Intuitively, dropout training mimics, simultaneously and jointly, training an ensemble of neural networks with varying deactivated unit configurations, all while utilizing shared weights and parameters to the dense model (Hinton et al., 2012; Srivastava et al., 2014) . However, it is non-trivial to explicitly assemble sub-models1 , and a single model characterized by scaled parameters without dropout, i.e., the dense model, is employed for the practical evaluation period. It introduces a subtle but non-negligible model distributional shift between training and evaluation stages.A variety of regularizers have been proposed to mitigate this issue. Sub-to-sub regularization paradigm imposes constraints between pairs of sub-models in the training phase. The primary objective of this constraint is to ensure consistency across different sub-models, thereby maintaining the expectation of a unified, coherent model for evaluation. Examples of such regularizers include L 2 distance (Zolna et al., 2018) , Kullback-Leibler (KL) divergence for two random sub-models (Liang et al., 2021) or worse-case sub-models (Xia et al., 2023) . Another line of research introduces the dense-to-sub regularization (Ma et al., 2017) . This approach imposes constraints from the divergence of dense and sub-models throughout the training process, ensuring consistency across the pair of dense-sub models. However, existing methodologies consistently employ an active update strategy for sub-models throughout the training process, with a predominant focus on minimizing sub-models loss. Those approaches fail to eliminate the model distributional shift, which is an intrinsic consequence of the expectation operation employed during evaluation.In this study, we propose a novel dense-to-sub regularization approach, named Model Regularization for Dropout (MoRe-Drop), to entirely mitigate the model distributional shift. MoReDrop exclusively updates the dense model, thereby circumventing the expectation operator to guarantee a uniform model configuration across both training and inference stages, with a primary emphasis on its loss function. Moreover, MoReDrop introduces a novel regularization term, endowed with a bounded property, into the loss function. This term is designed to quantify the discrepancy in outputs between the dense model and a randomly sampled sub-model for the same mini-batch dataset.Remarkably, the regularizer functions as a toolkit, enabling the dense model to exploit the advantages of dropout at each gradient iteration, but without explicit dropout implementation. Conversely, the sub-models receive passive updates via parameter sharing with the dense model over the course of training. The overall framework and training details of MoReDrop are shown in Fig. 1 . However, same as other methods, MoReDrop requires an additional forward pass in the sub-model, which is computationally intensive for parameter-heavy deep layers (Devlin et al., 2019; Dosovitskiy et al., 2021) . To alleviate this, MoReDropL is introduced, reducing the computational demand to a single last-layer forward pass by applying dropout solely to the final shared-parameter layer. Intuitively, MoRe-DropL trades a marginal loss in generalization for significantly lower computational costs.We assess our proposed methods across various models and tasks, containing image classification, image generation tasks, and language understanding. Experiments show that our proposed methods allow for a higher dropout rate, which potentially further improves the performance but avoids the distributional shift. We also observe that MoReDrop consistently delivers superior performance compared to state-ofthe-art baselines. Surprisingly, MoReDropL also surpasses previous methods in many tasks though it trades off the model generalization ability for computation efficiency. 
related work: 
 Dropout and its Variants. Regularization plays a pivotal role in preventing overfitting in deep learning and largescale models. A multitude of regularization techniques has been proposed to address this issue, including but not limited to weight decay, dropout, batch normalization, noise addition, early stopping, and label smoothing (Simonyan & Zisserman, 2015; Ioffe & Szegedy, 2015; Poole et al., 2014; Yao et al., 2007; Szegedy et al., 2016) . Among these, dropout stands out as a particularly effective method due to its simplicity and broad applicability in different domains (Hinton et al., 2012; Srivastava et al., 2014) . For different model architectures, different dropout methods a variety of dropout methods have been proposed. These consist of DropConnect (Wan et al., 2013) for fully connected layers, SpatialDropout (Tompson et al., 2015) and DropBlock (Ghiasi et al., 2018) for convolutional neural Networks, DropPath (Larsson et al., 2017) for ResNet, and DropHead (Zhou et al., 2020) for Transformer models. In addition to its role as a regularization method to prevent overfitting, dropout has also been utilized as a data augmentation technique (DeVries & Taylor, 2017; Zhong et al., 2020) , further contributing to its effectiveness and versatility. (Liang et al., 2021) Passive Active Sub-to-sub KL G WordReg (Xia et al., 2023) Passive Active Sub-to-sub KL G ELD (Ma et al., 2017) Passive Active Dense-to-subL 2 G MoReDrop (ours)Active Passive Dense-to-sub Modified L 1 (Eqn. ( 4)) 0Model Distributional Shift. Prior work has revealed that dropout brings the inconsistency between training and inference stages, specifically, the model distributional shift.There are primary two categories to address this issue: (1) Sub-to-sub regularization paradigm aims to maintain the consistency between a pair of sub-models in the training process. In this paradigm, Fraternal Dropout (FD) (Zolna et al., 2018) employs L 2 distance on hidden states, R-Drop (Liang et al., 2021) utilizes the on two sampled sub-models with dropout, and Worst-Case Drop Regularization (Wor-dReg) (Xia et al., 2023) holds the same inspiration with R-Drop but firstly find the two worse-case sub-models.(2) dense-to-sub regularization paradigm, i.e., Expectation Linear Dropout (ELD) (Ma et al., 2017) , which maintains the consistency between a pair of dense-sub models as training progresses. MoReDrop belongs to the dense-to-sub regularization paradigm while still holding a significant difference.Knowledge Distillation. Our research delves into the regularization of outputs between two distinct models: a dense model and a sub-model employing dropout. This approach resonates with the principles of knowledge distillation and the teacher-student training paradigm (Hinton et al., 2015; Romero et al., 2014; Yim et al., 2017; Allen-Zhu & Li, 2020; Chen et al., 2020; Tarvainen & Valpola, 2017) . Contrary to prior research in knowledge distillation, our approach does not explicitly define which model serves as the teacher or the student. In our framework, the dense model is expected to provide a stable baseline performance, with further improvements realized through regularization between the dense and sub-models. This dual functionality allows either model to assume the role of teacher or student. However, our primary objective is to enhance the generality and robustness of the dense model, positioning it as the student in this context. Notably, our methodology does not introduce additional parameters or a pre-trained teacher model during the training process, positioning it as a form of selfdistillation (Xu & Liu, 2019; Zhang et al., 2019; Mobahi et al., 2020; Allen-Zhu & Li, 2020) . 
preliminaries: 
 Notation The training set, denoted as D, consists of pairs {(x 1 , y 1 ), . . . , (x N , y N )}, where N signifies the total number of pairs in D. In this context, each pair (x i , y i ) in D is typically considered an independent and identically distributed (i.i.d.) sample drawn from the respective distributions of X ∈ X and Y ∈ Y, respectively.Consider a DNN, denoted by M, consisting of L hidden layers, with X and Y representing the input and output, respectively. Each layer in the network is indexed by l, which spans from 1 to L. The output vector from the l th layer is signified by h (l) . In this setup, the network's input is specified as h (0) = x, and the final network output is h (L) . The network M is characterized by a set of parameters collectively symbolized by θ = {θ l : l = 1, . . . , L}. Here, θ l encapsulates the parameters associated with the l th layer. With slight abuse of notation, we indicate l(θ) as the loss function.Dropout In the naïve dropout formulation (Hinton et al., 2012; Srivastava et al., 2014) , each layer is associated with Γ (l) , a vector composed of independent Bernoulli random variables. Each of these variables has a probability p l of taking the value 0 and a probability 1 -p l of assuming the value 1. This is analogous to independently deactivating the corresponding neuron (effectively setting each weight to zero) with a probability p l . We introduce a set of dropout random variables, denoted by S = {Γ (l) : l = 1, . . . , L}, where Γ (l) corresponds to the dropout random variable for the l th layer. We can represent the deep neural network M i as:h (l) = f l (h (l-1) ⊙ γ (l) ),where ⊙ denotes the element-wise product, and f l represents the transformation function for the l th layer. For instance, if the l th layer is a fully connected layer with a weight matrix W , a bias vector b, and a sigmoid activation function σ(x) = 1/(1 + exp(-x)), then the transformation function is defined as f l (x) = σ(W x + b). To justify the connection between the dense model and the sub-model, we represent M i as the sub-model derived from M through the application of dropout, i can be any number to represent different sub-models. We also use h (l) (x, s; θ) to denote the output of the l th layer given the input x and the dropout value s, under the parameter set θ.Conceptually, dropout seeks to train an ensemble of exponentially many neural networks concurrently, with each network corresponding to a unique configuration of deactivated units, while sharing the same weights or parameters (Hinton et al., 2012; Hara et al., 2016) . We denote the loss function with dropout as l(S; θ).Dropout implicitly forms an ensemble of neural networks via weight sharing during the training phase. However, for evaluation, a deterministic dense model without dropped neurons is employed to approximate the ensemble operation. This approximation results in a model distributional shift, noted as G, between training and evaluation when dropout is used:EQUATIONwhere the LHS of the minus signifies the ideal ensemble model with dropout for evaluation, which is represented by a dense model with expected activate units on the RHS of the minus. 
model regularization for dropout: 
 In this section, we delve into the specifics of our proposed algorithm which distinctly focuses on actively updating the dense model with dense-to-sub regularization, MoReDrop and MoReDropL. The overall high-level structure is depicted in Fig. 1 . We then proceed to discuss the chosen regularizer with its analysis and practical implementation of MoReDrop. Finally, we draw a comparison between our algorithm and the prior methods to alleviate the model distributional shift, elucidating why our proposed algorithm exhibits superior performance, even under high dropout rates.In supervised learning, for standard dense model training, the learning objective is to minimize the following negative log-likelihood function:EQUATIONFor dropout training, the optimization object additionally incorporates the marginalization of the dropout variables (Wang & Manning, 2013; Srivastava et al., 2014; Ma et al., 2017) :EQUATIONThe pivotal element driving the distributional shift is the disparity between model configurations during training and inference. During training, there is an active updating of submodels, while inference is conducted with a deterministic model. This approach implicitly defines an expectation term and consequently introduces a model distributional shift (Ma et al., 2017; Liang et al., 2021) , as delineated in Eqn. (1). To mitigate this discrepancy, we introduce a novel method, termed Model Regularization for Dropout (MoReDrop).Our approach begins by conducting gradient backpropagation exclusively on the dense model, prioritizing the loss from this configuration without dropout, shown in Eqn. ( 2).Unlike conventional methods that pursue model consistency solely during training, MoReDrop ensures model consistency applied throughout both the training and inference stages. This approach aligns with traditional dense training, thus nullifying the model distributional shift: G = 0.However, the advantages of dropout have been forsaken.To incorporate these benefits, our algorithm imposes constraints on the pair consisting of the dense model M and a randomly sampled sub-model M i . This is achieved through passive updating-without gradient backpropagation by exploiting their shared-parameter relationship with the dense model.R = g(l(S; θ) -l(θ)),Typically, both the KL divergence and L 2 divergence are unbounded and may outweigh the primary loss function from the dense model, particularly in scenarios characterized by high dropout ratios. Moreover, these divergences exhibit heightened sensitivity to the coefficient settings.In the present study, we adopt the function g = (exp(α • x) -1)/(exp(α • x) + 1), a variant of the Logistic Sigmoid function to confine the output within the range (-1, 1) and g(0) = 0, where α serves as an exponential-inner coefficient for R. This specific formulation of the regularization term offers two notable advantages over other divergences or distance functions. Firstly, its bounded nature imparts robustness to the loss function against varying dropout rates p, rendering it possible to train a near-optimal model even at high values of p. Second, the incorporation of the coefficient α into the exponential function standardizes the hyperparameter search space across various tasks, with the dense model providing a stable "baseline" performance, upon which further enhancements are realized through this regularization. Specifically, the regularization term we employed is as follows:EQUATIONand the final optimization objective in our algorithm is:EQUATIONwhere it is bifurcated into two principal elements: the primary one being the loss of the dense model, which guarantees model consistency for both training and inference; and a supplementary regularization term R, incorporated to draw upon the advantages offered by dropout, particularly generalization abilities. Importantly, MoReDrop conducts gradient backpropagation exclusively on the dense model, yet it still capitalizes on the advantages of dropout models.An ablation study on various regularization forms, as detailed in Appendix B.7, showcases the superiority of our designed R. Nevertheless, we recognize the potential for more effective loss functions and consider their exploration an avenue for future research.As the concrete form of R in Eqn. ( 4) cannot guarantee a point-wise lower bound to 0, i.e., ∃ x, y, s.t. g(l(S; θ)l(θ)) ≤ 0, which results in l(S; θ) -l(θ) ≤ 0. In theory, this could potentially lead to hacking solutions, such as an infinitely large l(θ) alongside a near-zero l(S; θ), against the primary optimization objective to minimize l(θ). However, that scenario is precluded because (1) the upper bound nature of g(•) and the coefficient α limit the magnitude of R; and (2) due to the passive update process of sub-models from the dense model, the value of l(S; θ) tends to stay proximate to l(θ) within a reasonable range.In our experiments, we find that the expectation of l(S; θ)l(θ) ≥ 0 holds across diverse dropout rate configurations, as shown in Fig. 7 . In contrast, the expected value of the concrete form of R over the set S, taken across the entire training dataset, is provably non-negative when l(S; θ)l(θ) ≤ 0, as established in Theorem 4.1 (see details in Appendix A), which further precludes undesirable solutions, i.e., l(θ) → ∞.Theorem 4.1. The regularizer R (Eqn. (4)), under the condition of l(S; θ) -l(θ) ≤ 0, maintains non-negativity over the expectation over S throughout the training process:E Si 1 N N i=1 exp(α • (l(S; θ) -l(θ))) -1 exp(α • (l(S; θ) -l(θ))) + 1 ≥ 0During every gradient update, we execute the forward operation twice using the same randomly sampled mini-batch dataset: once for the dense model and once for the submodel. Then, we calculate the loss function based on these two forward operations. We summarize our final algorithm in Algorithm 1 and Fig. 5) end for computational costs, we introduce a light variant of MoRe-Drop, termed MoReDropL. This version retains the guiding principle of MoReDrop, where the main loss comes from the dense model regularized by the interplay between the dense model and its sub-models. The key distinction between MoReDropL and MoReDrop lies in their network structure for utilizing dropout: MoReDropL employs dropout solely in the final layer, thereby circumventing additional matrix computations in forward, while MoReDrop applies dropout across all layers, necessitating extra matrix computations for all networks. Although MoReDropL sacrifices a degree of generalization capability, this compromise enables a significant reduction in computational burden.Our proposed method, MoReDrop and MoReDropL, exhibits similarities with various established approaches handling model distributional shift raised from the employment of dropout, as detailed in Table 1 . However, MoReDrop finds a distinct way to mitigate the model distributional shift and follows two principle ordering pipelines: a). Firstly, actively updating solely on the dense model during training to zero-forcing the model distributional shift b). Further, introducing the regularizer to embrace the benefits of dropout.In MoRe-Drop, model configurations during training and inference are identical to ensure G = 0 and passively update submodels. In contrast, alternative methods that actively update sub-models incur an implicit expectation and thus G > 0, despite various regularization attempts to minimize it. Furthermore, the dense model actively updating imparts a notable resilience to variations in dropout ratios, which is inherently independent of both p and α, thereby ensuring stable base performance under different coefficient configurations. Motivation for the regularizer. With the same formulation from the mathematical side to minimize the output discrepancy, the regularizer of other approaches aligns with the learning objective to reduce the model distributional gap, either dense-to-sub or sub-to-sub. Conversely, the purpose of R in MoReDrop is to leverage the advantages of dropout.Limitation of sub-to-sub regularization.Sub-to-sub regularization, such as R-Drop, potentially struggles with the polynomial growth of model pairs, particularly under high dropout rates. Thus, sub-to-sub regularization methods may inadvertently compromise model generalization in favor of maintaining sub-model consistency due to their larger search spaces. To confirm this, a systematic analysis through the training loss curves is presented in Section 5.4. 
experiments: 
 To underscore the wide-ranging applicability of our proposed method, we conducted a thorough evaluation spanning distinct machine learning domains, i.e., general language understanding, image classification, and image generation with different backbones algorithms (Section 5.1, Section 5.2, and Section 5.3). In the Section 5.4, we present a systematic comparison of our dense-to-sub regularization with R-Drop sub-to-sub regularization. This comparative study provides a crucial understanding of the superior performance of MoReDrop. Further experimental details, training time, and ablations on hyperparameters and forms of regularization are provided in Appendix B.To ensure a fair comparison, we retained the same common training hyperparameter setting as in the baseline models, such as epochs and batch size. We prioritize R-Drop as the baseline due to its efficiency and superiority over others, following the same setting as R-Drop. The absence of certain baseline results is a deliberate choice with different reasons, detailed in Appendix B.8.Benchmark Datasets. Our evaluation begins with natural language understanding tasks, for which we apply our proposed methods on the standard development sets of the General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) . The GLUE benchmark includes eight unique tasks, all of which involve text classification or regression. The distinct characteristics of each task provide a comprehensive and robust testing ground for our proposed methodology. The evaluation metrics for the eight tasks and experiment details are shown in Appendix B.2.Model & Training. We utilize three publicly available pre-trained models: BERT-base (Devlin et al., 2019) and RoBERTa-base (Liu et al., 2019) and DeBERTaV3-xsmall (He et al., 2020; 2021) as our foundation models for finetuning. As BERT-base, RoBERTa-base and DeBERTaV3xsmall all employ standard dropout, we could directly apply our method and use the original models for comparison. For MoReDrop, we performed a parameter sweep, setting the dropout probability p to span from 0.1 to 0.9 and α to take on values from the set {0.1, 0.5, 1, 2}, uniformly applying this sweeping across all subsequent domains.Results. We present the final performance in Table 2 averaged by 5 independent seeds. MoReDrop outperforms the baselines across all tasks, showing averaged improvements of approximately 1.0%, 1.3% and 1.1% on BERTbase, RoBERTa-base and DeBERTaV3-xsmall models, respectively. Our findings also indicate that MoReDropL enhances average performance over the baseline. Interestingly, MoReDropL outperforms MoReDrop on some tasks, notably achieving a 2% performance margin on the RTE task. We hypothesize that applying dropout to all layers may disrupt beneficial features in pre-trained layers, while limiting dropout to the last layer preserves these features, enabling more effective task-specific fine-tuning. This will be investigated in future work.Benchmark Datasets. Our image classification experiments were conducted on three well-recognized benchmark datasets: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) . The CIFAR-10 and CIFAR-100 datasets both consist of low-dimensional pixel images, with the primary distinction between them being the number of categories they feature, as indicated by their respective names. Conversely, the ImageNet dataset presents a significantly greater challenge, encompassing more than 1, 000 categories.Model & Training. To offer a scalable comparison of MoReDrop in the domain of image classification, we utilize three models: a small model with 1.2 million parameters (ResNet-18) (He et al., 2016) , a medium model with 86 million parameters (ViT-B/16) (Dosovitskiy et al., 2021 ) and a large model with 307 million parameters (ViT-L/16). Note that the standard ResNet-18 does not incorporate dropout, while the default configuration of the vanïlla ViT-B/16 and ViT-L/16 set p = 0.1. For the ResNet-18, our baselines comprise: (1) DropBlock (Ghiasi et al., 2018) , which mitigates overfitting by dropping continuous regions of neurons, and (2) DropPath (Larsson et al., 2017) , which zeroes out an entire branch in the neural network during training, aiming to achieve the same goal as DropBlock. It is worth noting that ResNet incorporates batch normalization as a technique to combat overfitting, disrupting the training process (Ioffe & Szegedy, 2015) . As for the ViT-B/16 and ViT-L/16 algorithm, we incorporate R-Drop (Liang et al., 2021) to alleviate model distributional shift, a goal akin to that of MoReDrop. In all baselines, we set the dropout rate to 0.1, as recommended by the original papers, and this rate has been found to yield the best performance compared to other dropout rates in our evaluation, as shown in Appendix Fig. 3 .Results. The results displayed in Section 5.2 represent averages obtained from 5 independent seeds and experimental details are shown in Appendix B.3. When integrated with DropPath and DropBlock for the ResNet-18 model, MoRe-Drop consistently delivers superior performance on both the CIFAR-10 and the more challenging CIFAR-100 datasets. Notably, when paired with DropBlock, MoRe-Drop realizes a significant increase in accuracy compared to both the original ResNet-18 and DropBlock, with improvements of approximately 1% in the CIFAR-10 dataset and 1.8% in the more challenging CIFAR-100 dataset. The consistently improved performance across different dropout methods, i.e., DropPath and DropBlock, attests to the general applicability of MoReDrop.With the backbone of ViT-B/16 and ViT-L/16, we find that MoReDrop outperforms the vanïlla model and its variant with R-Drop over three degrees of challenges tasks. The smaller margin gained (< 1%) from MoReDrop and MoRe-DropL, compared with the backbone of ResNet-18, is attributed to the saturated performance by its ViT-B/16 and ViT-L/16 backbones. This observation underscores the superiority and scalability of our proposed algorithm on varying challenging tasks. Also, we observe that R-Drop necessitates approximately 3x the number of training epochs to converge on the ImageNet dataset, yet its final performance is not on par with our methods (both MoReDrop and MoReDropL). We attribute this to the compromise of model expressiveness for maintaining sub-model consistency in R-Drop with sub-to-sub regularization (detailed analysis in Section 5.4).Benchmark Datasets. Our evaluation extends to image generation tasks, utilizing the CIFAR-10 dataset for conditional image synthesis. The detailed introduction of diffusion models is in Appendix B.4. To measure the quality of generated images, we calculate the Fréchet Inception Distance (FID) (Heusel et al., 2017) . Model & Training. In our comparative study, we utilize the state-of-the-art generative model EDM (Karras et al., 2022) . Due to computational constraints, we preserved the original dropout ratio p of 0.13 as specified in EDM, and set α = 1 based on our empirical observations from other tasks, without further sweeping. Both EDM and EDM with MoReDrop were trained for 250 epochs. Results. The experimental results, as presented in Table 4 , using deterministic sampling with 18 and 30 steps, respectively, reveal that the application of MoReDrop consistently yields a reduction in FID. This pattern emphasizes the efficacy of MoReDrop in improving the quality and diversity of generated images. While systematic hyperparameter tuning is likely to refine and possibly enhance MoReDrop's performance on image generation tasks, we designate this avenue of exploration as future work due to computational budget constraints.To further investigate the edges of the dense-to-sub regularization in MoReDrop, we perform a comprehensive comparative loss analysis between MoReDrop (dense-tosub) and the state-of-the-art sub-to-sub model regularization approach, R-Drop. In Fig. 2 zero-forcing discrepancy2 of R-Drop. The discrepancy of training curves between dense and sub-models in Fig. 2 , under the same algorithm, approximates the model discrepancy in function space. While R-Drop presents negligible divergence between its dense and sub-models (blue lines), an observed divergence of those is shown in MoReDrop (purple lines). This finding indicates that a high level of consistency among sub-models may compromise model expressivity, leading to incomparable performance. Additionally, a key factor contributing to this inferior performance is the inherent expectation operator used in R-Drop during the evaluation process.To answer it, we analyze the loss curves from dense and sub-models across varying dropout rates of R-Drop in Fig. 2 (Middle). We observe that the loss gap approaches zero irrespective of the dropout rate with p ∈ [0.1, 0.3, 0.5]. However, this is concurrent with increased loss and decreased performance, shown in Fig. 4 , suggesting a strong performance compromise while constraining sub-model pairs. We hypothesize the loss of generalization in sub-to-sub regularization along R-Drop arises from the rising challenge to maintain consistency. In contrast, MoReDrop dense-to-sub regularization approach successfully preserves model generalization ability. This is evidenced by the observed positive correlation between the dropout rate p and the loss gap, as illustrated in Fig. 2 (Right) and Fig. 4 . 
conclusion: 
 We present MoReDrop, an efficient method designed to counter model distributional shift in dropout models, capitalizing on dropout benefits, without dropping during gradient backpropagation. MoReDrop solely updates the dense model, maintaining uniformity across training and inference phases and focusing on its loss function optimization. MoReDrop then integrates a regularization loss originating from the output divergence from the dense model and submodels. We highlight that the sub-models only passively update due to shared parameters to the dense model. For computational efficiency, MoReDropL, a variant of MoRe-Drop, limits the dropout and matrix multiplication to the final layer, balancing efficiency with generalization. Our experimental results across a variety of tasks and domains consistently demonstrate that both MoReDrop and MoRe-DropL achieve state-of-the-art performance in the majority of tasks and their strong robustness. Significantly, MoRe-DropL not only reduces computational demands but also outperforms MoReDrop on some tasks, such as theRTE task of the GLUE benchmark from language understanding domains. This finding paves the way for future research to delve into the joint effects of last-layer dropout and finetuning, while simple last-layer fine-tuning can match or outperform state-of-the-art approaches in image domains (Kirichenko et al., 2022; Le et al., 2023) . 
 
 Appendix: Our paper studies the model distributional shift problem in dropout models. By thinking a step further about the fundamental reason for the model distributional shift, our paper has the potential to encourage researchers to solve existing problems from a new perspective, distant from prior methods. This work also has potential impacts on the field of last-layer pre-training for giant models.Theorem A.1. The regularizer R (Eqn. (4)), under the condition l(S; θ) -l(θ) ≤ 0, maintains non-negativity over the expectation over S throughout the training process:E Si 1 N N i=1 exp(α • (l(S; θ) -l(θ))) -1 exp(α • (l(S; θ) -l(θ))) + 1 ≥ 0Proof. Given f (x) = (e ax -1)/(e ax + 1) is convex (f ′′ (ax) ≥ 0) with a > 0 when x ≤ 0. We can apply Jensen's Inequality to Eqn. (4) under the condition l(S; θ) -l(θ) ≤ 0:E Si 1 N N i=1 exp(α • (l(S; θ) -l(θ))) -1 exp(α • (l(S; θ) -l(θ))) + 1 ≥ exp E Si 1 N N i=1 α • (l(S; θ) -l(θ)) -1 exp E Si 1 N N i=1 α • (l(S; θ) -l(θ)) + 1We aim to establish the following inequality as the denominator is non-negative:exp E Si 1 N N i=1 α • (l(S; θ) -l(θ)) -1 ≥ 0,which simplifies to:EQUATIONwhere α is omitted as it holds positive. For a tractable approximation for dropout variable p, we use Bayes' rule to express the parameterized conditional probability of the output y given the input x and p under parameter θ:p(y | x; θ) = S p(y | x, s; θ)p(s)dµ(s).Next, we reformulate the loss function for sub-models incorporating dropout (as in Eqn. 3):E Si 1 N N i=1 l(S; θ) = -E Si 1 N N i=1 log p(y i |x i , S i ; θ) = - S N i=1 p(s i ) 1 N N i=1 log p(y i |x i , s i ; θ) dµ(s 1 ) . . . dµ(s N ) = - 1 N N i=1 S p(s i ) log p(y i |x i , s i ; θ)dµ(s i ).Given that log(•) is a concave function, Jensen's Inequality yields: S p(s) log p(y|x, s; θ)dµ(s) ≤ log S p(s)p(y|x, s; θ)dµ(s).ThusE Si 1 N N i=1 (l(S; θ)) ≥ - 1 N N i=1 log S p(s i )p(y|x, s; θ)dµ(s i ) = 1 N N i=1 l(θ).Consequently, the proof is finished as Eqn. ( 6) is satisfied with the condition l(S; θ) -l(θ) ≤ 0. Dropout Rate 𝑝 Accuracy (%) Baseline Setting & Experiments Design. While the vanïlla ResNet-18 does involve the dropout technique, we take DropBlock (Ghiasi et al., 2018) and DropPath (Larsson et al., 2017) as our baselines. DropBlock mitigates overfitting by dropping continuous regions of neurons, while DropPath zeroes out an entire branch in the neural network during training, as opposed to just a single unit, making it a perfect match for ResNet. We set the dropout rate to 0.1, as recommended by the original papers for both DropBlock and DropPath, as well as the best dropout rate referring in our experiments(as shown in Fig. 3 ). This rate has been determined to yield the best performance in comparison to other dropout rates. Further details about these experiments can be found in Table 14 .In contrast, we adopt R-Drop as the comparison in ViT-B/16 and ViT-L/16, which minimizes the bidirectional KL-divergence of the output distributions of any pair of sub-models sampled from dropout in model training. As both ViT-B/16 and ViT-L/16 incorporate standard dropout in their architecture, MoReDrop is seamlessly applicable to these models.For MoReDropL, as applied in Appendix B.2, we employed standard dropout before the final linear layer.Fine-tuning Details. On ResNet-18, we fine-tuned the CIFAR-10, CIFAR-100 while ViT-B/16 and ViT-L/16 on CIFAR-10, CIFAR-100 and ImageNet datasets. As the datasets vary in size and complexity, we employed a dynamic dropout rate and used a set of different values for the parameter α. The exact values of the dropout rate and α were determined based on the specific characteristics of each task. The chosen values for each task and model can be found in The exact hyperparameters for each task and model can be found in Table 14 . For the CIFAR-10 and CIFAR-100 tasks, we employed the SGD optimizer with a learning rate of 1e -2, while for the ImageNet task in ViT-B/16, we utilized the Adam optimizer with a learning rate of 1e -4 and 5e -5 for ViT-L/16. For data augmentation, we exclusively utilized random cropping for the CIFAR-10 and CIFAR-100 tasks. However, for the ImageNet dataset, we adopted RandAugment (Cubuk et al., 2020) .Introduction. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020) are a novel class of generative models that produce high-quality samples, excelling in tasks such as image and audio generation. Inspired by the natural diffusion process, these models convert data into noise through a series of steps and then learn to reverse this process. By predicting and cancelling out the noise incrementally, the models reconstruct the original data, generating new samples. While diffusion models are powerful in capturing complex data patterns and ensuring the realism of the samples, they often require significant computational power due to the numerous steps.Experimental Setup. We employ one of the state-of-the-art generative models, EDM (Karras et al., 2022) , as our training backbone. Since EDM uses standard dropout, MoReDrop can be readily applied. Due to computational resource constraints, we maintained the original dropout ratio p = 0.13 and assigned α = 1. In this section, we conduct an ablation study to evaluate the impact of hyperparameters in MoReDrop, specifically the dropout ratio p and the regularization weight α. The sets explored for p and α are {0.1, 0.3, 0.5, 0.7} and {0.1, 0.5, 1, 2}, respectively. As illustrated in Fig. 5 , MoReDrop exhibits significant robustness to hyperparameter variations. Most combinations yield performance superior to the baseline, with the exception of certain extreme combinations such as (p, α) = (0.7, 2) on the CIFAR-10 dataset using the ResNet-18 backbone, and (p, α) = (0.7, 1)/(0.5, 2)/(0.5, 1) on the MRPC task. Generally, when tuning MoReDrop for application in other domains, we recommend prioritizing adjustments to p due to its significant influence on performance.Typically, higher p values, e.g., 0.5, are associated with less precise predictions, shown in Fig. 4 . However, MoReDrop challenges this norm by exhibiting superior performance even at high dropout, such as p = 0.7. This notable achievement can be ascribed to two main contributing factors.Firstly, the performance in MoReDrop primarily relies on traditional dense training for gradient updates, upon which further enhancements are expected through this regularization. Secondly, the inherent bounded nature of R, along with the weight factor α, ensures that its influence on the primary optimization target remains controlled. To counter underperforming submodels with high dropout rates p, lowering α is crucial for diminishing the influence of R, thus preserving the performance anchored in the dense model. Conversely, a higher coefficient α is beneficial at a lower dropout ratio p, capitalizing on In prior research, specifically in the contexts of ELD and FD, the L 2 distance on hidden states was employed as a regularization loss function. However, this approach diverges significantly from the primary training objective, which is to minimize the negative loglikelihood over the model's output distribution. R-Drop introduced the use of KL-divergence between output probability distributions. While this method imposes a strong constraint on the model, affecting its generalization capabilities, it also incurs considerable computational costs.To address these challenges, we propose to use a loss scalar to quantify the discrepancy and then project this into a relative space, adjustable through a hyperparameter α. Additionally, we exploit the bounded nature of the sigmoid(•) function, within [0, 1], to reshape it into g(x) = e x -1 e x +1 = 2 • (sigmoid(x) -1 2 ), forming our regularization loss function. Fig. 6 illustrates the behavior of g(x) for various values of α, highlighting how the function's shape is modulated to control the impact on the regularization term.We conduct numerical experiments to demonstrate the efficacy of different formulations of g(•) For practicality and simplicity in implementation, we adopt ResNet-18 as our model backbone. We configure four distinct experimental scenarios: ResNet-18-2D, which incorporates 2 dropout layers. The tables above reveal distinct performance trends across different g(•) functions. Specifically, g(x) = e x -1 e x +1 outperforms in all settings, g(x) = x demonstrated inferior performance compared to both g(x) = x 2 and g(x) = e x -1 e x +1 . Notably, g(x) = x 2 and g(x) = e x -1 e x +1 demonstrate comparable effectiveness under conditions of low dropout rates. However, as dropout rates increase to 0.7, 0.9, the function g(x) = e x -1 e x +1 exhibits superior robustness in these extreme conditions. This leads to the conclusion that g(x) = e x -1 e x +1 is not only adaptable to varying scenarios but also demonstrates enhanced performance and robustness across different dropout rates.Further investigation focuses on the role of the hyperparameter α within our regularization function, examining whether it is more effective in the form of α • g(x) or g(α • x). To this end, an experiment was conducted using ResNet-18-4D as the backbone, setting p = 0.5, and observing performance variations across different values of α. From the table, we observe that both α • g(x) and g(α • x) deliver comparable performance under low α value. However, with the increment of α, g(α • x) performs more robust compared to g(α • x), meanwhile considering various application scenarios, it becomes evident that a bounded loss function like g(α • x) offers more versatility in addressing a range of issues. Therefore, this paper opts for the use of g(α • x) in the proposed framework. Nevertheless, it is crucial to recognize the potential for discovering more effective loss functions. Future research will be directed towards exploring these alternatives to further enhance the performance and adaptability of our model.The absence of certain experimental results was a deliberate design choice. We provide a detailed explanation as follows:Why R-Drop is our main baseline for comparison? Our decision to use R-Drop as the main baseline is influenced by its proven computational efficiency and benchmarking superiority over ELD (Ma et al., 2017) and FD (Zolna et al., 2018) , as shown in the R-Drop paper. This choice follows the R-Drop paper, which compares R-Drop and FD based on FD's established advantages over ELD, thereby focusing our comparisons on R-Drop versus MoReDrop.Regarding WordReg (Xia et al., 2023) , the absence of publicly available code and detailed experimental information hinders direct comparison. We have reached out to the authors of WordReg via email to inquire further without a response. On the other hand, WordReg and R-Drop share a similar paradigm in using regularization to mitigate model distributional shifts. Conversely, MoReDrop represents a novel paradigm, aiming to harness the advantages of dropout through regularization while the models distributional shift is 0 through actively updating the dense model.Why is R-Drop excluded? We observed that R-Drop requires significant time (3x-10x) to converge, which is consistent with the experimental details shown in Table 8 of the R-Drop paper. To make a fair comparison to the established baseline, On the other hand, MoReDrop can guarantee coverage as the parameter optimization anchors the dense model. Our comprehensive analysis contrasts MoReDrop's dense-to-sub approach with R-Drop's sub-to-sub methodology, detailed in Section 5.4, underscores our method's superior scalability. We also present a new experiment under ViT-B + dropout head (p=0.1), a dropout variant by dropping attention heads in a multi-head attention mechanism [3] . It is similar to DropPath and DropBlock, dropping some structural features of the model. As shown in Table 10 , adding the R-Drop leads to catastrophic failure even not comparable to the backbone. 2nd Workshop on Advancing Neural Network Training: Computational Efficiency, Scalability, and Resource Optimization, Proceedings of the 41 st International Conference on Machine Learning, Vienna, Austria. PMLR 235, 2024.We note sub-models as their corresponding dense model employed dropout with shared parameters.The 'discrepancy' is specifically constrained to the training process. In contrast, 'model distributional shift' refers to differences arising from varying model configurations during training and evaluation phases. 


Decision: **Strengths**
- Updating the dense model and incorporating dropout regularization through loss function are interesting differentiations from prior work.
- The proposed idea is intuitive, easy to implement, and promising. 
- The evaluation is fairly extensive and includes a decent explanation of the results.  
  
**Weaknesses**
- Evaluation is missing important baselines
- The writing could be improved in multiple places, as detailed by the reviewers. 

**Summary***
I think novelty of the idea and the decent evaluation makes this an interesting work for the community.