Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World
Authors: Bowen Lei
Dongkuan Xu
Ruqi Zhang
Bani Mallick
Venue: Accepted by TMLR ()

Abstract: Sparse training has emerged as a promising method for resource-efficient deep neural networks (DNNs) in real-world applications. However, the reliability of sparse models remains a crucial concern, particularly in detecting unknown out-of-distribution (OOD) data. This study addresses the knowledge gap by investigating the reliability of sparse training from an OOD perspective and reveals that sparse training exacerbates OOD unreliability. The lack of unknown information and the sparse constraints hinder the effective exploration of weight space and accurate differentiation between known and unknown knowledge. To tackle these challenges, we propose a new unknown-aware sparse training method, which incorporates a loss modification, auto-tuning strategy, and a voting scheme to guide weight space exploration and mitigate confusion between known and unknown information without incurring significant additional costs or requiring access to additional OOD data. Theoretical insights demonstrate how our method reduces model confidence when faced with OOD samples. Empirical experiments across multiple datasets, model architectures, and sparsity levels validate the effectiveness of our method, with improvements of up to \textbf{8.4\%} in AUROC while maintaining comparable or higher accuracy and calibration. This research enhances the understanding and readiness of sparse DNNs for deployment in resource-limited applications. Our code is available on: \url{https://github.com/StevenBoys/MOON}.

Published in Transactions on Machine Learning Research (03/2024)
Embracing Unknown Step by Step: Towards Reliable Sparse Training in Real World
Bowen Lei Deportment of Statistics Teaas ABM University
bowenlei@stat..amaa www.::/ttttttttt.c.mm
Dongkuan Xu
dru27@ncsuu edu
Department of Computer Science North Carolina State University
Ruqi Zhang
rugiz@purdue. edu
Department of Computer Science
Purdue University
Bani Mallick
bmallick@sttt..en wom.aliitttttttt...oommmomm edu
Deportment of Statistics
Texas A6M Universily
Reviewed on OpenReview: https: // openreview. net/ forum? id= Db5c.3Waj9E
Abstract
Sparse training has emerged as a promising method for resource-effiieet deep neural networks (DNNs) in real-world applications. However, the reliability of sparse models remains crucial concern, particularly in detecting unknown out-of-distribution (OOD) data. This study addresses the knowledge gap by investigating the reliability of sparse training from OOD perspective and reveals that sparse training exacerbates OOD unreliability. The lack of unknown information and the sparse constraints hinder the effective exploration of weight space and accurate disferentiation between known and unknown knowledge. To tackle these challenges, we propose a new unknown-aware sparse training method, which incorporates loss modification, auto-tuning strategy, and voting scheme guide weight space exploration and mitigate confusion between known and unknown information without incurring significant additional costs requiring access to additional OOD data. Theoretical insights demonstrate how method reduces model confidence when faced with OOD samples. Empirical experiments across multiple datascts, model architectures, and sparsity levels validate the elfectiveness of our method, with improvements of up to 8.4% in AUROC while maintaining comparable or higher accuracy and calibration. This research enhances the underst anding and readiness of sparse DNNs for deployment resource- limited applications. Our code available on: http://hhhbbccc//S/oooorrr/////
1 Introduction
Sparse training is popular method for achieving resource efficiency in deep neural networks (DNNs), and it is recciving growing attention when considering the resource-limited real-world applications of DNNs [Bellec
et al. 2017 Evei et al.) 2020 Yuan et. al., 2021). By maintaining sparse weights throughout the training process, sparse training accelerates the training DNNs, saves training memory, and produces sparse models with dense performance levels (Mocanu al.) 2018 Dettmers & Zettlemoyer 2019 Evei al. 2020 Liu et al. (2022a). which has been applied an increasing number of tasks (Yuan al. 2021 Sokar al.) 2021
Bibikar et al. 2022 There remains, however, critical question that needs to be answered before sparse

---

Published in Transactions on Machine Learning Research (03/2024)
80
80
00D Rallability (AUROC)
92
Derse
000 Rellability (AUROC)
92
Dense
Bellabillng (AUROC)
Denst RigL
Momei (NUROC)
Dense SET
91
RigL
91
SET
78
78
90
76
76
69
89
74
74
88
88
c00
Goo
72
80%
90%
95%
80%
90%
95%
80%
95%
80%
50%
95%
Sparsity level
Sparsity level
Sparsity level
Sparsity level
(a) CIFAR-I0, 10, RigL
(b) CIFAR-10, 10, SET
(c) CIFAR-100, Rigl. (d) CIFAR- 100, SET
Figure 1:
OOD reliability (measured by AUROC (%), the higher the better) of the ResNet-18 produced by
dense and sparse training (RigL& & SET) on CIFAR- 10 100. Compared to dense training (black line), sparse training (blue bar) has a smaller AUROC, indicating sparse training exacerbates the unreliability on OOD data. Our MOON (red bar) improves AUROC and OOD detection.
training can be safely introduced in real-world applications: how reliable the sparse model is in the real world?
Reliable models in world scenarios possess the ability to recognize situations in which they are likely to be wrong, such as identifying unknown OOD data, and refrain from providing incorrect responses (Wang el al. 2021b; Vaze et al., 2021 Du et al. 2022 However, the investigation of OOD detection within the context of sparse training remains unexplored, creating an important knowledge gap that hinders comprehensive assessment of the readiness of DNNs for deployment. Models with weak OOD detection are prone to feign undorstanding and provide arbitrary guesses on unrecognizable OOD data, leading to safety concerns (Liu
et al. 2020 Hsu et al.) 2020 Du et al. 2022 Wang et al.] 2021b) in scenarios, e.g., automated healthcare and se I-rriiing cars (Rasheed et. al. 2022 De Silva & Mycroft 2022).
In this work, we investigate for the first time the reliability of sparse training from the OOD perspoctive, and find that sparse training exacerbates the OOD unreliability. Figure | compares AUROC (i.e., a metric for OOD detection, the higher the better) (Yang et al. 2022) between dense and sparse training, e-g., RigT. (Evei) et al. 2020) and SET (Mocanu et al. 2018 The AUROC is smaller for sparse training (blue bar) compared to dense training (black line), indicating that sparse training weakens the OOD detection capacity. In order to address the OOD unreliability of sparse training, we raise the following questions:
Question 1. Why sparse training ezacerbates the OOD unreliability in DNNs
The underlying determinant of weak OOD detection is the lack of unknown information, and sparse constraints hinder the effective communication of unknown concepts with DNNs. In order to recognize OOD samples, the model needs to distinguish between the known and the unknown. However, it has been shown that DNNs tend to focus on the known due to the close nature of the training process and ignore the rest. (Bendale & Boult 2016 |Padhy et al.) 2020: Wang et al., 2021b). There are studies trying to encourage DNNs to learn more about the unknown. But they usually focus on dense training. When moving to sparse training, the sparsity cuts off update routes and produces spurious local minima (Evei et al. 2019 Sun & Li, 2021 He et al.] 2022), which makes the exploration of the weight space more challenging, and thus more difficult. search for and find reliable models (Lei al.] 2023b). Thus, the sparse training exacerbates the OOD unreliability in DNNs.
Question 2. How can we leverage wnknown information under sporse constroints to effectively guide weight space exploration to OOD-reliable state?
To improve OOD reliability, it is crucial to utilize unknown information to guide model training. The sparsity constraint increases the difficulty of exploring the weight space and prevents the unknown information from guiding the model training an OOD-reliable state. Efforts have been made to exploit unknown information in dense DNNs (Thulasidasan et al., 2019 Wang et al., 2021b Roy et al.) 2022). However, DAC (Thulasidasan et al. 2019) is limited to the treatment to label noise and lacks a comprehensive guide to OOD detection. EOW-Softman (Wang et al., 2021b) only indirectly and inadequately exploits the unknown and incurs considerable additional costs. Moreover, HOD (Roy et al., (202)) require access to OOD data, which is often a non-trivial task. To address these issues and adapt to the challenges in sparse training, we propose a new autier-cxposuur-free training loss that extracts unknown information from hard ID samples and then directly tells the DNNs this unknown information, which gives an effective guide for weight space exploration under sparse constraints with only slight increase in computational and storage burden.
2

---

Published in Transactions on Machine Learning Research (03/2024)
Question 3. How can address the confusion between known and unknown information in sporse training to avoid providing misleading guidance for weight space exploration?
Despite the possibility of communicating unknown information, sparse DNNs are more likely to confuse known and unknown information, resulting in poor performance and reliability. DNNs show performance degradation in sparse training with high sparsity (Mocanu et al., 2018 Evei et al. 2020; Lei et al. 2023b), especially in the early training stage, which indicates the challenge of understanding knowable content under sparsity constraints. Therefore, to avoid confusion and provide proper guidance for training, we also design an auto-tuning strategy that allows the DNN to mainly discover the known in the early stage and embrace the unknown in the later stage step by step. In addition, we propose a voting scheme that combines information from multiple models for a more comprehensive understanding of the known and unknown.
In summary, our contributions are summarized as follows:
We for the first time investigate how sparse training influences OOD reliability and analyze the hidden reasons. We find that sparse training exacerbates the OOD unreliability of DNNs.
We propose a new mgdel-agnostic unknowg--ware sparse training method (MDOX) to improve real- world reliability, which allows the model to be aware of what it does not know via a loss modification, auto-tuning strategy, and a voling scheme at little additional cost.
We provide theoretical insights on how our MOON method reduces the confidence of the model when faced with OOD samples and improves OOD reliability.
Empirically, we conduct extensive experiments multiple benchmark datasets, model architectures, and sparsity. Our MOON improves the AUROC by up to 8.4% and successfully upholds the accuracy, calibration, and other pertinent metrics associated with sparse training, thereby complementing existing research and further enhancing the efficacy and applicability of the sparse training paradigm.
2 Related Work
Sparse Training. As models continue to grow in size, there is an increasing focus on sparse training, where sparse weights are maintained during training to accommodate resource-limited real-world deployments (Mo-
canu et al. 2018: Bellec al., 2018: Huang et al. 2023). To find sparse model with good performance, the sparse topology is updated by pruning and growth steps after every fixed number of iterations (Evei
et al. 2020 Huang et al. 2022). A variely of sparse training methods have been invesl and various
pruning and growth criteria, such as weights and gradient magnitude, have been developed (Mostafa & Wangg
2019 Dettmers & Zettlemoyer 2019 Evci ct al.) 2020 Jayakumar et al. 2020 Lin ct al.
2021b Ozdenizci 2023a Existing
& Legenstein 2021 Zhou et al. 2021 Schwarz et al. 2021 Yin et al. 2022 Lei el al.,
work mainly targets saving more resources and at the same time maintaining accuracy. However, despite
saving resources, sparse training can result in more difficult training due the cut-off update routes and the generation of spurious local minima, raising safety concerns both ID and OOD data.
Confidence Calibration. DNNs' reliability on ID data is increasingly important, which usually refers to whether the confidence of DNNs on ID data is well calibrated (Guo et al.) 2017] Nixon et al. 2019 Zhang
ct al. 2020 Wu et al.
2023 ). Prior work has shown that DNNs tend to be over-confident (Guo et al.
2017: Rahaman et al.
2021 Patel et al. 2022 Zhu et al., 2023),
) misleading humans and raising safety
concerns. Rescarch has been conducted to improve confidence calibration, where widely-usod methods include
lemperature scaling (Guo al.] 2017), Mixup (Zhang et al.,
2017). label smoothing (Szegedy et al. 2016),
and Bayesian methods (Gal & Ghahramani 2016 Ashukha et al.
2020 Recently, ID reliability in sparse training has also been studied in Sup-tickets (Yin et al. 2022) and Cigl. Lei et al. 2023b).
OOD Detection & Open-set Classification. For real-world (asks, reliability on OOD data, i.e., OOD detection capability, is also critical due to the presence of unknown categories. It is also referred to as open-set classification when ID classification is required as well (Yang et al., 2021bb Vaze et al. (021). When these unknown categories are given, a reliable model should detect them and refuse to answer, rather than giving random guesses. Research has been devoted to OOD detection, including post-processing methods (Hendrycks)
3

---

Published in Transactions on Machine Learning Research (03/2024)
& Gimpel
2016
Liang et al. 2017 Liu et al. 2020), training time regularization (DeVries & Taylor 2018
Hsu et al. 2020
2022), and training with outlier exposure (Hendrycks et al. 2018 Yu & Aizawa Du et al.
2019
2021a). However, existing OOD reliability studies have mainly focused on dense training and have not yet fully explored sparse training settings.
Yang et al.
OOD reliability of sparse DNNs. Despite the lack of exploration of sparse training, there are studies on the OOD reliability of sparse DNNs. The effect of random pruning on the sparse DNNs with fixed sparsity patterns is studied (Liu al., 2022b), while sparse training has dynamic sparsity patterns that provide better performance compared to fixed patterns. It is also investigated how to obtain deep ensembles that outperform dense models through sparse training, which requires non-trivial additional cost (Liu et al. 2021a Thus, generating a single sparse network through sparse training remains irreplaceable when resources are limited. In addition, efforts have been devoted to OOD reliability of pruning methods, which start with a dense model and gradually prune the weights, reducing the difficulty of exploring the weight space (Ayle et al. 2022 Cheng et al., 2023). In contrast, sparse training maintains a high level of sparsity throughout the training process, which cuts off' large portion of the route and produces more spurious local minima, thus making weight space exploration more challenging and having different properties from pruning methods (Chen et al. 2022b
Extra Dimension. Adding an extra dimension to the output of the K-way classification model has been studied and used to relax closed-worts world setting to open-world setting (Hsu et al., 2020 Mozannar & Sontag Wang et al. 2021b; Verma & Nalisnick 2022 Liu et al.) 2023). Specifically, the close-world setting assumes the test data has the same distribution as the training data, which is not always true in the real world. The open-world setting, on the other hand, allows for unknown classes in the test data. Previous work has used extra dimension to abstain from making decisions improve OOD detection in dense DNNs (Wang et al., 2021b Roy et al.
2020:
noisy labels (Thulasidasan et al., 2019 to
2022 However, they do not directly exploit the unknown information and incur extra costs (Wang et al.
2021b/, or require exposure to OOD data (Roy et al. 2022). In addition, they all ignore how to add extra dimensions in sparse training.
3 Method
We propose a new sparse training method MOON to improve the real-word reliability of DNNs, i.e., to provide effective detection for OOD data under sparsity constraints with comparable or higher ID accuracy and reliability. Importantly, MOON can be seamlessly integrated as a plug-in method and combined with existing OOD detection methods. Specifically, we add an extra dimension in the output probability and propose new unknown-aware loss with an auto-tuning strategy to gradually encourage DNNs to consider more unknown information. Then, we combine information from multiple models into the output model via a voting scheme so that the output model has a comprehensive view of what is known and what is unknown, providing reliable predictions.
3.1 Tell Model What It Doesn't Know
In general, given the data W where E (1,...,A K}, the model is forced to classify new input data into K predefined classes by the widely used cross-entropy loss, even if the model is uncertain about the class of the samples or has never seen this class before (Wang et al. 2021b). We argue that ignorance of unknown information during weight space exploration, ie., the sample for which the model gives incorrect predictions (Ø: * v.), the cause of this unreliability. Sparse constraints further make the exploration more challenging and thus require better guidance from unknown information. To solve these problems, we utilize K+1-way formulation and propose new unknown-aware loss to tell the model what is unknown to it during sparse training. Unlike existing methods, we directly tell DNNs the unknown, providing an effective guide to cope with the challenges posed by sparsity.
To store the unknown information, inspired by previous work on dense DNNs with extra dimensions (Thulasidasan et al. 2019 Wang al. 2021b Roy et al. 2022), we add an extra dimension in the softmax formulation, allowing the model to provide a K + 1-dimensional output probability I (x;)) (pi, : -KR,K+1) PK+1) for each H., unlike the previous K-way formulation. The first K dimensions
K+1-way formulation:
4

---

Published in Transactions on Machine Learning Research (03/2024)
(P1,-.., ,Pk) represent the model's beliefs about class attribution, while the K + 1-th dimension Px+1 is designed to store unknown information. Ideally, the model gives larger probabilities for the true class dimension Py: when the model knows the input samples very well. Otherwise, the model provides larger PK+1 and smaller (P1,'- .Pk) to avoid unreliable random guesses in the K classes.
.KK)
Unknown-aware Loss Modification: To incorporate unknown information into the model, we design an unknown-aware loss shown in Eq. ( On the one hand, when the model knows the sample x, well and makes correct predictions (i.e., Di Vi): we choose the broadly-used cross-entropy loss for zy to encourage larger py On the other hand, if the model does not have enough understanding of harder sample 2, and makes wrong predictions (i.c., Di # yi), we use new-designed loss to increase reliability. Specifically, in the new loss, we add a larger weight (1 + T+=pxe)) to encourage the model to focus more on the samples it predicts incorrectly. To reduce the loss, the model can determine the correct class (i.e., give a larger Py,) provide a larger PK+1 such that (P1, .P)) becomes smaller, indicating that the model is uncertain about its prediction.
log Py
yi y
L(r,) 11-110100
(1)
-(1 wppx-
log py,
U. +
3.2 Do not Frustrate Model in Early Stage
We find that telling the model too much about the unknowns at the beginning can lead to more severe unreliability (see Section 5.7), especially in cases where the weight space is insufficiently explored (e.g,, highly sparse models). It is as if too much emphasis on unknowns frustrates the model and prevents it from finding out what can know. Thus, inspired by (Thulasidasan et al. 2019), we design an auto-tuning strategy for w in the unknown-award loss (i.e., Eq. (I) to step-by-step inform the model about the unknown information, which is outlined in Algorithm] [ In this way, Py; and pk+1 can better help the model learn and store both the known and unknown information.
In the early stage (i.e., epoch t <<= Te), we fix w to zero so that the new loss is equivalent to the
Algorithm 1: w Auto-tuning Scheduler Input: Total epoch T, current epoch f, unknown-free epoch Te, probability {pt) K+1 unknown-free loss L., init factor r, final weight wy, smoothing factor a.
cross-entropy loss and no unknown information is emphasized. DNNs usually learn casy samples in the early stage of training and tend to fit the hard ones later (Wang et al., 2021a). Thus, in the begin- ning, the model does not learn the data well and usually makes many wrong predictions. With in- sufficient exploration of the weight space, a large w will unnocossarily put more emphasis on harder samples and prevent the model from learning easy samples, leading to poor reliability, The model is like new learner, and it is advisable not to point out its mistakes too much, in case it gets frustrated.
Output: The weight for the unknown-aware loss.
Initialization:
w = 0, 3 and 8=
if < T. then
Prepare information for the initial weight:
B (1 -1-210 ((x+))))(xx)
Use moving average 3 = (1 - a) + aB else if 5 = -1 then
After the early stage (i.e,, epoch & > Te), we start telling the model what is unknown. We keep w as a constant at each epoch and increase it from the initial value wa the final value wy after each epoch via a linear scheduler. The initial wy is based on the B (1 Pk+1 ((x;) and calculated in the carly stage. On the one hand, if we get smaller PK+1 and larger L(x;), we can know that the model hasn't end learned the data well while it is over-confident about
Calculate the initial weight wi
ã and wi
Calculate the step size å = wy of the scheduler
T-T.
else
Calculate weight at epoch t: w w. + (c)
its wrong predictions. Thus, we will set a larger 33 and wy, in this case, to emphasize more on the unknown for reliability. On the other hand, we get larger Pk+1 and smaller L(x;), we can know that the model learns the data well and is cautious about its output. Therefore, we can encourage the model to find out more about what it can know with smaller 3 and we For the initial factor r, it is used to control the initial value
5

---

Published in Transactions on Machine Learning Research (03/2024)
Wy to avoid w being too large. For the final Wy, it is pre-determined byper-parameter, which is usually larger than wi- At each epoch after Te, we set linear increase scheduler to = W; (4 TT) å where the step size 8 = (wy - wx) - Te), allowing more attention on the unknown at the later stage of the training.
3.3 Vote for Known and Unknown: Weight Averaging
We further design a voting strategy that combines information from multiple models to give the output model a more comprehensive sense of what it knows and what it does not know. Specifically, at the end of the proposed unknown-aware training (e.g., after 80% training epochs). we collect models at each epoch and use the weight averaging method (lzmailov et al. 2018: Wortsman et al., 2022) to uniformly average them into one output model. In this way, we let multiple models decide what known and unknown together, allowing fuller understanding of the learned knowledge and more reliable predictions.
4 Theoretical Insight
We provide a couple of theoretical insights to demonstrate that our model-agnostic unknown-aware sparse training method MOON can produce more reliable predictions in the face of outts--dist (OOD) data. The unknown information is extracted from hard in-distribution (ID) data, which explains the oullier-exposure-fre property of MOON. Detailed proofs are presented in the Appendix. Without loss of generality, suppose the goal is a wwo-way classification based on (zr, I where yi E (1,2). We have a deep neural network f= g o h, which can be seen as a composition of feature mapping function h and the softmax classification function g.
We first show how we can extract the unknown from the hard ID data. The following analysis is based on Assumption 4.1 which has been widely used in several studies of OOD detection (Lee et al. 2018 Ahuja
ct al. 2019 Morteza & Li 2022).
Assumption 4.1. (Gaussian Mixture Feature Space): The feature mapping function h maps the input ID data to Gaussian mixture (n, En) + 02N(u2, Σ2). Specifically, when y 1, we have h(x)) ~ N(1, En). And when y = 2, we have h(z) ~ NN2,, 22).
We define the definition of unreliability for a set of ID data.
Definition 4.2. (Unreliability) For samples whose features are around h((o), i.c., D(co) = {x,,); [h(x)
h(zo)]l < co). the predictions are unreliable around vp if for any 0 << e < co, we can find > 0 such that
E max M(((z); He, Ex)] - [ = y}1 > n7 cE(1.2)
(2)
Remark 4.3. Definition 4.2 gives a mathematical expression for the ID unreliability in feature space (i.c., the discrepancy between confidence and accuracy) around h(x0)). The first term ED(c) [max.ee(1.2] M(x;; Mes Ex)] denotes the expected confidence around h(zo), and the second term ED = y}] refers to the expected accuracy. For a threshold (e.g., 7), we consider predictions to be unreliable if the discrepancy between confidence and accuracy cannot be reduced below 7) no matter how we alter This extends the definition of unreliability from the global aspect to the local region around h(zo) in feature space and helps to describe unreliability within hard samples.
Insight 4.4. (Unreliability) Suppose we have (#1, from class and D2 = {(x 9): (1(x)) -(x1))l < <,, = 2} from class 2. Then, unrcliability can occur around h(z)).
Remark 4.5. The setting in Insight 1.4 is common for the hard ID samples, resulting in unreliability. Samples with large feature differences within two classes are easy to classify, with both high accuracy and confidence. While samples with minor differences can puzzle the model, with low accuracy and high confidence, resulting in over-confidence.
Insight 4.6. (Hard-ID Reliability) Suppose we have the same (#1, v1) and D2 as in Insight 4.4 If the model is trained with our MOON method and the extra dimension successfully stores the unknown information, the unreliability can be solved, i.e., for any 17 > O, we can find 0 << e < eo such that
Epoa max N(h(()) Me, Ed] - EDDe (1 = yll < <. (E(2))
(3)
6

---

Published in Transactions on Machine Learning Research (03/2024)
Remark 4.7. Insight 1.sight/466 shows that our MOON method reduces the confidence level and decreases the discrepancy, solving the unreliability issue when the model is faced with some hard ID samples.
We further show how OOD detection can benefit from the unknow information from hard ID data.
Insight 4.8. (OOD Reliability)) Suppose we achieve Hard-ID Reliability in Insight 4.6 with our MOON method, we can have lower confidence on OOD data, implying stronger OOD detection.
Remark 4.9. Insignt 4.8 show that OOD detection can be enhanced by the unknown information extracted from the hard ID data. Existing work has found that ID data contain information for the OOD samples (Du et al. 2022). Tt found that ID samples from the low-likelihood region of the class-conditional distribution, i.e, hard-to-detect ID data, are similar to OOD samples in the feature space, which is similar to the scenario described in Insight [.4 Although do not use OOD data in the training, these hard-to-detect ID samples can be viewed pseudo-OOD data. With MOON method, the model knows which hard ID samples it does not know and provides lower confidence for these samples (i.c., pscudo-OOD data). Since the roal OOD data are neighbors of the pseudo-OOD data in the feature space, the model also gives a low confidence for the real OOD data.
5 Experiments
We perform comprehensive empirical evaluation of our MOON where we add our MOON to the training of multiple benchmark datasets, model architectures, and sparsities, and compare it with the original sparse training results.
Datasets & Model Architectures: For in-distribution (ID) data, we include four benchmark datasets:
MNIST DDng 2012), CIFAR-10 and CIFAR-100 (Krizhevsky et al. 2009) and ImageNet-2000 2012 (Russakovsky et al. 2015). For out-of-distribution (OOD) data, follow the setups of OpenOOD (Yang al. 2022) (more details in Appendix B.1). For model architectures, we choose ResNet-18 to train MNIST, CIFAR-10, and CIFAR-100, and choose ResNet-50 to train ImageNet-2012 (He et al. 2016). We repeat all experiments 3 times and calculate the mean and standard deviation.
Training Settings: For sparse training, we study two widely-used methods, e.g., RigL Evci et al, 2020 and
SET Mooaau et al. 2018), and a variely sparsities, including 80,, 90%, 95%, and 99%, which sufficiently reduces the cost and is of great interest in real-world deployments. We also examine the performance of dense training to show the broad applicability of our method (sce Appendix A.3).
We follow the settings of OpenOOD (Yang et al. 2022 We choose SGD with momentum and use the cosine annealing learning rate scheduler. We train all the models for 100 epochs. For the batch size, we set it to 128 in MNIST, CIFAR-10, and CIFAR-100, and 32 in ImageNet.
Implementations:
To measure OOD detection capability, we chose AUROC and FPR-95 (Yang el al. 2022]. For AUROC, it calculates the area under the receiver operating characteristic (ROC) curve, where a larger AUROC suggests better OOD detection. For FPR-95, measures the false positive rate (FPR) true positive rate (TPR) of 95%, where lower FPR-95 implies better OOD detection. To compare ID reliability, we choose ECF (uuo et al. 2017) and test accuracy. ECE measures the discrepancy between confidence and true accuracy, with lower ECE indicating higher ID) reliability.
Comparison Metrics:
5.1 0OD Detection in Sparse Training
In this section, we show that our MOON can improve OOD detection in sparse training, including 80%, 90%, 95%, and 99% sparsities using RigL. Since our MOON requires additional OOD data and training, to fairly compare the OOD reliability, we incorporate our MDON into the widely-used post-process OOD detection methods MSP Hendrycks & Gimpel 2016) and compare the results MSP.
For CIFAR-10 based models, we examine its OOD detection capacity on two near OOD data, ie., CIFAR- 100 (Chandola et al. 2009) and TIN (Han et al., (022)), and four far OOD data, i.c., MNIST (Deng) 2012), SVHN (Netzer et al. 2011), Texture (Ahmed & Courville, 2020), Places365 (Guo et al. 2017). As shown in Table 1. the columns named "NearOOD" and "FarOOD" represent the average detection scores
7

---

Published in Transactions on Machine Learning Research (03/2024)
Table 1: Comparison of OOD detection by AUROC (%) (t) between MOON + MSP (M-MSP) and MSP for
CIFAR-10 in sparse training. Our MOON leads to larger AUROC each OOD data of CIFAR-10, showing its ability to improve reliability on OOD data.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACIS365
FxROOD
MSP
67.43 (0.2)
88.92 (0.2)
88.17 (0.2)
92.92 (0.8)
91.81 (0.7)
89.35 (0.3)
89. 44 (0.2)
90.88 (0.5)
80%
M-MSP
89.93 (0.1)
91.14 (0.1)
90.53 (0.1)
95.00 (0.7)
93.82 (0.5)
91.80 (0.2)
90.82 (0.3)
92.86 (0.4)
MSP
87.78 (0.2)
89.12 (0.1)
88.45 (0.1)
91.82 (0.9)
91.48 (0.5)
87,90 (0.3)
88, 93 (0.2)
90.03 (0.4)
90%
M-MSP
88.92 (0.2)
90.34 (0.2)
89.63 (0.2)
95.39 (0.8)
93.09 (0.4)
91.09 (0.4)
89.92(0.2)
92.37 (0.4)
MSP
56.04 (0.1)
87.30 (0.1)
86.67 (0.1)
93.74 (0.6)
86.09 (0.8)
85.26 (0.3)
87.68 (0.2)
88.19 (0.5)
95%
M-MSP
(0.1)
89.39 (U.2)
(0.1)
94.76 (0.7)
93.35 (0.8)
89.86 (0.3)
88.51 (0.3)
91.62 (0.5)
MSP
80.94 (0.2)
82.75 (0.2)
81.84 (0:2)
88.34 (0.8)
85.75 (0.7)
83.43 (0.2)
80.45 (0.2)
84.49 (0.5)
99%
M-MSP
83.75 (0.1)
85.11 (0.1)
84.43 (0.1)
91.96 (0.8)
89.01 (0.8) 86.44 (0.3)
83.38 (0.2)
87.70 (0.5)
of the near OOD data and the far OOD data, respectively. As shown in Table 1 the AUROC value drops with increasing sparsity, which implies the unreliability issues in sparse training. Our MOON obtains a larger AUROC on each OOD data of CIFAR-10, where the improvement of AUROC can be up to 8.4%. This indicates MOON 's ability to improve the reliability on OOD data in sparse training.
For the model trained on Image Net-20122 we examine its OOD detection ability on four near OOD data,
i.e., Species (SP) (Torralba et al. 2008). iNaturalist (IN) (Shorten & Khoshgoftaar 2019 Openlmage-O
(00) (Li et al. 2021), Net- (10) (Sun et al. 2022), and two far OOD data, i.e., Texture (TX) (Ahmed & Courville 2020 MNIST (MN) Deng 2012). As shown in Figure 2 the red and blue bars represent our
MOON and MSP, respectively. We can see that our MOON provides larger AUROC and smaller FPR-95 almost
on all of the OOD data of ImageNet-2012, showing its effective OOD detection in sparse training.
To show the general applicability of our method, we also incorporate our MOON into other broadly-used post- processing OOD detection methods, e.g..
MSP
MSP
MDON+ MSP
MCON+MSP
99.5
AUROC
FPR-95
ODIN (Liang et al. 2017), EBO (Liu ct al. 2020), KNN (Sun et al. 2022
99.0
98.5
and KLM (Hendrycks et al. (022), and compare the results with those using only ODIN and EBO. We take sparse models on CIFAR-10 using RigL as an exam- ple. First, we consider 99% sparsity while considering ODIN and EBO as baselines. As shown in Table 2 our MOON leads to larger AUROC and smaller FPR-95 on each OOD data compared to ODIN and EBO. Then, for KNN and KLM, we also experiment on different sparsity levels in- cluding 80%, 90%, 95%, and 99%. As shown in Table 3 our MOON leads to larger AUROC on each OOD data compared to KNN and KLM. This shows that the ability of our MOON to improve OOD reliability is consistent under sparsity constraints for different OOD detection methods.
98.0
SP
IN
00
10
TX
MN
SP
IN
00
10
TX
MN
OOD
Data
OOD
Data
(a) AUROC (%) (t)
(b) FPR-95 (%) (4)
Figure 2: Comparison of OOD detection by AUROC (%) (t) and FPR-95 (%) (L) between MOON +MSP and MSP for ImageNet-
2012 using RigL (90%). MOON leads to larger AUROC and smaller FPR-95 on OOD data of ImageNet-2012, showing improved OOD
reliability in sparse training.
5.2 Comparison with Calibration Method
Prior work shows that calibration methods can help with OOD detection in dense DNNs (Wang et al. 2021b; Yang et al.) 2022). To show the ineffectiveness of existing calibration methods for OOD detection in sparse training, we also compare our MOON to calibration methods including temperature scaling (Lee ct al. 2018), Mixup (Zhang et al.) (201)), and CigL (Lei al., (023b) using RigL (90%) for OOD detection capability. For our MOON, we report the results of "MOON ODIN'. As shown in Figure 3 the red and blue hexagons
8

---

Published in Transactions on Machine Learning Research (03/2024)
Table 2: Comparison of OOD detection by AUROC (%) (1) FPR-95 (%) (4) between MOON + HODIN
(M-ODIN), MOON +EBO (M-EBO) and baseline post process methods (i.e., ODIN and EBO) for CIFAR-10 using RigL (99% spansity). Our MOON leads to larger AUROC and smaller FPR-95 on each OOD data, showing ability improve reliability on OOD data in sparse training.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
PLACES165
FAROOD
FPR-05 AUROC
ODIN
$1.09 (0.2)
56.98 (0.3)
85.54 (0.2)
96.82 (0.5)
89.11 (0.7)
87.47 (0.3)
85,49 (0.2)
89.72 (0.4)
M-ODIN
90.59 (0.3)
92.64 (0.2)
91.61 (0.2)
99.21 (0.7)
93.91 (0.8)
93.26 (0.2)
93.35 (0.2)
94.93 (0.5)
EBO
83.73 (0.3)
56.93 (0.3)
85.33 (0.3)
94.68 (0.6)
85.71 (0.5)
85.09 (0.2)
85.44 (0.2)
87.73 (0.4)
M-EBO
85.64 (0.2)
88.55 (0.3)
87.09 (0.2)
97.10 (0.7)
87.35 (0.4)
87.11 (0.3)
88.55 (0.3)
90.03 (0.4)
ODIN
67.99 (0.8)
61.54 (0.5)
64.76 (0.6)
19.40 (1.6)
65.32 (1.7)
56.58 (0.6)
62.56 (0.7)
50.97 (1.1)
M- ODIN
61.23 (0.8)
56,00 (0.6)
58.62 (0.7)
11.58 (1.8)
57.67 (2.1)
51.81 (0.5)
54.09 (0.7)
43.79 (1.3)
EBO
69.42 (0.7)
61.15
65.29 29 (0.6)
34.49 (1.4)
78.00 (1.2)
67.64 (0.6)
61.23 (0.7)
60.36 (1.0)
M-EBO
(0.7)
55.85 (0.5)
59.58
16.37 (1.6)
77.70 (0.7)
(0.7)
51.48 (0.8)
51.60 (0.9)
Table Comparison of OOD detection by AUROC (%) (t) between MOON +KNN, MOON +KLM and baseline post-process methods (i.e., KNNN and KLM) for CIFAR-10 using RigL (80%, 90%, 95%, 99% sparsity). Our MOON leads to larger AUROC on each OOD data, showing its ability to improve reliability on OOD cata sparse training.
CIFAR-100
TIN
NearOOD
MNIST
SVHN
Texture
Places365
FarOOD
KNN
91.12
90.31
91.11
93.76
69
90.83
92.85
+ KNN
95.58
92.47
80%
KLM
77.29
78.98
78.14
83.65
80.29
77.89
MOON - KLM
78.92
81.92
80.42
89.29
86.44
84.78
80.23
85.18
KNN
88.49
90.54
89.51
92.93
92.12
92.06
89.98
91.77
MOON + KNNN
89.40
91.32
90.36
95.04
93.97
92.99
91.73
93.43
90%
KLM
77.60
79.66
78.63
87.9
85.58
80.93
76.64
82.76
MOON KLM
79.21
81.29
80.25
89.38
89.94
83.33
80.04
85.67
KNN
88.10
89.87
88 .9.98
94.85
93.10
92 60
89.50
92.51
MOON + KNNN
88.36
90.34
89.35
94.98
93.62
92.91
90.51
93.01
95%
KLM
76.56
78.34
77.45
87.84
78.86
77.67
76.15
80.13
-KLM
81.26
80.16
89.64
84.04
83.59
79.35
84.16
KNN
92.69
82.96
99%
MOON + KNNN
82.97
84.78
83.88
96.62
95.61
89.15
83.03
91.11
KLM
72.27
74.47
73.37
78.44
63.96
74.22
73.64
73.06
MOON KLM
75.40
76.63
76.01
85.03
82.32
78.51
74.90
80.
represent the FPR-95 of MOON and calibration methods, respectively. We can see that the red hexagons are smaller than the blue hexagons, indicating better OOD detection from MOON compared to calibration methods.
5.3 ID Test Accuracy and Reliability in Sparse Training
In this section, we show that, in addition to improving OOD detection, our MDON can also maintain or improve the accuracy and reliability on ID data. We incorporate our MOON into sparse training (RigT.) and compare it with the original sparse training results without MOON.
For ID Test Accuracy, shown in Figure 4 (a)-(b), the red and blue curves represent our MOON +RigL and RigL, respectively. The red curve is usually higher equal to the blue curve, implying that our MOON can maintain comparable or higher test accuracy on the ID data.
For ID Reliability, we choose expected calibration error (ECE) (Guo et al. 2017 a widely-used measure of the discrepancy between a model's confidence and true accuracy, where lower ECE indicates better confidence calibration and higher reliability. As shown in Figure ! (c)-(d), the red and blue curves represent RigT+ and RigT,, respectively, where the colored ares represent the 95% confidence intervals. The

---

Published in Transactions on Machine Learning Research (03/2024)
SVHN
MNIST
SVHN
MNIST
SVHN
MNIST
TempScale
Mixup MOON
CigL
MOON
MOON
Texture
TIN
Texture
TIN
Texture
TIN
Places365 CIFRR-100
Places365
CIFAR-100
Places365
CIFRR-100
(a) Temperature Sealing (90%)
(b) Mixup (90%)
(c) Cigl (90%)
Figure 3: Comparison of OOD detection by FPR-95 (%) (4) CIFAR-10 between MOON and other calibration methods using RigL (90%). The red hexagons (MOON ) are smaller than the blue hexagons (other calibration methods), indicating a better OOD detection using MOON compared to (a) Temperature Scaling, (b) Mixup, and (c) CigL.
78
0.035
0.10
94
Accursyy (8))
62
Accuracy 122
72
LC1 valuee
0.025
ECE value
70
0.000
0.06
88
66
MOON +RigL
MOON Rigl
0.010
MOON- Rigl
0.02
MOON-Riah
86
0%
80%
90%
95%
55%
0%
80%
50%
95%
95%
0%
80%
50%
95%
95%
80%
50%
95%
95%
Sparsity
Sparsity
Sparsity
Sparsity
(a) Test ACC (CIFAR-10)
(b) Test ACC (CIFAR-100)
(c) BCE (CIFAR-10)
(d) BCE (CIFAR-100)
Figure Comparison of performance on ID data by ECE (L) and test accuracy (ACC) (%) (t) between
MOON and RigL. Our MOON leads to smaller ECE and maintains comparable or higher test accuracy ID data, showing its ability to improve reliability on ID data.
red curve usually lower than the blue curve. This shows that our MOON reduces the ECE, implying an improvement in the ID reliability of MOON.
Table 4: Comparison of reliability of ID and OOD data between MOON +MSP (M-MSP) and MSP for CIFAR-10 in sparse training using SET. Our MOON leads to smaller ECE and comparable or higher accuracy (ACC) (%) ID data, and smaller FPR-95 (%) each OOD data compared to bascline method MSP, showing its ability to improve reliability on ID and OOD data.
ID DATA
OOD DATA (FPR-95 (%) (1))
BCB(1)
ACC (t)
CIFAR-100
TIN
MNIST
SVHN
TEXTURE
PLACES365
MSP
0.031 (0.002) 94.03 (0.2)
64.01 (0.5) 60.64 (0.4)
46.62 (1.2)
66.43 (1.4)
64.50 (0.7) 60.63 (0.4)
50%
M-MSP
0.020 (0.001)
94.80 (0.2)
62.72 (0.4)
58.63 (0.3)
47.63 (1.5)
60.68 (1.2)
59.10 (0.8)
59.39 (0.3)
MSP
0.032 (0.001) 93.94 (0.2)
64.70 (0.3) 62.52 (0.4)
50.45 (1.6)
67.50 (1.7)
64.13 (0.6) 63.33 (0.4)
90%
M-MSP
0.019 (0.001)
94.44 (0.3)
63.33 (0.3) 60.15 (0.5)
43.02 (1.2)
58.81 (2.2)
(0.6)
60.79 (0.4)
MSP
0.035 (0.002)
(0.3)
67.62 (0.5)
(0.4)
51.79 (1.7)
(1.5)
(0.7)
63.44 (0.3)
95%
M-MSP
(0.003)
(0.3)
63.69 (0.5) 60.20 (0.5)
(1.9)
62.02 (1.1)
(0.8)
(0.4)
0.025 (0.001)
(0.2)
74.98 (0.4) 73.27 (0.5)
54.12 (1-5)
74.44 (1.2) 74.36 (0.7) 74.03 (0.3)
99%
M-MSP
0.008 (0.002)
88.67 (0.2)
71.88 (0.5) 69.59 (0.5)
45.14 (1.4) 71.10 (1.1) 65.73 (0.9) 71.61 (0.3)
10

---

Published in Transactions on Machine Learning Research (03/2024)
SVHN
MNNST MOON w/o LM
SVHN
MNNST MOON w/o AT
SVHN. ...ISSTTT
MOON w/o VT
MOON
MOON
MOON
90
50
Texture
TIN
Texture
TIN
Texture
TIN
Places365 CHAR-1100
Places365 CIFAR-100
Places365.
CHFRR-100
(a) MOON w/o LMM (80%)
(b) MOON w/o AT (80%)
(c) MOON w/o VT (80%)
Figure 5: Ablation studies: comparison of FPR-95 (%) (4) between MOON MOON w/o LM, MOON w/o AT,
and MOON w/o VT on OOD data of CIFAR-10. MOON produces lower FPR-95 compared to the other three methods.
Table 5: Comparison of OOD detection by AUROC (%) (f) between MOON +MSP (M-MSP), Norm (L-MSP), and VOS+MSP (V-MSP) for CIFAR-10 in sparse training. Our MOON leads to larger AUROC on each OOD data of CIFAR-10, showing its ability to improve OOD reliability.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACes365
FAROOD
L-MSP
87.71
89.60
88.66
94.74
93.29
89.36
90.79
92.05
80%
V-MSP
87.37
90.19
M-MSP
L-MSP
$7.81
89.41
88.61
95.37
91.26
90.24
89.85
91.68
90%
V-MSP
$7.48
88.03
92. 92.97
92.17
88.93
88.81
90.53
M- MSP
88.92
90.34
89.63
95.39
93.09
91.09
89.92
92.37
L-MSP
57.35
88. 58
87.96
93.36
86.35
87.79
88. 19
89.03
95%
V-MSP
87.12
67.98
87.55
93.64
86. 86.40
28.10
87.89
89.01
M-MSP
88.04
89.39
88.71
94.76
93.35
89.86
88.51
91.62
L-MSP
$2.60
84.52
83.56
91.75
87
84.16
82.68
86.54
99%
V-MSP
82.22
83.09
82.65
84.75
$9.99
82.75
82.26
84.69
M-MSP
83.75
85.
84.43
91.96
89.01
86.44
83.38
87.70
5.4
Comparison with Other Sparse Training Method
Apart from the comparison based on RigL, we also include sparse training results using SET MMoanu et al. 2018) on CIFAR-10 at 80%, 90%, 95%, 99% sparsities. As shown in Table on OOD data, our MOON also leads to smaller FPR-95, indicating improved OOD reliability from our MOON. On ID data, our MOON brings smaller ECE and comparable or higher accuracy (ACC), implying comparable or improved ID reliability and performance. This shows our MOON is effective in different sparse training methods.
5.5 Comparison with Other Training-ting time Regularization Method
Since loss modification be viewed experiments on other dense training- regularization methods including LogitNorm (Wei el al. 2022a) and VOS (Du et al. 2022). As shown in Table 5 our method MOON has larger AUROC compared to LogitNorm and VOS, indicating better OOD reliability from our method in sparse training.
type of training-time regularization, also add additional
5.6 Comparison with Other ditrr-dimension Method
Since our loss modification also belongs to the extra-dimension method, we further compare our MOON with another extra-dimension method, i.c., DAC (Thulasidasan al. (2019) regarding the reliability of ID and OOD data. As shown in Table our MOON also provides larger AUROC when given OOD data. When faced with ID data, our MOON leads to smaller FCE. and comparable higher accuracy (ACC). This suggests that.
11

---

Published in Transactions on Machine Learning Research (03/2024)
Table Comparison of reliability of ID and OOD data between MOON and DAC for CIFAR-10 and CIFAR- 100.
Our NOON leads to smaller ECE and comparable or higher accuracy (ACC) on ID data, and larger AUROC on each OOD data, showing its ability to improve reliability on ID and OOD data.
ID DATA
OOD DATA (AUROC (1))
DATA
METHOD
BCE(4)
ACC(1)
CIFAR- 100/10
TIN
MNIST
SVHN
TEXTURE
PLACES365
DAC
0.0302
91.43
86.17
87.96
89.92
87.26
59.32
88.48
CIPAR-10
MOON
0.0239
94.97
89.93
91.14
96.00
93.82
91.80
90.82
DAC 0.0573
76.03
76.99
80.54
74.43
77.70
77.89
77.86
CIFAR-
MOON
0.0583
77.65
78.82
82.38
80.28
81.57
77.02
79.65
Table 7: Ablation studies: comparison of ECE (L) and accuracy (ACC, 1) between MOON MOON w/o LM,
MOON w/o AT, and MOON w/o VT on CIFAR-100. MOON produces lower FCE values and comparable or higher accuracy.
80%
90%
95%
99%
METHOD
ECE (4)
ACC (%)
ECE (1)
ACC (%)
BCE
ACC (%) (1)
ECE (J)
ACC (%)
w/o LM
0.0272
94.68
0.0282
94.40
0,0307
93.44
0.0207
89.63
w/o AT
94.81
0.0258
94.51
0.0308
0.0200
w/o
0.0296
92.97
0.0256
91.81
0.0096
MOOK
0.0218
91.77
0.0207
94.53
0.0173
93.70
0.0094
89.66
our MOON makes better use of the additional dimension to encourage the model think about the unknown, thus improving real-world reliability while maintaining ID performance and reliability.
5.7 Ablation Studies
We do ablation studies to demonstrate the importance of cach component in our MOON where we train CIFAR-100 using our MOON without the unknown-aware loss modification (MOON w/o LM), MOON without the auto-tuning strategy (MOON w/o AT), and MOON without the average-based voting scheme (MOON w/o VT), respectively. The OOD detection shown by FPR-95 for each OOD data CIFAR-100 in Figure and the ID results are summarized in Table
MOON w/ /o LM: As shown in Figure (a), the blue hexagon (MOON w/o LM) is larger than the red hexagon (MOON), indicating decrease in OOD reliability without the unknown-aware loss modification. Table 7 also shows an increased ECE and decreased ID reliability in MOON w/o LM.
MOON w/o AT: Similarly, as depicted in Figure 5 (b), the blue hexagon (MOON w/o AT) is larger than the red hexagon (MDON), indicating a decrease in OOD reliability the absence the auto-tuning strategy. Table also shows an increased ECE and decreased ID reliability in MOON w/o AT.
MOON w/o VT: As summarized in Figure (c), the blue hexagon (MOON w/o VT) is larger than the red hexagon (MOON), indicating a decrease in OOD reliability when seerage-based voting scheme is present. Table further shows decreased accuracy and increased ECE in MOON w/o VT.
6 Conclusion and Limitations
We investigate for the first lime the reliability of sparse training from an OOD perspective, which jointly considers OOD reliability and efficiency and has important implications for real-world DNN applications. We observe that sparse training causes more severe unrcliability, with useful unknown information being ignored in the training. Thus, we propose new unknown-aware sparse training method, MOON, to enhance OOD detection under sparsity constraints, with little additional cost and no extra OOD data requirement.
12

---

Published in Transactions on Machine Learning Research (03/2024)
We provide theoretical insights to demonstrate the lower confidence brought by MOON on OOD samples. We also conduct extensive experiments multiple benchmark datasets, architectures, and sparsities, showing that our MOON can improve AUROC by up to 8.4%. Our MOON can be used as a drop-in replacement for existing sparse training methods, which fills an important gap in sparse training and improves the effectiveness and applicability of the sparse training paradigm.
Despite the enhanced OOD reliability achieved in sparse training, MOON does not address this matter comprehensively from broader reliability perspective, including robust generalization (Silva & Najafirad
2020
Ozdenizei & Legenstein 2021; Chen et al. 2022a and adaptation (Emam et al. 2021 Alayrac et al,
2022
Hu et al. (022), which are equally vital aspects of reliability and deserve further invostigation (Tran et al. 2022 Additionllyy, the effectiveness of MOON remains unexplored in the context of widely employed
large generative models (Ouyang et al.) 2022 Xu et al., 2023; Croitoru et al. 2023). On the one hand, as model size expands, novel changes model characteristies may arise (Wei et al. 2022b). On the other hand, given the prevalence of non-classffication tasks within LLMs, the exploration of how evaluate their reliability remains an uncharted domain.
References
Faruk Ahmed and Aaron Courville. Detecting semantic anomalies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3162, 2020.
Nilesh A Ahuja, Tbrahima Ndiour, Trushant Kalyanpur, and Omesh Tickoo. Probabilistic modeling of deep features for out-of-distribution and adversarial detection. arXiv preprint 1r..i:1990.111 2019.
Jcan- Can--Batiste Alayrac, Jeff Donahuc, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 16 -277666, 2022.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty cstimation and ensembling in deep learning. arXiv preprint arXiv:2002.064 70, 2020.
Morgane Ayle, Bertrand Charpentier, John Rachwan, Daniel Zügner, Simon Geisler, and Stephan Günnemann. On the robustness and anomaly detection of sparse neural networks. arXiv preprint arXiv:2207.042: 042277, 2022.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. arXiv preprinl ar Xiv: (tf::1711105113 2017.
Guillaume Bellec, David Kappel, Wolfgang Maass, and Robert Legenstein. Deep rewiring: Training very sparse deep networks. International Conference on Learning Representations (ICLR), 2018.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1563 1572 2016.
Sameer Bibikar, Haris Vikalo, Zhangyang Wang, and Xiaohan Chen. Federated dynamic sparse training:
Computing less, communicating less, yet learning better. In Proceedings of the AAAT Conference Artificial Intelligence, volume 36, pp. 6080 6088, 2022.
Varun Chandola, Arindam Banerjee, and Vipin Kumar. Anomaly detection: A survey. ACM computing surveys (CSUR), 41(3):1-58, 2009.
Tianlong Chen, Zhenyu Zhang, Pengjun Wang, Santosh Balachandra, Haoyu Ma, Zehao Wang, and Zhangyang Wang. Sparsity winning twice: Better robust generaliztion from more efficient training. arXiv preprint (:Xi::2202.09844 2022a.
Tianlong Chen, Zhenyu Zhang, Jun Wu, Randy Huang, Sijia Liu, Shiyu Chang, and Zhangyang Wang. Can you win everything with a lottery ticket? Transactions on Machine Learning Research, 2022b.
13

---

Published in Transactions on Machine Learning Research (03/2024)
Zhen Cheng, Fei Zhu, Yu-Yao Zhang, and Cheng-Lin Liu. Average of pruning: Improving performance and stability of out-o--diit detection. ar preprint orXin:2303.0122 0100000,..011.00 2023.
Florinel-Alin Croitoru, Vlad Hondru, Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey. IEEE Transactions on Pottern Analysis and Machine Intelligence, 2023.
Lavindra De Silva and Alan Mycroft. Toward trustworthy programming for autonomous concurrent systems. AI & SOCIETY, pp. 1-3, 2022.
Li Deng. The mnist database of handwritten digit images for machine learning research [best of the webj. IEFE signal processing magazine, 29(6):141-142, 2012.
Tim Dettmers and Luke Zettlemoyer. Sparse networks from scratch: Faster training without losing performance. arXiv preprint arXi::1907.0488 2019.
Terrance DeVries and Graham W Taylor. Learning confidence for networks. arXiv preprint ar 1802 04865, 2018.
detection in neural
Xucfeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't know by virtual outlier synthesis. arxiv preprinl or 2202:2200.0011 01197, 2022.
Zeyad Ali Sami Emam, Hong-Min Chu, Ping-Yes Yeh Chiang, Wojeiech Czaja, Richard Leapman, Micah Goldblum, and Tom Goldstein. Active learning at the imagenet scale. arXiv preprint arXiv:2111.12880 2021.
Utku Evci, Fabian Pedregosa, Aidan Gomez, and Erich Elsen. The difficulty of training sparse neural networks. arXiv preprint 1rXi::1906.10792, 2019.
Utku Evci, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In International Conference on Machine Learning, pp. 2943 2352. PMI.R, 2020.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In indernational conference machine learning, pp. 1050 1059. PMLR, 2016.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMLR, 2017.
Songqiao Han, Xiyang Hu, Hailiang Huang, Mingqi Jiang, and Yue Zhao. Adbench: Anomaly detection
benchmark. arXiv preprint arxiv:2206. 09426, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pottern recognition, pp. 770-778, 2016.
Zheng He, Zeke Xie, Quanzhi Zhu, and Zengchang Qin. Sparse double descent: Where network pruning aggravates overfitting, In International Conference on Machine Learning, pp. 8635 8659. PMLR, 2022.
Dan Hendryeks and Kevin Gimpel. A bascline for detecting misclassified and out-of-distribution examples in
neural networks. ar preprint 1rXiv:1610.00 02116, 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprint arxiv:1812 04606, 2018
Dan Hendryeks, Steven Basart, Mantas Mazeika, Andy Zou, Joe Kwon, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. In International Conference on Machine Learning, pp. 8759 8773, PMLR, 2022.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-o--distributton image without learning from out-of distribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
14

---

Published in Transactions on Machine Learning Research (03/2024)
Shell Xu Hu, Da Li, Jan Stühmer, Minyoung Kim, and Timothy M Hospedales. Pushing the limits of simple pipelines for few-shot learning: External data and fine-tuning make difference. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9068 9077, 2022.
Shaoyi Huang, Bowen Lei, Dongkuan Xu, Hongwu Peng, Yue Sun, Mimi Xie, and Caiwen Ding. Dynamic sparse training via balancing the exploration-exxloitttion trade-off. ar preprint ar Xit:2211.11666, 2022.
Shaoyi Huang, Haowen Fang, Kaleel Mahmood, Bowen Lei, Nuo Xu, Bin Lei, Yue Sun, Dongkuan Xu, Wujic Wen, and Caiwen Ding. Neurogenesis dy dynamics-inspired spiking neural network training acceleration. arxiv preprint arXiv:2304. 12214, 2023.
Pavel Tzmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1833.05.4 054077 2018.
Siddhant Jayakumar, Razvan Pascanu, Jack Rae, Simon Osindero, and Erich Elsen. Top-kast: Top-k always sparse training. Advances in Nearal Information Processing Systems, 33:20744 20754, 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical reporl, Universily of Toronlo, 2009.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of- distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Bowen Lei, Dongkuan Xu, Ruqi Zhang, Shuren He, and Bani K Mallick. Balance is essence: Accelerating sparse training via adaptive gradient correction. arXiv preprint arXiv:2301.0571 2023a.
Bowen Lei, Ruqi Zhang, Dongkuan Xu, and Bani Mallick. Calibrating the rigged lottery: Making all tickets reliable. International Conference on Learning Representations (ICLR), 2023b.
Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 9664-9674, 2021.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint arXiv:1 006.02690, 2017.
Chuang Liu, Xueqi Ma, Yinbing Zhan, Liang Ding, Dapeng Tao, Bo Du, Wenbin Hu, and Danilo Mandic. Comprehensive graph gradual pruning for sparse training in graph neural networks. arXiv preprint w:Xiu:2207.08629 2022a.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola. Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. arXiv preprint Xrxi::2106 14568, 2021a.
Shiwei Liu, Lu Yin, Decebal Constantin Mocanu, and Mykola Pechenizkiy. Do we actually need dense ave-parameterrizatonn? in-time over-parameterization in sparse training. In International Conference Machine Learning, pp. 6989 7000. PMLR, 2021b.
Shiwei Liu, Tianlong Chen, Xiaohan Chen, Li Shen, Decebal Constantin Mocanu, Zhangyang Wang, and Mykola Pechenizkiy. The unreasonable efTectiveness of random pruning: Return of the most naive baseline for sparse training. arXiv preprint arXix:2202. 02649, 2022b.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Ensrgy-based out-of-distribution detection. Advances in Newral Information Processing Systems, 33:21464-21475, 2020.
Yixin Liu, Kaize Ding, Huan Liu, and Shirui Pan. Good-d: On unsupervised graph -ut-of-distribution detection. In Proceedings of the Sizteenth ACM International Conference on Web Search and Data Mining, pp. 339 347, 2023.
15

---

Published in Transactions on Machine Learning Research (03/2024)
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H. Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science, Nature communications, 9(1):1 12, 2018.
Peyman Morteza and Yixuan Li. Provable guarantees for understanding out-of-distribution detection, In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 7831 7840, 2022.
Hesham Mostafa and Xin Wang. Parameter efficient training of deep convolutional neural networks by dynamic sparse reparamelerization. In International Conference on Machine Learning, pp. 4646 4655. PMLR, 2019.
Hussein Mozannar and David Sontag. Consistent estimators for learning to defer to an expert. In International Conference on Machine Learning, pp. 7076 7087. PMLR, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Jeremy Nixon, Michael W Dusenberry, Linchuan Zhang, Ghassen Jerfel, and Dustin Tran. Mcasuring calibration in deep learning. In CVPR Workshops, volume 2, 2019.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 27744, 2022.
Ozan Özdenizci and Robert Legenstein. Training adversarially robust sparse networks via bayesian connectivity sampling. In International Conference on Machine Learning, pp. 8314-8324. PMLR, 2021.
Shreyas Padhy, Zachary Nado, Jie Ren, Jeremiah Liu, Jasper Snock, and Balaji Lakshminarayanan. Revisiting one-vs-all classifiers for predictive uncertainty and dittof-distribution detection in neural networks. arXiv preprint arXiv:2007.05134, 2020.
Kanil Patel, William Beluch, Kilian Rambach, Michael Pfeiffer, and Bin Yang. Improving uncertainty of deep learning-basse object classification on radar spectra using label smoothing. In 2022 IEEE Radar Conference (RadavConfe2)) Conf22), pp. 1-6. TEEE, 2022.
Rahul Rahaman et al. Uncertainty quantification and deep ensembles. Advances in Neural Information Processing Systems, 14:20063-20075 2021.
Khansa Rasheed, Adnan Qayyum, Mohammed Ghaly, Ala Al-Fuqaha, Adeel Razi, and Junaid Qadir. Explainable, trustworthy, and ethical machine learning for healtheare: A survey. Computers in Biology and Medicine, pp. 106043, 2022.
Abhijit Guha Roy, Jie Ren, Shekoofeh Azizi, Aaron Loh, Vivek Natarajan, Basil Mustafa, Nick Pawlowski, Jan Freyberg, Yuan Liu, Zach Beaver, et al. Does your dermatology classifier know what it doesn't 't know? detecting the long-tail of unseen conditions. Medical Image Analysis, 75:102274, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211 22:2222,2222211 2015.
Jonathan Schwarz, Siddhant Jayakumar, Razvan Pascamu, Peter E Latham, and Yee Teh. Powerpropagation: A sparsity inducing weight reparamcterisation. Advances in Neural Information Processing Systems, 34: 28889 28903, 2021.
Connor Shorten and Taghi M Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of big dala, 6(1):1 48, 2019.
Samuel Henrique Silva and Peyman Najafirad. Opportunities and challenges in deep learning adversarial robustness: A survey. Xix preprint arXw:20007.00753, 2020.
16

---

Published in Transactions on Machine Learning Research (03/2024)
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocamu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. arXiv preprint orXin:2106.42177 2021.
Yiyou Sun and Yixuan Li. On the effectiveness of sparsification for detecting the deep unknowns. arXiv preprint ar Xix: 1.09805, 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. arXiv preprint arXiv:2204.0 06507, 2022.
Varun Sundar and Rajat Vadiraj Dwaraknath. Ireproducibility report] rigging the lottery: Making all tickets winners. arXiv preprint arXie:2108.1571 15767, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shiens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
David Martinus Johannes Tax. One-class classification: Concept learning in the absence of counter-examples. 2002.
Sunil Thulasidasan, Tanmoy Bhattacharya, Jeff Bilmes, Gopinath Chemnupati, and Jamal Mohd- Yusof. Combating label noise in deep learning using abstention. arXie preprint 1905,1995..094 2019.
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nouparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(11):1958-1970, 2008.
Dustin Tran, Jeremiah Liu, Michael W Dusemberry, Du Phan, Mark Collier, Jie Ren, Kehang Han, Zi Wang, Zelda Mariet, Huiyi Hu, et al. Plex: Towards reliability using pretrained large model extensions. arXiv preprint XrXi::2207.0 07411, 2022.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. arXiv preprint arXiv:2110.0620 062007 2021.
Rajeev Verma and Eric Nalisnick. Calibrated learning to defer with one-vs-all classifiers. In International Conference on Machine Learning, pp. 22184 22202. PMLR, 2022.
Deng-Bao Wang, Lei Feng, and Min-Ling Zhang, Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. Advances in Newral Information Processing Systems, 34:11809 11820, 2021a.
Yezhen Wang, Bo Li, Tong Che, Kaiyang Zhou, Ziwei Liu, and Dongsheng Li. Energy-based open-world uncertainty modeling for confidence calibration. In Proceedings of the TEEE/CVF International Conference Computer Vision, pp. 9302 9311, 2021b.
Hongxin Wei, Renchunzi Xic, Hao Cheng, Lei Fong, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In International Conference on Machine Leurning, pp. 23631 23644. PMLR, 2022a.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Demny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.0 01222200077882 06882, 2022b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of iult fine-tuned models improves accuracy without increasing inference time. Tn International Conference on Machine Learning, pp. 23965-23998 PMLR, 2022.
Longfeng Wu, Bowen Lei, Dongkuan Xu, and Dawei Zhou. Towards reliable rare category analysis on graphs via individual calibration. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2629 2638, 2023.
17

---

Published in Transactions on Machine Learning Research (03/2024)
Binfeng Xu, Zhiyuan Peng, Bowen Lei, Subhabrata Mukherjee, Yuchen Liu, and Dongkuan Xu. Rewoo: Decou- pling reasoning from observations for efficient augmented language models. arviv preprint ar Xrv: 18323, 2023.
Jingkang Yang, Haoqi Wang, Litong Feng, Xiaopeng Yan, Huabin Zheng, Wayne Zhang, and Ziwei Liu. Semantically coherent out-of-distribution detection. In Proceedings of the IEEE/CVF International Conference on Compuler Vision, pp. 8301 8309, 2021a.
Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey.
arXiv preprinl XrXi::210.0 10000.1134,4 2021b.
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, et al. Openood: Benchmarking generalized out-of-distribution detection. Xrxiv preprint 0r1::2210000224 2022.
Lu Yin, Vlado Menkovski, Meng Fang, Tianjin Huang, Yulong Pei, Mykola Pechenizkiy, Decebal Constantin Mocanu, and Shiwei Liu. Superposing many tickets into one: A performance booster for sparse neural network training, arXiv preprint WXXv:2205.15322 2022.
Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximun classifier discrepancy. In Proceedings of the IEEE/CVE international conference on computer vision, pp.
2019.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, et al. Mest: Accurate and fast memory-cconomic sparse training framework the edge. Advances in Neural Information Processing Systems, 34:20838 20850, 2021.
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, and David Lopez-Paze mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 0tt:::1710000444 09,12, 2017.
Jize Zhang, Bhavya Kailkhura, and T Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, pp. 11117-11128. PMLRR, 2020.
Xiao Zhou, Weizhong Zhang, Hang Xu, and Tong Zhang. Effective sparsification of neural networks with global sparsity constraint. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. -599,-608, 2021.
Dongyao Zhu, Bowen Lci, Jie Zhang, Yanbo Fang, Yiqun Xie, Ruqi Zhang, and Dongkuan Xu. Rethinking data distillation: Do not overlook calibration. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1835-4945, 2023.
18

---

Published in Transactions on Machine Learning Research (03/2024)
A Appendix: Additional Experimental Results
A.1 More Sparse Training Results
To further demonstrate the elfectiveness of MOON apart from the comparison of AUROC (1) CIFAR-10 in the main manuscript, we also include more comparison of FPR-95 (4) between M and MSP for CIFAR-10 in sparse training using RigL. As shown in Table g our MOON usually leads to smaller FPR-95, indicating more effective OOD detection from our MOON compared to baseline methods.
Table Comparison of OOD delection by FPR-95 (4) between MOON+MSP and MSP for CIFAR-10 in sparse training using RigL. Our MOON leads to larger AUROC each OOD data of CIFAR-10, showing its ability to improve reliability on OOD data.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACES365
FAROOD
MSP
62.97
59.71
61.34
48.16
53.70
60.35
58.59
55.20
80%
MOON MSP
59.31
56.04
57.68
39.12
50.71
56.28
56.49
50.65
MSP
63.71
61.51
62.62
51.85
58.33
66.76
61.43
60.31
90%
MOON MSP
62.70
58.18
60.44
33.67
54.65
57.32
58.41
51.01
MSP
70.38
66.96
68.67
44.19
74.57
73.01
65.37
64.28
95%
MSP
39.26
52.72
63.44
63.38
54.70
MSP
75.53
60.45
77.77
71.54
76.49
71.56
99%
MSP
70.38
67.05
68.72
44.34
72.08
62.75
68.54
61.93
To show that the improvement from our MODN is consistent across different sparse training mothods, apart from the comparison based RigL, we also include more sparse training results using SET (Mocanu el. al., 2018). As shown in Tables 9 our MOON usually leads to larger AUROC, indicating more effective OOD detection from our MOON compared to baseline methods.
Table 9: Comparison of OOD detection by AUROC (f) between MOON+ and MSP for CIFAR-10 in sparse training using SET. Our MOON leads larger AUROC on each OOD data of CIFAR-10, showing its ability to improve reliability on OOD data.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACeS365
FAROOD
MSP
87.36
88.81
$8.09
93.41
88.70
$5.51
88.77
80.85
80%
HDOM+ MSP
89.52
90.81
90.16
93.64
92.47
91.25
90.46
91.96
MSP
87.93
89.06
88.49
93.54
90.26
85.40
88.37
90.19
90%
MSP
88.85
90.06
89.45
94.29
92.34
90.70
89.61
91.74
MSP
87.32
94.95
89.13
87.07
87.62
89.69
95%
MSP
93.78
91.96
88.56
MSP
81.79
82. 41
52.10
90.13
81.31
81.77
85.07
99%
MDON MSP
83.10
83.84
83.47
91.13
89.14
84.92
81.90
86.77
Prior work shows that calibration methods can help with OOD detection in dense DNNs Wang et al. 2021b Yang al. 2022). To show the ineffectiveness of existing calibration methods for OOD detection sparse training, we also compare our MOON to other confidence calibration methods including lemperature scaling (Lee et al.] 2018), Mixup {Zhang et al. 2017), and CigL (Lei et al. 2023b) using RigL for OOD detection capability in 99% sparsity. As shown in Figure D. the red and blue hexagons represent the FPR-95 of NOОN and other calibration methods, respectively. We can see that the red hexagons are smaller than the blue hexagons, indicating better OOD detection from MOON compared to other calibration methods.
To chock the effectiveness of our MOON at low sparsity level, we add additional experiments at 50%, 60%, and 70% sparsity on CIFAR-10 and CIFAR-100. The detailed results are summarized in Figure Our MOON can consistently improve the OOD reliability at. different sparsity levels.
19

---

Published in Transactions on Machine Learning Research (03/2024)
SVHN
MNIST
SVHN
MNIST
SVHN
MNIST
TempScale
Mixup MOON
CigL
MOON
MOON
Texture
TIN
Texture
TIN
Texture
TIN
Places365 CHAR-1100
Places365
CIFAR-100
Places365
CIFAR-100
(a) Temperature Sealing (99%)
(b) Mixup (99%)
(c) Cigl (99%)
Figure 6: Comparison of OOD detection by FPR-95 (4) CIFAR-10 between MOON and other calibration methods using RigL (90%). The red hexagons (MOON) are smaller than the blue hexagons (other calibration methods), indicating a better OOD detection using MOON compared to (a) Temperature Scaling, (b) Mixup,
and (c) CigL.
80
000 Rellability (AUROC)
Dense
000 Rell iity (AUROC)
....eeneee...
Rigl.
RigL
91
Ours
78
Durs
90
26
74
88
72
50%
60%
70%
80%
90%
95%
50%
60%
70%
80%
90%
95%
Sparsity level
Sparsity level
(a) 10 RigL.
(b) CIFAR-100, RigL
Figure 7: OOD reliability (measured by AUROC (%), the higher the better) of the ResNet-18 produced by dense and sparse training (RigL. & SET) on CIFAR-10/100, Compared dense training (black line), sparse training (blue bar) has smaller AUROC, indicating sparse training exacerbates the unreliability OOD data. Our MOON (red bar) improves AUROC and OOD detection.
Table 10: Comparison of OOD detection by FPR-95 (FPR, 4) & AUROC (AUC, t) between MOON and MSP for MNIST. Our MOON leads to smaller FPR-95 and larger AUROC on each OOD data, showing its ability to improve reliability on O0D) data.
NOTMNIST
MNIST
NEAROOD
TEXTURE
CIFAR1O
TIN
PLACEs365
FAROOD
AUC FPR
7.43
MOON MSPP
3.57
7.08
5.32
1.76
0.91
1.41
1.32
1.35
MSP
80.20
93.69
91.45
98.55
98.55
98.75
95.20
98.51
MO02 + MSP
98.07
96.54
97.31
98.86
99.18
99.08
99.12
99.05
A.2 More Results MNIST
To show that the improvement from our MOON consistent across different benchmark datasets, apart from the comparison on CIFAR-10, CIFAR-1000 and ImageNet-2012, we also include more results on MNIST. As shown in Tables 10 our MOON usually leads larger AUROC and smaller FPR-95, indicating more effective OOD detection from MOON compared to baseline methods.
20

---

Published in Transactions on Machine Learning Research (03/2024)
Table 11: Comparison of OOD detection by FPR-95 (4) & AUROC (1) between MOON and baseline post.
process methods for CIFAR-10. Our MOON leads to smaller FPR-95 and larger AUROC each OOD data, showing its ability to improve reliability on OOD data.
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACes365
FxiOOD
MSP
62.01
60.69
61.35
58.59
51.87
59.89
57.64
57.00
NOOX + MSP
61.01
56.87
58.94
40.91
48.25
51.31
57.56
49.50
ODIN
59.09
59.06
59.07
36.23
67.9
51.10
50.51
51.41
MOON +DDIN
48.47
42.24
45.36
8.10
33.30
32.58
40.64
28.65
EBO
51.46
45.02
48.24
44.50
44.94
48.32
41.88
44.91
HOO++EBO
43.35
14.98
21.34
36.75
38.51
87.11
86.62
86.87
89.91
90.88
88.72
89.03
89.64
AUROC
NDOX+ MSP
89.06
90.61
89.83
94.11
93.83
92.30
89.97
92.55
ODIN
77.68
77.33
77.51
90.91
73.32
80.70
82.55
81.87
MOOK ODIN
87.91
90.26
89.08
98.34
98.37
92.97
90.17
93.71
EBO
86.15
88.58
87.36
90.59
88.39
86.85
89.60
$8.86
MOON + ERO
90.05
92.36
91.20
97.30
96.15
93.34
92.04
94.71
A.3 OOD Detection in Dense Training
In this section, show the broad applicability of our MOON, we show our MOON can improve OOD detection in dense training. The results of baseline methods are based the scores reported in (Yang et al. 2022
For the model trained on CIFAR-10, we examine its OOD detection ability on two near OOD data (i.e.. CIFAR-100, TIN) and four far OOD data (i.e., MNIST, SVHN, Texture, Places365). As shown in Table II the columns named 'NearOOD" and "FarOOD" represent the average detection scores of the near OOD dala and the far OOD data, respectively. We can see that our MOON provides smaller FPR-95 and larger AUROC for different near and far OOD data compared MSP, ODIN, and EBO, where the reduction of FPR-95 can be up to 77.6% and the improvement of AUROC can be up to 27.3%. This demonstrates that our MOON can perform OOD detection more effectively compared to the original methods.
For the model trained on ImageNet-2012, we examine its OOD detection ability on four near OOD data (i.e., Species, iNaturllis, OpenImage-O, ImageNet-O) and two far OOD data (i.e., Texture, MNIST). More details about the OOD data are in Appendix B.T As shown in Figure g the red and blue bars represent our MOON and MSP, respectively. We can see that our MOON provides larger AUROC and AUPR, and smaller FPR-95 almost on all of the OOD data of ImageNet-2011 showing its effective OOD detection.
000 Data
0.00 Dasa
CO0 DaCa
(a) AUROC (1)
(b) AUPR (1)
(c) FPR-95 (4)
Figure 8: Comparison of OOD detection by AUROC (t), AUPR (t), and FPR-95 (4) between MOON+MSS and MSP for 2m222. Our MOON leads to larger AUROC and AUPR, and smaller FPR-95 on each OOD data of TmageNet-2012,, showing its ability to improve reliability OOD data.
21

---

Published in Transactions on Machine Learning Research (03/2024)
Table 12: More Ablation Studies for wy by AUROC (1) in MOON with 95% sparse ResNet-18 for CIFAR- R-
CIFAR-100
TIN
NEAROOD
MNIST
SVHN
TEXTURE
PLACEs365
FarOOD
10 0.1
83.57
85.25
84.41
86.49
86.47
86.35
54.73
85.76
AUROC
10y 0.5
86.25
87.07
87.66
93.46
91.02
87.80
85.11
89.35
WEy 1.0
88,04
89.39
88.71
94.70
93.35
89.86
88.51
91.62
Wy 2.0
86.45
87.77
87.11
89.96
91.44
89.24
86.05
89.17
Wy 84.78
85.84
86.31
94.69
91.44
86
84.65
89.36
A.4 More Ablation Studies for w
We add additional ablation studies for wy to test the sensitivity of wy. As summarized in Appendix G, we set Wy = for CIFAR-10. We add experiments of 95% sparse ResNet-18 with wy = 0.., 2, 2,4. 4. The AUROC
results are summarized in the table below. We also include them in Appendix A.4. As shown in the Table 12 when we increase wy from 0.1 to 4, OOD reliability first increases and then decreascs after While OOD reliability may be degraded when wy is loo small or too large, OOD reliability is not degraded much when wy is between 0.5 and 2. The proper wy value be found with simple hyperparameter tuning.
A.5 Additional Incremental Ablation Study
We add an incremental ablation study in which each component of the method is incremented to the previous component to better intuitively understand the utility of each component. We train 95% sparse ResNet-18 18 on CIFAR-10 with MOON + RigL. We find that the OOD's ' reliability decreases as we gradually remove one of its components.
For MOON w/o LM, we set wy as 0.5, 0.001, and 0.00001, respectively, to incrementally remove the loss modification component. The AUROC (%) results for MOON w/o LM are summarized in the Table 13 Wy 1 refers to our MOON.
Table 13: Additional incremental ablation study for loss modification (LM) by AUROC (t) in MOON with 95% sparse ResNet-18 18 for CIFAR-10.
My
10
0.001
Mfy
NEAROOD
88.71
86.97
86.32
85.76
FAROOD
91.62
90.18
89.41
87.76
For MOON w/o AT, we shorten T. to 3, 2, and 1, respectively, to incrementally remove the auto tuning component. The AUROC (%) results for MOON w/o AT are summarized in the Table 14 T, = refers to our MOON.
Table 14: Additional incremental ablation study for auto tuning (AT) by AUROC (t) in MOON with 95% sparse ResNet-18 for CIFAR-10.
T:
5
T.
3
T.
2
T.
NEAROOD
88.71
85.51
85.33
85.17
FAROOD
For MOON w/o VT, we shorten the length of epochs using the voting scheme the end of training to 10, 5, and 2 epochs, respectively, incrementally remove the voting scheme component. The AUROC (%) results for MOON w/o LM are summarized in the Table 15 Vote time 20 refers to MOON.
22

---

Published in Transactions on Machine Learning Research (03/2024)
Table 15: Additional incremental ablation study for voting scheme (VT) by AUROC (f) MOON with 95%
sparse ResNet-18 18 for CIFAR-10.
VOTE
20
10
VOTE TIME
VOTE TIME
NEAROOD FAROOD
84.77
86.70
B Appendix: Additional Details about Experiment Settings
B.1 OOD Benchmark Description
The choice of out-of-distribution (OOD) Benchmarks is according to OpenOOD YYnng et al. 2022). The distribution here refers to the "label distribution'. Following the common practice of constructing OOD detection benchmarks, we treat the entire dataset as in-distribution (ID) and then gather several datasets that are not correlated with any ID category as OOD datasets. We use four OOD benchmarks, which are named after ID datasets, including MNIST (Deng) 2012), CIFAR-10 (Krizhevsky al. 2009). CIFAR- 100 (Krizhevsky al., 2009). and ImageNet (Russakovsky et al. 2015). As mentioned by OpenOOD (Yang ct al. 2022), compared to the ID dataset, the nOD dataset has only semantic shifts, while the far OOD further contains significant covariate (domain) shifts. The detailed description of each benchmark is follows.
For MNIST (Deng. 2012 it is a 10-class handwriting digit dataset with 60k images for training and
10k for test. Its near-OOD datasets include NOTMNIST (Yang et al. 2021b) and MNSS (Hsu
et al. 2020). The far-OOI) datasets include Texture (Ahmed & Courville 2020). CIFAR-10 TTxx 2002) and
Tiny ImageNet (Han al. 2022 and Places-365 (Guo al. (017)).
For CIFAR-10 Krizhevsky et al. 2009), it is 10-class dataset for general object classification with 50k training images and 10k for test. Its ncar-OOD datasets include CIFAR-100 (Chandola et al. 2009) and Tiny IinymmagNNet (Han et al. 2022), where 1,207 images are removed from Imagenee since they belong to
CIFAR- 10 classes (Yang et al. 2022). The far-OOD datasets include MNIST (Deng) 2012). SVHN (Netzer et al. 2011), Texture (Ahmed & Courville 2020), and Places365 (Guo et al. 2017 with 1,305 images removod due semantic overlaps.
For CIFAR-100 Krizhevsky et al. 2009 it is 100-class dataset for general object classification with 50k training images and 10k for test. Its near-OOD datasets include CIFAR-10 (Tax 2002) and TinylmageNet (Han) ct al. 2022,, where 2,502 images are removed from TinyImageNet due to the overlapping semantics with
R-FAR-100 classes (Yang et al., 2022). Its far-OOD include MNIST (Deng 2012). SVHN (Netzer et al.)
2011h, Texture (Ahmed Courville 2020), and Places365 (Guo et al. 2017 with 1,305 images removed due semantic overlaps.
For ImageNet, it is large-scale image classification dataset. with 1000 classes. OpenOOD (ann al. 2022) build the OOD dataset, where they use 10k subset of Species (orralla et al. 2008) with 713k images, Naturalist (Shorten & Khoshgoftaar
2019) with 10k images, ImageNet-O (Li et al. 2021 with 2k images, and OpenImage-O) (Sun et al. 2022) with 17k images as near-OOD datasets, and use Texture (Ahmed &
Courville 2020), MNIST (Deng 2012) as far-OOD. All images belonging to the ID classes are deleted.
B.2 Computing Resources
The training and evaluation are mainly on NVIDIA Quadro RTX 6000. For CIFAR-10 and CIFAR-100, training ResNet-18 18 for 100 epochs with our MOON using GPU takes around 1.5 hours. For ImageNet- 12, training ResNet-50 for 100 epochs with MOON using 1 GPU takes around 6 days.
23

---

Published in Transactions on Machine Learning Research (03/2024)
C Appendix: Theoretical Analysis
In this section, we provide detailed proof of our insights in Section the main manuscript. The proof is based on Assumption C.I. which describes the characteristic of the feature space.
Assumption C.1. (Gaussian Mixture Feature Space): The feature mapping function h maps the input ID data to a Gaussian mixture V (M,, Mi) + v2N(uz, E2). Specifically, when y = 1, we have h(x) N(,, En). And when y 2, have f(x) ((22, E2).
C.1 Insight 4.4: Unreliability
(Unreliability) suppose we have (11, from class and D2 = {(z, v):11(()) h(x1))l <. 22 from class 2. Then, unreliability can occur around h(x1).
Proof:
The final output probability composition of the feature mapping function and the softmax classification function g- As know, the softmax function is Lipschitz continuous with Lipschitz constant bboundd by 1.
Because of the definition of D2, we can know that
l1g(h(x)) g(h(x1))l cl(((z) - h(x1)l-
(4)
Suppose g(h(ar)) (111, and g(h(xz2) = {221, for any E D2. Then can know that Il P21 S Ce.
(5)
Without loss of generality, suppose xı predicted correctly, i.c, have Pıi 0.5. Then, we have
P2i 2 P11 ce > 0.5, c < 0. Then, with small 6, the samples in D2 will have P21 > 0.5 and be incorrectly predicted to class Thus, for any eo < e, we can define D2(e0) and have its expected accuracy as:
(6)
D2(e0) = {zz y): [1((z) - h(x1))l < co. 2}, EDi(co) [1 = y}1 = EDI(cO) [1 = 2
(7)
(8)
In this case, suppose the samples near 31 are D(co). And D(eo) can be divided into (e(e)) and D2(50). D; ((., [|(x)) h(x1)ll < co. y = i}. The ratio of the size of D2 and D2 can be approximated by
D1 w1 M(h(z1): pı, En)
D2 W2 (((((z); p2, M2) where 11 + w2 =
(9)
Then, we can calculate the unreliability level following the Definition 4.2:
Ep(ol max ((((())) Ex]l - EpCe) =
cE(1,2)
= -i EDittl max ((h(r)) Ex)l En, yt1 cE(1.2)
+ w0
EDa max ((h(z): Me: Ex)l ED.(C) [10 = =yyl cE(1.2)
wi EDi(el max N(h(T): -1) + w2 Epatei max (() (E(1.2)
(E(1.2)
2 Pi1 CE
(10)
Then, we choose ø1 which is not very close to in such that wn Pn 17. Then, will have:
Edol max W(h(z); Ed] EDG = yHl >> C w >
(11)
cE(1.2
The proof is complete.
24

---

Published in Transactions on Machine Learning Research (03/2024)
C.2 Insight 4.6: Hard-ID Reliability
(Hard-ID Reliability) Suppose we have the same (I1,21) and D2 as in Insight 4.4. II the model is trained with our MOON method and the extra dimension successfully stores the unknown information, the unreliability can be solved, i.c.,
Epgol max ((h(z); Ex)] - 1 = y11 < n cE(1.2)
(12)
Proof:
Following the notations in the proof of Insight 4.4, we know that unreliability around æ1 has two parts. One is from D and the other is from D2. And the majority of the unreliability comes from Dz. Thus, in order to reduce the discrepancy between confidence and accuracy, we need to mainly reduce the part of D2, which is:
Ep max (V((()): Ee)l - (n = ytl cE(1.2)
(13)
Suppose the two classos of data are balanced. Since æ1 is predicted correctly, we can know that wi > W2 and Dii>> (D2..
In this case, if the model can find a better feature mapping that does not have such a discrepancy for any set of data D, then there is no safety problem. However, finding such a suitable feature mapping h is too difficult, which is usually the case, can only change the softmax classification function g. In sparse training, the update route is cut off, generating spurious local optimization, leading to more challenging exploration of the weight space and thus inability to find suitable feature mapping h.
If we choose the widely-used cross entropy loss L which in our case will take the form as below:
L(D) = L(D1) + L(22)
Σ
log Pai
log Pre
iED1
ieDz
= Σ
logPa1 P-1 -
Σ
log(1 - Pa)
(14)
iED;
iED;
Since is difficult to find better mapping h, we assume is fixed in this case. If c is small, then the gradient of samples in D(e) will be similar to that of Φ1. Suppose g(h(21)) (pi. p5). Then, we can have:
L(D) = Σ
log Pai - -E log(1 - Pa))
(15)
iED
iEDs
- D, log pi |D2] log(1 Pa)
(16)
And the gradient of loss will be:
(D22 + (D1l) pi |
VL(D) x
(17)
pi rf)
[Di Thus, to reduce the loss, pt will stuck at [Da and can not reduce the discrepancy, implying severe unreliability in D.
If we choose our MOON there will be three dimensions in the output probability and g(h(x1)) = (pi.pi.Po p.). And the new loss will take the form as below:
L(D)= L(D)) L(D2)
Σ
logPa Pii
Σ (1
w
Pc2
1 + tpps
tED
iEDz
Σ iED
log Pii
Σ (1
w + uPis
log(1 - Pai)
DEDs
- Di log
[12
log(1 -pi)
(18)
wp
25

---

Published in Transactions on Machine Learning Research (03/2024)
Then, the gradient of the loss will be below
VI(D) (111l + |D2
D2l) pi - Dil
14wp3
(19)
pi pf)
We can see that in order to minimize the loss, we update not only pi but also P3; thus providing more degrees of freedom for the model to learn reliable predictions. Instead of increasing pi and being stuck in an unreliable state, the model can increase Pa when it encounters hard-ID samples. Suppose we have nonzero ps after the parameter update. We can calculate the new unreliability level following the Definition 4.2:
ED(e) max ((h(z)))e Ee)l ED(O) =yll cE(1.2)
=w EDr max N(h(z)) Ex)] - ED (0 = =w}]] (E(1.2)
u2 EDi max ((h(z)) Mo: 2x,1 EDI 11 =n1] 6E(1.2)
wi (o) max ((((2)) Me: Σe)l cE(1.2)
-1) --]1-1)
Epstol max Ee, cE(1.2)
w2
(p1 + - Pg 1) w2(111 CE p3))
PM + p9
(20)
Since the extra dimension (i.e., p5) successfully stores the unknown information, we can assume that p3 is reasonably large, i.e. p3 Pm1 w1 17. Then, the unreliability level can be reduced to a value less than
y
Epol max W(h(z); Ee)l ED [+ y}l $ P11 P3 w1 < n (E(1.2)
(21)
The proof is complete.
C.3
Insight 4.8: OOD Reliability
(OOD Reliability) Suppose we achieve Hard-ID Reliability in Insight 4.8 with MOON method, can have lower confidence OOD data, implying stronger OOD detection.
Proof:
We know that hard-to-detect OOD data Dy usually have fcatures close to those hard ID data D2, which is similar the scenario described Insight 4.4. Otherwise, the features of the hard-to-detect OOD data are very different from those of hard ID data, then the OOD data becomes easy to detect without the OOD unreliability.
Thus, these hard-to-detect ID samples can be viewed as pseudo-OOD data. And the (h(x2): E D2) are close to {M(zs): Ds}. Then we can follow the proof of Insignt 4.6 and show that:
Epa()) max ((((()):)e, Ee)] is reduced. cE(1.2)
(22)
In this way, we able have smaller confidence in the hard-to-detect OOD data D3, which helps to enhance the OOD reliability. This describes how extract unknown information from hard ID data, explaining the outhier-epposure-free property of MOON.
Appendix:
Limitations of Our MOON
D.1 More General Reliability
Despite the enhanced OOD reliability achieved in sparse training, MOON does not address this matter comprehensively from broader reliability perspective, including robust generalization (iilaa & Najafirad
26

---

Published in Transactions on Machine Learning Research (03/2024)
2020
O Ozdenizci & Legenstein 2021 Chen et al. 2022al and adaptation [Emam et al. 2021 Alayrac et al.
2022
Hu et al. 2022), which are equally vital aspects of reliability and deserve further investigation (ran
et al. 2022).
D.2 Reliability in Large Generative Models
Additionally, the effectiveness of MOON remains unexplored in the context of widely employed large generative models (Ouyang et al. 2022 Croitoru et al. 2023) On the one hand, as model size expands, novel changes in model characteristics may arise (Wei et al., 2022b). On the other hand, given the prevalence of non- classification tasks within LLMs, the exploration of how evaluate their reliability remains an uncharted domain.
E
Broader Impact
This work likely encourage other future work on more general reliability in sparse training achieve efficiency and comprehensive reliability in real-world DNN deployments.
In addition, large generative models develop, both computational efficiency and decision reliability important aspects of their application. This work can increase the progress of efficient and reliable large generative models.
F FLOPs Analysis
In this section, add FLOPs analysis to MOON. We use 80% sparse ResNet-50 Image-Net-2011 as example. The training FLOPs using RigL are about 7.4c17 (Evci et al. 2020 For our MOON, the increased FLOPs come from three components, i.., loss modification, auto tuning, and voting scheme.
(i) For loss modification and auto-tuning.
In the early stage where w 0, we have about 1.3e5 FLOPs in each iteration because of the computation of B.
After the early stage, from the calculation of and the new loss, have about 2.6e5 FLOPs in each iteration.
Thus, the loss modification and auto tuning introduce additional 2.3e15 FLOPs.
(ii) For voting scheme,
We start doing weight averaging in 80% of the training epoch and have about 1.8e9 FLOPs from the weight averaging. In each epoch after the 80% training epoch, we also have about 2.4e15 FLOPs from the batchNorm buffer updates. Thus, the voting scheme introduces an additional 4.8e16 FLOOP.
Overall, our MOON introduces an additional 5.0e16 FLOPs, which about 6.8% of the original RigL's FLOPs. The main cause of the increase in FLOPs is the update of the batchNorm buffer. To further reduce the increase of FLOPs, we can reduce the update frequency of the batchNorm buffer.
We also conduct experiments to compare the training time using our MOON and baseline methods. We train 80%, 90%, 95%, and 99% sparse ResNet-18 on both CIFAR-10 and CIFAR- using MOON + RigL and RigL. We denote the training time of our benchmark method, RigL, overhead our MOON adds the training process. The results are summarized in the Table 16 17 see that MOON adds less than 5% overhead.
help understand exactly what
We can
27

---

Published in Transactions on Machine Learning Research (03/2024)
Table 16: Training time comparison using our MOON + RigL, and RigT. with 80%, 90%, 95%, and 99% sparse
ResNet-18 18 on both CIFAR-10.
CIFAR-10 (80%)
CIFAR-10 (90%)
CIFAR-10 (95%)
CIFAR-10 (99%)
RIGL
KOOK + RICL
1.015
1.012
1.039
1.016
Table 17: Training time comparison using our MOON RigL, and RigT. with 80%, 90%, 95%, and 99% sparse
ResNet-18 18 on both CIFAR-100.
CIFAR-10 (80%)
CIFAR-10 (90%)
CIFAR-10 (95%)
CIFAR-10 (99%)
RiGL.
woOk + RicL
1.099
1.046
1.048
1.013
G Hyperparameter Tuning
The hyperparamcters of the baseline method follow their original settings (Evei et al. 2020 Yang al.
2022 The hyperparameters of our MOON are described as follows.
(i) For the learning rate, following (Ynng et al. 2022 we use the cosine annealing scheduler, where the maximum learning rate is set to 1.2 and the minimum learning rate is set to 0.08.
(ii) For the fraction of weights removed and added (pruning ratio) in each sparse pattern update, we follow the original settings in (Evci et al. 2020: Sundar & Dwaraknath 2021
The initial fraction is 0.3.
This fraction decays in the mask update step by cosine annealing.
The score is set to zero after 70% of the training cycles.
(ii) For the weight distribution, follow the original scttings in Evei et al. 2020 Sundar & Dwaraknath 2021 and initialize the mask by Erdos-Renyi-Keroo (ERK).
(iv) For wy and
For MNIST, we set wy and as 0.01 and 64, respectively.
For CIFAR-10, we set wy and r as 1 and 64, respectively.
For CIFAR-100, we set wy and r as and 64, respectively.
For Nee2212,, we set wy and r as 0.001 and 64, respectively.
H
Feature Visualization
One of the assumptions on which our theory basod that the feature distribution of a DNN feature extractor is Gaussian mixture. Additional experiments are added support this assumption. Specifically, train ResNet-18 CIFAR-10 with 80%, 90%, 95% and 99% sparsity and collect features before the last layer. Then, to visualize the feature distribution, we use the PCA method to reduce the dimensionality and plot the histogram of the first component.
As shown in Figure 9 12, the feature distribution shows a high density in the center and then decreases towards the two sides across all classos and sparsity levels. This characterization consistent with Gaussian distribution. Therefore, the overall distribution for all classes can be approximated by a Gaussian mixture distribution, supporting assumption.
28

---

Published in Transactions on Machine Learning Research (03/2024)
Cloa
Cass
dess 2
170
200
1250
000
620
3800
000
-0.3
120
Class 0
(b) Class
(c) Class
300
1230
1230
40
330
652
(d) Class 3
(e) Class 4
(f) Class 5
3.0
1300
1350
1200
250
2.
0.2
(g) Class 6
(h) Class
(i) Class S
class 9
823-
Class 9
Figure 9: Feature distribution visualization on CIFAR-10 for 80% sparse ResNet-18. The feature distribution shows a high density in the center and then decreases towards the two sides across all classes.
29

---

Published in Transactions on Machine Learning Research (03/2024)
Cloa
Cass
dess
90
200
-1.
-2.0 -2.4
125
Class 0
(b) Class
(c) Class
240
200
790
400
140
0.5
38
(d) Class 3
(e) Class 4
(f) Class 5
1490
12:0
323
460
355
0.5
5.5
0.68
1.53
6.79 195
15%
67
(g) Class 6
(h) Class
(i) Class 8
class 9
2030
1350
130
3800
350
-0.35
Class 9
Figure 10: Feature distribution visualization on CIFAR-10 for 90% sparse ResNet-18. The feature distribution shows a high density in the center and then decreases towards the two sides across all classes.
30

---

Published in Transactions on Machine Learning Research (03/2024)
Cloa
Cass
dess
1290
60
630
330
240
-1.4
-25
-82
Class 0
(b) Class
(c) Class
2009
1200
1890
40
00
(d) Class 3
(e) Class 4
(f) Class 5
14.0
1800
312
1200
300
3000
120
353
1.5
4%
1.50
1.2%
65
1.0
(g) Class 6
(h) Class
(i) Class S
class 9
330
Class 9
Figure 11: Feature distribution visualization on CIFAR-10 for 95% sparse ResNet-18. The feature distribution shows a high density in the center and then decreases towards the two sides across all classes.
31

---

Published in Transactions on Machine Learning Research (03/2024)
clos
cass
des
200
200
3030
630
79
1030
300
239
Class O
(b) Class
(c) Class
373
120
300
180
790
230
150
11
(d) Class 3
(e) Class 4
(f) Class 5
200
210
1200
300
300
300
430
2.5
20
65
24
(g) Class 6
(h) Class
(i) Class S
class 9
830
Class 9
Figure 12: Feature distribution visualization on CIFAR-10 for 99% sparse ResNet-18. The feature distribution shows a high density in the center and then decreases towards the two sides across all classes.
32


Reviews:
[{"review_id": "20se48vtDi", "reviewer": "Reviewer_Ww7e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Exhaustive empirical results on datasets with varying complexity.\n2. The method is post-hoc and modular and therefore easy to integrate with existing strategies for sparse training (as is also illustrated in the experiments)\n3. I found the presentation clear in the sense that the assumptions for the theoretical intuitions are clearly stated with proofs in the Appendix. The experimental settings are also clearly described.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors explore the phenomenon of reliability of OOD detection in DNN models trained with sparsity constraints.The authors hypothesize that sparse DNN models might suffer from OOD unreliability due to their constrained weight space exploration (as many of the weights go to zero leading to limited exploration in the weight space). The authors then present a uncertainty aware loss formulation for training sparse DNNs by introducing an extra dimension in the softmax classification layer of DNNs which is representative of the uncertainty in the output prediction. Based on the training stage (early or late), an auto-tuning strategy for tuning the weight for this loss is also presented. Finally, the authors combine multiple models (using weight averaging) from different training epochs to get a final prediction.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Questions:\n\n1. While the theoretical insights are intuitive, as a non-expert in this field, I wanted to know about the practical usefulness of Assumption 4.1. To assume the distribution of features from a DNN feature extractor to be a gaussian mixture with the same number of classes seems a bit simplified. It would be great if the authors can comment on this. It seems like this assumption can be validated empirically by clustering the last layer features for a pretrained DNN?\n\n2. For weight averaging, what strategy does the method use? Is it a traditional uniform averaging or weighted averaging of some kind (for instance, with more preference for models at later epochs.) This should be clarified in Section 3.3.\n\n3. From the ablation experiments in Section 5.7, it looks like the method suffers most without voting. Is there any intuition behind this? Also I would encourage the authors to present an incremental ablation where each component of the method is added incrementally over the previous component which could provide more intuition on the utility of each component.\n\nMinor weaknesses\n\n1. “However, it has been shown that DNNs tend to focus on the known due to the close nature of the training process and ignore the rest”-Missing citations here.\n\n2. Typo: Under “Unknown-aware Loss Modification: Specifically, in the new loss, we add a larger weight”, there is a mismatch in the expression for the weight from Eq 1 and the expression for the weight in the inline equation in this line.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "uieD62cJrE", "reviewer": "Reviewer_KgVE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of adding an extra label that means \"unknown\" and reweighing the samples based on p_{unknown} is intuitive.\n2. Performance of MOON is good according to the experiments.\n\n\nWeaknesses:\n\n1. K+1-ary softmax has been adopted in calibrated learning, such as \"[ICML2020] Consistent Estimators for Learning to Defer to an Expert\" and \"[ICML2022] Calibrated Learning to Defer with One-vs-All Classifiers\". The author has not mentioned these relevant works.\n2. It is not clear how the theoretical insights in Section 4 associates with the MOON algorithm. For example, I cannot see why the author picked (1 + w_i / (1 + w_{K+1})) as the weights by reading Section 4.\n3. I cannot understand the proof of the Insight 4.4 (Appendix C.1) and Insight 4.6 (Appendix C.2). Specifically, in Appendix C.1, I cannot understand why equation (6) is correct. I can see that $p_{21} \\geq p_{11} - c \\epsilon$ . Since $p_{11} > 0.5$ , this gives $p_{21} \\geq p_{11} - c \\epsilon \\gt 0.5  - c \\epsilon$ . I cannot see why $p_{21}$ is larger than 0.5. The equation (14) in Appendix C.2 does not seem to be correct. $\\log p_{i2}$ should be equal to $\\log (1 - p_{i1})$ and does not equal to $1 - \\log p_{i1}$.\n4. The sensitivity of w_f is unclear. The author reported the hyper-parameter choice of w_f in Appendix G but there is no ablation study.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed a new training loss for multi-class classification called MOdel-agnostic unknOwN-aware sparse training (MOON). Training with MOON will boost the network's ability in detecting out-of-distribution (OOD) samples. MOON augmented the label space to include an extra label that means \"unknown\". This turns the K-ary classification problem to K+1-ary classification problem. To ensure the model can assign proper probabilities to the \"unknown\" class, MOON adjusts the cross-entropy loss by assigning a larger weights $(1 + w_i / (1 + w_{K+1}))$ to hard samples that model gave incorrect predictions. In addition, the author proposed to gradually adjust the weights during training to give the model enough time to learn the easy samples. MOON can be combined with other sparse training and OOD detection methods. Experiments show that MOON consistently boosts the OOD detection performance.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The author needs to mention related work in calibrated learning and also compare with those methods.\n2. Clarify the connection between the theoretical insights and the MOON algorithm.\n3. Clarify the potential errors in the proof.\n4. Conduct ablation study on the w_f.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sb5nGChjPV", "reviewer": "Reviewer_yuD9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed MOON method effectively leverages unknown information under sparse constraints to improve real-world reliability in DNNs\n\n- The K+1-way formulation allows the model to distinguish between known and unknown information, and the unknown-aware loss modification provides an effective guide for weight space exploration during sparse training.\n\n- The theoretical insights provided in the paper offer a deeper understanding of the reliability issues and the effectiveness of the MOON method in addressing them.\n\n- The experimental results demonstrate the improved OOD detection capabilities of the MOON method.\n\nWeaknesses:\n\n- The explanation of key algorithm 1 requires further clarification. Specifically, it is unclear whether the auto-tuning of $w$ occurs after each epoch or during each sample's training. If it's the former, the meaning and role of the probabilities $p_i, 1\\le i \\le K+1$ need clarification. Additionally, the initial factor $r$ is mentioned but not updated or explained in the context of the algorithm. The expression $L(p_1,\\cdots,p_{K+1})$ is also confusing, as $L$ is initially described as operating on $x_i$ in equation (1).\n\n- The paper repeatedly claims that the MOON method introduces minimal cost compared to other methods, but this assertion lacks empirical support. It would be beneficial to include experiments or data showing the specific overheads MOON adds to the training process and how these compare to other existing methods.\n\n- The OOD setting discussed seems limited to classification problems, with the unknown-aware loss constrained to cross-entropy loss. It would be valuable to explore whether there's a more general framework capable of incorporating the high-level concepts of this method, applicable to a broader range of contexts.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission presents a new sparse training method called MOON, which aims to improve the reliability of deep neural networks (DNNs) in detecting out-of-distribution (OOD) data. It identifies the problem and proposes to leverage unknown information under sparse constraints to effectively guide weight space exploration to an OOD-reliable state. Experiments show that MOON can effectively enhance the reliability of OOD detection in DNNs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "see the weaknesses part above.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the reliability of sparse training from the out-of-distribution (OOD) perspective. The paper reveals that sparse training exacerbates OOD unreliability and proposes a training method to enhance sparse training for detecting OOD data. The training method includes the modifying of the loss function by introducing an extra dimension in the softmax layer, incorporating an auto-tuning strategy and a voting (model averaging) scheme. The effectiveness of the methods is empirically evaluated across a variety of dataset, model architectures and sparsity levels, alongside theoretical insights.

The paper received recommendations of one accept, one leaning accept and one leaning reject. Overall, reviewers acknowledge several strengths of the paper:

1). Exploring a new area, i.e. reliability of OOD detection with sparse training.

2). The proposed methods are sound and intuitive.

3). The empirical evaluation is extensive, demonstrating consistent improvement in OOD detection reliability across various benchmarks.

Reviewers raised two main concerns: 

1). The provided theory seemed overly simplified and somewhat confusing. 

2). The novelty of the paper is somewhat limited as it primarily used existing methods for sparse training. 

During rebuttal, the authors provided clarification on the theory and additional data to justify the assumptions. The authors also conducted additional experiments, including ablation experiments and an assessment of training overhead, to further justify their claims and understanding of the methods.

In summary, while the novelty is somewhat limited and the proposed method at this stage can be only used for classification problems, this paper is technically solid and potential benefits in the domain of sparse training. Therefore, I recommend acceptance.
Decision: Accept (poster)