SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[36] Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. MEST: accurate and fast memory-economic sparse training framework on the edge. CoRR, abs/2110.14032, 2021.
[37] Jize Zhang, Bhavya Kailkhura, and T. Yong-Jin Han. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning, 2020.
[38] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask, 2019.
A Summary of Notations 15
B Robust Loss Optimization in DRO 15
B.1 Robust Loss Optimization 15
B.2 Hyperparameter settings 16
C Proofs of Main Theoretical Results 16
C.1 Proof of Lemma 1 16
C.2 Proof of Theorem 2 17
D Experimental Details and Additional Results 18
D.1 Detailed Dataset Description 18
D.2 Hardware Details for Experimentation 19
D.3 Single-view and Multi-view Examples 19
D.4 Additional Result on Cifar10 and Cifar100 19
D.5 Additional Baseline Results on Sparse Training Methods 20
D.6 Additional Baseline Results on TinyImageNet 20
D.7 Performance from Ensemble Members 20
D.8 Comparison with Common Calibration Techniques 20
D.9 Ablation Study 21
D.10 Parameter Size and Inference Speed 23
D.11 Diversity on Sparse Sub-networks 23
D.12 Qualitative Analysis 23
E Broader Impact, Limitations, and Future Work 24
E.1 Broader Impact 25
E.2 Limitations and Future Works 25
F Source Code 25
Organization of Appendix
In this appendix, we first present a table summarizing the major notations used by the main paper in Appendix A. Next, we provide detailed information about the training process and hyperaprameters setting B. We provide the detailed proof of Lemma 1 and Theorem 2 in Section C. After that, we provide additional experimental details and results in Appendix D. Finally, we discuss the broader impacts, limitations, and future work of our DRE technique in Appendix E. The link to the source code can be found in the end of the Appendix.
Table 5 below shows the major notations used in the main paper. We further assign each notation into one of four major categories: dataset, DRO formulation, sparse training, and theoretical results.
Table 5: Symbols with Descriptions.
Symbol Group	Notation	Description
Dataset	X	Set of training images
Y	Set of training class labels
C	Total classes
ŷ	Predicted class label
N	Total number of training samples
D	Dimensionality of each data sample
DRO	Df	f-divergence
η	Parameter controlling size of uncertainty set in DRO framework
zn	Weight associated with nthdata sample
Sparse Training	M	Number of sparse sub-networks
K	Density of the given network
Θ	Parameter associated with given neural network
p	Confidence associated with predicted class
l(xn, Θ)	Loss associated with nthdata sample
Theoretical Results	β	Learning rate of the given network
P	Total number of patches in each data sample
d	Dimensionality of each patch
vc,l	Major lthfeature associated with class c
L	Total number of features in each class class
DS	Collection of single-view data samples
DN	Collection of multi-view data samples
U	Collection of features
H	Number of convolution layers
Fc(x)	Logistic output for the cthclass for the data sample x
Pvc,l	Collection of patches containing feature vc,l in sample xj
SOFTc	Softmax output for class c
B Robust Loss Optimization in DRO
In this section, we first provide a detailed description on how we optimize the robust loss function in (1). We then explain how to set the uncertainty set by choosing a proper hyperparameter.
B.1 Robust Loss Optimization
The optimization problem specified in (1) involves an inequality constraint so directly solving it may incur a higher computational overhead. Therefore, we consider a regularized version of the robust loss to train each base learner by using the following loss:
$$
\mathcal {L} ^ {\text {R o b u s t}} = \max  _ {\mathbf {z} \geq \mathbf {0}, \mathbf {z} ^ {\top} \mathbb {1} = 1} \sum_ {n = 1} ^ {N} z _ {n} l _ {n} (\Theta) - \lambda D _ {f} \left(\mathbf {z} | | \frac {\mathbb {1}}{N}\right) \tag {9}
$$
where  $l_{n}(\Theta) = l(\mathbf{x}_{n},\Theta)$ . Solving the above maximization problem leads to a closed-form solution for  $\mathbf{z}^*$  as shown by the following lemma:
Lemma 3. Assuming that  $D_{f}$  is the KL divergence, then solving (9) leads to the following solution
$$
\mathcal {L} ^ {\text {R o b u s t}} = \sum_ {n = 1} ^ {N} z _ {n} ^ {*} l _ {n} (\Theta) \tag {10}
$$
where  $z_{n}^{*}$  is given by
$$
z _ {n} ^ {*} = \frac {\exp \left(\frac {l _ {n} (\Theta)}{\lambda}\right)}{\sum_ {j = 1} ^ {N} \exp \left(\frac {l _ {j} (\Theta)}{\lambda}\right)} \tag {11}
$$
It can be verified that there is a one-to-one correspondence between  $\eta$  in (2) and  $\lambda$  in (9). Given their roles in the corresponding equations, a large  $\eta$  implies a small  $\lambda$  and a small  $\eta$  implies a large  $\lambda$ .
B.2 Hyperparameter settings
The hyperparameter in the regularization term is chosen based on the difficulty of a dataset. Specifically, for DRE, we always consider the  $\lambda \rightarrow \infty$  for the first sparse sub-network which is equivalent to Expected Risk Minimization (ERM). For the second and third sub-networks, we choose this hyperparameter based on the difficulty of data samples. It should be noted that we need to set higher  $\lambda$  values for more difficult datasets as difficult samples are more common on those datasets. Using this notion, for Cifar10, we choose small  $\lambda$  values so that the model can focus on the difficult samples that are few. For this, we choose  $\lambda = 10$  for the second sparse sub-network and  $\lambda = 500$  for the third sparse sub-network. Considering Cifar100 is more difficult, we would have more difficult samples and therefore higher  $\lambda$  value is preferred. For this, we choose  $\lambda = 50$  for the second sparse sub-network and  $\lambda = 500$  for the third one. In the case of TinyImageNet, we have many difficult samples and therefore we choose relatively large  $\lambda$  values. Specifically, we choose  $\lambda = 100$  for the second sparse sub-network and  $\lambda = 1,000,000$  for the third sparse sub-network.
C Proofs of Main Theoretical Results
In this section, we provide detailed proofs of the theoretical results presented in the main paper.
Proof. For  $y_{n} = c$ , with respect to data sample  $\{\mathbf{x}_n, y_n\}$ , the gradient can be evaluated as
$$
- \nabla_ {\Theta_ {c, h}} l (\Theta ; \mathbf {x} _ {n}, y _ {n}) = [ 1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {n})) ] \sum_ {p \in [ P ]} \operatorname {R e L U} [ \langle \Theta_ {c, h}, \mathbf {x} _ {n} ^ {p} \rangle ] \mathbf {x} _ {n} ^ {p} \tag {12}
$$
Assume that the given sample has a major feature  $\mathbf{v}_{c,l}$ , taking dot product with respect to  $\mathbf{v}_{c,l}$  on both side of (12) leads
$$
\left\langle - \nabla_ {\Theta_ {c, h}} l \left(\Theta ; \mathbf {x} _ {n}, y _ {n}\right), \mathbf {v} _ {c, l} \right\rangle = \left[ 1 - \mathrm {S O F T} _ {c} \left(F \left(\mathbf {x} _ {n}\right)\right) \right] \sum_ {p \in [ P ]} \left\langle \operatorname {R e L U} \left[ \left\langle \Theta_ {c, h}, \mathbf {x} _ {n} ^ {p} \right\rangle \right] \mathbf {x} _ {n} ^ {p}, \mathbf {v} _ {c, l} \right\rangle \tag {13}
$$
Let's further assume that the feature set is orthonormal:  $\forall c, c', \forall l \in [L], ||\mathbf{v}_{c,l}||_2 = 1$  and  $\mathbf{v}_{c,l} \perp \mathbf{v}_{c',l'}$  when  $(c, l) \neq (c', l')$ . Using  $\mathbf{x}^p = a^p\mathbf{v}_{c,l} + \sum_{\mathbf{v}' \in \cup \backslash \mathbf{v}_c} \alpha^{p,\mathbf{v}'}\mathbf{v}' + \epsilon^p$  given in (4), we have
$$
\left. \langle - \nabla_ {\Theta_ {c, h}} l (\Theta ; \mathbf {x} _ {n}, y _ {n}), \mathbf {v} _ {c, l} \rangle = [ 1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {n})) ] \left(\sum_ {p \in \mathcal {P} _ {v, l} (\mathbf {x} _ {n})} \operatorname {R e L U} [ \langle \Theta_ {c, h}, \mathbf {x} _ {n} ^ {p} \rangle a ^ {p} ] + \sum_ {p \in [ P ]} \langle \epsilon^ {p}, \mathbf {v} _ {c, l} \rangle\right) \right. \tag {14}
$$
It should be noted that the term i.e.,  $\sum_{v^{\prime}\in \cup \backslash \mathbf{v}_{c}}\alpha^{p,v^{\prime}}\langle \mathbf{v}^{\prime},\mathbf{v}_{c,l}\rangle$  becomes zero due to the orthogonal properties of the feature set. Let us represent the second term by  $\kappa$ :  $\sum_{p\in [P]}\langle \epsilon^p,\mathbf{v}_{c,l}\rangle = \kappa$ . Then, we have
$$
\left\langle - \nabla_ {\Theta_ {c, h}} l (\Theta ; \mathbf {x} _ {n}, y _ {n}), \mathbf {v} _ {c, l} \right\rangle = (1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {n}))) \left(\sum_ {p \in \mathcal {P} _ {v, l} (\mathbf {x} _ {n})} \operatorname {R e L U} [ \langle \Theta_ {c, h}, \mathbf {x} _ {n} ^ {p} \rangle a ^ {p} ] + \kappa\right) \tag {15}
$$
Furthermore, let us define  $V_{c,h,l}(\mathbf{x}_j) = \sum_{p\in \mathcal{P}_{\mathbf{v}_{c,l}}(\mathbf{x}_j)}\mathrm{ReLU}\bigl (\langle \Theta_{c,h},\mathbf{x}_j^p\rangle a^p\bigr)$  then above equation further reduces to following
$$
\left\langle - \nabla_ {\Theta_ {c, h}} l (\boldsymbol {\theta}; \mathbf {x} _ {n}, y _ {n}), \mathbf {v} _ {c, l} \right\rangle = \left(1 - \operatorname {S O F T} _ {c} (F (\mathbf {x} _ {n}))\right) \left(V _ {c, h, l} (\mathbf {x} _ {n}) + \kappa\right) \tag {16}
$$
Recall the above equation is the gradient with respect to the  $n^{th}$  data sample. Considering the gradient with respect to all data samples with  $y_{n} = c$ , and let us consider the total loss, where the weight  $z_{n}$  of each loss is assigned according to a distribution specified by the uncertainty set  $\mathcal{U}$ . Then, the total gradient is
$$
\left\langle - \nabla_ {\Theta_ {c, h}} l (\Theta ; \mathbf {X}, \mathbf {Y}), \mathbf {v} _ {c, l} \right\rangle = \max  _ {\mathbf {z} \in \mathcal {U}} \sum_ {n = 1} ^ {N} z _ {n} \left[ \mathbb {1} _ {y _ {j} = c} \left(V _ {c, h, l} \left(\mathbf {x} _ {n}\right) + \kappa\right) \left(1 - \mathrm {S O F T} _ {c} (F \left(\mathbf {x} _ {n}\right))\right) \right] \tag {17}
$$
Now using the standard gradient update rule with  $\beta$  being the learning rate, we have
$$
\left. \left\langle \Theta_ {c, h} ^ {t + 1}, \mathbf {v} _ {c, l} \right\rangle = \left\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \right\rangle + \beta \max  _ {\mathbf {z} \in \mathcal {U}} \sum_ {n = 1} ^ {N} z _ {n} \left[ \mathbb {1} _ {y _ {j} = c} \left(V _ {c, h, l} \left(\mathbf {x} _ {n}\right) + \kappa\right) \left(1 - \mathrm {S O F T} _ {c} (F \left(\mathbf {x} _ {n}\right))\right) \right] \right. \tag {18}
$$
Let  $\mathbf{x}_k\in \mathcal{D}_N^S$  be the most difficult sample having  $\mathbf{v}_{c,l}$  as the main feature. Also, consider  $\mathbf{x}_n\in \mathcal{D}_N^M$  to be the easy sample with  $y_{n} = c,y_{k} = c$ . Then, we have
$$
[ 1 - \mathsf {S O F T} _ {c} (F (\mathbf {x} _ {k})) ] \geq [ (1 - \mathsf {S O F T} _ {c} (F (\mathbf {x} _ {n})) ], \forall n \in [ 1, N ], n \neq k, y _ {n} = c \tag {19}
$$
Using above property, we can write the following using (18)
$$
\begin{array}{l} \left\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \right\rangle + \beta \max  _ {\mathbf {z} \in \mathcal {U}} \sum_ {n = 1} ^ {N} z _ {n} \left[ \mathbb {1} _ {y _ {j} = c} \left(V _ {c, h, l} (\mathbf {x} _ {n}) + \kappa\right) \left(1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {n}))\right) \right] \\ \leq \left\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \right\rangle + \beta N z _ {k} \left(1 - \operatorname {S O F T} _ {c} \left(F (\mathbf {x} _ {k})\right)\right) \tag {20} \\ \end{array}
$$
On the r.h.s., we have  $z_{n} = \frac{1}{N}$  for ERM, which assigns equal weights to all samples. Under the assumption of  $N_{\mathbf{v}_{c,l}} \ll N_{\cup \setminus \mathbf{v}_{c,l}}$ , the contribution of the  $N_{\mathbf{v}_{c,l}}$  on overall gradient will be negligible. In contrast, for the DRO framework, using (11), we have
$$
z _ {k} = \frac {1}{\sum_ {j = 1 , j \neq k} ^ {N} \exp \left(\frac {l _ {j} (\Theta) - l _ {k} (\Theta)}{\lambda}\right) + 1} \tag {21}
$$
Since  $l_{k}(\Theta) > l_{j}(\Theta), \forall \lambda > 0, \lambda \neq \infty$ , we have  $z_{k} > \frac{1}{N}$ . Using r.h.s. of (20) and incorporating  $z_{k} = \frac{1}{N}$  for ERM and  $z_{k} > \frac{1}{N}$ , we have
$$
\{\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \rangle + \beta (1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {k}))) \} _ {E R M} \leq \{\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \rangle + \beta (1 - \mathrm {S O F T} _ {c} (F (\mathbf {x} _ {k}))) \} _ {\text {R o b u s t}} \tag {22}
$$
This subsequently leads to the following:
$$
\{\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \rangle \} _ {\text {R o b u s t}} > \{\langle \Theta_ {c, h} ^ {t}, \mathbf {v} _ {c, l} \rangle \} _ {E R M}; \forall t > 0 \tag {23}
$$
Let  $\mathbf{x} \in \mathcal{D}_S^N$  from class  $c$  with  $\mathbf{v}_{c,l}$  as the main feature and  $\mathbf{v}'$  as the dominant feature learned through the memorization. Also consider  $\mathbf{v}'$  to be the main feature characterizing class  $k$ . Then for any class  $c'$ , we can define the following
$$
\mathrm {S O F T} _ {c ^ {\prime}} (\mathbf {x}) = \frac {\exp \left(F _ {c ^ {\prime}} (\mathbf {x})\right)}{\sum_ {j \in [ C ]} \exp \left(F _ {j} (\mathbf {x})\right)} \tag {24}
$$
In the above equation,  $F_{c'}(\mathbf{x})$  can be written as
$$
F _ {c ^ {\prime}} (\mathbf {x}) = \sum_ {h \in [ H ]} \sum_ {p \in [ P ]} \operatorname {R e L U} \left[ \left\langle \Theta_ {c ^ {\prime}, h}, \mathbf {x} ^ {p} \right\rangle \right] \tag {25}
$$
Substituting  $\mathbf{x}^p$  from (4), we have
$$
F _ {c ^ {\prime}} (\mathbf {x}) = \sum_ {h \in [ H ]} \sum_ {p \in [ P ]} \operatorname {R e L U} \left[ a ^ {p} \left\langle \Theta_ {c t, h}, \mathbf {v} _ {c, l} \right\rangle + \sum_ {\mathbf {v} ^ {\prime} \in \cup \backslash \mathbf {v} _ {c}} \alpha^ {p, \mathbf {v} ^ {\prime}} \left\langle \Theta_ {c t, h}, \mathbf {v} ^ {\prime} \right\rangle + \left\langle \Theta_ {c t, h}, \epsilon^ {p} \right\rangle \right] \tag {26}
$$
Substituting  $c^\prime$  by  $k$  , we have
$$
F _ {k} (\mathbf {x}) = \sum_ {h \in [ H ]} \sum_ {p \in [ P ]} \operatorname {R e L U} \left[ a ^ {p} \left\langle \Theta_ {k, h}, \mathbf {v} _ {c, l} \right\rangle + \sum_ {\mathbf {v} ^ {\prime} i n \cup \backslash \mathbf {v} _ {c}} \alpha^ {p, \mathbf {v} ^ {\prime}} \left\langle \Theta_ {k, h}, \mathbf {v} ^ {\prime} \right\rangle + \left\langle \Theta_ {k, h}, \epsilon^ {p} \right\rangle \right] \tag {27}
$$
In case of ERM, the  $\mathbf{v}_{c,l}$  signal is fairly weak during the training process due to  $N_{\mathbf{v}_{c,l}} \ll N_{\cup \backslash \mathbf{v}_{c,l}}$ . Therefore, the term  $\langle \Theta_{k,h}, \mathbf{v}_{c,l} \rangle$  is negligible. Also, the last term  $\langle \Theta_{k,h}, \epsilon^p \rangle$  is also small as this corresponds to the Gaussian noise. For the second term  $\exists \mathbf{v}'$  for which  $\langle \Theta_{k,h}, \mathbf{v}' \rangle$  is very high because of the spurious correlation. In contrast, for the robust loss, using Lemma 1, the model learns a stronger correlation with the true class parameter and therefore  $\langle \Theta_{c,h}, \mathbf{v}_{c,l} \rangle$  is high. As such, both terms  $\langle \Theta_{k,h}, \mathbf{v}_{c,l} \rangle$  as well as  $\langle \Theta_{k,h}, \mathbf{v}' \rangle$ ,  $\forall v'$  becomes low. As a result, we have
$$
\left\{F _ {k} (\mathbf {x}) \right\} _ {E R M} > \left\{F _ {k} (\mathbf {x}) \right\} _ {R o b u s t} \tag {28}
$$
Substituting this inequality to (24), we have
$$
\left\{\mathrm {S O F T} _ {k} (\mathbf {x}) \right\} _ {\text {R o b u s t}} <   \left\{\mathrm {S O F T} _ {k} (\mathbf {x}) \right\} _ {E R M} \tag {29}
$$
This completes the proof of Theorem 2.
D Experimental Details and Additional Results
In this section, we first provide a detailed description of datasets used in our experimentation followed by hardware description of our experimentation. We then provide examples of single-view and multi-view data samples. Next, we provide additional experimental results and baselines on Cifar10 and Cifar100 datasets. After that, we provide additional baseline results TinyImageNet. We also compare individual ensemble members' performance followed by our model performance with different calibration techniques commonly used in dense networks. Then, we perform an in-depth ablation study. Parameter size and inference speed are discussed in the subsequent subsection. We also further investigate the diversity of the sparse subnetworks. Finally, we provide detailed qualitative analysis to support our proposed claim.
D.1 Detailed Dataset Description
For general classification setting, we consider Cifar10, Cifar100 [12], and TinyImageNet [14] datasets. For the out of distribution setting, we consider corrupted version of Cifar10 and Cifar100, which are named as Cifar10-C and Cifar100-C [10], respectively. Finally, for open-set detection, we leverage SVHN [21] as the open-set dataset. The detailed description of each dataset is given below:
- Cifar10 consists of total 10 classes, each consisting of 5,000 training samples and 1,000 testing (evaluation) samples. Each image is a colored image with size  $32 \times 32$ .
- Cifar100 consists of 20 super classes where each super-class consists of 5 classes resulting into total 100 classes. Each class consists of 500 training samples and 100 testing samples. Each image is a colored image with size  $32 \times 32$ .
- TinyImageNet consists of 200 classes with 1,000,000 samples where each class has 500 training images, 50 validation images, and 50 test images. Each image is a colored image with size  $64 \times 64$ .
- Cifar10-C consists of fifteen different types of corruptions applied on the Cifar10 clean testing dataset where each corruption has 5 severity levels, ranging from 1 to 5 with 1 being least severe and 5 being most severe. The corruptions include Gaussian noise, shot noise, impulse noise, defocus blur, forsted glass blur, motion blur, zoom blur, snow, frost, fog, brightness, contrast, elastic, pixelate, and JPEG.
- Cifar10-C. consists of fifteen different corruptions applied on the Cifar100 clean testing dataset.
- SVHN consists of 10 classes with digit 1 as class 1, digit 9 as class 9 and digit 0 as class 10. These are original, variable-resolution, colored house-number images with character level bounding boxes. We use this dataset as the open-set dataset in our experimentation.
Multi View
(Present: Headlight, Missing: Tire, Door handle)
Headlight
(Present: Headlight, Tire, Door handle)
(Present: Tire, Door handle Missing: Headlight)
Figure 5: Examples of single-view and multi-view samples.
Door handle
D.2 Hardware Details for Experimentation
All experimentations are conducted using NVIDIA RTX A6000 GPU with 48GB memory requiring 300 Watt power. For GPU, CUDA Version: 11.6, Driver Version: 510.108.03, and NVIDIA-SMI: 510.108.03 is used. In terms of CPU, our experimentation uses an Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz with a 64-bit system and an x86_64 architecture.
D.3 Single-view and Multi-view Examples
Figure 5 show the three example images, where the first image is a representative single-view data sample whereas the last two are multi-view samples. In this example, we consider three major features for cars: i.e., Tire, Headlight, and Door handle. As only headlight feature is present in the first image, it belongs to the single-view category. For the second and third images, multiple features are presented and therefore we regard those images as multi-view data samples.
Table 6: Accuracy and ECE performance with  ${15}\%$  density for Cifar10 and Cifar100 dataset.
Training Type	Approach	Cifar10	Cifar100
ResNet50	ResNet101	ResNet101	ResNet152
ACC	ECE	ACC	ECE	ACC	ECE	ACC	ECE
Dense†	94.82	5.87	95.12	5.99	76.40	16.89	77.97	16.73
Dense Training	L1 Pruning	93.88	5.69	94.23	5.88	75.53	15.52	75.83	15.78
LTH	92.97	4.03	93.15	5.69	74.36	15.13	74.77	15.22
DLTH	95.15	6.21	95.65	6.96	77.98	16.24	78.23	16.54
Mixup	93.22	4.02	93.38	5.68	74.48	15.10	74.68	15.16
Sparse Training	CigL	92.25	4.67	93.34	4.59	77.88	10.16	77.27	10.62
DST Ensemble	89.57	2.10	88.64	1.34	64.57	9.76	64.75	9.27
Sup-ticket	94.65	3.20	94.95	3.09	78.68	10.16	78.95	10.32
Mask Training	AdaBoost	94.07	5.65	94.76	5.14	75.98	23.55	76.28	24.27
EP	94.41	3.90	94.42	4.07	75.66	14.79	76.05	14.79
SNE	94.85	3.05	94.96	3.18	76.82	11.12	77.23	11.63
DRE	94.87	1.71	94.74	1.34	75.86	4.90	76.46	5.81
D.4 Additional Result on Cifar10 and Cifar100
Table 6 shows the experimental result on Cifar10 and Cifar100 datasets with a  $15\%$  density. As shown, the proposed technique has a far superior performance in terms of the ECE score compared to the competitive baselines. This is consistent with the results with a  $9\%$  density as presented in the main paper, which further justifies the effectiveness of our proposed technique.
D.5 Additional Baseline Results on Sparse Training Methods
Apart from the baselines included in the main paper, there are other sparse training methods without the need for iterative pruning/growing [15, 19, 33, 36]. However, as all these methods primarily focus on pushing the accuracy up to the original dense networks, they still suffer from a severely overfitting behavior, leading to a poor calibration performance as shown in Table 7.
Table 7: ECE and Accuracy for other sparse baselines.
Approach	Cifar10	Cifar100
9%	15%	9%	15%
ACC	ECE	ACC	ECE	ACC	ECE	ACC	ECE
SNIP [15]	93.45	4.10	94.12	3.86	52.99	10.97	54.40	10.56
Dynamic Sparse [19]	91.11	5.73	92.09	5.33	70.06	15.56	71.23	14.29
GraSP [33]	92.73	4.78	93.16	4.54	72.60	17.07	73.07	15.95
MEST [36]	92.94	4.67	93.50	4.40	72.19	16.47	73.49	15.81
EP	94.20	3.97	94.41	3.90	75.07	14.62	75.66	14.79
DRE	94.60	0.7	94.87	1.71	74.68	1.20	75.86	4.90
D.6 Additional Baseline Results on TinyImageNet
As mentioned in the main paper, the computational issue (i.e., memory overflow) makes it impossible to run sparse learning techniques i.e., CigL [16], DST Ensemble [18], and Sup-ticket [35] on the ResNet101 and WideResNet101 architectures to make a fair comparison. Therefore, in this section, we pick a lower capacity model (ResNet50) and compare the performance. Even for the ResNet50 architecture, CigL still runs into the memory overflow issue with a batch
size of 128. Furthermore, lowering the batch size (e.g., 16) makes the training process extremely slow even using a  $48Gb$  GPU, where each training epoch takes more than half an hour, making model training extremely difficult. Therefore, we did not report the performance of CigL. It should be noted that CigL can be trained on CIFar10 and CIFar100 because of lower dimension of the input images and we have already reported its performance in the main paper. Table 8 shows the performance of DRE along with those from DST Ensemble and Sup-ticket on ResNet50. It is clear that DRE achieves better performance compared to these baselines.
Table 8: Additional baseline results on TinyImageNet using ResNet50 with  $\mathcal{K} = 15\%$ .
Training Type	Approach	ACC	ECE
Sparse Training	DST Ensemble	72.00	2.94
Sup-ticket	68.68	10.96
Mask Training	DRE	71.57	1.51
D.7 Performance from Ensemble Members
We investigate how performance varies in different sparse sub-networks. We use Cifar100 as an example and Table 9 report the individual sub-network performance on both accuracy and ECE. While each sparse sub-network is a relatively weaker learner (which is expected), they contribute to the final ensemble model in a complementary way, leading to a better ECE score as well as accuracy.
Table 9: Different sub-networks performance on Cifar100 dataset.
Subnetworks	ResNet101	ResNet152
ACC	ECE	ACC	ECE
Subetwork 1 (3%)	68.22	14.35	69.65	13.31
Subetwork 2 (3%)	69.03	1.39	70.00	3.39
Subetwork 3 (3%)	72.86	11.96	70.24	14.78
DRE	74.68	1.20	74.37	2.09
D.8 Comparison with Common Calibration Techniques