SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Complexity analysis on  $\mathcal{D}_{\mathrm{out}}^e$ . With the above lemma and the assumptions of informative extrapolation, we can derive the results to understand the benefit of our proposed DivOE.
Consider the given classifier defined as  $\theta_{n_1,n_2}^* = \frac{1}{n_1 + n_2} (\sum_{i = 1}^{n_1}x_i^1 -\sum_{i = 1}^{n_2}x_i^2)$ , assume each  $x_{i}^{1}$  is drawn i.i.d. from  $\mathcal{D}_{\mathrm{in}}$  and each  $x_{i}^{2}$  is drawn i.i.d from  $\mathcal{D}_{\mathrm{out}}^e$ , and assume the signal/noise ratio is  $\frac{||\mu||}{\sigma} = r_0\gg 1$ , the dimensionality/sample size ratio is  $\frac{d}{n} = r_1$ , as well as exist some constant  $\alpha < 1$ . By decomposition, we can rewrite  $\theta_{n_1,n_2}^* = \mu +\frac{n_1}{n_1 + n_2}\theta_1 + \frac{n_2}{n_1 + n_2}\theta_2$  with the following  $\theta_{1}$  and  $\theta_{2}$ :
$$
\theta_ {1} = \frac {1}{n _ {1}} \left(\sum_ {i = 1} ^ {n _ {1}} x _ {i} ^ {1}\right) - \mu , \quad \theta_ {2} = \frac {1}{n _ {2}} \left(- \sum_ {i = 1} ^ {n _ {2}} x _ {i} ^ {2}\right) - \mu , \tag {22}
$$
Since  $\theta_{1} \sim \mathcal{N}(0, \frac{\sigma^{2}}{n_{1}}\mathcal{I})$ , we have that  $||\theta_{1}||^{2} \sim \frac{\sigma^{2}}{n_{1}}\mathcal{X}_{d}^{2}$  and  $\frac{\mu^{T}\theta_{1}}{||\mu||} \sim \mathcal{N}(0, \frac{\sigma^{2}}{n_{1}})$  to form the standard concentration bounds as:
$$
\mathbb {P} \left(\left\| \theta_ {1} \right\| ^ {2} \geq \frac {\sigma^ {2}}{n _ {1}} \left(d + \frac {1}{\sigma}\right)\right) \leq e ^ {- \frac {d}{8 \sigma^ {2}}}, \quad \mathbb {P} \left(\frac {\left| \mu^ {T} \theta_ {1} \right|}{\left\| \mu \right\|} \geq (\sigma \| \mu \|) ^ {\frac {1}{2}}\right) \leq 2 e ^ {- \frac {n _ {1} \| \mu \|}{2 \sigma}} \tag {23}
$$
The distribution of  $\theta_{2}$  can be treated as a truncated distribution of  $\theta_{1}$  as  $x_{i}^{2}$  drawn i.i.d. from the extrapolated  $\mathcal{D}_{\mathrm{out}}^e$ . Without losing the generality, we replace  $n_1$  with  $n$ , and have the following inequality with a finite positive constant  $a$ :
$$
\mathbb {P} \left(\left\| \theta_ {2} \right\| ^ {2} \geq \frac {\sigma^ {2}}{n _ {1}} \left(d + \frac {1}{\sigma}\right)\right) \leq a e ^ {- \frac {d}{8 \sigma^ {2}}} \tag {24}
$$
According to Lemma B.2, we can have that  $|\mu^T\theta_2| \leq ||\mu||^2 + \frac{\sigma^2(\alpha - \tau(\epsilon))}{2}$ . Now we can have  $||\theta_1||^2 \leq \frac{\sigma^2}{n} (d + \frac{1}{\sigma})$ ,  $||\theta_2||^2 \leq \frac{\sigma^2}{n} (d + \frac{1}{\sigma})$ ,  $\frac{|\mu^T\theta_1|}{||\mu||} \leq (\sigma ||\mu||)^{\frac{1}{2}}$  simultaneously hold and derive the following recall the decomposition,
$$
\left| \left| \theta_ {n _ {1}, n _ {2}} ^ {*} \right| \right| ^ {2} = \left| \left| \mu + \frac {n _ {1}}{n _ {1} + n _ {2}} \theta_ {1} + \frac {n _ {2}}{n _ {1} + n _ {2}} \theta_ {2} \right| \right| ^ {2} \leq \frac {\sigma^ {2}}{n} \left(d + \frac {1}{\sigma}\right) + \left| \left| \mu \right| \right| ^ {2}, \tag {25}
$$
$$
\left| \mu^ {T} \theta_ {n _ {1}, n _ {2}} ^ {*} \right| \geq \frac {1}{2} \left(\left| \left| \mu \right| \right| ^ {2} - \sigma^ {\frac {1}{2}} \left| \left| \mu \right| \right| ^ {\frac {3}{2}} - \frac {\sigma^ {2} (\alpha - \tau (\epsilon))}{2}\right). \tag {26}
$$
With the above inequality derived in Eq. (25) and Eq. (26), we can have the following bound with the probability at least  $1 - (1 + a)e^{-\frac{n_1n}{8\sigma^2}} - 2e^{-\frac{n_1||\mu||}{2\sigma}}$
$$
\frac {\mu^ {T} \theta_ {n _ {1} , n _ {2}} ^ {*}}{\sigma | | \theta_ {n _ {1} , n _ {2}} ^ {*} | |} \geq \frac {| | \mu | | ^ {2} - \sigma^ {\frac {1}{2}} | | \mu | | ^ {\frac {3}{2}} - \frac {\sigma^ {2} (\alpha - \tau (\epsilon))}{2}}{2 \sqrt {\frac {\sigma^ {2}}{n} (d + \frac {1}{\sigma}) + | | \mu | | ^ {2}}} \tag {27}
$$
Since  $\mathrm{FPR}(\theta_{n_1,n_2}^*) = \operatorname{erf}\left(\frac{\mu^T\theta_{n_1,n_2}^*}{\sigma||\theta_{n_1,n_2}^*||}\right)$  is monotonically decreasing, as the lower bound of  $\frac{\mu^T\theta_{n_1,n_2}^*}{\sigma||\theta_{n_1,n_2}^*||}$  will increase as the constraint from the extrapolated  $\mathcal{D}_{\mathrm{out}}^{e}$  changed accordingly in our DivOE, the upper bound of  $\mathrm{FPR}(\theta_{n_1,n_2}^*)$  will decrease. From the above analysis, we can observe the benefit of DivOE when conducting the informative extrapolation.
Additional Discussion with Extrapolation Condition. The extrapolation of outlier data towards inliers may not always be possible. Similar to the condition analysis in [Fang et al., 2022], the informative extrapolation that targets better characterize the decision boundary between ID and OOD also needs some requirements or realization assumptions. Intuitively, it mainly relies on auxiliary outliers and specific extrapolation techniques. For the former, the original outliers should not be far away from the ID sample; For the latter, the extrapolation techniques should be able to add enough perturbation or semantic changes to the original outlier. It is worthy further exploring the condition in a theoretical way by referring the analysis in Fang et al. [2022].
C Additional Experimental Results and Further Discussion
In this section, we provide more experiment results from different perspectives to characterize our proposed DivOE. First, we introduce the additional experimental setups for the empirical verification in previous figures and our learning framework. Second, we provide more analyses of the manipulated data via informative extrapolation. Finally, comprehensive results with discussions are provided.
C.1 Additional Experimental Setups
Figure 1. In the right-panel of Figure 1, we conduct experiments to evaluate the OOD detection performance of different OE-based methods with the gradually reduced number of auxiliary outliers. We first sample a particular number of the least informative auxiliary outliers to simulate the different cases in collecting those surrogate OOD data. With the highest OOD scores, we sample the selected auxiliary outliers from the whole auxiliary dataset. Specifically, we choose the negative boundary
score generated by POEM [Ming et al., 2022] as the OOD score. For the fair comparison, we adjust the size of the mini-batch for OOD samples so that all methods actually train on  $90\%$  of the least informative auxiliary dataset by random sampling, or their custom sampling strategies during every epoch while keeping the number of ID training data unchanged.
Figure 2. In the left-middle panel of Figure 2, we evaluate the OOD detection performance of Energy-OE [Liu et al., 2020] with auxiliary outliers with different levels of diversity. To simulate the collected auxiliary outliers from different levels of diversity, we first sample the 40000 least informative outliers from Tiny-ImageNet [Le and Yang, 2015] according to the boundary score generated by POEM [Ming et al., 2022]. Then we choose 10000 informative outliers in different but not-overlapped boundary score intervals, namely [0, 10000], [10000, 20000], [30000, 40000], [40000, 50000], [50000, 60000], and combine them with the 40000 least informative outliers to form 5 groups of auxiliary outliers with different levels of diversity. In the two right panels of Figure 2, we present TSNE visualization of feature embedding (taken at the penultimate layer of the network) on ID, the original outliers, and extrapolated outliers after the first training epoch in DivOE, with extrapolated ratio  $r = 0.5$ , extrapolate epsilon  $\epsilon = 0.05$  and optimization step number  $k = 5$ .
Figure 3. In the two left panel of Figure 3, we keep the extrapolated ratio  $r$  fixed to 0.5 and optimization step number  $k$  fixed to 5. In the right-middle panel, we compare ID/OOD distributions based on Energy score after training with DivOE, with hyperparameters  $r = 0.5, \epsilon = 0.05$ , and  $k = 5$ .
Figure 4. In Figure 4(a), the extrapolated epsilon  $\epsilon$  is set to 0.5 and optimization step number  $k$  is set to 5. In Figure 4(b), we fix the extrapolated ratio  $r$  to 0.5 and optimization step number  $k$  to 5.
Training details. We conduct all major experiments on pre-trained WideResNet [Zagoruyko and Komodakis, 2016] with 40 depth and 2 widen factor and fix the number of fine-tuning epochs to 10, following the previous research work [Hendrycks et al., 2019, Liu et al., 2020]. The models are trained using stochastic gradient descent [Kiefer and Wolfowitz, 1952] with Nesterov momentum [Duchi et al., 2011]. We adopt Cosine Annealing [Loshchilov and Hutter, 2017] to schedule the learning rate, which begins at 0.001. We set the momentum and weight decay to be 0.9 and  $10^{-4}$  respectively for all experiments. The size of the mini-batch is 128 for both ID samples and OOD samples when training and 200 for both when testing.
Hyper-parameter of DivOE. Throughout the paper, all the experiments with DivOE are conducted with extrapolated ratio  $r = 0.5$ , extrapolate epsilon  $\epsilon = 0.05$ , and optimization step number  $k = 5$  unless stated otherwise.
C.2 Analyses on the Extrapolated Data
In this part, we further compare the extrapolated outliers with the given auxiliary outliers during the fine-tuning process of OE. In Figure 6, we adopt the TSNE visualization to check the feature representation of the extrapolated outliers during training. Through the comparison with the given outliers (represented by the blue circles), the extrapolated outliers (represented by the orange cricles) manipulated by our introduced target (e.g., Eq. (6)) can better cover the decision area between ID and OOD data. Note that the extrapolated OOD samples are optimized to be closer to the ID samples (represented by the green circles), which indicates this extrapolated portion can perform better on more accurately shaping the decision boundary of the binary classification task for ID and OOD.
In Figure 8, we present the energy score of the extrapolated outliers compared with the original ones. It can be found that the newly synthesized outliers can generally easier to be recognized as ID samples than the original auxiliary outliers. In Figure 9 and Table 28, we report the loss differences corresponding to the maximization part that verifies our operation indeed achieves the conceptual idea of diversified outlier exposure. Intuitively, we visualize the extrapolated outliers in Figure 10. The extrapolated samples still keep the semantic information that belongs to the OOD class space.
In addition, we also conduct another quantitative investigation on the extrapolated outliers with the original ones. In Table 6, we adopt the Maximum Mean Discrepancy (MMD) [Borgwardt et al., 2006] to check the distance between the distributions of the given auxiliary outliers with the extrapolated ones. By varying the specific  $\epsilon$  that is adopted in the multi-step optimization (e.g., Eq. (9)), we can
(a) Epoch 1
(b) Epoch 2
(c) Epoch 3
(d) Epoch 4
Figure 6: TSNE visualization of CIFAR10 ID data and the extrapolated outliers generated during OE.
(e) Epoch 5
observe that the manipulated outliers with large  $\epsilon$  perform large discrepancy with the original outliers. It is intuitive and reasonable with more changes on semantic levels as shown in the left panel of Figure 3. However, as previously illustrated in the left-middle panel of Figure 3, we also find the larger  $\epsilon$  may not always indicates more informative outliers as the semantic information may be lost when the pixel perturbations are arbitrarily large. It is worth to be further explored in the future.
Table 6: MMD value between original outliers and the extrapolated outliers with different  $\epsilon$  in DivOE
ε	0.01	0.025	0.05	0.075	0.1	0.15	0.5
MMD value	0.0155	0.0176	0.0208	0.0233	0.0253	0.0286	0.0439
C.3 Discussion on Using PGD-based Attack
Note that, PGD is a replaceable choice in the framework of DivOE. To better understand the effect of these attack-based noise, we compare it with other inner-attack methods and show that PGD with the uniform distribution loss is generally the better way to achieve higher performance.
We would like to explain that DivOE was proposed as a general framework for diversifying outlier exposure with partially conduct informative extrapolation, which allows any appropriate noise or augmentation techniques to be incorporated. To show the effectiveness of PGD, we conduct the following experiments. On the one hand, we compare the random noise, re-adversarial perturbation, and adversarial perturbation to justify its rationality. The results are summarized in the Table 7 and show the adversarial perturbation achieve better performance than other types of perturbations. On the other hand, we compare with other adversarial attacks and data augmentation methods for inner-maximization part in Table 8 and show that PGD with the uniform distribution loss is generally a better way to achieve higher performance due to the optimization originality.
How would  $\alpha$  play an effect in DivOE As for  $\alpha$ , it is a step size adopted in the informative extrapolation. To better understand its effect, we conducted the experiments by changing the step size in the PGD attack in Table 9. Since the step size is only related to the optimization of the constrained
Table 7: Rationality of adopting PGD attack in the inner maximization.
Dataset	Method	Perturbation Direction & method	FPR95	AUROC	AUPR	ID-ACC
CIFAR100	OE	27.67	91.89	97.93	75.41
DivOE	Re-PGD (most uninformative)	27.36	92.02	97.98	75.62
DivOE	Random Noise	26.16	92.39	98.06	75.56
DivOE	Mixup	25.70	92.46	98.07	76.06
DivOE	PGD (most informative)	24.80	92.91	98.22	75.24
CIFAR10	OE	13.76	97.53	99.43	94.51
DivOE	Re-PGD (most uninformative)	13.20	97.59	99.43	94.80
DivOE	Random Noise	13.19	97.60	99.43	94.81
DivOE	Mixup	12.87	97.59	99.42	94.83
DivOE	PGD (most informative)	11.66	97.82	99.48	94.66
Table 8: DivOE incorporating different attacked noise
Dataset	Method	Attack Method	FPR95↓	AUROC↑	AUPR↑	ID-ACC↑
CIFAR100	OE	-	27.67	91.89	97.93	75.41
DivOE	Random	26.16	92.39	98.06	75.56
DivOE	FGSM	25.58	92.60	98.14	75.59
DivOE	PGD	24.80	92.91	98.22	75.24
DivOE	C&W	24.20	93.02	98.25	75.18
DivOE	TRADES	24.88	92.85	98.20	75.13
DivOE	Mixup	25.70	92.46	98.07	76.06
DivOE	CutMix	28.46	91.70	97.89	75.62
DivOE	AugMix	25.40	92.60	98.12	75.58
CIFAR10	OE	-	13.76	97.53	99.43	94.51
DivOE	Random	13.19	97.60	99.43	94.81
DivOE	FGSM	12.71	97.70	99.45	94.78
DivOE	PGD	11.66	97.82	99.48	94.66
DivOE	C&W	11.73	97.74	99.46	94.64
DivOE	TRADES	12.47	97.67	99.44	94.54
DivOE	Mixup	12.87	97.59	99.42	94.83
DivOE	CutMix	13.95	97.54	99.42	94.78
DivOE	AugMix	12.41	97.82	99.49	94.71
maximization, the effects on the final performance of DivOE are not significant. By enlarging the step size, we may save more extra time on the inner maximization to reduce the time cost relatively, which is beneficial for algorithm deployment.
Table 9: The effect of  $\alpha$  in training of DivOE
Dataset	Method	Attack Method	Attack Steps	FPR95↓	AUROC↑	AUPR↑	ID-ACC↑
CIFAR-100	OE	-	-	27.67	91.89	97.93	75.41
DivOE	PGD (alpha=0.005)	20	25.02	92.69	98.15	75.27
DivOE	PGD (alpha=0.01)	10	25.08	92.75	98.17	74.82
DivOE	PGD (alpha=0.02)	5	24.41	92.99	98.23	75.34
DivOE	PGD (alpha=0.04)	3	25.26	92.71	98.17	75.44
CIFAR-10	OE	-	-	13.76	97.53	99.43	94.51
DivOE	PGD (alpha=0.005)	20	11.92	97.78	99.47	94.57
DivOE	PGD (alpha=0.01)	10	12.20	97.61	99.42	94.59
DivOE	PGD (alpha=0.02)	5	12.11	97.80	99.47	94.64
DivOE	PGD (alpha=0.04)	3	12.23	97.76	99.46	94.79
Visual illustration of Extrapolation Process. From the algorithmic perspective, the adversarial perturbation with uniform distribution loss is a non-target perturbation. The intuition is to perturb the OOD sample to be more like ID regarded by the model. Maximizing uniform distribution loss has no requirement for each OOD sample to be perturbed into some specific ID classes. Thus, it is actually automatically decided by the gradient for the specific perturbation directions. To better
$\epsilon = 0.005$
$\epsilon = 0.01$
$\epsilon = 0.1$
$\epsilon = 0.5$
Figure 7: Prediction on the perturbed outlier with different extrapolation strength  $e$ .
understand the perturbation effect, we conduct the case study on the specific example in Figure 7 for their specific prediction results during the adversarial perturbation process. It can be found that after the adversarial perturbation, the prediction result of the second example will be concentrate on one specific class with high confidence. By enlarging the perturbation strength, the example will be perturbed to be confidently predicted on different classes.
C.4 Rationality of Partially Extrapolation
Considering the initial motivation to diversify outliers, we explore to extrapolate more potential OOD distribution beyond the original auxiliary ones. For the unperturbed OOD samples used in training, we hope the training can still learn the knowledge from the original auxiliary outliers, for the perturbed OOD samples, we hope the training can find different but informative outliers beyond the original ones to improve the generalization of outlier exposure.
Formally, Eq. (5) is an instance-level realization of Eq. (4) that actually contains the perturbed part and the unperturbed part. The deduction from Eq. (4) to Eq. (5) requires to merge the same terms in Eq. (4) first, and then specifically set with a subsequent rescaling. Finally, we have the formulation that a portion of data to conduct maximization part for diversifying the existing data distribution and the remaining other portion of data corresponds to learning on the original outliers.
To better understand the specific effects of different samples, it can be referred to the left panel of Figure 4, in which we show the OOD detection performance of DivOE under different portions of perturbed data during training. Generally, using a small part of perturbed samples can achieve the extrapolation target and improve the performance. However, if we totally perturb all the data, it is even worse than the original OE.
Intuitively, we hope the extrapolation can realize the diversification target and cover more potential OOD distributions beyond the original auxiliary ones. Under such high-level guidance, generally, the extrapolation ratio is expected to be not too large to totally change the whole original auxiliary distribution, which is empirically demonstrated to be negative for OE. As for diversified strength, we kindly refer to the left middle panel of Figure 3, which serves as a search space for synthesizing different outliers using different. In practical hyper-parameter tuning, we may generally suggest starting with the original OE baseline (i.e., extrapolation ratio equals 0 and diversified strength equals 0) and enlarging the extrapolation ratio or diversified strength step by step, finding a stage where the OOD detection performance on the validation set will not be further enhanced. For the coarse-grained augmentations and specific OOD scores, it may be further decided by other.
For four extrapolation factors, we re-summarize the previous results and discuss the general guidance for tuning. First, MSP score and PGD are empirically demonstrated to be the appropriate choice for extrapolation, which can be fixed (refer to the results in Figure 4(c)). Second, extrapolation ratio and strength (i.e.,  $\epsilon$ ) have general trends either individually or jointly considering them in Table 10. The performance of the extrapolation ratio is monotonous regarding different  $\epsilon$ . The large values would degrade performance due to the loss of diversity on the auxiliary outliers. Then, the extrapolation strength could be tuned after the other factors are set to appropriate choices. Our empirical results
show that very small or large  $\epsilon$  should be avoided, and epsilons from 0.05 to 0.1 can generally achieve satisfactory results by referring previous experimental results.
Table 10: DivOE with different extrapolation ratios and strengths.
Dataset	Method	Extrapolation ratio r	Extrapolation strength ε	FPR95↓
CIFAR-100	OE	-	-	27.67
DivOE	1	0.01	30.52
DivOE	0.5	0.01	29.06
DivOE	0.25	0.01	26.21
DivOE	0.1	0.01	24.57
DivOE	1	0.05	43.21
DivOE	0.5	0.05	24.80
DivOE	0.25	0.05	24.21
DivOE	0.1	0.05	23.54
DivOE	1	0.125	49.69
DivOE	0.5	0.125	24.94
DivOE	0.25	0.125	25.36
DivOE	0.1	0.125	24.77
C.5 Extra Experimental Results and Discussions
In this part, we present the extra experimental results in the following perspectives to characterize the learning framework of DivOE for the OOD detection task.
Experiments on Add-on Modules. For adding other methods to Table 2, we conduct experiments accordingly and summarize the results in Table 11. The results show that DivOE can consistently improve them on the original basis compared with other counterparts across different setups.
Table 11: Extra comparison with various add-on methods.
Dataset	Method	FPR95↓	AUROC↑	AUPR↑	ID-ACC↑
CIFAR-100	OE	27.67	91.89	97.93	75.41
DOE	30.69	93.13	98.39	71.43
MixOE	35.26	90.31	97.58	75.74
DivOE	24.80	92.91	98.22	75.24
Energy	27.86	90.70	97.40	75.41
Energy+DOE	44.91	88.80	97.22	73.79
Energy+MixOE	38.14	85.12	95.40	73.87
Energy+DivOE	24.71	92.04	97.82	75.25
CIFAR-10	OE	13.76	97.53	99.43	94.51
DOE	7.39	98.29	99.58	92.82
MixOE	13.57	97.47	99.39	94.66
DivOE	11.66	97.82	99.48	94.66
Energy	18.15	90.90	96.34	94.02
Energy+DOE	9.65	97.70	99.37	94.83
Energy+MixOE	31.90	83.72	93.50	93.72
Energy+DivOE	16.03	92.23	96.87	94.53
Experiments on Fine-grained OOD Test Sets. In order to further understand the effectiveness of our DivOE on different OOD datasets, we report the fine-grained results of our experiments on CIFAR-10/CIFAR-100 with 5 OOD datasets (Textures, Places365, iSUN, LSUN-r, LSUN-c). The results on the 5 OOD datasets demonstrate the general effectiveness of our DivOE.
In Tables 12 and 13, we summarize the fine-grained detection performance. The results reflect the performance improvement of each method compared with those representative baselines. Our DivOE can significantly reduce the FPR95 score on some OOD testing sets in which those previous baselines do not perform well (e.g., Textures and Places365), and achieve comparable performance on the rest ones. Note that it is general to have some tradeoffs [Hendrycks et al., 2019, Chen et al., 2021, Ming et al., 2022, Fang et al., 2022] on the performance change when using different methods under the OE framework. DivOE shows a good balance in gaining significant improvement without sacrificing the other performance too much via the informative extrapolation. On the other hand, the significant
Table 12: OOD detection performance on CIFAR-10 [Krizhevsky, 2009] as ID. All methods are trained on the same backbone. Values are percentages.  $\uparrow$  indicates larger values are better, and  $\downarrow$  indicates smaller values are better. (averaged by multiple trials)
Method	OOD dataset	ID ACC
Textures	Places365	iSUN	LSUN-r	LSUN-c
FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑	FPR95↓	AUROC↑
MSP	59.27	88.50	60.77	88.13	56.79	89.53	53.60	91.02	30.55	95.68	94.84
ODIN	54.84	80.28	48.89	85.66	28.68	93.85	23.17	95.05	9.96	97.90	94.84
Mahalanobis	16.99	96.69	69.96	82.54	31.62	94.44	32.41	94.70	30.50	95.06	94.84
Energy	53.17	85.40	40.22	89.91	34.60	92.44	28.48	93.85	8.12	98.35	94.84
GradNorm	73.17	58.11	78.27	60.46	70.28	70.82	65.63	73.23	11.41	97.02	94.84
OE	21.81	96.15	44.86	91.77	0.00	100.00	0.00	100.00	0.99	99.85	94.56
Energy (w. Daux)	33.59	85.47	58.39	68.32	0.00	100.00	0.00	100.00	0.31	99.83	94.50
VOS	55.45	81.72	46.89	87.02	4.21	99.13	33.33	92.47	39.16	90.82	95.02
NTOM	32.78	86.30	59.03	67.50	0.00	100.00	0.00	100.00	1.17	99.38	95.06
POEM	34.73	85.28	57.82	67.93	0.00	100.00	0.00	100.00	0.43	99.74	95.10
MixOE	22.23	95.74	39.45	92.86	0.26	99.92	0.27	99.92	4.40	99.21	95.00
DivOE (Ours)	12.28	97.73	44.97	91.51	0.07	99.98	0.02	99.99	3.03	99.47	94.56