SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We visualize the generated outlier images by interpolating token embeddings from different classes in Figure 9. The result shows that interpolating different class token embeddings tends to generate images that are still in-distribution rather than images with semantically mixed or novel concepts, which is aligned with the observations in Liew et. al. [43]. Therefore, regularizing the model using such images is not effective for OOD detection (Table 2).
Figure 9: Visualization of the generated outlier images by interpolating token embeddings from different classes. We show the results with different interpolation weights  $\alpha$ .
E Visualization of the Outlier Generation by Adding Noise
As in Table 2 in the main paper, we visualize the generated outlier images by adding Gaussian and learnable noise to the token embeddings in Figure 10. We observe that adding Gaussian noise tends to generate either ID images or images that are far away from the given ID class. In addition, adding learnable noise to the token embeddings will generate images that completely deviate from the ID data. Both of them are less effective in regularizing the model's decision boundary.
F Comparison with Training w/ real Outlier Data.
We compare with training using real outlier data on CIFAR-100, i.e., 300K Random Images [4], which contains 300K preprocessed images that do not belong to CIFAR-100 classes. The result shows that DREAM-OOD (FPR95:  $40.31\%$ , AUROC:  $90.15\%$ ) can match or even outperform outlier exposure with real OOD images (FPR95:  $54.32\%$ , AUROC:  $91.34\%$ ) under the same training configuration while using fewer synthetic OOD images for OOD regularization (100K in total).
(a) Generated outliers by adding Gaussian noise for ID class Beaver
(b) Generated outliers by adding Gaussian noise for ID class Apron
(c) Generated outliers by adding Gaussian noise for ID class Strawberry
(d) Generated outliers by adding learnable noise for ID class Beaver
Figure 10: Visualization of the generated outlier images by adding Gaussian and learnable noise to the token embeddings from different classes.
(e) Generated outliers by adding learnable noise for ID class Apron
(f) Generated outliers by adding learnable noise for ID class Strawberry
G Visualization of Generated Inlier Images
We show in Figure 11 the visual comparison among the original IMAGENET images, the generated images by our DREAM-ID, and the generated ID images using generic prompts "A high-quality photo of a [cls]" where "[cls]" denotes the class name. Interestingly, we observe that the prompt-based generation produces object-centric and distributionally dissimilar images from the original dataset. In contrast, our approach DREAM-ID generates inlier images that can resemble the original ID data, which helps model generalization.
H Experimental Details for Model Generalization
We provide experimental details for Section 4.3 in the main paper. We use ResNet-34 [42] as the network architecture, trained with the standard cross-entropy loss. For both the CIFAR-100 andImagenet-100 datasets, we train the model for 100 epochs, using stochastic gradient descent with the cosine learning rate decay schedule, a momentum of 0.9, and a weight decay of  $5e^{-4}$ . The initial learning rate is set to 0.1 and the batch size is set to 160. We generate 1,000 new ID samples per class using Stable Diffusion v1.4, which result in 100,000 synthetic images. For both the baselines and our method, we train on a combination of the originalImagenet/CIFAR samples and synthesized ones. To learn the feature encoder  $h_\theta$ , we set the temperature  $t$  in Equation (2) to 0.1. Extensive ablations on hyperparameters  $\sigma$  and  $k$  are provided in Appendix J.
I Implementation Details of Baselines for Model Generalization
For a fair comparison, we implement all the data augmentation baselines by appending the original IMAGENET-100 dataset with the same amount of augmented images (i.e., 100k) generated from different augmentation techniques. We follow the default hyperparameter setting as in their original papers.
Figure 11: Visual comparison between our DREAM-ID vs. prompt-based image generation on four different classes.
- For RandAugment [46], we set the number of augmentation transformations to apply sequentially to 2. The magnitude for all the transformations is set to 9.
- For AutoAugment [47], we set the augmentation policy as the best one searched on IMAGENET.
- For CutMix [48], we use a CutMix probability of 1.0 and set  $\beta$  in the Beta distribution to 1.0 for the label mixup.
- For AugMix [49], we randomly sample 3 augmentation chains and set  $\alpha = 1$  for the Dirichlet distribution to mix the images.
- For DeepAugment [50], we directly use the corrupted images for data augmentation provided in their Github repo  ${}^{3}$  .
- For MEMO [51], we follow the original paper and use the marginal entropy objective for test-time adaptation, which disentangles two distinct self-supervised learning signals: encouraging invariant predictions across different augmentations of the test point and encouraging confidence via entropy minimization.
Methods	IMAGENET	IMAGENET-A	IMAGENET-V2
Original (no aug)	87.28	8.69	77.80
RandAugment	87.56	11.07	79.20
AutoAugment	87.40	10.37	79.00
CutMix	87.64	11.33	79.70
AugMix	87.22	9.39	77.80
DREAM-ID (Ours)	88.46±0.1	12.13±0.1	80.40±0.1
We also provide the comparison in Table 5 with baselines that are directly trained by applying the augmentations on IMAGENET without appending the original images. The model trained with the images generated by DREAM-ID can still outperform all the baselines by a considerable margin.
J Ablation Studies on Model Generalization
In this section, we provide additional analysis of the hyperparameters and designs of DREAM-ID for ID generation and data augmentation. For all the ablations, we use the IMAGENET-100 dataset as the in-distribution training data.
Ablation on the variance value  $\sigma^2$ . We show in Table 6 the effect of  $\sigma^2$  — the number of the variance value for the Gaussian kernel (Section 3.2). We vary  $\sigma^2 \in \{0.005, 0.01, 0.02, 0.03\}$ . A small-mild variance value  $\sigma^2$  is more beneficial for model generalization.
Table 5: Model generalization performance (accuracy, in %), usingImagenet-100 as the training data. The baselines are implemented by directly applying the augmentations onImagenet-100.
σ2	IMAGENET	IMAGENET-A	IMAGENET-V2
0.005	87.62	11.39	78.50
0.01	88.46	12.13	80.40
0.02	87.72	10.85	77.70
0.03	87.28	10.91	78.20
Ablation on  $k$  in calculating  $k$ -NN distance. In Table 7, we analyze the effect of  $k$ , i.e., the number of nearest neighbors for non-parametric sampling in the latent space. In particular, we vary  $k = \{100, 200, 300, 400, 500\}$ . We observe that our method is not sensitive to this hyperparameter, as  $k$  varies from 100 to 500.
Table 6: Ablation study on the variance value  ${\sigma }^{2}$  in the Gaussian kernel for model generalization.
k	IMAGENET	IMAGENET-A	IMAGENET-V2
100	88.51	12.11	79.92
200	88.35	12.04	80.01
300	88.46	12.13	80.40
400	88.43	12.01	80.12
500	87.72	11.78	80.29
Table 7: Ablation study on the  $\overline{k}$  for  $k$ -NN distance for model generalization.
<sup>3</sup>https://github.com/hendrycks/imagenet-r/blob/master/DeepAugment
K Computational Cost
We summarize the computational cost of DREAM-OOD and different baselines on IMAGENET-100 as follows. The post hoc OOD detection methods require training a classification model on the ID data  $(\sim 8.2\mathrm{h})$ . The outlier synthesis baselines, such as VOS  $(\sim 8.2\mathrm{h})$ , NPOS  $(\sim 8.4\mathrm{h})$ , and GAN  $(\sim 13.4\mathrm{h})$  incorporate the training-time regularization with the synthetic outliers. Our DREAM-OOD involves learning the text-conditioned latent space  $(\sim 8.2\mathrm{h})$ , image generation with diffusion models  $(\sim 10.1\mathrm{h}$  for 100K images), and training with the generated outliers  $(\sim 8.5\mathrm{h})$ .
L Software and hardware
We run all experiments with Python 3.8.5 and PyTorch 1.13.1, using NVIDIA GeForce RTX 2080Ti GPUs.