FodFoM: Fake Outlier Data by Foundation Models Creates Stronger Visual Out-of-Distribution Detector

Authors: jiankang  chen
ling  deng
zhiyong  gan
wei-shi  zheng
ruixuan  wang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 14:01:15

Abstract: Out-of-Distribution (OOD) detection is crucial when deploying machine learning models in open-world applications. The core
challenge in OOD detection is mitigating the model’s overconfidence on OOD data. While recent methods using auxiliary outlier
datasets or synthesizing outlier features have shown promising OOD detection performance, they are limited due to costly data col-
lection or simplified assumptions. In this paper, we propose a novel OOD detection framework FodFoM that innovatively combines
multiple foundation models to generate two types of challenging fake outlier images for classifier training. The first type is based
on BLIP-2’s image captioning capability, CLIP’s vision-language knowledge, and Stable Diffusion’s image generation ability. Jointly
utilizing these foundation models constructs fake outlier images which are semantically similar to but different from in-distribution
(ID) images. For the second type, GroundingDINO’s object detection ability is utilized to help construct pure background images by blur-
ring foreground ID objects in ID images. The proposed framework can be flexibly combined with multiple existing OOD detection
methods. Extensive empirical evaluations show that image classifiers with the help of constructed fake images can more accurately
differentiate real OOD image from ID ones. New state-of-the-art OOD detection performance is achieved on multiple benchmarks.
The source code will be publicly released.

Main paper: 
 introduction: 
 Out-of-distribution (OOD) detection aims at identifying test samples that are outside the distribution of training data [25, 46, 48] . The ability to detect OOD data is crucial for AI systems, as misclassifying such OOD data as in-distribution (ID) could have severe consequences in some applications like autonomous driving, intelligent healthcare, and industrial manufacturing.To effectively address the OOD detection problem, most existing methods [1, 16, 28, 51] rely on models trained solely on ID data. Some of them analyze potential differences between ID and OOD samples by examining outputs at various locations in the model, such as the softmax output layer, the penultimate or other intermediate layer. Any observed difference could be employed to help design a more effective OOD detection score function [26, 53, 64, 69] . Some other methods like SSD+ [47] and CIDER [35] demonstrate that encouraging the model to learn more compact ID representations in the representation space improves OOD detection performance. However, models with these OOD detection methods still exhibit overconfidence in predicting OOD data as ID class(es) [59] and often struggle with effective OOD detection due to the lack of OOD information during model training. To better address this issue, some methods like outlier exposure (OE) [17] utilize auxiliary OOD samples to train the model and improve decision boundaries between ID and OOD samples. Considering the difficulty in obtaining real auxiliary OOD samples, various generative methods have been proposed to generate fake OOD data, e.g., synthesizing fake OOD images by adversarial generative networks (GANs) [23, 36] or constructing fake OOD features in the feature space as in VOS [12] and NPOS [55] . However, it is challenging for GANs to stably generate even simple images, and the generalizability of VOS and NPOS is limited by the simplified assumption in feature distribution and the complex sampling process. Although the recently proposed Dream-OOD [11] can learn to generate realistic and semantically rich OOD images based on a diffusion model, the only usage of class names to control the generation process affects the scalability.Recently, foundation models [4] with extensive prior knowledge and strong generalization capabilities have been applied in various domains. For example, BLIP-2 [27] possesses powerful image-text generation ability [19, 50] by combining vision and language models. CLIP [41] , which holds strong text-image alignment ability, has been adopted by many large-scale models [29, 42] . Stable Diffusion [44] is known for its ability to use conditional controls to generate photo-realistic images. Besides, being capable of detecting everything, GroundingDINO [30] has been used in open-world.To fully leverage the robust capabilities of the foundation models and address the limitations of current fake OOD data generation methods, we propose a novel framework called FodFoM (Fake outlier data by Foundation Models) for model training. Our framework generates two types of fake OOD data to enhance OOD detection performance. The first type is generated by combining BLIP-2, CLIP, and Stable Diffusion. Specifically, we utilize BLIP-2 and class names to generate exhaustive descriptions for training images of each ID class. These descriptions are then encoded into the textual space using CLIP's text encoder. Based on these text embeddings of ID classes and corresponding mean text embedding of each class, fake OOD text embeddings which are similar to the ID text embeddings are constructed, and such fake OOD text embeddings are then utilized as conditions to guide the Stable Diffusion model to generate challenging fake OOD images that closely resemble the ID classes. The second type is generated using GroundingDINO. In particular, GroundingDINO's capability to "detect everything" is leveraged to identify ID objects in training images. After blurring the detected ID object regions in ID images, manipulated background images without clear ID objects can be obtained. Such manipulated images are highly correlated with the original ID images and used as the second type of challenging fake OOD images. By training the model with these two types of challenging fake OOD images along with ID images, the model can learn to better differentiate OOD images from ID images. Extensive experiments demonstrate that our proposed method achieves superior OOD detection performance across a wide range of benchmarks. The contributions of this study are summarized below.• A novel framework FodFoM is proposed which utilizes the collaboration of BLIP-  
method: 
 In the OOD detection task, the model is expected to identify whether or not a new data belongs to one of the learned ID classes. OOD detection can be viewed as a binary classification task. A scoring function 𝑆 𝜆 is usually designed based on the model's output at a particular layer for OOD detection, where 𝜆 is a threshold. In the testing phase, any new sample resulting in a larger score than 𝜆 is determined as ID, otherwise it is determined as OOD.Our framework FodFoM generates fake OOD images in two ways (Figure 2 ). 1 ○ The first way employs three foundation models, i.e., BLIP-2, CLIP, and Stable Diffusion, to innovatively help construct challenging fake OOD images. Basically, textual description of each ID image is first generated by the image-to-text model BLIP-2, and the combination of the textual description and a simple classspecific prompt is then sent to the CLIP's text encoder to obtain the text embedding for the ID image. Based on text embeddings of all images for each ID class, challenging fake OOD text embeddings outside the cluster of ID text embeddings are innovatively constructed and then used as conditions to control Stable Diffusion to generate challenging fake OOD images. These fake OOD images are similar to but different from ID images, therefore can help the classifier find a more compact decision boundary around each ID class to improve the OOD detection ability. 2○ The second way employs the foundation model GroundingDINO to help construct background images as fake OOD images. The foreground object in each ID image is first detected by GroundingDINO, and then the foreground region is blurred to create a background image for the ID image. Considering that OOD images sharing similar background regions with ID images are often misclassified as ID classes [22, 43] , such background images as fake OOD images will help train a classifier which can better focus on foreground region during inference. Together with training images of ID classes, these two types of fake OOD images can help train a classifier model having better OOD detection performance.The first way to generate fake OOD images is based on foundation models BLIP-2, CLIP, and Stable Diffusion. This generation method is motivated by Chen et al. [6] and Ming et al. [34] 's suggestion that more challenging OOD data help the model find a better decision boundary between ID and OOD classes, as well as by the question "How to generate effective fake OOD images using as much ID semantic information as possible?". Built on BLIP-2's powerful imageto-text capability and Stable Diffusion's strong image-generation ability with text as conditions, the proposed method can generate fake OOD images whose semantic contents are similar to but different from ID images. It consists of three components which are detailed below.An initial caption description is generated by BLIP-2 for each ID image, denoted by BLIP2(x 𝑖 ) for the 𝑖-th training ID image x 𝑖 . Note that although BLIP-2 can generate rich and precise content descriptions for ID images, it does not discriminate the names of ID classes at a fine-grained level. For example, in the upper left part of Figure 2 , BLIP-2 can recognize bird but the output caption is not fine-grained to Brambling or Little blue heron, and can recognize snake but not fine-grained to Night snake. To fine-grain the ID class information in the description to guide Stable Diffusion in generating ID-similar OOD images in the next stage, a simple class-specific prompt 𝑃 (x 𝑖 ), i.e., "This is a photo of 𝑐𝑙𝑎𝑠𝑠_𝑛𝑎𝑚𝑒 (x 𝑖 ) and", is prefixed to the initial caption description for each image, where 𝑐𝑙𝑎𝑠𝑠_𝑛𝑎𝑚𝑒 (x 𝑖 ) denotes the name of the image class for image x 𝑖 . The resulting final textual description 𝑇 𝑖 for image x 𝑖 isEQUATIONwhere [𝑎, 𝑏] denotes the concatenation of two sentences 𝑎 and 𝑏.Diffusion to generate effective fake OOD images that look like ID images, some textual descriptions similar to the ID description are needed. Considering that the textual conditions accepted by Stable Diffusion are vectorized and based on the textual semantic space of CLIP, the text embedding for 𝑇 𝑖 is generated by mapping the description 𝑇 𝑖 into CLIP's textual semantic space via CLIP's text encoder T as follows,EQUATIONWith the powerful textual representation ability from the CLIP's text encoder and the desired text-image alignment ability of CLIP, the text embedddings of ID images are often clustered for each ID class and separated between ID classes in the textual semantic space. For each ID class, any new text embedding which is close to but outside the embedding cluster of the ID class does not belong to this ID class and meanwhile will probably be away from any other ID class. Such new embeddings will be generated as fake OOD text embeddings, with the generation process detailed below.For each ID class, the first step is to identify the text embeddings of the ID images which are located at the periphery region of the associated embedding cluster. Formally, for the 𝑐-th ID class, let 𝝁 𝑐 denote the mean of text embeddings over all the ID images of this class, and J 𝑐 denote the index set of all selected text embeddings from the 𝑐-th class such that for any 𝑗 ∈ J 𝑐 , the cosine similarity (also refer to Supplementary Section 4 for other similarity measurements) between the text embedding t 𝑗 and the mean embedding 𝝁 𝑐 is not higher than a threshold 𝑠 𝛼 . 𝑠 𝛼 is automatically determined such that the size of the index set J 𝑐 accounts for 𝛼% of the total ID images of the 𝑐-th class, where 𝛼 is a hyperparameter. The selected text embeddings associated with the index set J 𝑐 are relatively less similar to the mean embedding 𝝁 𝑐 and therefore located at the periphery region of the embedding cluster for the 𝑐-th class.Then, for each text embedding t 𝑗 associated the index 𝑗 ∈ J 𝑐 , a new text embedding t 𝑜 𝑗 close to t 𝑗 but slightly outside the cluster of the 𝑐-th class can be obtained byEQUATIONwhere 𝝎 𝑗 = (t 𝑗 -𝝁 𝑐 )/||t 𝑗 -𝝁 𝑐 || is the unit vector from the mean embedding 𝝁 𝑐 to the text embedding t 𝑗 , and 𝛾 is the step length hyperparameter. The constructed text embeddings {t 𝑜 𝑗 } for each ID class are collected together to form the set of fake OOD text embeddings. As Figure 3 (Right) demonstrates, similarities between constructed fake OOD text embeddings and the mean embedding are smaller than those for most true ID text embeddings as expected.Generation by Stable Diffusion. The constructed fake OOD text embeddings are used as conditions to guide Stable Diffusion to generate challenging fake OOD images. Stable Diffusion is able to control image generation using data from other modalities as conditions [38, 45] . For text modality, text embeddings from the CLIP's text encoder are employed as conditions. Therefore, the constructed fake OOD text embeddings in the textual semantic space of the CLIP's text encoder can be directly used as conditions to control Stable Diffusion to generate images whose visual contents are consistent with the semantic information of the fake OOD text embeddings. Since fake OOD text embeddings are semantically similar to but different from text embeddings of ID images, the generated images from Stable Diffusion with the fake OOD text embeddings as conditions would also be similar to but different from ID images, therefore can be used as challenging fake OOD images.The other way to generate fake OOD images is based on Ground-ingDINO which has impressive capability in detecting open set objects by leveraging additional texts as queries. This method is inspired by Kim et al. [22] and Ren et al. [43] 's discovery that OOD images sharing similar background with ID images often mislead the model into thinking that these OOD images are semantically similar to ID images and thus incorrectly predicting such OOD images as ID classes with high confidence.To allow the classifier model to focus more on the foreground objects during model training while being aware of the background information, pure background images are constructed as fake OOD images based on ID images and powerful object detection capability of GroundingDINO (Figure 2 , lower left). Specifically, each ID training image along with the textual class name of the image is fed to GroundingDINO to obtain the bounding box containing the foreground object of the ID class. Then, the image region within the bounding box is blurred by a mean filtering operator, resulting in a background image without clear ID object information. For those ID images whose foreground objects occupy the majority of the whole image region, background images would contain much less background information after blurring foreground regions. Therefore, only when area of the foreground region is smaller than a predefined proportion 𝛽% (e.g., 50%) of the whole ID image, the ID image is used to construct a background image as described above.The fake OOD images generated by the above two methods are used together with the ID images to train the classifier's image encoder 𝑓 , the classifier head ℎ, and the projection layer 𝑔 (Figure 2 , lower right). In order for the classifier to distinguish between the challenging OOD images and images of various ID classes, the fake OOD images are considered as an additional class. If there are totally 𝐶 ID classes, then the output of the classifier is (𝐶 + 1)-dimensional, and the commonly used cross-entropy loss L 𝐶𝐸 is adopted to train the image encoder 𝑓 and the classifier head ℎ.In addition, to further separate the individual ID classes and the the class of fake OOD images in the feature space, the supervised contrastive loss L 𝑆𝐶 is adopted to train the model following the idea of SupCon [21] as follows,EQUATIONwhere 𝑁 is total number of training ID images and fake OOD images, A(i) represents all the sample indices in the mini-batch that includes the sample with index i, and P(i) is the subset of A(i) in which all the corresponding samples share the same class label as that of the sample with index 𝑖. z = 𝑔(𝑓 (x)) is the projected visual embedding for the input image x, where the structure of the projection layer 𝑔 is Linear-BatchNorm-ReLU-Linear and 𝑠 (z 𝑖 , z 𝑝 ) represents the cosine similarity between the two embeddings z 𝑖 and z 𝑝 . 𝜏 is the temperature scaling factor. The two loss functions L 𝐶𝐸 and L 𝑆𝐶 help separate ID images from OOD images at two different levels, and the combination of them is utilized to optimize the classifier model, i.e., by minimizing the joint loss function L,EQUATIONwhere coefficient 𝜆 is used to balance the two loss terms.Once the model is well-trained, the image encoder and the classifier head are used to determine whether any new image is ID or OOD.Here the Energy-based OOD detection score from the post-hoc method ReAct [51] is adopted, although other post-hoc methods can also be used. The Energy is defined asEQUATIONwhere ℎ 𝑖 (𝑓 (x)) represents the 𝑖-th logit generated for image x after passing through the image encoder 𝑓 and the classifier head ℎ. The Energy score [31] is defined as the negative Energy, i.e., -𝐸 (x), with larger score indicating the image is more likely from certain ID class. The OOD detection score from ReAct achieves a better performance by pruning high-activation values of the features from the penultimate layer before calculating the Energy score. Note that only the logit values of the 𝐶 ID classes are used to calculate the OOD detection score, and the potential utilization of the logit associated with the OOD class has also been discussed in Supplementary Section 3. 
experiments 3.1 experimental setup: 
 Our method is extensively evaluated on three OOD detection benchmarks, including two small-scale CIFAR benchmarks [24] and one large-scale ImageNet benchmark [55] . Each benchmark contains one ID training set, one ID test set and multiple OOD test sets. For the two CIFAR benchmarks, CIFAR10 and CIFAR100 are respectively used as ID dataset, and SVHN [37] , LSUN-R [62] , LSUN-C [62] , iSUN [61] , Textures [8] , Places365 [68] are used as OOD test sets for performance evaluation. For the ImageNet benchmark, Ima-geNet100 is used as the ID dataset, and four test OOD datasets, iNaturalist [56] , SUN [60] , Places [68] , and Textures [8] , are used for evaluation. The categories of OOD datasets are disjoint from those of the ID dataset. Please see Supplementary Section 1 for more dataset details.We follow the common practice as previous studies [35, 51, 55] to use ResNet18 or ResNet34 [14] as the model backbone on the CIFAR10 and CIFAR100 benchmarks, and ResNet50 or ResNet101 on the ImageNet100 benchmark. For generation of the first type of fake OOD images, BLIP-2 leveraging OPT-2.7b [67] , CLIP-L/14 based on ViT-L/14 as CLIP's Text Encoder, and Stable Diffusion v1.4 are used. For generation of the second type of fake OOD images, GroundingDINO with the Swin-T [32] backbone is used. The hyperparameter 𝛼 is set to 30 for CIFAR10 and CIFAR100 benchmarks, and 20 for ImageNet100 benchmark. The step size 𝛾 is respectively set to 1e-5, 5e-5 and 9e-5 for ImageNet100, such that each selected periphery text emebdding generates three fake OOD text embeddings. For CIFAR benchmarks, 𝛾 is respectively set to 3e-5, 6e-5, 9e-5, 1.2e-4, 1.5e-4, such that each selected text embedding generates five fake OOD text embeddings. The Mean filtering function with kernel size 50 is used to blur foreground regions during background image generation. The proportion hyperparameter 𝛽% is set to 50%. The dimension of the hidden layer in the projection layer is the same as that of input to the projection layer, and the output layer of the projection layer is 128-dimensional.During model training, each ID image or fake OOD image was randomly cropped and resized to 32 × 32 pixels for the CIFAR datasets or 224 × 224 pixels for the ImageNet100 training set, while maintaining the aspect ratio within a scale range of 0.2 to 1. Random horizontal flipping, color jittering, and grayscale transformation were applied to each image. The stochastic gradient descent optimizer with momentum 0.9 and weight decay 1e-4 was used to train each model up to 200 epochs on three ID training datasets. The initial learning rate was 0.05, with warming up from 0.01 to 0.05 in the first 10 epochs when the batch size was larger than 256, and it was decayed by a factor of 10 at the 100-th and 150-th epoch on both CIFAR benchmarks and at the 100-th, 150-th, 180-th epoch on ImageNet100 benchmark. The batch size was set to 512 for CIFAR benchmarks and 128 for ImageNet100 benchmark. The temperature 𝜏 was set to 0.1 and coefficient 𝜆 was set to 1 for all experiments. During testing, only random cropping and resizing were performed on each test image. All experiments were run on NVIDIA GeForce A30 GPUs.The two widely adopted metrics FPR95 and AUROC were used for performance evaluation. FPR95 represents the false positive rate of OOD test samples when the true positive rate of ID test samples reaches 95%, and AUROC is the area under the receiver operating characteristic curve. The lower FPR95 and higher AUROC indicate better OOD detection performance. The performance on each OOD test set with respect to the corresponding ID test set and the average performance over all OOD test sets were reported.Table 2 : OOD detection performance on the CIFAR100 benchmark with ResNet34 backbone and the ImageNet100 benchmark with ResNet50 and ResNet101 backbones. Averaged performances over six OOD datasets for the CIFAR benchmark and four OOD datasets for the ImageNet100 benchmark were reported. 1 summarizes the performance comparison of our method with various strong baselines on the CIFAR10 and CIFAR100 benchmarks. These baselines contain multiple post-hoc methods, which do not require model retraining, including MSP [16] , Mahalanobis [26] , ODIN [28] , DICE [52] , ViM [58] , Energy [31] , BATS [69] , DICE+ReAct [52] , Re-Act [51] , FeatureNorm [63] and LINe [1] . In addition, since our method is based on model training, our method is also compared with various powerful re-training methods, including CSI [54] , SSD+ [47] , VOS [12] , LogitNorm [59] , NPOS [55] and CIDER [35] .As Table 1 shows, our method achieves state-of-the-art performance on four of the six OOD datasets and superior average performance on both benchmarks. For example, on the CIFAR10 benchmark with ResNet18 (Table 1 , upper half), our method's average performance outperforms the best baseline CIDER by a large margin (FPR95 8.43% vs. 14.64%, AUROC 98.33% vs. 97.10%). Similar results were observed on the CIFAR100 benchmark with ResNet18 (Table 1 , lower half), with ResNet34 backbone (Table 2 , left; also see detailed performance on CIFAR benchmarks with ResNet34 on six OOD datasets in Supplementary Table 1 ), and on ImageNet100 benchmark with both ResNet50 and ResNet101 backbones (Table 2 , center and right; also see detailed performance on each OOD dataset in Supplementary Table 2 ). All results supports that the classifiers trained with the constructed fake OOD images learn better decision boundaries for effective OOD detection. As one recent study [3] found, most of the currently used test OOD datasets have an unclean issue, i.e., the OOD test datasets actually contain more or less images of objects belonging to ID classes in ImageNet-1k [9] . To further validate the effectiveness of our method, we used ImageNet100 as the ID dataset and two clean and challenging OOD test sets NINCO [3] and OpenImage-O [58] . As shown in Table 3 , our method again outperforms these competitors on both OOD test sets, further confirming the effectiveness of constructed fake OOD images for OOD detection.Ablation analysis was performed to validate the effectiveness of key components in the proposed framework for OOD detection. As Table 4 shows, the addition of L 𝑆𝐶 (row 2) improves the performance of the original baseline (row 1), indicating that optimization in the feature space is helpful for OOD detection. The introduction of background images generated by GroundingDINO ('BG-OOD', row 3) or ID-similar fake OOD images generated by Stabel Diffusion (with BLIP-2 and CLIP) ('SD-OOD', row 4) outperforms the baseline without any components (row 1) or with L 𝑆𝐶 (row 2), resulting in the current state-of-the-art OOD detection performance in AUROC. The performance is further improved when both types of fake OOD images were involved (row 5). When either type of fake OOD images is combined with L 𝑆𝐶 (row 6 or row 7), the model performs better than that without L 𝑆𝐶 (row 3 or row 4), further confirming the effectiveness of L 𝑆𝐶 . Inclusion of all fake OOD images and L 𝑆𝐶 (last row) achieves the new state-of-the-art performance, supporting that constructed fake OOD images and supervised contrastive learning in the feature space are helpful for learning better decision boundaries for OOD detection.Our proposed framework is insensitive to the value choice of hyperparameters, including the temperature factor 𝜏, the loss coefficient 𝜆, and the threshold 𝛼 for selecting periphery text embeddings. As shown in Figure 4 , on both CIFAR10 and CIFAR100 benchmarks the performance of our method is stable and better than best baseline (CIDER both in AUROC and FPR95 for CIFAR10; ReAct in AUROC, NPOS in FPR95 for CIFAR100) for moderate adjustments of 𝜏 in the range [0.01, 4], 𝜆 in the range [0.05, 2] and 𝛼 in the range [10, 30] . Another merit of our method is its flexible combination with various post-hoc methods. As shown in Table 5 , when applying the OOD detection scores from different post-hoc methods to our framework, each of these post-hoc methods gains a huge boost in performance compared to the original one. Please also refer to Supplementary Section 4 for flexible choice of image captioning module in our framework.As demonstrated in Figure 5 , by combining the strengths of BLIP-2, CLIP, and Stable Diffusion with the construction of fake OOD text embeddings, fake OOD images are generated which are semantically related to ID images but do not belong to ID classes. Therefore, such generated images can serve as challenging fake OOD images to effectively improve the model's OOD detection capability. Please also refer to Supplementary Figure 2 for the constructed demonstrative background images as the other type of challenging fake OOD images. 
related work: 
 Multiple lines of research have been investigated for OOD detection. One line of research performs OOD detection by designing a score function based on certain observed difference in output information of the pre-trained model between ID and OOD samples, such as the softmax output [16, 28] , logits output [15, 31] , gradient information [7] , features of the penultimate or medium layer [26, 51, 53, 58, 63, 64, 69] , or sparsification operations on model weights or feature outputs [1, 10, 52] . The developed methods are also called post-hoc methods. Another line of research deals with OOD detection with additional regularization during model training. Among these methods, GODIN [18] is proposed to decompose confidence scoring during training with a modified input pre-processing method. SSD+ [47] uses supervised contrastive loss to optimize the model, proving that more compact ID representations are beneficial for OOD detection. LogitNorm [59] performs normalization on the logit of last classifier layer to mitigate overconfidence in predicting OOD as ID classes. CIDER [35] is a hyperspherical prototype learning method that allows ID data to form class prototypes during training and keeps each ID class of data close to the corresponding prototype, achieving state-of-the-art performance on the CIFAR10 benchmark. However, performance of these methods is often limited due to learning without any OOD information. In contrast, our method constructs fake OOD data similar or related to ID data to achieve superior performance.Compared to those methods which do not use any OOD data during model training, inclusion of certain OOD training data during model training has proven to be beneficial in further improving model's OOD detection performance [20, 34] . A group of straightforward methods directly utilize additional auxiliary OOD datasets to regularize model training [17, 34] . However, collection of real OOD datasets is time-consuming and costly, making these methods ineffective when real OOD data are unavailable. In order to address this limitation, another group of methods try to construct spurious OOD data to help optimize the model, such as generating OOD images based on GANs [23, 36] , constructing fake OOD images based on the image's own transformations [54, 63] , and constructing pseudo-OOD features in the feature space [12, 55] . This group of methods bring OOD knowledge to the model to certain degree and enhance the model's OOD detection ability. However, these methods are often constrained, e.g., by the instability of training GANs [23, 36] or by the simplified assumption of conditional Gaussian distribution in the feature space [12, 55] . The recently proposed Dream-OOD [11] can generate fake OOD images by learning a text-conditioned latent space based on single-word ID class names and utilizing Diffusion Model in low-likelihood regions in the latent space (same as NPOS [55] ), but pre-learning of the text-conditioned space and the inability of handling multi-word class names make it challenging to generalize this method. In contrast, our method needs no such pre-learning and can construct more semantically informative descriptions which are then used to help construct fake OOD text embeddings in the latent space of CLIP's text encoder.Recently, a variety of foundation models have been proposed, such as CLIP [41] , ChatGPT [39] , BLIP-2 [27] , DINO [5] , Ground-ingDINO [30] , and Stable Diffusion [44] . These foundation models contain rich prior knowledge and have been applied to various tasks, such as few-shot learning [65, 66] , open-set object recognition [40] , and image retrieval [2] etc. In addition, OOD detection methods based on CLIP [13, 33, 49, 57] and Stable Diffusion [11] are also becoming more popular very recently. In this study, a novel framework for OOD detection is designed which can generate fake OOD images by leveraging rich knowledge and strong downstream generalization capability from multiple foundation models. 
conclusion: 
 In this study, a novel OOD detection framework called FodFoM is proposed by jointly utilizing multiple foundation models to generate fake OOD images. Specifically, two sets of fake OOD images are constructed, with one set generated by innovatively utilizing BILP-2, CLIP, Stable Diffusion, and the other set generated by utilizing GroundingDINO. Both sets of fake OOD images are similar to but different from ID images, thus helping the classifier model find more compact decision boundaries around ID classes. Sufficient empirical evaluations confirm that the generated fake OOD images based on these foundation models are effective to improve the model's ability in OOD detection, with state-of-the-art performance achived on multiple benchmarks. Moreover, FodFoM can be easily and flexibly combined with existing post-hoc methods. We expect that this study will inspire future research in better utilizing foundation models' rich and distinct knowledge to synthesize OOD data for enhanced OOD detection. 
 
 Appendix:  


Decision: This paper aims to leverage the power of foundation models to address the OOD detection problem. The basic idea is to generate higher-quality outliers to ensure that the model have a better OOD-detection ability. However, the experiments still need improvement, as many outlier-exposure methods are not included in the current version. We encourage merging more baselines' results into the paper in the next round.

TPC addendum: This paper overall has positive reviewer ratings. Reviewers are also generally happy with the rebuttal. Reviewers also raised some issues regarding lack of comparisons with some existing methods. Authors must incorporate these studies into their final version. The TPCs decide to accept this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces a novel OOD detection framework that generates two types of challenging fake outlier images. The first one uses multiple foundation models: BLIP-2, CLIP, and Stable Diffusion to generate fake images, while the second one uses DinoGround to blur the object as the OOD images. These fake images are semantically similar to but distinct from in-distribution (ID) images, enhancing the model's ability to distinguish between ID and OOD data. The proposed method achieves state-of-the-art OOD detection performance across multiple benchmarks, demonstrating its effectiveness in improving classifier robustness in open-world applications.
strength_weakness: strength_weakness: •	The authors find a good way to combine the foundation models to create OOD images and use it to train OOD detection in classifiers,

•	The proposed method does not require any auxiliary dataset for OOD detection.

•	The paper is well-written as it is clear and easy-to-understand

•	The authors did a comprehensive experiment on CIFAR datasets compared to previous OOD detection methods.
limitations: limitations: •	In Fig.5, the authors show some qualitative results of the ood images generated with the proposed method. However, some images are ambiguous between ID and OOD. For example, the strawberries on the cake, and the keyboard on the calculator. Is there a way to guarantee the OOD images are truly OOD? Is there a quantitative way to measure such?

•	The foundation models are usually pretty large and slow, especially combining three of them altogether. The efficiency comparison, such as memory usage or time required to create synthetic OOD images for training, is not discussed in the paper.

•	Though using a series of Foundational Models can achieve OOD image generations, it could also potentially accumulate their biases or noise. Are there failed cases of generated images?

•	Minor issue:
In Fig. 5, the last row is labeled leaf beetles while the images seem to be keyboard.
paper_summary: paper_summary: The work presents a novel framework called FodFoM for OOD detection. The method addresses the challenge of model overconfidence on OOD data by generating two types of challenging fake outlier images using a combination of multiple foundation models (BLIP-2, CLIP, Stable Diffusion, and GroundingDINO). These images are used to train classifiers to better differentiate between ID (In-Distribution) and OOD images. Extensive empirical evaluations show that this method achieves state-of-the-art OOD detection performance on multiple benchmarks.
strength_weakness: strength_weakness: - Adopting foundation models leads to a strong capability for OOD detection of the classifier.

- The paper includes extensive empirical evaluations on multiple benchmarks, demonstrating the superiority of the proposed method over existing baselines.

- Generation of both ID-similar and background OOD images is intriguing.
limitations: limitations: - Lack of proof in the authors' assertion "These fake OOD images are similar to but different from ID images, therefore can help the classifier find a more compact decision boundary around each ID class to improve the OOD detection ability." One must prove what has been said, it would be meaningful to observe how the decision boundary moves for each of the adopted techniques.

- Some additional clarification on the novelty of the proposed method, and how these differentiate from the context of other works that generate OOD data for training purposes would be beneficial.

- Some sections could benefit from additional clarity, particularly in explaining the integration of multiple models.
paper_summary: paper_summary: The paper introduces FodFoM (Fake outlier data by Foundation Models), a novel framework for improving Out-of-Distribution (OOD) detection. The core idea is to utilize multiple foundation models to generate two types of challenging fake outlier images for training classifiers. The first type of fake outlier images is created by combining BLIP-2's image captioning, CLIP's vision-language alignment, and Stable Diffusion's image generation capabilities. These images are semantically similar but distinct from in-distribution (ID) images. The second type involves using GroundingDINO's object detection to blur foreground ID objects in ID images, creating manipulated background images.
strength_weakness: strength_weakness: Innovative Use of Foundation Models: The framework cleverly combines the strengths of various foundation models (BLIP-2, CLIP, Stable Diffusion, GroundingDINO) to generate challenging fake OOD images, enhancing the OOD detection capability.

Improved OOD Detection Performance: Extensive experiments show that the proposed method achieves state-of-the-art performance across multiple benchmarks.

Flexibility: The framework can be combined with multiple existing OOD detection methods, providing versatility.

Automatic Fake OOD Image Generation: The approach automates the generation of fake OOD images, reducing the need for costly and time-consuming data collection.
limitations: limitations: The framework involves multiple sophisticated models, which might increase the computational complexity and the difficulty of implementation.

It seems that there are many other methods that can ease the requirement of OOD data, such as DAL [1] and [2], which I think should be included in the discussion. 

The suggested method, all in all, involves the auxiliary OOD data during training. Therefore, the authors should compared with outlier exposure and its variants. 


[1] Learning to Augment Distributions for Out-of-distribution Detection

[2] Contrastive Training for Improved Out-of-Distribution Detection

[2] Contrastive Training for Improved Out-of-Distribution Detection
paper_summary: paper_summary: The paper proposes a new framework called FodFoM for Out-of-Distribution (OOD) detection. The existing methods for OOD detection either require a lot of data to train on, or they make simplifying assumptions about the data. FodFoM addresses these limitations by using a combination of different foundation models to generate fake outlier images for training. The paper shows that using these fake images can improve the accuracy of OOD detection on multiple benchmarks.
strength_weakness: strength_weakness: 1. Novelty: The paper proposes a new approach to generate fake OOD data for training OOD detection models. This approach leverages the capabilities of multiple foundation models (BLIP-2, CLIP, Stable Diffusion, and GroundingDINO) to create more realistic and challenging fake OOD images compared to previous methods (e.g., GANs).
2. Clarity: The paper offers a clear writing and follows a logical flow.
3. Adequate evaluation: The paper mentions extensive evaluations on multiple OOD detection benchmarks and claims to achieve state-of-the-art performance.
limitations: limitations: -The generation process of fake OOD text embeddings raises concerns about potential overlap with existing ID class clusters. Given the fine-grained categories within ImageNet, how do the authors ensure these newly generated embeddings remain distinct from established clusters?

-The paper mentions leveraging GroundingDINO for background manipulation. However, it's unclear how the authors address potential recognition errors made by GroundingDINO. Are these error samples included in the training data, or is there a correction mechanism employed?

-While a subset might be used for efficiency, how do the authors ensure the chosen subset adequately reflects the fine-grained categories within ImageNet? Is there a discussion on the generalizability of the findings to the full dataset?

-The paper proposes using semantically similar text embeddings for generating OOD images. However, a potential concern exists regarding the model's generalization on in-distribution (ID) data. Did the authors investigate the impact of this approach on ID data performance? Are there any relevant experiments to support the findings?