LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition
Authors: Haoxuan Qu
Xiaofei Hui
Yujun Cai
Jun Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep learning
Open-set object recognition
Large models
Training-free

Abstract: Open-set object recognition aims to identify if an object is from a class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge is how to reduce the reliance on spurious-discriminative features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named Large Model Collaboration (LMC) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs to effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available \href{https://github.com/Harryqu123/LMC}{here}.

LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition
Haoxuan Qu"
Xiaofei Hui SUTD
SUTD
Singapore
Singapore
sg
xiaofei heaffeihhu@@sutd.c edu. sg
Yujun Cai
Jun Liu f SUTD
Meta
U.S. yujuncai@meta. cownncciimmtt.ccmm
Singapore
jun_ [im_liu@sutd.ede edu. sg
Abstract
Open-set object recognition aims to identify if object is from class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge how reduce the reliance spurroonsdiierrimnnaate features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named Large Model Collaboration (LMC) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available here.
1 Introduction
Open-set object recognition aims to identify if an object is from a closed-set class that has appeared during training, or an open-set class that has not been encountered the training set. a crucial task since open-ser classes always exist in the real world [63], and misidentifying open-ser classes into closed-set ones can lead to severe consequences in many practical applications [2], such as autonomous driving [19], robotics system [54], malware classification Due to importance, open-set object recognition has received lot of research attention recently and various methods have been proposed [14, 36, 28, 6, 66, 56, 12].
[18], and medical analysis [15].
To perform open-set object recognition well, some existing works [34, 40] have noted that a key challenge is how to reduce the reliance spurious-discriminn discriminative features. Specifically, spurious- discriminative feature refers a feature that satisfies the following two requirements:
it discriminative between closed-set class and other closed-set classes; 2) it is shared between this closed-set class and certain open-set classes. As suggested by previous works [34, 40], while such spurious ciriiiouuddsriimintiii features can facilitate classification within the closed set, in open-set recognition, relying on such features can make closed-set and open-set classes to be easily confused
Bott authors contributed equally the work. CCorrepoondigg Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

with each other, and thus bring negative effects in accurately identifying open-set objects. For example, as shown in Fig. 1, given closed set of classes including bighom, tabby cat, and hog. within these closed-set classes, "with horn" as feature of bighorn can be used to discriminate bighorn from other classes (tabby cat and hog). However, relying on "with horn" to perform open-set object recognition, bighorn can be easily confused with potential open-set classes such as ox and gazelle that are also "with horn". Therefore, it can be difficult to correctly identify ox and gazelle as open-set objects. To avoid open-set objects from being misidentified into closed-set classes, common type of methods is to simulate additional virtual open-set samples or classes [14, 36, 67, 12, 34]. Specifically, among this type of methods, various external knowledge bases have been used, such as pre-trained Glo embeddings [67] and extra datasets [12]. Yet, despite the considerable efforts made and various architectures specifically designed for this task, the open-set object recognition task remains challenging. This is because, in the real world, various different sorts of open-set classes can appear [1]]. Moreover, given trained classifier, to further equip it with the ability to identify open-set objects, most existing open-set recognition methods require an extra training process. Especially when the classifier has been deployed, conducting such extra training process can be inconvenient, as it needs both additional training time and access to training data. Note that the training data can be no longer accessible during deployment g., due privacy concerns [35, 22)).
Recently, large models containing very rich implicit knowledge [17] have served as powerful external knowledge bases in many areas [13]. For example, ChatGPT [38] possesses rich common-sense knowledge, and has been applied in various natural language processing tasks [42, 20]. DALL-E [47], which holds strong image-generative ability, has been used in different generation tasks [32, 26]. CLIP [45] has been leveraged for visual recogmition tasks [72, 41] since can align relevant image-text pairs. Besides, being capable of aligning rel- evant image patches, DINO [5] has
closed-set samples
open-set testing samples hom)
bighern (wtth horn)
Expected set probability
Predicted closed-set
probability of DIAS: 0.94
tabby cat |wihout
gazelle
Expected closed prababilty:
hog
|without hamn)
Predicted cloved-set
probability of DIAS: 0.67
Figure I: Illustration of spurious-discriminatie fea- ture example. As shown, given "with horn" spurious- discriminative feature of the closed set, DIAS [34] as re- cently proposed open-set recognition method misidentifies ox and gazelle (both with horn) as closed-set classes with high probability.
been applied in areas [55, 1] such as image retrieval.
Inspired by that large models often contain rich knowledge, here we aim investigate leveraging such rich knowledge to handle the open-set objecl recognition task. However, using large models to perform open-set object recognition is non-trivial due to the following challenges: 1) It can be difficult to directly leverage an existing large model to effectively perform open-set object recogmition. Take CLIP as an example, being able align an image with closed set of classes and thus finding out the most relevant class in the set, CLIP has been applied in various visual recognition tasks, such as zero-shot and few-shot image classification [45]. However, CLIP cannot handle the open-set object recognition task effectively by itself. This is because, various sorts of open-set classes exist in the real world, and it is difficult know which class should the testing image be aligned with when using CLIP. 2) Meanwhile, as the knowledge in large models is often implicit [57, 3]. it can be challenging to extract the desired knowledge (e.g.. in our case, the knowledge that can facilitate open-set object recognition) from large models successfully [57]. To handle these challenges, in this work, with the observation that different large models pre-trained through different paradigms can contain distinct knowledge, first propose novel open-set object recognition framework named Large Model Collaboration (LMC). As shown in Fig. 2, the proposed LMC performs open-set set object recognition in training-fire free manner by leveraging rich and distinct knowledge from different fff-the-shell large models complementarily. Moreover, LMC is also incorporated with several novel designs to facilitate extracting implicit knowledge in large models effectively.
Overall, to better perform open-set object recognition, LMC aims to reduce the reliance of this task on sparious-dixerimineee discriminative features. To achieve this, LMC proposes to simulate additional virtual open-set classes that share the spurious-discriminntive features through the collaboration of different large models. Specifically, in LMC, motivated by that ChatGPT possesses rich common-senss knowledge, we first adopt ChatGPT to expand the list of closed-set set classes with additional virtual
2

---

open-set classes. During this process, to extract implicit knowledge from ChatGPT effectively, we propose to guide ChatGPT with intermediate reasoning, while also enabling ChatGPT to perform self-checking on whether it covers as many spurious-discrimnntii inspired by that images can catch different details that are complementary to the text domain [60], in addition to representing classes in the expanded list with their names in the text domain, we propose to further use image information to better represent these classes. Specifically, to facilitate extracting image information from large models, we propose to incorporate our framework with novel cyclic module, which collaborates ChatGPT, DALL-E, and CLIP through a cross-assessing mechanism to generate diverse images for each class. Once both the text and image information have been extracted for each class in the expanded list, during inference, we propose to use the image-text aligmment ability of CLIP and the image-image aligmment ability of DINO in collaboration to perform open-set object recognition on testing images.
features as possible. Moreover,
The contributions of our work are summarized as follows. 1) We propose LMC,, a novel framework that can handle the open-set object recognition task in a training-free manner, via collaborating different off-the-shelf pre-trained large models and leveraging their knowledge in a complementary manner. 2) We introduce several designs in LMC to extract both text and image information from large models effectively. 3) On the evaluated benchmarks, LMC achieves state-of-the-ar performance.
2 Related Work
Open-set Object Recognition. Due to the wide range of applications, open-set object recogmition has received a lot of research attention [48, 4, 50, 14, 63, 39, 68, 53, 51, 7, 6, 33, 23, 56, 61, 19, 58, 70, 31, 71, 49, 21]. Bendale and Boult [4] made the first attempt of applying deep neural networks into the task of open-set recognition. Specifically, they first showed that simply thresholding on the softmax probability cannot yield a robust open-set recognition, and thus proposed Openmax based on the Extreme Value Theory. Ge et al. [14] further proposed G-Openmax to generate unknown samples using generative models. After that, Neal et al. [36] showed the effectiveness of generating images that do not belong any of the closed-set classes but look similar to those images in the training dataset. Later on, OpenGAN, which uses real open-set images to perform model selection, was proposed by Kong et al. [23]. Vaze et al. [56] proposed to well-train the closed-set classifier so that can perform both closed-set image classification and open-set recogmition well. Besides, Esmacilpour et al. [12] proposed to train an image description generator using an extra image captioning dataset and use the trained generator to improve the open-set object recognition performance.
Different from existing approaches, in this work, from novel perspective, we investigate how to collaborate different large models to perform open-set object recognition in a training-free manner. Besides, we also introduce several designs extract implicit knowledge from large models effectively.
Large Models. Recently, a variety of large models have been proposed, such as ChatGPT [38], DALL-E [47], CLIP [45], and DINO [5]. Such large models contain rich knowledge and have been studied in various tasks [20, 65, 69, 32, 64, 27, 1], such as language translation [20]. video generation [32], and image retrieval [1]. In this work, we design a new framework to handle the open-set object recognition task by cross cross-assessing among large models and leveraging the rich and distinct knowledge from different large models in complementary manner.
3 Proposed Method
Given a testing image from Y. Y., where Y. represents closed-set classes in the training set and Y. represents open-set classes that are not included in the training set, the goal of open-set object recognition is to identify if the image is from a closed-set class or an open-set class. To recognize open-set objects accurately, as suggested by previous works [40, 34], a key challenge is: how to reduce the reliance on inspired by that large models can hold very rich while distinct implicit knowledge due to their different training paradigms, we propose a novel open-set object recognition framework LMC. Specifically, as shown in Fig. 2, LMC first extracts the implicit knowledge from different off- th-thh-shelf large models to simulate additional virtual open-set classes that share the spurions-discriminaie features. After that, LMC utilizes these virtual open-set classes to make the puuriousdiiccriminatie features "less discriminative" during its inference process.
features. To better tackle this challenge, in this work,
3

---

Below. we first describe how LMC extracts implicit knowledge from different large models to effectively simulate virtual open-set classes. After that, we introduce the inference process of LMC.
3.1 Virtual Open-set Class Simulation
In the proposed LMC, to reduce the reliance on spurious- discriminative fea- tures, we aim to utilize the rich implicit knowledge of different large models to simu- late additional virtual open-set classes. Specifically, motivated by that knowledge from different modali- ties (e.g., text and image) can contain different details [60], we aim to leverage implicit knowledge of large models in both the text domain and the image domain to facilitate simulation. To achieve this, we first simulate the names of virtual open-set classes and expand the list of closed-set classes with these virtual open-set classes. We then further generate diverse images for each class in the list to describe each class better.
selfcheediing
module
Mitiual open-set clesss
ChatGPT
Danngeco, st
CharGPT
closed ast casses [nabbe, dolohin... ..]
CLIP
DALL-E
testing images arbitrary classes
open-set object recogrition results
concatenate incorporate
Figure 2: Illustration of the proposed LMC framework.
Simulating names of virtual open-set classes. Inspired by that ChatGPT holds rich common-sense knowledge, here we aim to simulate names of virtual open-set classes by asking ChatGPT questions. Specifically, to perform such a simulation in high quality, we find that, it is necessary to 1) ensure that ChatGPT can well understand the question, and 2) encourage ChatGPT to cover many spurious- discriminative features as possible. To achieve these, we introduce the following two designs in our framework.
Inspired by chain-of -ffthuughh [59] that large language models can better understand the given instruction when they are guided with intermediate rea- soning, to ensure that ChatGPT can well understand the question, we here also guide ChatGPT with intermediate reasoning. Specifically, we find that an effective way to guide ChatGPT to ask ChatGPT the following three questions for each closed-set class: 1) "Given list of classes [classes], can you describe the visual features of each class in the list?", [classes] represents the names of the closed-set classes; 2) "What are the discriminative visual features of class [class]
Given a lss of casses [aadybug dragonfly, goose, .... can you describe the steual featuree of each class the list?
Sum Ladybug is small mund beetle that as typically red or mange with back spols. Its body aovered in a heed shiny snell Dragonfly is
What are the deecriminaivve visual features dass ladybug sompar red with other dasses in the lis??
Compared with other classes in the list, lacybuza have several diserminaaiv navie visual features including: color (adybups are typicallyy red orange with slack spots); shape (ladybuug have a round shape)
Can you lst other classes that alse chare these detoem visual featuree?
here some other clesses that shere some the decriminatie vocual faveturss ledyluuge
Tortoise beetle a lype of beelle that has wimler body shrep
hard, shiny whnel to tedyytuge, and some spaciass have simbir bright volone and markings.
Ladybird spider specios spider that hass simbar coloneiio spotied pertern in bedyhugge
where
Figure 3: Formulation of language command for ChatGPT in step-by-step manner with intermediate rationales involved.
compared with other classes the list?", where [class] represents the closed-set class that the current set of questions is asked for; 3) "Can you list other classes that also share these discriminative visual features?", In Fig. 3, we illustrate an example when the above three questions are asked for the class ladybug. As shown, by formulating the language command in this way, besides the final answer to question 3), ChatGPT is also demanded generate intermediate rationales that lead to the final answer through the first two questions. This can help ChaiGPT to better understand our final purpose, i.e., simulating virtual open-set classes that share spurious-discriminn features. In Particular, in Fig. 3, tortoise beetle and ladybird spider are the simulated virtual open-set classes that share spurious- disrriiinliseriiinaatiiee features with the closed-set class ladybug. Through the above three questions, we can simulate the names of virtual open-set classes

---

that share spurious-discrimmnativ discriminative features with a single closed-set class. To simulate the names of virtual open-set classes for all closed-set classes, we can simply ask the above three questions for each class in the closed set either sequentially, or concurrently via different ChatGPT instances.
In addition, to simulate a comprehensive list of virtual open-set classes to better cover the spurious- discriminative features, we aim to further enable ChatGPT to perform self-checking. In other words, we aim to guide ChatGPT to re-think if there are other spurious-discriminn discriminative features that are missed. To achieve this, for each closed-set class, after asking the above three questions, we expand the list of classes in question 1) with the obtained virtual open-set classes and re-ask ChatGPT the three questions. Note that by expanding the list with the obtained virtual open-set set classes, we exclude the spurious spurious-discrimiinn discriminative features that are already covered and thus force ChatGPT look for features that are not discovered yet. This iteration of self-checking terminates when ChatGPT stops proposing new virtual open-set classes or a maximum number of cycle times is reached. After performing self-checking. inspired by [34] that all sorts of classes can appear in the real world, we also follow [34] to simulate virtual open-set samples that are less similar to closed-set samples (sharing less sparious-discrimingg discriminative features) via asking ChatGPT: "Given a list of classes [classes], can you name classes that are not similar to them?". Note after expanding the list of closed-set classes with simulated virtual open-set classes through applying the above designs, those spurious-disee distimiiiiiiiiinaiiiee features that are originally discriminative among the closed-set classes would be "less discriminative" in the expanded list.
cnurioos-discruminaati
Generating diverse images and cross-assessing. Above we simulate the names of virtual open-set classes Y._.. Here, inspired by that images can catch different details that can be complementary to the text domain [60], we aim to describe both the closed-set classes and the simulated virtual open-set classes in more detail by further generating diverse images for each class E Y, U Yo_.- These generated images will be utilized in the inference process of LMC to reduce its reliance on aparious-discriminatii features.
Specifically,, for each class y E Y. U Yo_o. we generate diverse images for class y through the following two steps: 1) we first ask ChatGPT to generate diverse text descrip- tions as: "Can you write [K] diverse descriptions, each describing a different scene about the class [y] ?" where K is hyperparameter representing the number of de- scriptions generated per class. 2) After that, leveraging the image-generative ability of DALL-E [47], we ask DALL-E to generate images based on these descriptions for each class as:
* = D(LL- where k E {L,..., K}
(I)
where di denotes the k-th description generated by ChatGPT for class y, and it denotes the image generated by DALL-E based on dt.. While we can directly use the images generated above during inference, as shown in Fig. 5 which demonstrates number of such images, we realize that some of such images may not hold a high enough quality to accurately represent their desired classes (e.g.. ih may not represent class y accurately). Thus, directly using the above images can lead to suboptimal model performance during inference.
To address this problem, referring the images that cannot accurately represent their desired classes as the less accurate images, we aim to automatically detect these less accurate images, refine their corresponding descriptions, and regenerate them accordingly. To achieve this, inspired by that humans can refine their understanding based on others' feedback, here we want to investigate, can a large model also refine its output based on another large model's 's feedback? In particular, we propose a cyclic assessing module, where CLIP acts as the "feedback provider", and provides ChatGPT with feedback about the less accurate images to help refine their corresponding descriptions.
Specifically, denote tt an image generated for class y, and D the text descriptions "a photo of [class]" for all the classes (i.e., Y. U Y. The aross-assessing module operates by iteratively conducting the following three steps: 1) Detection. To detect the less accurate images, we first leverage the strong image-text alignment ability of CLIP to align it with D as:
Passess = softmar (CIIP vis (1t) (CLIP text (D))²
(2)
where CLIP pis denotes the CLIP's visual encoder, and CLIP text denotes the CLIP's text encoder. The image E is determined as less accurate if compared class y, it is more aligned towards another class u (i.e., instead of class y. another class u corresponds to the highest probability value in Paeeess). 2) Refinement. Next, if tt is determined as a less accurate image, in order for its corresponding
5

---

description dr to describe class y more properly, we ask ChatGPT to refine d based on feedback from CLIP as: "This description [d]]] seems more like class [u] to me. Can you refine it to enhance the characteristics of class [y]?". 3) Regeneration. Lastly, the image i is regenerated through DALL-E based on the refined text description.
Through conducting the above three steps iteratively, during every iteration, we can leverage CLIP to provide specific feedback for ChatGPT so that ChatGPT can refine the descriptions more toward the direction of our desire. With these refined descriptions, images that can well represent each class will be generated. The above iteration terminates when all the generated images are most aligned with their desired classes or a maximum number of cycle times is reached. Note after reaching the maximum number of cycle times, discard those images that are still less accurate.
3.2 Inference Process
Above we expand the list of closed-set classes with additional virtual open-set classes, and generate diverse images for each class in the expanded list. In this section, we discuss how our framework utilizes the expanded list and the generated images to make the spurious-disct distiiiiiiiiiiiiee features "less discriminative" during its inference process. Note our framework does not require any training process before its inference, which can facilitate its convenient usage in real-world applications.
Specifically, during inference, inspired by the strong image-text alignment ability of CLIP [45] and the strong image-image alignment ability of DINO [5], we propose to adopt both CLIP and DINO to perform open-set object recognition. Note CLIP and DINO are used in collaboration since they can complement each other well. To be more concrete, by aligning the testing image with the names of all the classes in the expanded list (i.e., Y. U Yo_.), with the overall concept of each class. Meanwhile, by leveraging DINO to align the testing image with the generated images of each class in image patch level, we better match the testing image with the detailed local features of each class.
first adopt CLIP to match the testing image
Before entering the inference process, with the notice that certain CLIP and DINO features are used repeatedly across different testing images, we first pre-store these features to keep the efficiency of the inference process. Specifically, denoting [C CLIPteart (D) the CLIP text feature of all classes, we first store /CLIP which has already been computed the above Eq. 2. Moreover, denoting DINO the DINO's s encoder and Ly the set of images generated from class y through the cyclic cross-assessing module, we compute the DINO visual feature for class as fy fDINO DINO via (Ty) and store fo /DINO for each class y Y. YY...
After pre-storing these features, we then introduce the inference process of the proposed LMC framework. Specifically, given a testing image iteat. utilizing pre-stored features, LMC first aligns destt with the names of all the classes in Y. U Yu_o through CLIP as:
PCLIP = so ttmar (CLIP Cic.PPisstteeee (ftest)) ( (((UP)T
where Per. IP denotes the softmax probability derived from CLIP. At the same time, LMC also aligns lestt with the generated images through DINO as:
PDINO so somfmarr (1 y E Y.YUU., YuYoot)) where II average (frees) (DINO)
where Io is derived from aligning iteat with Ly through DINO, and PDINO denotes the softmax probability correspondingly calculated for all the classes in Y.YY, oo- After deriving PCLIP and PDINO. LMC then incorporates them to determine whether itest is from the closed-set classes or open-set classes as:
Pine aPCLIP + (1 a))DDIO, S = =mxx (Pino()lireat)) DEY.
where a is hyperparameter denoting the incorporation weight, Pine denotes the softmax probability derived from incorporating PeLIP and PDINO- (yne(y/ denotes the probability value that class y corresponds to in Pincs and S E [0, 1] is the closed-set set score measured by our framework. Following [34], we represent as the maximum probability value among the closed-set classes. By performing open-set object recognition in the above way, open-set images would no longer be easily misidentified into the closed-set classes due to the contained spurious-discrimnnen expanding the list of closed-set classes with the virtual open-set classes, in the expanded list, besides feature can held by some virtual ope classes as well. Then, when an open-set testing image containing spurious- discrmiinative features is
features. This is because, after
a certain closed-set class, a ppariou-discriminaate

---

Table 1: The AUROC results on the detection of closed-set set and open-set samples. Results are
averaged over five random dataset splits following [6, 56].
Method
Venue
CIFARIO
CIFAR+10
CIFAR+50
TinylmageNet
Methods that involve training process
OSRCI [36]
ECCV 2018
69.9 3.8
83.8
82.7 +
58.6
C2AF [39]
CVPR 2019
89.5
95.5
93.7 +
74.8
RPL. [7]
ECCV 2020
90.1
97.6
96.8
80.9 =
OpenHybrid [68]
ECCV 2020
95.0
96.2
95.5 +
79.3
CVAECapOSR [16]
ICCV 2021
83.5 2.3
88.8 1.9
88,9
71.5 1.8
ARPL [6]
TPAMI 2021
90.1 + 0.5
96.5 + 0.6
94.3 + 0.4
76.2 0.5
ARPL+CS [6]
TPAMI 2021
91.0 0.7
97.1 + 0.3
95.1 0.2
78.2 + 1.3
PMAL [31]
AAAI 2022
95.1 +
97.8
96,9 +
83.1 +
ZOC [12]
AAAI 2022
93.0 + 1.7
97.8 1 0.6
97.6 0.0 0.0
84.6 + 1.0
MLS [56]
ICLR 2022
93.6 1
97.9 1
96.5
83.0 +
DIAS [34]
ECCV 2022
85.0 2.2
92.0 + 1.1
91.6 0.7
73.1 1.5
Class-ince lus-inccusion [8]
ECCV 2022
94.8 =
96.1 +
95.7
78.5
[DL[30]
TPAMI 2022
85.7 + 1.3
89.1 + 1.4
88.3 0.0 0.0
76.4 1.7
ODL+ [30]
TPAMI 2022
88.5 - 1.3
91.1 + 0.8
90.6 0.0
74.6 0.8
CSSR [19]
TPAMI 2022
91.3 =
96.3 -
96.2
82.3
RCSSR [19]
TPAMI 2022
91.5 +
96.0 +
96.3 +
81.9
Methods that involve
training process
Softmax
93.7
96.5 0.6
95.1
83.5 2.7
Ours
96.6 0.3
98.9 0.7
98.5 +
86.7
Table 2: The OSCR results for open-set object recognition. Results are averaged over five random dataset splits following [6].
Method
Venue
CIFARIO
CIFAR+10
CIFAR+50
TinylmageNet
Methods that involve training process
GCPL [62]
CVPR 2018
84.3 + 1.7
91.0 = 1.7
88.3 = I.I
59.3
+ 5.3
RPL
ECCV 2020
85.2 1.4
91.8 + 1.2
89.6
+ 0.9
53.2
+ 4.6
ARPL [6]
TPAMI 2021
86.6 1.4
93.5 +
0.4
62.3
3.3
ARPLACS [6]
TPAMI 2021
87.9 1.5
94.7 - 0.7
0.7
92.9
+ 0.3
65.9 + 388
ODL [30]
TPAMI 2022
84.8
1.4
92.5 + 1.0
89.8
+ 0.7
64.3 + 3,2
ODL+ [30]
TPAMI 2022
86.9 + 1.5
93.2 + 0.3
90.3
+ 0.2
59.2 + 2.1
Methods that involve extra training process
Softmax
90.1
1.1
94.0 + 1.4
92.8 1.1
75.1
5.0
Ours
93.6
1.5
96.8 + 0.7
96.4 + 0.4
80.6 + 3.4
aligned with the expanded list, will tend approach both the closed-set and virtual open-set classes that contain these features at the same time, rather than approaching only certain closed-set classes. Therefore, after softmax normalization, the closed-set score of the open-set testing image will no longer be easily overestimated due to the spurious sparious-discriminns discriminative features contained in the image.
Experiments
To evaluate the effectiveness of our proposed framework LMC, we conduct experiments on four evaluation protocols including CIFAR CIFAR+10, CIFAR+50, and TinylmageNet. We conduct our experiments on an RTX 3090 GPU.
4.1 Datasets and Evaluation Metric
CIFARIO. CIFARIO [24] a ten-class object recognition dataset. On this dataset, following [6, 56], 6 closed-set classes and open-set classes are randomly sampled for evaluation each time.
CIFAR+10 & CIFAR+50. Following [6. 56], we also evaluate method on CIFAR+10 and CIFAR+50, which are extensions the CIFARIO evaluation protocol. Specifically, following [6. 56), for the CIFAR+N experiments, 4 classes are randomly sampled from CIFAR- 10 as the closed-set classes and N non-overlapping classes are randomly sampled from CIFAR-100 [24] as the open-set classes for evaluation (N- 10 for CIFAR+10 and N 50 for CIFAR+50).
[9] subset of ImageNet and contains 200 classes. Following [6. 56], each time, 20 closed-set classes and 180 open-set classes randomly sampled.
7

---

Evaluation metric. Following [6, 56], we use the following two metries: AUROC and OSCR [10]. Among these two metrics, AUROC focuses on evaluating model performance on identifying open-set samples, and OSCR measures the trade-off between classification accuracy and open-set recognition performance. More details especially about the OSCR metric are in the supplementary.
4.2 Implementation Details
In LMC, we integrate rich and distinct knowledge from off- off-the-shelf large models including ChatGPT, DALL-E, CLIP, and DINO in a complementary manner. Specifically, for ChatGPT, we use GPT-3,5- turbo. For CLIP, we ViT-B/32 for its visual encoder, and the transformer architecture described in [46] for its text encoder. For DINO, we use the second version of DINO [37] with ViT-B/14. We set the maximum number of cycle times for ChatGPT's self-checking to 3. To generate diverse images for each class through the proposed cyclic cross-assessing module, we set the number of diverse detailed descriptions K generated for each class 10, and the maximum number of cycle times for cyclic cross-assessing to 3. Besides, we set a in incorporating PCLIP and PDINO to 0.6.
4.3 Experimental Results
We report the AUROC results and the OSCR results respectively in Tab. 1 and Tab. Specifically, besides comparing with existing open-set object recognition methods that assume access to open-set classes, in these two tables, we also compare our framework with Softmax as a baseline that collaborates large models (CLIP and DINO) and involves no extra training process. Particularly, Softmax does not simulate any virtual open-set classes. It treats the ensemble of CLIP and DINO as closed-set classifier and computes the maximum probability value of the classifier's ' output as its score S in the same way as Eq. to Eq. (more details about this baseline are in the supplementary). As shown, compared to state-of-the-ar open-set set recognition methods and the baseline Softmax, framework achieves superior performance across different evaluation protocols, demonstrating the effectiveness of our framework.
4.4 Ablation Studies
In this section, we conduct extensive ablation experiments on the TinyImageNet evaluation protocol. In particular, the experiments, we report the AUROC metric averaged over five dataset splits. More ablation studies such as experiments w.r.t. byperparameters are in the supplementary.
Evaluation the two alignments performed by LMC. In
Table 3: Evaluation on the two alignments performed by LMC. Method
LMC, as shown in Fig. 2, during inference, we align the testing image with the images representing each class through DINO, as well as align the testing image with the name of each class through CLIP. Above, we already evaluate our framework as a whole. Here, we further assess the following two variants of our framework, i.e., the variant (w/o aligning with names) that only aligns the testing image with the images representing each class, and the variant (w/o aligning with images) that only aligns the testing image with the name of each class. As shown Tab. 3, ignoring either alignment would lead to performance drop compared to our framework. This demonstrates the effectiveness of our framework in concurrently aligning the testing image with the names and the generated images.
AUROC
w/o aligning with names
84.4
w/o aligning with images
82.1
LMC
86.7
Evaluation on the two designs for class name simulation. To simulate names of virtual open- set classes in high quality, in our framework,
Table 4: Evaluation on the two designs for class name simulation.
Method
AUROC
we introduce two designs formulate language commands for ChatGPT properly. i.c., guiding ChatGPT with intermediate reasoning and enabling ChatGPT to perform self-checking. Here, to assess the effectiveness of the two designs introduced, we test three variants: 1) w/o intermediate reasoning & self-checking: this variant uses neither of the two designs and simulates names virtual open-set classes for each closed-set class via directly asking ChatGPT the following question: "Given list of classes [classes], can you name other classes that share visual features with [class] while these shared features are discriminative in the list?", where [class] represents a closed-set class and [classes]
w/o intermediate reasoning & self-checking
84.1
w/o intermediate reasoning
85.6
w/o self-checking
85,9
LMC
86.7
8

---

represents the list of closed-set classes. 2) w/o intermediate reasoning: this variant enables ChatGPT perform self-checking but does not guide ChatGPT with intermediate reasoning. Specifically, in this variant, after asking ChatGPT the question in variant 1), we expand [classes] in this question with the obtained virtual open-set classes and re-ask ChatGPT this question iteratively until ChatGPT stops proposing new virtual open-set classes or maximum number of cycle times is reached. As with our framework, we set the maximum number of cycle times to 3 here as well. 3) w/o self-checking: this variant guides ChatGPT with intermediate reasoning but does not perform self-checking. Specifically, in this variant, we only ask the three questions proposed in Sec. 3.1 once but not in an iterative manner. As shown in Tab. 4, our proposed framework consistently outperforms all three variants. This demonstrates (1) the effectiveness of the intermediate reasoning design, which can lead ChatGPT to better understand the asked question; (2) the effectiveness of the self-checking design, which can lead ChatGPT to simulate a more comprehensive list of virtual open-set classes.
Impact of the cyclic cross-assessing module. To ensure that the images generated by DALL-E can well represent their desired classes, in our framework, we propose cyclic cross-assessing module to refine the generated images iteratively. To validate this module, we consider three alternative ways to generate images:
Table 5: Evaluation on the ef- fectiveness of the cyclic cross- assessing module.
Method
AUROC
w/o cross-aseessing
84.5
1) w/o cross-assessing: this variant directly passes the gener- ated images to the inference process of our framework without checking or refining them. 2) Check and discard: this variant checks the generated images through CLIP and passes only those images that pass the checking (i.e., those images that are most aligned towards their desired classes) to the inference process of our framework. Note in this variant, those images that do not pass the checking are simply discarded and no refinement is involved. 3) Check and naively refine: this variant both checks and refines the generated images. However, in this variant, we directly ask ChatGPT refine the descriptions of those incorrect images, and the feedback from CLIP (i.e., a target refinement direction) is not involved in the refinement. As shown in Tab. 5, the performance of all three variants are worse than our method. This shows the effectiveness of our proposed cyclic cross-assessing module, which can leverage CLIP to provide specific feedback for ChatGPT, so that ChatGPT can refine the descriptions more toward the direction of our desire.
Check and discard
84.9
Check and naively refine
85.4
LMC
86.7
Overlap between the virtual open-set classes and the open-set classes used for evaluation. In our method, we simulate virtual open-set classes to lead the framework to less rely on apwrious-iiccrmmnaaie aim verify that the effectiveness of our framework does not come from just matching the open-set classes used for evaluation. To achieve this, we evaluate on two subsets of the open-set classes used for evaluation. Specifically, the first subset (subset A) includes all those open-set classes that are contained in the list of simulated virtual open-set classes Y._o. and the second subset (subset B) includes all the other open-set classes that are not contained in Yr_o- As shown in Tab. 6, compared Softmax as baseline, even for subset B, there is still significant performance improvement, demonstrating the effectiveness of our method even when Yr and Y, do not overlap at all. Moreover, we would like to point out that the virtual open-set classes and the open-set classes
Table 6: Analysis on the overlap between the virtual open-set classes and the open- set classes used for evaluation.
features. Here, we
AUROC
Method
Subset A
Subset B
(Sottaa))
83.5
83.6
LMC
93.1
86.5
closed-sstt samples
testing samples
goose (with beak)
king penguin (open-set) value S: LMC's prediction of 0..2
albatross (open-set) value 5: LMC's prediction of 0.188
ladybug without beak)
black stork (apen-set) 5:0
goose (closed-set) value o:
dragonfly without beak)
LMC's prediction of 0.19
LMC's prediction of 0098
Figure 4: Qualitative results on the TinyImageNet dataset. As shown, even when the testing image holds the ppurious-discriming if it is from closed-set or open-set classes. The above results are based on the fourth dataset split following [6]. More qualitative results are in the supplementary.
feature ('with beak" here), our framework can accurately identify

---

used for evaluation barely overlap. For example, for each data split of TinylmageNet, out of its 180 open-set classes used for evaluation, only no more than 6 classes (i.e., no more than 3.4% of classes) are within the list of virtual open-set classes. Yet, our method still achieves state-of-the-art performance. This further demonstrates that the effectiveness of our method does not naively come from matching the real open-set classes.
Qualitative results. Some qualitative results are shown in Fig. 4. As shown, our framework can perform open-set object recognition accurately even when the testing images contain spurious- discriminative features. This demonstrates that our framework can effectively reduce the reliance on crrrious-dicciiminatii features when performing open-set object recognition.
Visualization of images re-generated through the cross-assessing module. In 5, we visualize some images generated before and after passing through the pro- posed cross-assessing module. As shown, the images before regeneration cannot well represent their desired classes, whereas the images regenerated through the cross- assessing module can represent their de- sired classes more accurately. This demon- strates the effectiveness of the proposed cross-assessing module.
gown
bag
Betore regeneration throuugh srone- assessing:
After ongeraraliass through croru asdessing: gi
Figure Visualization of the images generated before and after passing through the cross-assessing module. More visualizations are in the supplementary.
Conclusion
In this paper, we have proposed novel open-set object recognition framework LMC, which collabo- rates different large models to recognize open-set objects. Specifically, in LMC, we utilize rich and distinct knowledge from different off-the-shell large models reduce the reliance of the proposed framework on spurions-discrimee discriminative features. Moreover, we also incorporate our framework with several designs effectively extract implicit knowledge from large models. Our framework achieves state-of-the-ar performance on four widely used evaluation protocols in a training-free manner. Besides, our method may potentially also (1) be applied other related tasks such as confidence estimation [44, 43] and uncertainty estimation [25, 29]; (2) inspire new ways for follow-up research. Below, we list a few possible new ways that follow-up research work can further investigate: (1) how to collaborate large models with conventional open-set object recognition methods further improve performance; (2) how to facilitate the usage of black-box large models our framework through black-box optimization algorithms such as the CMA Evolution Strategy [52]. We leave these as our future works.
Acknowledgments and Disclosure of Funding
This work was supported by the National Research Foundation Singapore under the Al Singapore Programme (Award Number: AISG-100E-2023-122
References
[1] Jon Almazán, Byungsoo Ko, Geonmo Gu, Diane Larlus, and Yannis Kalantidis. Granularity- aware adaptation for image retrieval over multiple tasks. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV, pages
389-406. Springer, 2022. 2,
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 06565, 2016. 1
[3] Trevor Ashby, Braden K Webb, Gregory Knapp, Jackson Searle, and Nancy Fulda. Personalized quest and dialogue generation in role-playing games: A knowledge graph-and language model- based approach. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-20, 2023. 2
10

---

[4] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016. 3
[5] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference Computer Vision (ICCV), 2021. 2,
[6] Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(1 1):8065-8081, 2021. 3,8,788.
[7] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In Computer Vision-ECCO 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part III 16, pages 507-522. Springer, 2020. 3,
[8] Wonwoo Cho and Jaegul Choo. Towards accurate open-set recognition via background-class regularization. In Computer Vision-ECCV 2022: 17th Europeun Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Purt XXV, pages 658-674. Springer, 2022.
[9] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 255. lece, 2009. 7
[10] Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. Advances in Neural Information Processing Systems, 31, 2018. 8
[11] Xuefeng Du Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't 't know from videos in the wild. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13678-13688, 2022. 2
[12] Sepideh Esma Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribttio detection based on the pre-trained model clip. Proceedings of the AAAI Conference Artificial
Intelligence, 36(6):6568- (66606666666666 Jun. 2022. 1, 2, 3,7
[13] Lin Geng Foo, Hossein Rahmani, and Jun Liu. Ai-generated content (aige) for various data modalities: A survey. arXiv preprint arXiv:2308. 14/77, 2023.
[14] Zongyuan Ge. Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference. BMVA Press, 2017.
1, 3
[15] Zongyuan Ge and Xin Wang. Evaluation of various open-set medical imaging tasks with deep neural networks. arXiv preprint arXiv:2/ 10. 10888, 2021.
[16] Yunrui Guo, Guglielmo Camporese, Wenjing Yang, Alessandro Sperduti, and Lamberto Bal- lan. Conditional variational capsule network for open set recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 103-111, 2021. 7
[17) Ziyu Guo, Yiwen Tang, Renrui Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding with gpt and prototype guidance. arXiv preprint arXiv:2303.1682) 2023. 2
[18] Mchadi Hassen and Philip K Chan. Learning a newrk--network-basedd representation for open set recognition. In Proceedings of the 2020 SIAM International Conference on Data Mining, pages 154-162. SIAM, 2020.
[19] Hongzhi Huang, Yu Wang, Qinghua Hu, and Ming-Ming Cheng. Class-specific semantic reconstruction for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 3,7
[20] Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.0877 2023. 2,3
[21] CEN Jun Di Luan, Shiwei Zhang. Yixuan Pei, Yingya Zhang, Deli Zhao, Shaojie Shen, and Qifeng Chen. The devil is in the classly-classified samples: Towards unified open-set recognition. In The Eleventh International Conference on Learning Representations, 2022. 3
[22] Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Samarasekera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. arXiv preprint Ar0iv:2303. /7/32, 2023.
2
11

---

[23] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822,
2021. 3
[24] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[25] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017. 10
[26] Kyle Lennon, Katharina Fransen, Alexander O' Brien, Yumeng Cao, Matthew Beveridge, Yamin Arefeen, Nikhil Singh, and Iddo Drori. Image2lego: customized lego set generation from images. arXiv preprint arXiv:2108. 08477, 2021. 2
[27] Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. Crowdelip: Unsupervised crowd counting via vision-language model. arXiv preprint arXiv:2304.04233 2023.
[28] Ting-En En Lin and Hua Xu. A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier. Knowledge -asedd Systems, 186: 104979, 2019.
[29] Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshmi- narayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-7512, 2020. 10
[30] Zhun-ga Liu, Yi-min Fu, Quan Pan, and Zuo-wei Zhang. Orientational distribution learning with hierarchical spatial attention for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022. 7
[31] Jing Lu, Yunlu Xu, Hao Li, Zhanzhan Cheng, and Yi Niu. Pmal: Open set recognition via robust prototype mining. Proceedings of the AAAI Conference on Artificial Intelligence, 36(2):1872-1880, Jun. 2022. 3,7
[32] Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv e-prinís, pages arXiv-2303, 2023. 2,
[33] Dimity Miller, Niko Sunderhauf, Michael Milford, and Feras Dayoub. Class anchor clustering: loss for bastance-based open set recognition. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pages 3570-3578, 2021. 3
[34] WonJun Moon, Junho Park, Hyun Seok Seong, Cheol-Ho Cho, and Jae-Pil Heo. Difficulty- aware simulator for open set recognition. Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV, pages 365-381.
Springer, 2022. 1, 3,5, 5, 6,7
[35] Gaurav Kumar Nayak, Ruchit Rawal, and Anirban Chakraborty. Dad: Data-free adversarial defense at test time. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3562-3571, 2022. 2
[36] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 613-628, 2018 3,7,
[37] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[38] Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022. 2,3
[39] Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the JEFE/CVV Conference on Computer Vision and Pattern Recognition, pages 2307-2316, 2019. 3,7
12

---

[40] Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Senerative-discrimnaatv feature representations for open-set recognition. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 1 1811-11820, 2020. 1.3
[41] Rui Qian, Yeqing Li, Zheng Xu, Ming- Hsuan Yang. Serge Belongie, and Yin Cui. Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint wwXiv:2007.07644, 2022. 2
[42] Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purposs natural language processing task solver? arXiv preprint 2DNv:2322.06476. 2023. 2
[43] Haoxuan Qu, Lin Geng Foo, Yanchao Li, and Jun Liu. Towards more reliable confidence estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 10
[44] Haoxuan Qu, Yanchao Li, Lin Geng Foo, Jason Kuen, Jiuxiang Gu, and Jun Liu. Improving the reliability for confidence estimation. In European Conference on Computer Vision, pages 391-408. Springer, 2022. 10
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-8763. PMLR, 2021. 2, 6
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.
Language models are unsupervised multitask learners. OpenAl blog, 18))9, 2019. 8
[47] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pages 8821-8831. PMLR, 2021. 2,5
[48] Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7)1175711772. 2013. 3
[49] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self supervisd outlier detection. In International Conference on Learning Representations, 2020. 3
[50] Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2911- 2916, 2017.
[51] Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific reports, 10(1):7146, 2020. 3
[52] Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-moder www.em-a-----eeviiiiie
In International Conference on Machine Learning, pages
PMLR, 2022. 10
[53] Xin Sun, Zhenning Yang, Chi Zhang, Keck- Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13480-13489) 2020. 3
[54] Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International journal of robotics research, 37(4-5):05-420, 2018.1
[55] Tuan Truong, Sadegh Mohammadi, and Matthias Lenga. How transferable are self- supervisdd features in medical image classification tasks? In Machine Learning for Health, pages 54-74. PMLR, 2021. 2
[56] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need? In International Conference on Learning Representations, 2022. 3, 8,8
[57] Fei- Yue Wang, Qinghai Miao, Xuan Li, Xingxia Wang, and Yilun Lin. What does chatgpt say: The dao from algorithmic intelligence to linguistic intelligence. IFFF/CAA Journal of Automatica Sinica, 17(3):575-579, 2023. 2
13

---

[58] Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. OpenAUC: Towards AUC-oriented open-set recognition. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[59] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. 4
[60] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
3. 3.5.5
[61] Albert Xu, Xiang Ren, and Robin Jia. Conal: Anticipating outliers with large language models. arXiv preprint arXiv:2211.157/88 2022. 3
[62] Hong-Ming Yang, Xu- Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional prototype learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1474-3482 2018. 7
[63] Ryota Yoshihashi, Wen Shao, Rei Kawakami. Shaodi You, Makoto lida, and Takeshi Naemura. Clasifficaioon-rconsttuuutii learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4016-4025, 2019. 1,
[64] Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci. Autolabel: Clip-based framework for open-set video domain adaptation. arXiv preprint arXivv23304.01 10, 2023. 3
[65] Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marein Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal rea- soning with language. In The Eleventh International Conference Learning Representations, 2023.
[66] Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for -o-mmnn detection with supervised contrastive learning. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Ithh International Joint Conference on Natural Language Processing ( (Volume 2: Short Papers), pages 570-878,77 2021.
[67] Zhiyuan Zeng, Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. Adversarial generative bistance-based classifier for robust out-of-domain detection. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7658-7662. IEEE, 2021. 2
[68] Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 102-117. Springer, 2020. 3,7
[69] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. urXiv preprint arXiv:2303. 02151, 2023. 3
[70] Xuelin Zhang, Xuelian Cheng, Donghao Zhang, Paul Bonnington, and Zongyuan Ge. Learning network architecture for open-set recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3):3362-
Jun. 2022. 3
[71] Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4401-4410, 2021.
[72] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-lante language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16816-16825 2022. 2
14


Reviews:
[{"review_id": "mxAPPZycaC", "reviewer": "Reviewer_Kpbj", "summary": "This paper proposes an open-set recognition framework, Large Model Collaboration (LMC), which collaborates several large models (ChatGPT, DALL-E, CLIP, and DINO) to make use of the rich implicit knowledge to reduce the reliance on spurious-discriminative features. The proposed framework consists of the following two stages:\n\n(i)\tThe first stage is to simulate virtual open-set classes, including simulating names and generating images. In order to improve the effectiveness of the simulated names for open-set classes, ChatGPT is asked with three designed questions and the corresponding intermediate rationales for obtaining names of new classes that share spurious-discriminative features with each closed-set class, and is guided to perform iterative self-check (where the new simulated class names are used to expand the name list, and the expanded name list is used to obtain new simulated class names) for covering as many spurious-discriminative features as possible. Then, ChatGPT is asked to generate diverse descriptions for each class in the expanded name list, and DALL-E is used to generate images based on these descriptions. In order to improve the accuracy of the generated images, a cyclic self-assessing module is proposed, where ChatGPT is provided with feedbacks from CLIP about the less accurate images to refine the descriptions generated by ChatGPT and DALL-E is used to re-generate images based on the refined descriptions. At this stage, an expanded list of both closed-set and simulated open-set classes and diverse generated images for each class in the list are obtained.\n\n(ii)\tThe second stage is for inference based on the expanded list and the generated images. At this stage, two alignments (image-text alignment by CLIP and image-image alignment by DINO) are performed for obtaining the corresponding scores to each testing image, and the weighted result of the two scores is used for open-set recognition. \n\n", "questions": "Please see the weaknesses above.", "limitations": "The authors did not adequately address the limitations. \nTwo additional suggestions are listed here:\n(1) It’s better to add the full name of LMC, i.e., Large Model Collaboration, in the abstract in line 6.\n\n(2) In Line 219, it is suggested to add a brief description of 5, although Fig. 5 is also described in Line 415.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe idea, which makes use of several large models for open-set recognition, is somewhat novel and interesting.\n\n(2)\tThe writing is clear and easy to follow. Some examples are well visualized.\n\n(3)\tExperimental results on four small-scale datasets (CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet) demonstrate that the proposed methods performs better than the comparative methods. \n", "weaknesses": "(1)\tLacking comparisons under the cross-dataset setup: The proposed framework is only evaluated under the standard setup where the open-set-class images and closed-set-class images are from a same dataset. Such experiments are insufficient, and in fact, many open-set recognition methods [15,21,36,45,55,63] have also been evaluated under the cross-dataset setup. \n\n(2)\tLacking comparisons on larger-scale datasets: in this paper, only four small-scale datasets (i.e., CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet) are used for evaluation. It is necessary to make comparisons on some larger-scale datasets, e.g., CUB and FGVC-Aircraft.\n\n(3)\tComparison with other image generative methods: One main contribution of the proposed framework is generating virtual open-set-class images. Many other works (e.g., [8,21,30]) also generate virtual images, and it is better to replace the image generation method used in the proposed framework with some of them, and make a further comparison.\n\n(4)\tLacking details of efficiency: One advantage of the training-free methods is its high efficiency. In the proposed framework, although there is no training stage, the pre-processing stage and the inference stage which is based on two alignments may also bring additional computational costs. Hence, it is suggested to provide details about the efficiency of the two stages.\n\n(5)\tLacking discussion about the ablation results: In Sec 4.4, several ablation studies are conducted. However, the authors only stated that the final results were improved, but didn’t give any discussion on the effect of each module. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">*Q1: Comparisons under the cross-dataset setup.*\n\n**A1:** Below, we evaluate our method under a common cross-dataset setup following [15,36,45,55,63]. In this setup, closed-set-class images are from CIFAR-10, and open-set-class images are from ImageNet-crop, ImageNet-resize, LSUN-crop, and LSUN-resize. We use the same evaluation metric F1-score as [15,36,45,55,63] under this setup.\n\n| Methods  | ImageNet-crop | ImageNet-resize | LSUN-crop | LSUN-resize |\n|---|---|---|---|---|\n| **CVAECapOSR [15]**| 85.7 | 83.4 | 86.8 | 88.2|\n| **GFROSR [36]**  | 82.1 | 79.2 | 84.3 | 80.5 |\n| **CGDL [45]**  | 84.0 | 83.2 | 80.6 | 81.2 |\n| **CROSR [55]**  | 73.3 | 76.3 | 72.0 | 74.9 |\n| **PROSER [63]**  | 84.9 | 82.4 | 86.7 | 85.6 |\n| **Ours**  | **88.0** | **86.0** | **91.5** | **93.5** |\n\nAs shown, under the cross-dataset setup, our method also achieves significant performance improvement over previous methods, further demonstrating the effectiveness of our method.\n\n>*Q2: Comparisons on larger-scale datasets, e.g., CUB and FGVC-Aircraft.*\n\n**A2:** Following your suggestion, below, we also evaluate our method on larger-scale datasets including CUB and FGVC-Aircraft, and compare our method with previous methods which report results on these datasets. We report the AUROC score and we follow the Easy/Hard data split of [48] for these two datasets.\n\n| Methods  | CUB-Easy | CUB-Hard | FGVC-Aircraft-Easy | FGVC-Aircraft-Hard | \n|---|---|---|---|---|\n| **ARPL+CS [6] (TPAMI 2021)**  | 83.5 | 75.5 | 87.0 | 77.7 |\n| **MLS [48] (ICLR 2022)**  | 88.3 | 79.3 | 90.7 | 82.3 |\n| **Ours**  | **90.0** | **81.4** | **92.1** | **84.1** |\n\nAs shown, on these larger-scale datasets, our method can also outperform previous methods and achieve SOTA performance on all splits.\n\n>*Q3: Replace the image generation method: Many other works (e.g., [8,21,30]) also generate virtual images.*\n\n**A3:** Following your suggestion, below, we compare our framework with three variants of our framework (**Our other parts + Image generation with [8]**, **Our other parts + Image generation with [21]**, **Our other parts + Image generation with [30]**).  In these three variants, we replace the image generation method used in our framework with the image generation method in [8], [21], and [30] respectively, while keeping the rest parts of our framework to be the same (e.g., in these three variants, we will still use ChatGPT to simulate a list of virtual open-set classes, and these classes will still be aligned with the testing image through CLIP during inference).\n\n| Methods  | CIFAR10 | CIFAR+10 | CIFAR+50 | TinyImageNet |\n|---|---|---|---|---|\n| **Our other parts + Image generation with [8]**  | 95.0 | 97.1 | 96.1 | 84.5 |\n| **Our other parts + Image generation with [21]**  | 94.3 | 96.7 | 95.3 | 83.7 |\n| **Our other parts + Image generation with [30]**  | 94.5 | 97.0 | 95.4 | 83.8 |\n| **Ours**  | **96.6** | **98.9** | **98.5** | **86.7** |\n\nAs shown, our framework with its original image generation method outperforms all three variants, showing the effectiveness of the image generation method proposed in our framework. \n\n>*Q4: Details of efficiency: it is suggested to provide details about the efficiency of the two stages.*\n\n**A4:** Below, we show the time of the two stages, i.e., (1) the pre-processing stage and (2) the inference stage. We evaluate the time cost of our framework on a consumable workstation with an RTX 3090 GPU. With respect to the pre-processing stage, the time cost of our framework ranges from around 10.9 minutes to 38.9 minutes across the evaluated datasets. Taking TinyImageNet as an example, on our workstation, the pre-processing stage of our framework takes 38.9 minutes. Note that our framework is free of training, while the training stage of previous SOTA methods often requires a long training time (e.g., ZOC [12] takes around 40 hours during its training stage when run on our workstation). This demonstrates the efficiency of our training-free framework. Moreover, with respect to the inference stage, we find that our method achieves real-time performance (around 0.02 seconds per image across the evaluated datasets).\n\n>*Q5: Discussion about the ablation results in Sec 4.4 on the effect of each module.*\n\n**A5:** Following your suggestion, we will add discussions to Sec 4.4 on the effect of each module. Below, we take Tab. 4 and Tab. 5 in Sec 4.4 as examples.\n\n(1) In Tab. 4, our framework that has a design to guide ChatGPT with intermediate reasoning outperforms the variant \"w/o intermediate reasoning\". This is because, the intermediate reasoning design can lead ChatGPT to better understand the asked question, and thus lead to performance improvement. Moreover, our framework with the self-checking design also outperforms the variant \"w/o self-checking\". The performance improvement is due to the effect of the self-checking module, which can lead ChatGPT to simulate a more comprehensive list of virtual open-set classes.\n\n(2) In Tab. 5, our framework that uses the cyclic self-assessing module outperforms all three variants, i.e., the \"w/o self-assessing\" variant, the \"Check and discard\" variant, and the \"Check and naively refine\" variant. Such improvement can be due to the effect of the cyclic self-assessing module, which can leverage CLIP to provide specific feedback for ChatGPT, so that ChatGPT can refine the descriptions more toward the direction of our desire. Note that all three variants generate images in some alternative way but do not use the cyclic self-assessing module.\n\nDue to space limitations here, we will add more detailed discussions w.r.t. all ablations to Sec 4.4 of paper.\n\n\n>*Q6: Two additional suggestions.*\n\n**A6:** Thanks for your suggestion. We will (1) add the full name of LMC, i.e., Large Model Collaboration, to the abstract, and (2) describe Fig. 5 in Line 219 in the revised version."}, {"review_id": "L1ShPrcI3L", "reviewer": "Reviewer_Qabu", "summary": "This paper proposed to incorporate several large models to solve the spurious-discriminative problem in open-set recognition. It first prompts ChatGPT to describe the known classes and generate a list of new classes that share the spurious-discriminative features with the known classes. Then it prompts DALL-E to generate the images of known and unknown classes. It also uses CLIP to detect the less accurate generated images and asks ChatGPT to refine the description and uses DALL-E to generate again. During inference, it uses both the CLIP model and DINO model to calculate the uncertainty scores and fuse their results. The experiments and ablation study are comprehensive. Overall, I think this paper provides a new alternative solution for open-set recognition, and the data generation process could be very useful for the industry.", "questions": "1. How much time does the data generation cost for different datasets?\n2. What if we use other pre-trained image models instead of DINO? Like ImageNet pre-trained or MoCo?", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The overall idea of using several large models for training-free open-set recognition is new and interesting. The pipeline of using ChatGPT to generate the new classes that share the spurious-discriminative features with the known classes is reasonable. The refinement procedure using CLIP and ChatGPT is very rigorous and effective.\n2. The inference pipeline that fuses the results of CLIP and DINO is also reasonable. It is somewhat similar to the method in the few-shot open-set recognition [1, 2], which also fuses the uncertainty score from two aspects. The authors may consider citing the related papers.\n3. The experiments and ablation studies are very comprehensive.\n\nReference:  \n[1] Ssd: A unified framework for self-supervised\noutlier detection. In ICLR, 2022.\n[2] THE DEVIL IS IN THE WRONGLY-CLASSIFIED SAMPLES: TOWARDS UNIFIED OPEN-SET RECOGNITION. In ICLR 2023.", "weaknesses": "The main pipeline is a kind of data generation process, which could not be done online. So although it is a good try to use several large models and coordinate them to generate wanted data, it is more like an engineering project rather than a research topic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">*Q1: May consider citing the related papers [1,2].*\n\n**A1:** We will cite [1,2] in the paper. Note that different from [1,2] which fuse the uncertainty score from two aspects (e.g., from FS-KNN and Softmax), we are the first to collaborate large models and leverage their rich and distinct knowledge to perform open-set object recognition. During inference, as mentioned in our paper, we collaborate two large models (CLIP and DINO) complementarily to match the testing image to both the overall concept and detailed local features of each class. \n\n>*Q2: The main pipeline is a kind of data generation process, which could not be done online. [...] it is more like an engineering project rather than a research topic.*\n\n**A2:** (1) **Online**: In open-set recognition, most existing works rely on a training process. Besides training, many works [8,13,21,30,32] also perform data generation (i.e., they need both training and data generation). The data generation process in these works (and also in our method) is not done **online**. In contrast to previous works, our framework performs **data generation** but does not require any training process. In other words, in this work, we, for the first time, propose a **training-free** framework to handle the open-set recognition task. This shows the advantage of our research work. As also mentioned by Reviewer BEEJ, \"this method possesses many advantages, such as being training-free and fast inference, that are important in real-world applications\".\n\n(2) **Research topic**: Open-set object recognition is a challenging research problem and has attracted lots of research attention. As mentioned by Reviewer zWHw, \"open-set object recognition is interesting and challenging\". \n\nTo perform open-set object recognition well, as also pointed out by Reviewer BEEJ, \"the main challenge in open-set recognition is that the chosen classifier may get confused by certain spurious-discriminative features\". To tackle this challenge, in our paper, inspired by that different large models often contain rich and distinct knowledge, we, for the first time, propose a framework to collaborate different large models to handle the open-set object recognition task. Reviewer BEEJ also mentions that, our work \"demonstrates a very clever usage of several off-the-shelf large models of different modalities\".\n\nHowever, utilizing and collaborating large models to perform open-set object recognition is challenging and is a research problem by itself that is worth exploring. To handle this problem and extract implicit knowledge in large models effectively, in our paper, we also incorporate our framework with several novel designs. During simulating names of virtual open-set classes, to simulate a more comprehensive list of virtual open-set classes to better cover the spurious-discriminative features, we propose to enable the large model to perform self-checking. Moreover, during generating images via DALL-E, to ensure that the images generated can well represent their desired classes, we also design an effective cyclic self-assessing module to perform cross-checking across different large models and refine the generated images iteratively. \n\nIn summary, open-set object recognition is challenging, and one of its main research challenges is to handle spurious-discriminative features (as mentioned by Reviewer BEEJ). In this work, to handle this research challenge, we, for the first time, propose a framework to collaborate large models to perform open-set object recognition. We also incorporate our framework with several new and effective designs to utilize and collaborate large models better. Our framework achieves SOTA performance on all the evaluated benchmarks. **These show that, our work can bring some new research contributions**.\n\n>*Q3: Data generation time cost.*\n\n**A3:** In our framework, data generation consists of two steps: (1) generating the names of virtual open-set classes and (2) generating diverse images w.r.t. each class. Below, we report the time cost on each of the above two steps across different datasets.\n\n| Methods | CIFAR+10 | CIFAR+50 | CIFAR10 | TinyImageNet |\n|---|---|---|---|---|\n| **Name generation**  | around 1.3 mins | around 1.3 mins | around 2.0 mins | around 5.2 mins |\n| **Image generation**  | around 9.4 mins | around 9.4 mins | around 9.5 mins | around 32.7 mins |\n\nAs shown, our framework does not need much time during both the above two steps. Taking TinyImageNet as an example, on our workstation with a single RTX 3090 GPU, the above two steps take 37.9 minutes in total. Note that after the above two steps, our framework is free of training, while the training stage of previous SOTA methods often requires a long training time (e.g., ZOC [12] takes around 40 hours during its training stage when run on our workstation).\n\n>*Q4: Other pre-trained image models instead of DINO like ImageNet pre-trained or MoCo.*\n\n**A4:** Below, we compare \"our framework with DINO\" with two other variants. In these two variants, we replace DINO with ImageNet pre-trained transformer and MoCo respectively, while keeping the rest parts of our framework to be the same. We report the AUROC score.\n\n| Methods  | CIFAR10 | CIFAR+10 | CIFAR+50 | TinyImageNet |\n|---|---|---|---|---|\n| **Previous SOTA** | 95.1 | 97.9 | 97.6 | 84.6 |\n| **ImageNet pre-trained transformer**  | 95.7 | 98.5 | 98.1 | 86.4 |\n| **MoCo**  | 95.2 | **98.9** | 98.0 | 85.8 |\n| **DINO**  | **96.6** | **98.9** | **98.5** | **86.7** |\n\nAs shown, no matter using DINO, ImageNet pre-trained transformer, or MoCo, our framework can consistently outperform previous SOTAs across different datasets. Moreover, our framework with DINO outperforms the other two variants. This can be because, as discussed in our paper, the collaboration of DINO with good capability in detailed local features, and CLIP with good capability in overall concepts, is complementary and can achieve good performance. "}, {"review_id": "nvt25IqqRi", "reviewer": "Reviewer_BEEJ", "summary": "This work demonstrates a very interesting and sophisticated usage of various large models, including ChatGPT, CLIP, DINO and DALL-E., to tackle the open-set recognition problem. The main challenge in open-set recognition is that the chosen classifier may get confused by certain spurious-discriminative features that are shared between the closed- and open-set classes. The main idea of this work is to leverage the text-to-image alignment capability of CLIP and image-to-image alignment capability of DINO with the aid of virtual open-set classes. The list of virtual open-set classes is elicited from LLM, i.e. ChatGPT, through carefully designed prompts that introduces intermediate reasoning and self-checking. Additionally, the descriptive text can then serve as prompts for DALL-E to synthesize the representative images of the corresponding open-set classes. During inference time, both the pre-trained CLIP and DINO models can then utilize the text class names and synthetic representative images to perform zero-shot (training-free) classification, and meanwhile reducing the influence of spurious-discriminative features. Evaluation on several benchmarks demonstrates new state-of-the-art performance.", "questions": "From the exemplar dialogs with ChatGPT, the conversation still seems a bit open-ended and unstructured. I am curious about how the authors completed the preparation of the virtual open-set list, synthesis of diverse representative images and pre-compute the CLIP and DINO features in just about 38.9 minutes (Appendix, Table 9). I assume that this is done programmatically. Are there any tricks involved here to simplify the post-processing of ChatGPT answers? As far as I know, the most important tunable parameter when calling OpenAI API is the temperature, which controls the diversity of the results that the LLM generates. How did you set this parameter? Does it affect the quality of ChatGPT's self-checking process?", "limitations": "The performance of LMC framework is apparently affected by the large models and there is no discussion about it. I am particularly curious about the potential failure modes of ChatGPT.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and very easy to follow. The additional information provided in the supplemental material is also very helpful in resolving most of the doubts I had when reading the main manuscript. In the beginning, I particularly felt that how the large models collaborate and self-improve to accomplish their jobs was a little magical and I wanted some more transparency after reading the main manuscript. I appreciate that the authors had put sufficient information in the supplemental material to make the whole work more technically sound and convincing.\n2. This work tackles the open-set recognition from the system-level and demonstrates a very clever usage of several off-the-shelf large models of different modalities. Although it didn't introduce new techniques to improve any of the models involved, it successfully combined them to achieve superior performance. From the practical point of view, this method possesses many advantages, such as being training free and fast inference, that are important in real-world applications.", "weaknesses": "1. Although large model collaboration is an interesting way of solving open-set recognition problem, it treats the large models as black boxes as it is. and it is perhaps not easy to come up with an even more sophisticated method to build a LMC system like this work, In some sense, I feel that it does not pave a new way for follow-up research, and yet I do not believe the open-set recognition problem is completely solved by the LMC approach.\n2. Although I don't think the evaluation of this work is insufficient, I think it is still worth validating the effectiveness of LMC framework on larger benchmarks such as the Semantic Shift Benchmark proposed in [48]. I doubt if the construction of virtual open-set classes by LLM will fail at a certain point as the number of fine-grained visually similar close-set classes grows. Another drawback of using LLM is that it is very difficult for human to \"fact check\" the answers and detect any content that is made up by LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">*Q1: Pave a new way for follow-up research.*\n\n**A1:** Thank you for pointing out our method *is an interesting way*. Below, we also discuss potential **follow-up research** in solving open-set recognition (OSR) that could be inspired by our work. In our method, we identify suitable large models, design schemes to extract implicit knowledge, and design collaboration of large models to solve OSR problem. This implies that follow-up research work can further investigate (1) how to identify more suitable large models, (2) how to better extract knowledge from large models, and (3) how to better collaborate large models for OSR. Besides, how to collaborate large models with conventional OSR methods to further improve performance with the complimentary ability can also be interesting. Moreover, despite the convenience of training-free, adding trainable modules to further improve the performance can be explored. This implies that our method could also inspire new ways for follow-up research. We will add more detailed discussions to paper.\n\n>*Q2: (1) Validate on larger benchmark Semantic Shift Benchmark. (2) Construction will fail at a certain point as the number of classes grows. (3) Difficult for human to \"fact check\".*\n\n**A2:** (1) **Larger benchmarks.** We evaluate on a larger benchmark Semantic Shift Benchmark [48] and report AUROC.\n\n|Methods|CUB-Easy|CUB-Hard|SCars-Easy|SCars-Hard|FGVC-Aircraft-Easy|FGVC-Aircraft-Hard|ImageNet-Easy|ImageNet-Hard|\n|-|-|-|-|-|-|-|-|-|\n|**MLS [48] (ICLR 2022)**|88.3|79.3|94.0|82.2|90.7|82.3|78.7|72.8|\n|**Ours**|**90.0**|**81.4**|**96.1**|**84.3**|**92.1**|**84.1**|**80.0**|**74.8**|\n\nAs shown, on larger benchmark that has many fine-grained classes, our method can also outperform the previous SOTA method on all splits.\n\n(2) **Construction of classes.** We admit that if there are huge amount of very similar fine-grained classes, construction of classes may yield weaker performance. However, even on large fine-grained datasets (Semantic Shift) with 1000 close-set classes, we observe that our method can still construct classes reliably and achieve SOTA performance, showing the robustness of our method. But we also need to mention that, if there are too many and too similar fine-grained classes, it will become very difficult for existing open-set recognition methods in common.\n \n(3) **Fact check.** To further show the efficacy of our designs, below, we also introduce **human fact check**. We invite 3 volunteers, each conducting fact checks of the construction of classes independently with access to internet and online bases (e.g., Wiki) to help their assessment. Then, volunteers further discuss the disagreed class names. We conduct above fact check on the constructed classes for CIFAR10. We observe that, without proposed designs (intermediate reasoning and self-checking), around 5.5% constructed classes are judged as low quality, while with proposed designs, only around 0.9% constructed classes are judged as low quality. This further shows (1) the efficacy of proposed designs, (2) though it is difficult to do very accurate fact check, we can still perform fact check to some extent.\n\n\n>*Q3: ChatGPT's conversation seems unstructured. How to complete preparation in just about 38.9 mins. Any tricks to simplify the post-processing of ChatGPT answers?*\n\n**A3:** As mentioned in Supp, we complete the preparation automatically with a single script. Using this script (Python), we can (1) structure ChatGPT's answers and (2) complete the preparation very efficiently and by the following processes:\n\n(1) Like online tutorial [a], we simply use ChatGPT to **structure the answers** by instructing ChatGPT: \"Please respond in a list and start each class with '--'\". Thus, the answers are structured.\n\n(2) As the script can run automatically, we can complete preparation conveniently. To further enhance the efficiency, in our script, we open 20 ChatGPT instances concurrently during simulating class names, and use 20 DALL-E instances concurrently to synthesize images. Using this way, on a consumable workstation with a single 3090 GPU, we can **finish the preparation in just about 38.9 mins**.\n\nWe will release code and script.\n\n[a] Zdnet. 7 advanced ChatGPT prompt-writing tips you need to know.\n\n\n>*Q4: How did you set the temperature? Does it affect self-checking?*\n\n**A4:** We set the temperature to its default value 1.0. Below, we test setting it to different values in self-checking.\n\n|Temperature|0.1|0.4|0.7|1.0|1.3|1.6|1.9|\n|-|-|-|-|-|-|-|-|\n|AUROC|86.1|86.4|86.6|86.7|86.7|86.3|86.0|\n\nWe observe that our method gets optimal results on 1.0 and 1.3 (1.0 used in paper). We also find that setting temperature to different values all outperform the variant w/o self-checking, showing the efficacy of self-checking.\n\n\n>*Q5: The performance is affected by the large models and there is no discussion about it. Curious about potential failure modes of ChatGPT.*\n\n**A5:** (1) **The performance is affected by the capability of each large model**. For example, GPT has different versions (e.g., GPT 3.0 and 3.5) with different capabilities. For GPT 3.0, AUROC on TinyImageNet is 85.2, while for GPT 3.5, AUROC is 86.7. This shows that the performance is affected by the large models. We will add more discussion to paper.\n\n(2) **Potential failure mode** of ChatGPT: We observe that when we ask ChatGPT a complex question (e.g., \"Given a list of classes [classes], please list some classes that share spurious-discriminative features with class [class]?\"), on some classes, ChatGPT fails to well understand the question and provides undesired answers: Sorry, as an AI language model, I cannot answer this question. This can be seen as a failure mode. Nevertheless, as discussed in Sec. 3 in paper, we design to guide ChatGPT step-by-step with intermediate reasoning. When adding this design, we well-handle this failure mode and observe that ChatGPT can stably generate good-quality answers.\n"}, {"review_id": "8dSwAyFrXe", "reviewer": "Reviewer_zWHw", "summary": "In this paper, the authors tackle the problem of open-set object recognition. The key challenge in open-set recognition problem they tackle is the model's reliance on spurious-discriminative features. To tackle this challenge, they propose an open-set object recognition system consisting of multiple pre-trained foundation models. They employ ChatGPT to generate virtual open-set categories that share spurious-discriminative features with closed-set categories. Then they generate images of the both closed-set and open-set categories by using DALL-E. During testing, they employ DINO and CLIP to get more accurate predictions by matching a test image to the generated images and, by matching a test image to category embeddings. They validate the proposed method on public benchmarks.", "questions": "Please address the weaknesses part. I have a few additional questions below.\n\nQ1: What is the performance if we learn a few parameters on top of the foundation models used? For example, what is the performance if we add some adapter layers on top of CLIP and/or DINO to fine-tune the model on the target datasets? Can we get an even stronger performance?\n\nQ2: Does the proposed method work well on more fine-grained recognition tasks? E.g., CUB or Oxford Flowers datasets. It seems that the proposed method heavily relies on ChatGPT’s common sense reasoning ability. What happens if ChatGPT cannot generate reasonable descriptions of certain fine-grained classes? Since the proposed method does not learn anything, it might fail if ChatGPT fails. I would like to listen to the authors' opinions and/or empirical validation on this issue.\n", "limitations": "I could not find limitation and broader impact part.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper has the following strengths.\n\nS1: The problem tackled, open-set object recognition is interesting and challenging.\n\nS2: The proposed approach is reasonable. Using generative models to leverage knowledge-base and image generation capability could improve open-set recognition performance by mitigating spurious correlations between the category and features.\n\nS3: The proposed method shows favorable performance on public benchmarks, without any fine-tuning.\n\nS4: The ablation experiments show the effectiveness of the proposed design choices. Chain-of-thought reasoning, self-checking, and CLIP-based feedback all contribute to the final performance. Furthermore, the performance improvement is not merely from the category overlap between the close-set and the virtual open-set as shown in Table 6. The improvement might come from mitigated spurious correlation.\n", "weaknesses": "This paper has the following weaknesses.\n\nW1: Although the proposed method is effective in open-set recognition, the technical contribution is somewhat limited. This is okay if the performance improvement compared to the baseline is very significant. However, the performance improvement compared to the baseline is not surprising given that the proposed method consists of four foundation models: GPT, DALL-E, CLIP, and DINO. For example, the proposed method shows 3.2 points AUROC improvement and 5.5 points OSCR improvement over the softmax baseline in TinyImageNet. \n\nW2: The authors claim the improved performance comes from mitigating the spurious correlation. However, there is no quantitative evaluation on the amount of spurious correlation mitigated. It would be better to show some quantitative evidence of how much the proposed method mitigates the spurious correlation.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">*Q1: Technical contribution.*\n\n**A1:** In this work, we propose a novel framework that collaborates large models and performs training-free open-set object recognition. Moreover, to effectively extract implicit knowledge from large models, we also incorporate several novel designs, such as iterative self-checking and cyclic self-assessing. To the best of our knowledge, we are the first to collaborate different large models to perform open-set object recognition.\n\n>*Q2: The proposed method consisting of four foundation models only outperforms baseline by 3.2 and 5.5 points.*\n\n**A2:** We would like to clarify, similar to our method, the softmax baseline we set up also collaborates foundation models (as described in Line 320-324 in paper). This softmax baseline can be actually regarded as a variant of our method without designs to extract implicit knowledge, and the performance improvement (3.2 and 5.5) shows the benefit of the designs to extract implicit knowledge, instead of fully showing the superiority of our method. Note that the baseline (with large models) outperforms previous SOTA methods [6] by 9.2 on OSCR, and our designs further improve over baseline by 5.5 on OSCR, showing that overall, our collaboration of large models and designs bring large improvement (9.2 + 5.5 = 14.7). Sorry for confusion caused. We will make it clearer in paper.\n\n>*Q3: Show some quantitative evaluation on the amount of spurious correlation mitigated.*\n\n\n**A3:** While it can be difficult to directly and precisely evaluate spurious correlation, below, we still do some analysis and show some quantitative evaluation. Take the open-set class Crane as an example, without our method's list of virtual open-set classes used, 40% of Crane are misrecognized as closed-set class Albatross (both with white feathers) and 26% of Crane are misrecognized as closed-set class Black Stork (both with long lags). While after adding virtual open-set classes such as Swan (also with white feathers like Albatross) and Avocet (also with long legs like Black Stork), only 4% of Crane are misrecognized as Albatross and 2% of Crane are misrecognized as Black Stork. Such reduced amounts (40% to 4% and 26% to 2%) can imply the amounts of spurious correlation mitigated. Similar to what was done above, we measure the reduced amount for all classes on the TinyImageNet protocol. We find that, the amount reduced (mitigated) on average is around 24% after applying our method, implying the efficacy of our method in mitigating spurious correlation.\n\n>*Q4: Add adapter layers on top of CLIP and/or DINO.*\n\n**A4:** We add two adapter (linear) layers on top of both CLIP and DINO, and fine-tune these layers on the training set of the target dataset while keeping other parts frozen. We fine-tune CLIP (or not) and fine-tune DINO (or not) to get the four variants below. We report AUROC.\n\n| Methods  | CIFAR10 | CIFAR+10 | CIFAR+50 | TinyImageNet |\n|-|-|-|-|-|\n| **CLIP & DINO**  | 96.6 | 98.9 | 98.5 | 86.7|\n| **Fine-tuned CLIP & DINO**  | 96.3 | 98.5 | 98.0 | 86.8 |\n| **CLIP & fine-tuned DINO**  | 96.4 | 98.4 | 98.2 | 86.8 |\n| **Fine-tuned CLIP & fine-tuned DINO**  | 96.2 | 98.3 | 97.9 | 86.9 |\n\nAs shown, on smaller-scale datasets (CIFAR10, CIFAR+10, and CIFAR+50), fine-tuning leads to slight performance drop, while on the relatively large TinyImageNet, fine-tuning leads to slight performance enhancement. Nevertheless, without any fine-tuning, our training-free method has already outperformed previous works largely.\n\n>*Q5: Fine-grained recognition tasks (CUB or Oxford Flowers datasets)*\n\n**A5:** Below, we evaluate on the more fine-grained datasets CUB and Oxford Flowers, and compare our method with previous SOTA methods. We report AUROC.\n\n| Methods  | CUB-Easy | CUB-Hard | \n|-|-|-|\n| **MLS [48]**  | 88.3 | 79.3 |\n| **Ours**  | **90.0** | **81.4** |\n\n| Methods  | Oxford Flowers |\n|-|-|\n| **DML [a]**  | 90.8 |\n| **Ours**  | **91.7** |\n\nAs shown, on these more fine-grained datasets, our method can also outperform previous SOTA methods. This further shows the efficacy of our method.\n\n[a] The Importance of Metric Learning for Robotic Vision: Open Set Recognition and Active Learning. ICRA, 2019.\n\n>*Q6: What happens if ChatGPT cannot generate reasonable descriptions of certain fine-grained classes?*\n\n\n**A6:** Among ChatGPT's generated descriptions, we observe that, reasonable descriptions are consistently generated for all the evaluated classes (including fine-grained classes), while for certain (fine-grained) classes, in rare cases, unreasonable descriptions are also generated. But note that in our work, we designed a cyclic self-assessing module to replace unreasonable descriptions with reasonable descriptions. We find that, before passing into the self-assessing module, 3% of descriptions are checked to be unreasonable, but none of the descriptions output by the module is checked to be unreasonable. This shows that ChatGPT has small probability to generate unreasonable descriptions, while our self-assessing module can further mitigate this problem. The above check is done by inviting 3 volunteers and passing the same 1000 descriptions generated from fine-grained classes in CUB to each of them. The 3 volunteers first make decisions independently and then discuss disagreed decisions. We will also add above analysis to Supp.\n\nThough not observed, if ChatGPT cannot generate any reasonable description of a certain fine-grained class, all generated (unreasonable) descriptions may fail to be replaced in self-assessing process and images can fail to be generated for this class. This can weaken the mitigation of spurious-discriminative features w.r.t. this class. Nevertheless, in our experiments, ChatGPT is observed to be able to generate reasonable descriptions for all the evaluated classes, and our method achieves SOTA performances on all splits and all datasets, showing the efficacy of our method.\n\n"}]

Author rebuttal:
We thank all reviewers for recognition of our contributions (Reviewer Kpbj:"novel and interesting"; Reviewer Qabu:"the overall idea is new and interesting"; Reviewer BEEJ:"demonstrates a very clever usage of several off-the-shelf large models of different modalities", "possesses many advantages, such as being training free and fast inference, that are important in real-world applications"; Reviewer zWHw: "the problem tackled is interesting and challenging", "favorable performance").

Meta-review: Four knowledgeable reviewers recommend acceptance of this paper citing it as a novel and valuable contribution for the research community. Two reviewers raised their rating after rebuttal and discussion. Authors should attend to main points in the reviews when preparing a final version. No basis to overturn reviews.
Decision: Accept (poster)