Title: LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep learning

Open-set object recognition

Large models

Training-free
Abstract: Open-set object recognition aims to identify if an object is from a class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge is how to reduce the reliance on spurious-discriminative features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named Large Model Collaboration (LMC) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs to effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available \href{https://github.com/Harryqu123/LMC}{here}.

LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition

Abstract

Open-set object recognition aims to identify if an object is from a class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge is how to reduce the reliance on spurious-discriminative features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named L arge M odel C ollaboration ( LMC ) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs to effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available here.

1 Introduction

Open-set object recognition aims to identify if an object is from a closed-set class that has appeared during training, or an open-set class that has not been encountered in the training set. It is a crucial task since open-set classes always exist in the real world [63], and misidentifying open-set classes into closed-set ones can lead to severe consequences in many practical applications [2], such as autonomous driving [19], robotics system [54], malware classification [18], and medical analysis [15]. Due to its importance, open-set object recognition has received a lot of research attention recently and various methods have been proposed [14, 36, 28, 6, 66, 56, 12].

To perform open-set object recognition well, some existing works [34, 40] have noted that a key challenge is how to reduce the reliance on spurious-discriminative features. Specifically, a spuriousdiscriminative feature refers to a feature that satisfies the following two requirements: 1) it is discriminative between a closed-set class and other closed-set classes; 2) it is shared between this closed-set class and certain open-set classes. As suggested by previous works [34, 40], while such spurious-discriminative features can facilitate classification within the closed set, in open-set recognition, relying on such features can make closed-set and open-set classes to be easily confused

with each other, and thus bring negative effects in accurately identifying open-set objects. For example, as shown in Fig. 1, given a closed set of classes including bighorn, tabby cat, and hog, within these closed-set classes, 'with horn' as a feature of bighorn can be used to discriminate bighorn from other classes (tabby cat and hog). However, relying on 'with horn' to perform open-set object recognition, bighorn can be easily confused with potential open-set classes such as ox and gazelle that are also 'with horn'. Therefore, it can be difficult to correctly identify ox and gazelle as open-set objects. To avoid open-set objects from being misidentified into closed-set classes, a common type of methods is to simulate additional virtual open-set samples or classes [14, 36, 67, 12, 34]. Specifically, among this type of methods, various external knowledge bases have been used, such as pre-trained GloVe embeddings [67] and extra datasets [12]. Yet, despite the considerable efforts made and various architectures specifically designed for this task, the open-set object recognition task remains challenging. This is because, in the real world, various different sorts of open-set classes can appear [11]. Moreover, given a trained classifier, to further equip it with the ability to identify open-set objects, most existing open-set recognition methods require an extra training process. Especially when the classifier has been deployed, conducting such an extra training process can be inconvenient, as it needs both additional training time and access to training data. Note that the training data can be no longer accessible during deployment (e.g., due to privacy concerns [35, 22]).

Recently, large models containing very rich implicit knowledge [17] have served as powerful external knowledge bases in many areas [13]. For example, ChatGPT [38] possesses rich common-sense knowledge, and has been applied in various natural language processing tasks [42, 20]. DALL-E [47], which holds strong image-generative ability, has been used in different generation tasks [32, 26]. CLIP [45] has been leveraged for visual recognition tasks [72, 41] since it can align relevant image-text pairs. Besides, being capable of aligning relevant image patches, DINO [5] has

closed-setsamples

open-settestingsamples

bighorn

ox

(with horn)

(withhorn)

Expected closed-setprabability:0

Predictedclosed-set

probabilityofDIAS:0.94

tabbycat

(withouthorn)

gazelle

（withhorn)

Expected closed-setprabability:0

hog

(withouthorn)

Predictedclosed-set

probabilityofDlAS:0.67

been applied in areas [55, 1] such as image retrieval.

Inspired by that large models often contain rich knowledge, here we aim to investigate leveraging such rich knowledge to handle the open-set object recognition task. However, using large models to perform open-set object recognition is non-trivial due to the following challenges: 1) It can be difficult to directly leverage an existing large model to effectively perform open-set object recognition. Take CLIP as an example, being able to align an image with a closed set of classes and thus finding out the most relevant class in the set, CLIP has been applied in various visual recognition tasks, such as zero-shot and few-shot image classification [45]. However, CLIP cannot handle the open-set object recognition task effectively by itself. This is because, various sorts of open-set classes exist in the real world, and it is difficult to know which class should the testing image be aligned with when using CLIP. 2) Meanwhile, as the knowledge in large models is often implicit [57, 3], it can be challenging to extract the desired knowledge (e.g., in our case, the knowledge that can facilitate open-set object recognition) from large models successfully [57]. To handle these challenges, in this work, with the observation that different large models pre-trained through different paradigms can contain distinct knowledge, we first propose a novel open-set object recognition framework named L arge M odel C ollaboration ( LMC ). As shown in Fig. 2, the proposed LMC performs open-set object recognition in a training-free manner by leveraging rich and distinct knowledge from different off-the-shelf large models complementarily. Moreover, LMC is also incorporated with several novel designs to facilitate extracting implicit knowledge in large models effectively.

Overall, to better perform open-set object recognition, LMC aims to reduce the reliance of this task on spurious-discriminative features. To achieve this, LMC proposes to simulate additional virtual open-set classes that share the spurious-discriminative features through the collaboration of different large models. Specifically, in LMC, motivated by that ChatGPT possesses rich common-sense knowledge, we first adopt ChatGPT to expand the list of closed-set classes with additional virtual

open-set classes. During this process, to extract implicit knowledge from ChatGPT effectively, we propose to guide ChatGPT with intermediate reasoning, while also enabling ChatGPT to perform self-checking on whether it covers as many spurious-discriminative features as possible. Moreover, inspired by that images can catch different details that are complementary to the text domain [60], in addition to representing classes in the expanded list with their names in the text domain, we propose to further use image information to better represent these classes. Specifically, to facilitate extracting image information from large models, we propose to incorporate our framework with a novel cyclic module, which collaborates ChatGPT, DALL-E, and CLIP through a cross-assessing mechanism to generate diverse images for each class. Once both the text and image information have been extracted for each class in the expanded list, during inference, we propose to use the image-text alignment ability of CLIP and the image-image alignment ability of DINO in collaboration to perform open-set object recognition on testing images.

The contributions of our work are summarized as follows. 1) We propose LMC, a novel framework that can handle the open-set object recognition task in a training-free manner, via collaborating different off-the-shelf pre-trained large models and leveraging their knowledge in a complementary manner. 2) We introduce several designs in LMC to extract both text and image information from large models effectively. 3) On the evaluated benchmarks, LMC achieves state-of-the-art performance.

2 Related Work

Open-set Object Recognition. Due to the wide range of applications, open-set object recognition has received a lot of research attention [48, 4, 50, 14, 63, 39, 68, 53, 51, 7, 6, 33, 23, 56, 61, 19, 58, 70, 31, 71, 49, 21]. Bendale and Boult [4] made the first attempt of applying deep neural networks into the task of open-set recognition. Specifically, they first showed that simply thresholding on the softmax probability cannot yield a robust open-set recognition, and thus proposed Openmax based on the Extreme Value Theory. Ge et al. [14] further proposed G-Openmax to generate unknown samples using generative models. After that, Neal et al. [36] showed the effectiveness of generating images that do not belong to any of the closed-set classes but look similar to those images in the training dataset. Later on, OpenGAN, which uses real open-set images to perform model selection, was proposed by Kong et al. [23]. Vaze et al. [56] proposed to well-train the closed-set classifier so that it can perform both closed-set image classification and open-set recognition well. Besides, Esmaeilpour et al. [12] proposed to train an image description generator using an extra image captioning dataset and use the trained generator to improve the open-set object recognition performance.

Different from existing approaches, in this work, from a novel perspective, we investigate how to collaborate different large models to perform open-set object recognition in a training-free manner. Besides, we also introduce several designs to extract implicit knowledge from large models effectively.

Large Models. Recently, a variety of large models have been proposed, such as ChatGPT [38], DALL-E [47], CLIP [45], and DINO [5]. Such large models contain rich knowledge and have been studied in various tasks [20, 65, 69, 32, 64, 27, 1], such as language translation [20], video generation [32], and image retrieval [1]. In this work, we design a new framework to handle the open-set object recognition task by cross-assessing among large models and leveraging the rich and distinct knowledge from different large models in a complementary manner.

3 Proposed Method

Given a testing image from Y c ∪ Y o , where Y c represents closed-set classes in the training set and Y o represents open-set classes that are not included in the training set, the goal of open-set object recognition is to identify if the image is from a closed-set class or an open-set class. To recognize open-set objects accurately, as suggested by previous works [40, 34], a key challenge is: how to reduce the reliance on spurious-discriminative features. To better tackle this challenge, in this work, inspired by that large models can hold very rich while distinct implicit knowledge due to their different training paradigms, we propose a novel open-set object recognition framework LMC . Specifically, as shown in Fig. 2, LMC first extracts the implicit knowledge from different off-the-shelf large models to simulate additional virtual open-set classes that share the spurious-discriminative features. After that, LMC utilizes these virtual open-set classes to make the spurious-discriminative features 'less discriminative' during its inference process.

Below, we first describe how LMC extracts implicit knowledge from different large models to effectively simulate virtual open-set classes. After that, we introduce the inference process of LMC.

3.1 Virtual Open-set Class Simulation

In the proposed LMC, to reduce the reliance on spuriousdiscriminative features, we aim to utilize the rich implicit knowledge of different large models to simulate additional virtual open-set classes. Specifically, motivated by that knowledge from different modali-

self-checking

cyclic cross-assessing module

virtual open-set classes

ChatGPT

[kangaroo, shark,...]

ChatGPT

closed-set classes

[rabbit, dolphin,...]

CLIP

DALL-E

DINO

testingimages

open-setobject

fromarbitrary

recognition

concatenate

classes

results

incorporate

CLIP

ties (e.g., text and image) can contain different details [60], we aim to leverage implicit knowledge of large models in both the text domain and the image domain to facilitate simulation. To achieve this, we first simulate the names of virtual open-set classes and expand the list of closed-set classes with these virtual open-set classes. We then further generate diverse images for each class in the list to describe each class better.

Simulating names of virtual open-set classes. Inspired by that ChatGPT holds rich common-sense knowledge, here we aim to simulate names of virtual open-set classes by asking ChatGPT questions. Specifically, to perform such a simulation in high quality, we find that, it is necessary to 1) ensure that ChatGPT can well understand the question, and 2) encourage ChatGPT to cover as many spuriousdiscriminative features as possible. To achieve these, we introduce the following two designs in our framework.

Inspired by chain-of-thought [59] that large language models can better understand the given instruction when they are guided with intermediate reasoning, to ensure that ChatGPT can well understand the question, we here also guide ChatGPT with intermediate reasoning. Specifically, we find that an effective way to guide ChatGPT is to ask ChatGPT the following three questions for each closed-set class: 1) 'Given a list of classes [classes], can you describe the visual features of each class in the list?' , where [classes] represents the names of the closed-set classes; 2) 'What are the discriminative visual features of class [class]

Given a listof classes[ladybug,dragonfly,goose,..],canyou describe

thevisualfeaturesofeachclassinthelist?

5

Sure.Ladybug isasmall,roundbeetle thatistypicallyredororangewith

black spots. Its body is covered in a hard, shiny shell. Dragonfly is...

Whatarethediscriminativevisualfeaturesofclassladybugcomparedwith

otherclassesinthelist?

S

Comparedwithotherclassesinthelist,ladybugshaveseveral

discriminativevisual featuresincluding:color(ladybugsaretypicallyred

or orange with black spots);shape(ladybugs have a round shape);..

Can you list other classes that also share these discriminative visual features?

S

Sure,herearesomeotherclassesthatsharesomeof thediscriminative

visualfeaturesofladybugs:

-Tortoisebeetle:a typeofbeetlethathasasimilarbodyshape and

hard,shiny shell to ladybugs,and some species have similar bright colors

and markings.

spotted pattern to ladybugs.

compared with other classes in the list?' , where [class] represents the closed-set class that the current set of questions is asked for; 3) 'Can you list other classes that also share these discriminative visual features?' . In Fig. 3, we illustrate an example when the above three questions are asked for the class ladybug. As shown, by formulating the language command in this way, besides the final answer to question 3), ChatGPT is also demanded to generate intermediate rationales that lead to the final answer through the first two questions. This can help ChatGPT to better understand our final purpose, i.e., simulating virtual open-set classes that share spurious-discriminative features. In Particular, in Fig. 3, tortoise beetle and ladybird spider are the simulated virtual open-set classes that share spurious-discriminative features with the closed-set class ladybug. Through the above three questions, we can simulate the names of virtual open-set classes

that share spurious-discriminative features with a single closed-set class. To simulate the names of virtual open-set classes for all closed-set classes, we can simply ask the above three questions for each class in the closed set either sequentially, or concurrently via different ChatGPT instances.

In addition, to simulate a comprehensive list of virtual open-set classes to better cover the spuriousdiscriminative features, we aim to further enable ChatGPT to perform self-checking. In other words, we aim to guide ChatGPT to re-think if there are other spurious-discriminative features that are missed. To achieve this, for each closed-set class, after asking the above three questions, we expand the list of classes in question 1) with the obtained virtual open-set classes and re-ask ChatGPT the three questions. Note that by expanding the list with the obtained virtual open-set classes, we exclude the spurious-discriminative features that are already covered and thus force ChatGPT to look for spurious-discriminative features that are not discovered yet. This iteration of self-checking terminates when ChatGPT stops proposing new virtual open-set classes or a maximum number of cycle times is reached. After performing self-checking, inspired by [34] that all sorts of classes can appear in the real world, we also follow [34] to simulate virtual open-set samples that are less similar to closed-set samples (sharing less spurious-discriminative features) via asking ChatGPT: 'Given a list of classes [classes], can you name classes that are not similar to them?'. Note after expanding the list of closed-set classes with simulated virtual open-set classes through applying the above designs, those spurious-discriminative features that are originally discriminative among the closed-set classes would be 'less discriminative' in the expanded list.

Generating diverse images and cross-assessing. Above we simulate the names of virtual open-set classes Y v _ o . Here, inspired by that images can catch different details that can be complementary to the text domain [60], we aim to describe both the closed-set classes and the simulated virtual open-set classes in more detail by further generating diverse images for each class y ∈ Y c ∪ Y v _ o . These generated images will be utilized in the inference process of LMC to reduce its reliance on spurious-discriminative features.

Specifically, for each class y ∈ Y c ∪ Y v _ o , we generate diverse images for class y through the following two steps: 1) we first ask ChatGPT to generate diverse text descriptions as: 'Can you write [ K ] diverse descriptions, each describing a different scene about the class [ y ]?' where K is a hyperparameter representing the number of descriptions generated per class. 2) After that, leveraging the image-generative ability of DALL-E [47], we ask DALL-E to generate images based on these descriptions for each class as:

where d k y denotes the k -th description generated by ChatGPT for class y , and i k y denotes the image generated by DALL-E based on d k y . While we can directly use the images generated above during inference, as shown in Fig. 5 which demonstrates a number of such images, we realize that some of such images may not hold a high enough quality to accurately represent their desired classes (e.g., i k y may not represent class y accurately). Thus, directly using the above images can lead to suboptimal model performance during inference.

To address this problem, referring the images that cannot accurately represent their desired classes as the less accurate images, we aim to automatically detect these less accurate images, refine their corresponding descriptions, and regenerate them accordingly. To achieve this, inspired by that humans can refine their understanding based on others' feedback, here we want to investigate, can a large model also refine its output based on another large model's feedback? In particular, we propose a cyclic cross-assessing module, where CLIP acts as the 'feedback provider', and provides ChatGPT with feedback about the less accurate images to help refine their corresponding descriptions.

Specifically, denote i k y an image generated for class y , and D the text descriptions 'a photo of [class]' for all the classes (i.e., Y c ∪ Y v _ o ). The cross-assessing module operates by iteratively conducting the following three steps: 1) Detection. To detect the less accurate images, we first leverage the strong image-text alignment ability of CLIP to align i k y with D as:

where CLIP vis denotes the CLIP's visual encoder, and CLIP text denotes the CLIP's text encoder. The image i k y is determined as less accurate if compared to class y , it is more aligned towards another class u (i.e., instead of class y , another class u corresponds to the highest probability value in p assess ). 2) Refinement. Next, if i k y is determined as a less accurate image, in order for its corresponding

description d k y to describe class y more properly, we ask ChatGPT to refine d k y based on feedback from CLIP as: 'This description [ d k y ] seems more like class [ u ] to me. Can you refine it to enhance the characteristics of class [ y ]?' . 3) Regeneration. Lastly, the image i k y is regenerated through DALL-E based on the refined text description.

Through conducting the above three steps iteratively, during every iteration, we can leverage CLIP to provide specific feedback for ChatGPT so that ChatGPT can refine the descriptions more toward the direction of our desire. With these refined descriptions, images that can well represent each class will be generated. The above iteration terminates when all the generated images are most aligned with their desired classes or a maximum number of cycle times is reached. Note after reaching the maximum number of cycle times, we discard those images that are still less accurate .

3.2 Inference Process

Above we expand the list of closed-set classes with additional virtual open-set classes, and generate diverse images for each class in the expanded list. In this section, we discuss how our framework utilizes the expanded list and the generated images to make the spurious-discriminative features 'less discriminative' during its inference process. Note our framework does not require any training process before its inference, which can facilitate its convenient usage in real-world applications.

Specifically, during inference, inspired by the strong image-text alignment ability of CLIP [45] and the strong image-image alignment ability of DINO [5], we propose to adopt both CLIP and DINO to perform open-set object recognition. Note CLIP and DINO are used in collaboration since they can complement each other well. To be more concrete, by aligning the testing image with the names of all the classes in the expanded list (i.e., Y c ∪ Y v _ o ), we first adopt CLIP to match the testing image with the overall concept of each class. Meanwhile, by leveraging DINO to align the testing image with the generated images of each class in image patch level, we better match the testing image with the detailed local features of each class.

Before entering the inference process, with the notice that certain CLIP and DINO features are used repeatedly across different testing images, we first pre-store these features to keep the efficiency of the inference process. Specifically, denoting f CLIP text = CLIP text ( D ) the CLIP text feature of all classes, we first store f CLIP text , which has already been computed in the above Eq. 2. Moreover, denoting DINO vis the DINO's encoder and I y the set of images generated from class y through the cyclic cross-assessing module, we compute the DINO visual feature for class y as f DINO y = DINO vis ( I y ) and store f DINO y for each class y ∈ Y c ∪ Y v _ o .

After pre-storing these features, we then introduce the inference process of the proposed LMC framework. Specifically, given a testing image i test , utilizing pre-stored features, LMC first aligns i test with the names of all the classes in Y c ∪ Y v _ o through CLIP as:

where p CLIP denotes the softmax probability derived from CLIP. At the same time, LMC also aligns i test with the generated images through DINO as:

where l y DINO is derived from aligning i test with I y through DINO, and p DINO denotes the softmax probability correspondingly calculated for all the classes in Y c ∪ Y v _ o . After deriving p CLIP and p DINO, LMC then incorporates them to determine whether i test is from the closed-set classes or open-set classes as:

where α is a hyperparameter denoting the incorporation weight, p inc denotes the softmax probability derived from incorporating p CLIP and p DINO, p inc ( y | i test ) denotes the probability value that class y corresponds to in p inc , and S ∈ [0 , 1] is the closed-set score measured by our framework. Following [34], we represent S as the maximum probability value among the closed-set classes. By performing open-set object recognition in the above way, open-set images would no longer be easily misidentified into the closed-set classes due to the contained spurious-discriminative features. This is because, after expanding the list of closed-set classes with the virtual open-set classes, in the expanded list, besides a certain closed-set class, a spurious-discriminative feature can be held by some virtual open-set classes as well. Then, when an open-set testing image containing spurious-discriminative features is

aligned with the expanded list, it will tend to approach both the closed-set and virtual open-set classes that contain these features at the same time, rather than approaching only certain closed-set classes. Therefore, after softmax normalization, the closed-set score S of the open-set testing image will no longer be easily overestimated due to the spurious-discriminative features contained in the image.

4 Experiments

To evaluate the effectiveness of our proposed framework LMC, we conduct experiments on four evaluation protocols including CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet. We conduct our experiments on an RTX 3090 GPU.

4.1 Datasets and Evaluation Metric

CIFAR10. CIFAR10 [24] is a ten-class object recognition dataset. On this dataset, following [6, 56], 6 closed-set classes and 4 open-set classes are randomly sampled for evaluation each time.

CIFAR+10 & CIFAR+50. Following [6, 56], we also evaluate our method on CIFAR+10 and CIFAR+50, which are extensions of the CIFAR10 evaluation protocol. Specifically, following [6, 56], for the CIFAR+ N experiments, 4 classes are randomly sampled from CIFAR-10 as the closed-set classes and N non-overlapping classes are randomly sampled from CIFAR-100 [24] as the open-set classes for evaluation ( N = 10 for CIFAR+10 and N = 50 for CIFAR+50).

TinyImageNet. TinyImageNet [9] is a subset of ImageNet and contains 200 classes. Following [6, 56], each time, 20 closed-set classes and 180 open-set classes are randomly sampled.

Evaluation metric. Following [6, 56], we use the following two metrics: AUROC and OSCR [10]. Among these two metrics, AUROC focuses on evaluating model performance on identifying open-set samples, and OSCR measures the trade-off between classification accuracy and open-set recognition performance. More details especially about the OSCR metric are in the supplementary.

4.2 Implementation Details

In LMC, we integrate rich and distinct knowledge from off-the-shelf large models including ChatGPT, DALL-E, CLIP, and DINO in a complementary manner. Specifically, for ChatGPT, we use GPT-3.5turbo. For CLIP, we use ViT-B/32 for its visual encoder, and the transformer architecture described in [46] for its text encoder. For DINO, we use the second version of DINO [37] with ViT-B/14. We set the maximum number of cycle times for ChatGPT's self-checking to 3. To generate diverse images for each class through the proposed cyclic cross-assessing module, we set the number of diverse detailed descriptions K generated for each class to 10, and the maximum number of cycle times for cyclic cross-assessing to 3. Besides, we set α in incorporating p CLIP and p DINO to 0.6.

4.3 Experimental Results

We report the AUROC results and the OSCR results respectively in Tab. 1 and Tab. 2. Specifically, besides comparing with existing open-set object recognition methods that assume no access to open-set classes, in these two tables, we also compare our framework with Softmax as a baseline that collaborates large models (CLIP and DINO) and involves no extra training process. Particularly, Softmax does not simulate any virtual open-set classes. It treats the ensemble of CLIP and DINO as a closed-set classifier and computes the maximum probability value of the classifier's output as its score S in the same way as Eq. 3 to Eq. 5 (more details about this baseline are in the supplementary). As shown, compared to state-of-the-art open-set recognition methods and the baseline Softmax, our framework achieves superior performance across different evaluation protocols, demonstrating the effectiveness of our framework.

4.4 Ablation Studies

In this section, we conduct extensive ablation experiments on the TinyImageNet evaluation protocol. In particular, in the experiments, we report the AUROC metric averaged over five dataset splits. More ablation studies such as experiments w.r.t. hyperparameters are in the supplementary.

Evaluation on the two alignments performed by LMC. In LMC, as shown in Fig. 2, during inference, we align the testing image with the images representing each class through DINO, as well as align the testing image with the name of each class through CLIP. Above, we already evaluate our framework as a whole. Here, we further assess the following two variants of our

framework, i.e., the variant ( w/o aligning with names ) that only aligns the testing image with the images representing each class, and the variant ( w/o aligning with images ) that only aligns the testing image with the name of each class. As shown in Tab. 3, ignoring either alignment would lead to performance drop compared to our framework. This demonstrates the effectiveness of our framework in concurrently aligning the testing image with the names and the generated images.

Evaluation on the two designs for class name simulation. To simulate names of virtual openset classes in high quality, in our framework, we introduce two designs to formulate language commands for ChatGPT properly, i.e., guiding ChatGPT with intermediate reasoning and enabling ChatGPT to perform self-checking.

Here, to assess the effectiveness of the two designs introduced, we test three variants: 1) w/o intermediate reasoning & self-checking: this variant uses neither of the two designs and simulates names of virtual open-set classes for each closed-set class via directly asking ChatGPT the following question: 'Given a list of classes [classes], can you name other classes that share visual features with [class] while these shared features are discriminative in the list?' , where [class] represents a closed-set class and [classes]

represents the list of closed-set classes. 2) w/o intermediate reasoning: this variant enables ChatGPT to perform self-checking but does not guide ChatGPT with intermediate reasoning. Specifically, in this variant, after asking ChatGPT the question in variant 1) , we expand [classes] in this question with the obtained virtual open-set classes and re-ask ChatGPT this question iteratively until ChatGPT stops proposing new virtual open-set classes or a maximum number of cycle times is reached. As with our framework, we set the maximum number of cycle times to 3 here as well. 3) w/o self-checking: this variant guides ChatGPT with intermediate reasoning but does not perform self-checking. Specifically, in this variant, we only ask the three questions proposed in Sec. 3.1 once but not in an iterative manner. As shown in Tab. 4, our proposed framework consistently outperforms all three variants. This demonstrates (1) the effectiveness of the intermediate reasoning design, which can lead ChatGPT to better understand the asked question; (2) the effectiveness of the self-checking design, which can lead ChatGPT to simulate a more comprehensive list of virtual open-set classes.

Impact of the cyclic cross-assessing module. To ensure that the images generated by DALL-E can well represent their desired classes, in our framework, we propose a cyclic cross-assessing module to refine the generated images iteratively. To validate this module, we consider three alternative ways to generate images: 1) w/o cross-assessing : this variant directly passes the generated images to the inference process of our framework without checking or refining them. 2) Check and discard : this variant

checks the generated images through CLIP and passes only those images that pass the checking (i.e., those images that are most aligned towards their desired classes) to the inference process of our framework. Note in this variant, those images that do not pass the checking are simply discarded and no refinement is involved. 3) Check and naively refine : this variant both checks and refines the generated images. However, in this variant, we directly ask ChatGPT to refine the descriptions of those incorrect images, and the feedback from CLIP (i.e., a target refinement direction) is not involved in the refinement. As shown in Tab. 5, the performance of all three variants are worse than our method. This shows the effectiveness of our proposed cyclic cross-assessing module, which can leverage CLIP to provide specific feedback for ChatGPT, so that ChatGPT can refine the descriptions more toward the direction of our desire.

Overlap between the virtual open-set classes and the open-set classes used for evaluation. In our method, we simulate virtual open-set classes to lead the framework to less rely on spurious-discriminative features. Here, we aim to verify that the effectiveness of our framework does not come from just matching the open-set classes used for evaluation. To achieve this, we evaluate it on two subsets

of the open-set classes used for evaluation. Specifically, the first subset ( subset A ) includes all those open-set classes that are contained in the list of simulated virtual open-set classes Y v _ o , and the second subset ( subset B ) includes all the other open-set classes that are not contained in Y v _ o . As shown in Tab. 6, compared to Softmax as a baseline, even for subset B, there is still a significant performance improvement, demonstrating the effectiveness of our method even when Y v _ o and Y o do not overlap at all. Moreover, we would like to point out that the virtual open-set classes and the open-set classes

closed-setsamples

testingsamples

goose

kingpenguin(open-set)

albatross(open-set)

(withbeak)

GTvalue ofS:0

GTvalue ofS:0

LMC'sprediction ofS:0.12

LMC'sprediction of S:0.18

ladybug

（withoutbeak)

blackstork(open-set)

goose(closed-set)

GTvalueof S:0

GTvalue of S:1

dragonfly

LMC'sprediction ofS:0.19

LMC'sprediction ofS:0.98

(withoutbeak)

used for evaluation barely overlap. For example, for each data split of TinyImageNet, out of its 180 open-set classes used for evaluation, only no more than 6 classes (i.e., no more than 3 . 4% of classes) are within the list of virtual open-set classes. Yet, our method still achieves state-of-the-art performance. This further demonstrates that the effectiveness of our method does not naively come from matching the real open-set classes.

Qualitative results. Some qualitative results are shown in Fig. 4. As shown, our framework can perform open-set object recognition accurately even when the testing images contain spuriousdiscriminative features. This demonstrates that our framework can effectively reduce the reliance on spurious-discriminative features when performing open-set object recognition.

Visualization of images re-generated through the cross-assessing module. In Fig. 5, we visualize some images generated before and after passing through the proposed cross-assessing module. As shown, the images before regeneration cannot well represent their desired classes, whereas the images regenerated through the crossassessing module can represent their desired classes more accurately. This demonstrates the effectiveness of the proposed cross-assessing module.

5 Conclusion

In this paper, we have proposed a novel open-set object recognition framework LMC, which collaborates different large models to recognize open-set objects. Specifically, in LMC, we utilize rich and distinct knowledge from different off-the-shelf large models to reduce the reliance of the proposed framework on spurious-discriminative features. Moreover, we also incorporate our framework with several designs to effectively extract implicit knowledge from large models. Our framework achieves state-of-the-art performance on four widely used evaluation protocols in a training-free manner. Besides, our method may potentially also (1) be applied in other related tasks such as confidence estimation [44, 43] and uncertainty estimation [25, 29]; (2) inspire new ways for follow-up research. Below, we list a few possible new ways that follow-up research work can further investigate: (1) how to collaborate large models with conventional open-set object recognition methods to further improve performance; (2) how to facilitate the usage of black-box large models in our framework through black-box optimization algorithms such as the CMA Evolution Strategy [52]. We leave these as our future works.

Acknowledgments and Disclosure of Funding

This work was supported by the National Research Foundation Singapore under the AI Singapore Programme (Award Number: AISG-100E-2023-121).

References

Jon Almazán, Byungsoo Ko, Geonmo Gu, Diane Larlus, and Yannis Kalantidis. Granularityaware adaptation for image retrieval over multiple tasks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIV , pages 389-406. Springer, 2022. 2, 3

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 , 2016. 1

Trevor Ashby, Braden K Webb, Gregory Knapp, Jackson Searle, and Nancy Fulda. Personalized quest and dialogue generation in role-playing games: A knowledge graph-and language modelbased approach. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , pages 1-20, 2023. 2

Desired class:

Academicgown

Christmas tree

Fox

Punchingbag

Before

regeneration

through cross-

assessing:

After

regeneration

through cross-

assessing:

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1563-1572, 2016. 3

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV) , 2021. 2, 3, 6

Guangyao Chen, Peixi Peng, Xiangqian Wang, and Yonghong Tian. Adversarial reciprocal points learning for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(11):8065-8081, 2021. 1, 3, 7, 8, 9

Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng, Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 507-522. Springer, 2020. 3, 7

Wonwoo Cho and Jaegul Choo. Towards accurate open-set recognition via background-class regularization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV , pages 658-674. Springer, 2022. 7

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009. 7

Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. Advances in Neural Information Processing Systems , 31, 2018. 8

Xuefeng Du, Xin Wang, Gabriel Gozum, and Yixuan Li. Unknown-aware object detection: Learning what you don't know from videos in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13678-13688, 2022. 2

Sepideh Esmaeilpour, Bing Liu, Eric Robertson, and Lei Shu. Zero-shot out-of-distribution detection based on the pre-trained model clip. Proceedings of the AAAI Conference on Artificial Intelligence , 36(6):6568-6576, Jun. 2022. 1, 2, 3, 7

Lin Geng Foo, Hossein Rahmani, and Jun Liu. Ai-generated content (aigc) for various data modalities: A survey. arXiv preprint arXiv:2308.14177 , 2023. 2

Zongyuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference . BMVA Press, 2017. 1, 2, 3

Zongyuan Ge and Xin Wang. Evaluation of various open-set medical imaging tasks with deep neural networks. arXiv preprint arXiv:2110.10888 , 2021. 1

Yunrui Guo, Guglielmo Camporese, Wenjing Yang, Alessandro Sperduti, and Lamberto Ballan. Conditional variational capsule network for open set recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 103-111, 2021. 7

Ziyu Guo, Yiwen Tang, Renrui Zhang, Dong Wang, Zhigang Wang, Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding with gpt and prototype guidance. arXiv preprint arXiv:2303.16894 , 2023. 2

Mehadi Hassen and Philip K Chan. Learning a neural-network-based representation for open set recognition. In Proceedings of the 2020 SIAM International Conference on Data Mining , pages 154-162. SIAM, 2020. 1

Hongzhi Huang, Yu Wang, Qinghua Hu, and Ming-Ming Cheng. Class-specific semantic reconstruction for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022. 1, 3, 7

Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. Is chatgpt a good translator? a preliminary study. arXiv preprint arXiv:2301.08745 , 2023. 2, 3

CEN Jun, Di Luan, Shiwei Zhang, Yixuan Pei, Yingya Zhang, Deli Zhao, Shaojie Shen, and Qifeng Chen. The devil is in the wrongly-classified samples: Towards unified open-set recognition. In The Eleventh International Conference on Learning Representations , 2022. 3

Nazmul Karim, Niluthpol Chowdhury Mithun, Abhinav Rajvanshi, Han-pang Chiu, Supun Samarasekera, and Nazanin Rahnavard. C-sfda: A curriculum learning aided self-training framework for efficient source free domain adaptation. arXiv preprint arXiv:2303.17132 , 2023. 2

Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 813-822, 2021. 3

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009. 7

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems , 30, 2017. 10

Kyle Lennon, Katharina Fransen, Alexander O'Brien, Yumeng Cao, Matthew Beveridge, Yamin Arefeen, Nikhil Singh, and Iddo Drori. Image2lego: customized lego set generation from images. arXiv preprint arXiv:2108.08477 , 2021. 2

Dingkang Liang, Jiahao Xie, Zhikang Zou, Xiaoqing Ye, Wei Xu, and Xiang Bai. Crowdclip: Unsupervised crowd counting via vision-language model. arXiv preprint arXiv:2304.04231 , 2023. 3

Ting-En Lin and Hua Xu. A post-processing method for detecting unknown intent of dialogue system via pre-trained deep neural network classifier. Knowledge-Based Systems , 186:104979, 2019. 1

Jeremiah Liu, Zi Lin, Shreyas Padhy, Dustin Tran, Tania Bedrax Weiss, and Balaji Lakshminarayanan. Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems , 33:7498-7512, 2020. 10

Zhun-ga Liu, Yi-min Fu, Quan Pan, and Zuo-wei Zhang. Orientational distribution learning with hierarchical spatial attention for open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022. 7

Jing Lu, Yunlu Xu, Hao Li, Zhanzhan Cheng, and Yi Niu. Pmal: Open set recognition via robust prototype mining. Proceedings of the AAAI Conference on Artificial Intelligence , 36(2):1872-1880, Jun. 2022. 3, 7

Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. arXiv e-prints , pages arXiv-2303, 2023. 2, 3

Dimity Miller, Niko Sunderhauf, Michael Milford, and Feras Dayoub. Class anchor clustering: A loss for distance-based open set recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3570-3578, 2021. 3

WonJun Moon, Junho Park, Hyun Seok Seong, Cheol-Ho Cho, and Jae-Pil Heo. Difficultyaware simulator for open set recognition. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXV , pages 365-381. Springer, 2022. 1, 2, 3, 5, 6, 7

Gaurav Kumar Nayak, Ruchit Rawal, and Anirban Chakraborty. Dad: Data-free adversarial defense at test time. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 3562-3571, 2022. 2

Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 613-628, 2018. 1, 2, 3, 7

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 8

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35:27730-27744, 2022. 2, 3

Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2307-2316, 2019. 3, 7

Pramuditha Perera, Vlad I. Morariu, Rajiv Jain, Varun Manjunatha, Curtis Wigington, Vicente Ordonez, and Vishal M. Patel. Generative-discriminative feature representations for open-set recognition. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11811-11820, 2020. 1, 3

Rui Qian, Yeqing Li, Zheng Xu, Ming-Hsuan Yang, Serge Belongie, and Yin Cui. Multimodal open-vocabulary video classification via pre-trained vision and language models. arXiv preprint arXiv:2207.07646 , 2022. 2

Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint arXiv:2302.06476 , 2023. 2

Haoxuan Qu, Lin Geng Foo, Yanchao Li, and Jun Liu. Towards more reliable confidence estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023. 10

Haoxuan Qu, Yanchao Li, Lin Geng Foo, Jason Kuen, Jiuxiang Gu, and Jun Liu. Improving the reliability for confidence estimation. In European Conference on Computer Vision , pages 391-408. Springer, 2022. 10

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021. 2, 3, 6

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019. 8

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning , pages 8821-8831. PMLR, 2021. 2, 3, 5

Walter J. Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E. Boult. Toward open set recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence , 35(7):1757-1772, 2013. 3

Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations , 2020. 3

Lei Shu, Hu Xu, and Bing Liu. Doc: Deep open classification of text documents. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 29112916, 2017. 3

Yu Shu, Yemin Shi, Yaowei Wang, Tiejun Huang, and Yonghong Tian. P-odn: Prototype-based open deep network for open set recognition. Scientific reports , 10(1):7146, 2020. 3

Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-model-as-a-service. In International Conference on Machine Learning , pages 20841-20855. PMLR, 2022. 10

Xin Sun, Zhenning Yang, Chi Zhang, Keck-Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13480-13489, 2020. 3

Niko Sünderhauf, Oliver Brock, Walter Scheirer, Raia Hadsell, Dieter Fox, Jürgen Leitner, Ben Upcroft, Pieter Abbeel, Wolfram Burgard, Michael Milford, et al. The limits and potentials of deep learning for robotics. The International journal of robotics research , 37(4-5):405-420, 2018. 1

Tuan Truong, Sadegh Mohammadi, and Matthias Lenga. How transferable are self-supervised features in medical image classification tasks? In Machine Learning for Health , pages 54-74. PMLR, 2021. 2

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: a good closed-set classifier is all you need? In International Conference on Learning Representations , 2022. 1, 3, 7, 8

Fei-Yue Wang, Qinghai Miao, Xuan Li, Xingxia Wang, and Yilun Lin. What does chatgpt say: The dao from algorithmic intelligence to linguistic intelligence. IEEE/CAA Journal of Automatica Sinica , 10(3):575-579, 2023. 2

Zitai Wang, Qianqian Xu, Zhiyong Yang, Yuan He, Xiaochun Cao, and Qingming Huang. OpenAUC: Towards AUC-oriented open-set recognition. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 3

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 4

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9653-9663, 2022. 3, 4, 5

Albert Xu, Xiang Ren, and Robin Jia. Conal: Anticipating outliers with large language models. arXiv preprint arXiv:2211.15718 , 2022. 3

Hong-Ming Yang, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. Robust classification with convolutional prototype learning. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3474-3482, 2018. 7

Ryota Yoshihashi, Wen Shao, Rei Kawakami, Shaodi You, Makoto Iida, and Takeshi Naemura. Classification-reconstruction learning for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4016-4025, 2019. 1, 3

Giacomo Zara, Subhankar Roy, Paolo Rota, and Elisa Ricci. Autolabel: Clip-based framework for open-set video domain adaptation. arXiv preprint arXiv:2304.01110 , 2023. 3

Andy Zeng, Maria Attarian, brian ichter, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, and Pete Florence. Socratic models: Composing zero-shot multimodal reasoning with language. In The Eleventh International Conference on Learning Representations , 2023. 3

Zhiyuan Zeng, Keqing He, Yuanmeng Yan, Zijun Liu, Yanan Wu, Hong Xu, Huixing Jiang, and Weiran Xu. Modeling discriminative representations for out-of-domain detection with supervised contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) , pages 870-878, 2021. 1

Zhiyuan Zeng, Hong Xu, Keqing He, Yuanmeng Yan, Sihong Liu, Zijun Liu, and Weiran Xu. Adversarial generative distance-based classifier for robust out-of-domain detection. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 7658-7662. IEEE, 2021. 2

Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 102-117. Springer, 2020. 3, 7

Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Hongsheng Li, Yu Qiao, and Peng Gao. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. arXiv preprint arXiv:2303.02151 , 2023. 3

Xuelin Zhang, Xuelian Cheng, Donghao Zhang, Paul Bonnington, and Zongyuan Ge. Learning network architecture for open-set recognition. Proceedings of the AAAI Conference on Artificial Intelligence , 36(3):3362-3370, Jun. 2022. 3

Da-Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2021. 3

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16816-16825, 2022. 2

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors tackle the problem of open-set object recognition. The key challenge in open-set recognition problem they tackle is the model's reliance on spurious-discriminative features. To tackle this challenge, they propose an open-set object recognition system consisting of multiple pre-trained foundation models. They employ ChatGPT to generate virtual open-set categories that share spurious-discriminative features with closed-set categories. Then they generate images of the both closed-set and open-set categories by using DALL-E. During testing, they employ DINO and CLIP to get more accurate predictions by matching a test image to the generated images and, by matching a test image to category embeddings. They validate the proposed method on public benchmarks.\n\nSTRENGTHS:\nThis paper has the following strengths.\n\nS1: The problem tackled, open-set object recognition is interesting and challenging.\n\nS2: The proposed approach is reasonable. Using generative models to leverage knowledge-base and image generation capability could improve open-set recognition performance by mitigating spurious correlations between the category and features.\n\nS3: The proposed method shows favorable performance on public benchmarks, without any fine-tuning.\n\nS4: The ablation experiments show the effectiveness of the proposed design choices. Chain-of-thought reasoning, self-checking, and CLIP-based feedback all contribute to the final performance. Furthermore, the performance improvement is not merely from the category overlap between the close-set and the virtual open-set as shown in Table 6. The improvement might come from mitigated spurious correlation.\n\nWEAKNESSES:\nThis paper has the following weaknesses.\n\nW1: Although the proposed method is effective in open-set recognition, the technical contribution is somewhat limited. This is okay if the performance improvement compared to the baseline is very significant. However, the performance improvement compared to the baseline is not surprising given that the proposed method consists of four foundation models: GPT, DALL-E, CLIP, and DINO. For example, the proposed method shows 3.2 points AUROC improvement and 5.5 points OSCR improvement over the softmax baseline in TinyImageNet. \n\nW2: The authors claim the improved performance comes from mitigating the spurious correlation. However, there is no quantitative evaluation on the amount of spurious correlation mitigated. It would be better to show some quantitative evidence of how much the proposed method mitigates the spurious correlation.\n\nQUESTIONS:\nPlease address the weaknesses part. I have a few additional questions below.\n\nQ1: What is the performance if we learn a few parameters on top of the foundation models used? For example, what is the performance if we add some adapter layers on top of CLIP and/or DINO to fine-tune the model on the target datasets? Can we get an even stronger performance?\n\nQ2: Does the proposed method work well on more fine-grained recognition tasks? E.g., CUB or Oxford Flowers datasets. It seems that the proposed method heavily relies on ChatGPT’s common sense reasoning ability. What happens if ChatGPT cannot generate reasonable descriptions of certain fine-grained classes? Since the proposed method does not learn anything, it might fail if ChatGPT fails. I would like to listen to the authors' opinions and/or empirical validation on this issue.\n\nLIMITATIONS:\nI could not find limitation and broader impact part.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work demonstrates a very interesting and sophisticated usage of various large models, including ChatGPT, CLIP, DINO and DALL-E., to tackle the open-set recognition problem. The main challenge in open-set recognition is that the chosen classifier may get confused by certain spurious-discriminative features that are shared between the closed- and open-set classes. The main idea of this work is to leverage the text-to-image alignment capability of CLIP and image-to-image alignment capability of DINO with the aid of virtual open-set classes. The list of virtual open-set classes is elicited from LLM, i.e. ChatGPT, through carefully designed prompts that introduces intermediate reasoning and self-checking. Additionally, the descriptive text can then serve as prompts for DALL-E to synthesize the representative images of the corresponding open-set classes. During inference time, both the pre-trained CLIP and DINO models can then utilize the text class names and synthetic representative images to perform zero-shot (training-free) classification, and meanwhile reducing the influence of spurious-discriminative features. Evaluation on several benchmarks demonstrates new state-of-the-art performance.\n\nSTRENGTHS:\n1. The paper is well-written and very easy to follow. The additional information provided in the supplemental material is also very helpful in resolving most of the doubts I had when reading the main manuscript. In the beginning, I particularly felt that how the large models collaborate and self-improve to accomplish their jobs was a little magical and I wanted some more transparency after reading the main manuscript. I appreciate that the authors had put sufficient information in the supplemental material to make the whole work more technically sound and convincing.\n2. This work tackles the open-set recognition from the system-level and demonstrates a very clever usage of several off-the-shelf large models of different modalities. Although it didn't introduce new techniques to improve any of the models involved, it successfully combined them to achieve superior performance. From the practical point of view, this method possesses many advantages, such as being training free and fast inference, that are important in real-world applications.\n\nWEAKNESSES:\n1. Although large model collaboration is an interesting way of solving open-set recognition problem, it treats the large models as black boxes as it is. and it is perhaps not easy to come up with an even more sophisticated method to build a LMC system like this work, In some sense, I feel that it does not pave a new way for follow-up research, and yet I do not believe the open-set recognition problem is completely solved by the LMC approach.\n2. Although I don't think the evaluation of this work is insufficient, I think it is still worth validating the effectiveness of LMC framework on larger benchmarks such as the Semantic Shift Benchmark proposed in [48]. I doubt if the construction of virtual open-set classes by LLM will fail at a certain point as the number of fine-grained visually similar close-set classes grows. Another drawback of using LLM is that it is very difficult for human to \"fact check\" the answers and detect any content that is made up by LLMs.\n\nQUESTIONS:\nFrom the exemplar dialogs with ChatGPT, the conversation still seems a bit open-ended and unstructured. I am curious about how the authors completed the preparation of the virtual open-set list, synthesis of diverse representative images and pre-compute the CLIP and DINO features in just about 38.9 minutes (Appendix, Table 9). I assume that this is done programmatically. Are there any tricks involved here to simplify the post-processing of ChatGPT answers? As far as I know, the most important tunable parameter when calling OpenAI API is the temperature, which controls the diversity of the results that the LLM generates. How did you set this parameter? Does it affect the quality of ChatGPT's self-checking process?\n\nLIMITATIONS:\nThe performance of LMC framework is apparently affected by the large models and there is no discussion about it. I am particularly curious about the potential failure modes of ChatGPT.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to incorporate several large models to solve the spurious-discriminative problem in open-set recognition. It first prompts ChatGPT to describe the known classes and generate a list of new classes that share the spurious-discriminative features with the known classes. Then it prompts DALL-E to generate the images of known and unknown classes. It also uses CLIP to detect the less accurate generated images and asks ChatGPT to refine the description and uses DALL-E to generate again. During inference, it uses both the CLIP model and DINO model to calculate the uncertainty scores and fuse their results. The experiments and ablation study are comprehensive. Overall, I think this paper provides a new alternative solution for open-set recognition, and the data generation process could be very useful for the industry.\n\nSTRENGTHS:\n1. The overall idea of using several large models for training-free open-set recognition is new and interesting. The pipeline of using ChatGPT to generate the new classes that share the spurious-discriminative features with the known classes is reasonable. The refinement procedure using CLIP and ChatGPT is very rigorous and effective.\n2. The inference pipeline that fuses the results of CLIP and DINO is also reasonable. It is somewhat similar to the method in the few-shot open-set recognition [1, 2], which also fuses the uncertainty score from two aspects. The authors may consider citing the related papers.\n3. The experiments and ablation studies are very comprehensive.\n\nReference:  \n[1] Ssd: A unified framework for self-supervised\noutlier detection. In ICLR, 2022.\n[2] THE DEVIL IS IN THE WRONGLY-CLASSIFIED SAMPLES: TOWARDS UNIFIED OPEN-SET RECOGNITION. In ICLR 2023.\n\nWEAKNESSES:\nThe main pipeline is a kind of data generation process, which could not be done online. So although it is a good try to use several large models and coordinate them to generate wanted data, it is more like an engineering project rather than a research topic.\n\nQUESTIONS:\n1. How much time does the data generation cost for different datasets?\n2. What if we use other pre-trained image models instead of DINO? Like ImageNet pre-trained or MoCo?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an open-set recognition framework, Large Model Collaboration (LMC), which collaborates several large models (ChatGPT, DALL-E, CLIP, and DINO) to make use of the rich implicit knowledge to reduce the reliance on spurious-discriminative features. The proposed framework consists of the following two stages:\n\n(i)\tThe first stage is to simulate virtual open-set classes, including simulating names and generating images. In order to improve the effectiveness of the simulated names for open-set classes, ChatGPT is asked with three designed questions and the corresponding intermediate rationales for obtaining names of new classes that share spurious-discriminative features with each closed-set class, and is guided to perform iterative self-check (where the new simulated class names are used to expand the name list, and the expanded name list is used to obtain new simulated class names) for covering as many spurious-discriminative features as possible. Then, ChatGPT is asked to generate diverse descriptions for each class in the expanded name list, and DALL-E is used to generate images based on these descriptions. In order to improve the accuracy of the generated images, a cyclic self-assessing module is proposed, where ChatGPT is provided with feedbacks from CLIP about the less accurate images to refine the descriptions generated by ChatGPT and DALL-E is used to re-generate images based on the refined descriptions. At this stage, an expanded list of both closed-set and simulated open-set classes and diverse generated images for each class in the list are obtained.\n\n(ii)\tThe second stage is for inference based on the expanded list and the generated images. At this stage, two alignments (image-text alignment by CLIP and image-image alignment by DINO) are performed for obtaining the corresponding scores to each testing image, and the weighted result of the two scores is used for open-set recognition.\n\nSTRENGTHS:\n(1)\tThe idea, which makes use of several large models for open-set recognition, is somewhat novel and interesting.\n\n(2)\tThe writing is clear and easy to follow. Some examples are well visualized.\n\n(3)\tExperimental results on four small-scale datasets (CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet) demonstrate that the proposed methods performs better than the comparative methods.\n\nWEAKNESSES:\n(1)\tLacking comparisons under the cross-dataset setup: The proposed framework is only evaluated under the standard setup where the open-set-class images and closed-set-class images are from a same dataset. Such experiments are insufficient, and in fact, many open-set recognition methods [15,21,36,45,55,63] have also been evaluated under the cross-dataset setup. \n\n(2)\tLacking comparisons on larger-scale datasets: in this paper, only four small-scale datasets (i.e., CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet) are used for evaluation. It is necessary to make comparisons on some larger-scale datasets, e.g., CUB and FGVC-Aircraft.\n\n(3)\tComparison with other image generative methods: One main contribution of the proposed framework is generating virtual open-set-class images. Many other works (e.g., [8,21,30]) also generate virtual images, and it is better to replace the image generation method used in the proposed framework with some of them, and make a further comparison.\n\n(4)\tLacking details of efficiency: One advantage of the training-free methods is its high efficiency. In the proposed framework, although there is no training stage, the pre-processing stage and the inference stage which is based on two alignments may also bring additional computational costs. Hence, it is suggested to provide details about the efficiency of the two stages.\n\n(5)\tLacking discussion about the ablation results: In Sec 4.4, several ablation studies are conducted. However, the authors only stated that the final results were improved, but didn’t give any discussion on the effect of each module.\n\nQUESTIONS:\nPlease see the weaknesses above.\n\nLIMITATIONS:\nThe authors did not adequately address the limitations. \nTwo additional suggestions are listed here:\n(1) It’s better to add the full name of LMC, i.e., Large Model Collaboration, in the abstract in line 6.\n\n(2) In Line 219, it is suggested to add a brief description of 5, although Fig. 5 is also described in Line 415.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}