SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haoxuan Qu *
haoxuan_qu@mymail.sutd.edu.sg
Yujun Cai
yujuncai@meta.com
Xiaofei Hui*
xiaofei_hui@sutd.edu.sg
Open-set object recognition aims to identify if an object is from a class that has been encountered during training or not. To perform open-set object recognition accurately, a key challenge is how to reduce the reliance on spurious-discriminative features. In this paper, motivated by that different large models pre-trained through different paradigms can possess very rich while distinct implicit knowledge, we propose a novel framework named Large Model Collaboration (LMC) to tackle the above challenge via collaborating different off-the-shelf large models in a training-free manner. Moreover, we also incorporate the proposed framework with several novel designs to effectively extract implicit knowledge from large models. Extensive experiments demonstrate the efficacy of our proposed framework. Code is available here.
Open-set object recognition aims to identify if an object is from a closed-set class that has appeared during training, or an open-set class that has not been encountered in the training set. It is a crucial task since open-set classes always exist in the real world [63], and misidentifying open-set classes into closed-set ones can lead to severe consequences in many practical applications [2], such as autonomous driving [19], robotics system [54], malware classification [18], and medical analysis [15]. Due to its importance, open-set object recognition has received a lot of research attention recently and various methods have been proposed [14, 36, 28, 6, 66, 56, 12].
To perform open-set object recognition well, some existing works [34, 40] have noted that a key challenge is how to reduce the reliance on spurious-discriminative features. Specifically, a spurious-discriminative feature refers to a feature that satisfies the following two requirements: 1) it is discriminative between a closed-set class and other closed-set classes; 2) it is shared between this closed-set class and certain open-set classes. As suggested by previous works [34, 40], while such spurious-discriminative features can facilitate classification within the closed set, in open-set recognition, relying on such features can make closed-set and open-set classes to be easily confused
*Both authors contributed equally to the work.
with each other, and thus bring negative effects in accurately identifying open-set objects. For example, as shown in Fig. 1, given a closed set of classes including bighorn, tabby cat, and hog, within these closed-set classes, "with horn" as a feature of bighorn can be used to discriminate bighorn from other classes (tabby cat and hog). However, relying on "with horn" to perform open-set object recognition, bighorn can be easily confused with potential open-set classes such as ox and gazelle that are also "with horn". Therefore, it can be difficult to correctly identify ox and gazelle as open-set objects. To avoid open-set objects from being misidentified into closed-set classes, a common type of methods is to simulate additional virtual open-set samples or classes [14, 36, 67, 12, 34]. Specifically, among this type of methods, various external knowledge bases have been used, such as pre-trained GloVe embeddings [67] and extra datasets [12]. Yet, despite the considerable efforts made and various architectures specifically designed for this task, the open-set object recognition task remains challenging. This is because, in the real world, various different sorts of open-set classes can appear [11]. Moreover, given a trained classifier, to further equip it with the ability to identify open-set objects, most existing open-set recognition methods require an extra training process. Especially when the classifier has been deployed, conducting such an extra training process can be inconvenient, as it needs both additional training time and access to training data. Note that the training data can be no longer accessible during deployment (e.g., due to privacy concerns [35, 22]).
Recently, large models containing very rich implicit knowledge [17] have served as powerful external knowledge bases in many areas [13]. For example, ChatGPT [38] possesses rich common-sense knowledge, and has been applied in various natural language processing tasks [42, 20]. DALL-E [47], which holds strong image-generative ability, has been used in different generation tasks [32, 26]. CLIP [45] has been leveraged for visual recognition tasks [72, 41] since it can align relevant image-text pairs. Besides, being capable of aligning relevant image patches, DINO [5] has been applied in areas [55, 1] such as i
Figure 1: Illustration of a spurious-discriminative feature example. As shown, given "with horn" a spurious-discriminative feature of the closed set, DIAS [34] as a recently proposed open-set recognition method misidentifies ox and gazelle (both with horn) as closed-set classes with high probability.
Inspired by that large models often contain rich knowledge, here we aim to investigate leveraging such rich knowledge to handle the open-set object recognition task. However, using large models to perform open-set object recognition is non-trivial due to the following challenges: 1) It can be difficult to directly leverage an existing large model to effectively perform open-set object recognition. Take CLIP as an example, being able to align an image with a closed set of classes and thus finding out the most relevant class in the set, CLIP has been applied in various visual recognition tasks, such as zero-shot and few-shot image classification [45]. However, CLIP cannot handle the open-set object recognition task effectively by itself. This is because, various sorts of open-set classes exist in the real world, and it is difficult to know which class should the testing image be aligned with when using CLIP. 2) Meanwhile, as the knowledge in large models is often implicit [57, 3], it can be challenging to extract the desired knowledge (e.g., in our case, the knowledge that can facilitate open-set object recognition) from large models successfully [57]. To handle these challenges, in this work, with the observation that different large models pre-trained through different paradigms can contain distinct knowledge, we first propose a novel open-set object recognition framework named Large Model Collaboration (LMC). As shown in Fig. 2, the proposed LMC performs open-set object recognition in a training-free manner by leveraging rich and distinct knowledge from different off-the-shelf large models complementarily. Moreover, LMC is also incorporated with several novel designs to facilitate extracting implicit knowledge in large models effectively.
Overall, to better perform open-set object recognition, LMC aims to reduce the reliance of this task on spurious-discriminative features. To achieve this, LMC proposes to simulate additional virtual open-set classes that share the spurious-discriminative features through the collaboration of different large models. Specifically, in LMC, motivated by that ChatGPT possesses rich common-sense knowledge, we first adopt ChatGPT to expand the list of closed-set classes with additional virtual
open-set classes. During this process, to extract implicit knowledge from ChatGPT effectively, we propose to guide ChatGPT with intermediate reasoning, while also enabling ChatGPT to perform self-checking on whether it covers as many spurious-discriminative features as possible. Moreover, inspired by that images can catch different details that are complementary to the text domain [60], in addition to representing classes in the expanded list with their names in the text domain, we propose to further use image information to better represent these classes. Specifically, to facilitate extracting image information from large models, we propose to incorporate our framework with a novel cyclic module, which collaborates ChatGPT, DALL-E, and CLIP through a cross-assessing mechanism to generate diverse images for each class. Once both the text and image information have been extracted for each class in the expanded list, during inference, we propose to use the image-text alignment ability of CLIP and the image-image alignment ability of DINO in collaboration to perform open-set object recognition on testing images.
The contributions of our work are summarized as follows. 1) We propose LMC, a novel framework that can handle the open-set object recognition task in a training-free manner, via collaborating different off-the-shelf pre-trained large models and leveraging their knowledge in a complementary manner. 2) We introduce several designs in LMC to extract both text and image information from large models effectively. 3) On the evaluated benchmarks, LMC achieves state-of-the-art performance.
Open-set Object Recognition. Due to the wide range of applications, open-set object recognition has received a lot of research attention [48, 4, 50, 14, 63, 39, 68, 53, 51, 7, 6, 33, 23, 56, 61, 19, 58, 70, 31, 71, 49, 21]. Bendale and Boult [4] made the first attempt of applying deep neural networks into the task of open-set recognition. Specifically, they first showed that simply thresholding on the softmax probability cannot yield a robust open-set recognition, and thus proposed Openmax based on the Extreme Value Theory. Ge et al. [14] further proposed G-Openmax to generate unknown samples using generative models. After that, Neal et al. [36] showed the effectiveness of generating images that do not belong to any of the closed-set classes but look similar to those images in the training dataset. Later on, OpenGAN, which uses real open-set images to perform model selection, was proposed by Kong et al. [23]. Vaze et al. [56] proposed to well-train the closed-set classifier so that it can perform both closed-set image classification and open-set recognition well. Besides, Esmaeilpour et al. [12] proposed to train an image description generator using an extra image captioning dataset and use the trained generator to improve the open-set object recognition performance.
Different from existing approaches, in this work, from a novel perspective, we investigate how to collaborate different large models to perform open-set object recognition in a training-free manner. Besides, we also introduce several designs to extract implicit knowledge from large models effectively.
Large Models. Recently, a variety of large models have been proposed, such as ChatGPT [38], DALL-E [47], CLIP [45], and DINO [5]. Such large models contain rich knowledge and have been studied in various tasks [20, 65, 69, 32, 64, 27, 1], such as language translation [20], video generation [32], and image retrieval [1]. In this work, we design a new framework to handle the open-set object recognition task by cross-assessing among large models and leveraging the rich and distinct knowledge from different large models in a complementary manner.
Given a testing image from  $Y_{c} \cup Y_{o}$ , where  $Y_{c}$  represents closed-set classes in the training set and  $Y_{o}$  represents open-set classes that are not included in the training set, the goal of open-set object recognition is to identify if the image is from a closed-set class or an open-set class. To recognize open-set objects accurately, as suggested by previous works [40, 34], a key challenge is: how to reduce the reliance on spurious-discriminative features. To better tackle this challenge, in this work, inspired by that large models can hold very rich while distinct implicit knowledge due to their different training paradigms, we propose a novel open-set object recognition framework LMC. Specifically, as shown in Fig. 2, LMC first extracts the implicit knowledge from different off-the-shelf large models to simulate additional virtual open-set classes that share the spurious-discriminative features. After that, LMC utilizes these virtual open-set classes to make the spurious-discriminative features "less discriminative" during its inference process.
Below, we first describe how LMC extracts implicit knowledge from different large models to effectively simulate virtual open-set classes. After that, we introduce the inference process of LMC.
3.1 Virtual Open-set Class Simulation
In the proposed LMC, to reduce the reliance on spurious-discriminative features, we aim to utilize the rich implicit knowledge of different large models to simulate additional virtual open-set classes. Specifically, motivated by that knowledge from different modali
Figure 2: Illustration of the proposed LMC framework.
ties (e.g., text and image) can contain different details [60], we aim to leverage implicit knowledge of large models in both the text domain and the image domain to facilitate simulation. To achieve this, we first simulate the names of virtual open-set classes and expand the list of closed-set classes with these virtual open-set classes. We then further generate diverse images for each class in the list to describe each class better.
Simulating names of virtual open-set classes. Inspired by that ChatGPT holds rich common-sense knowledge, here we aim to simulate names of virtual open-set classes by asking ChatGPT questions. Specifically, to perform such a simulation in high quality, we find that, it is necessary to 1) ensure that ChatGPT can well understand the question, and 2) encourage ChatGPT to cover as many spurious-discriminative features as possible. To achieve these, we introduce the following two designs in our framework.
Inspired by chain-of-thought [59] that large language models can better understand the given instruction when they are guided with intermediate reasoning, to ensure that ChatGPT can well understand the question, we here also guide ChatGPT with intermediate reasoning. Specifically, we find that an effective way to guide ChatGPT is to ask ChatGPT the following three questions for each closed-set class: 1) "Given a list of classes [classes], can you describe the visual features of each class in the list?", where [classes] represents the names of the closed-set classes; 2) "What are the discriminative visual features of class [class]
Figure 3: Formulation of language command for ChatGPT in a step-by-step manner with intermediate rationales involved.
compared with other classes in the list?”, where [class] represents the closed-set class that the current set of questions is asked for; 3) “Can you list other classes that also share these discriminative visual features?” In Fig. 3, we illustrate an example when the above three questions are asked for the class ladybug. As shown, by formulating the language command in this way, besides the final answer to question 3), ChatGPT is also demanded to generate intermediate rationales that lead to the final answer through the first two questions. This can help ChatGPT to better understand our final purpose, i.e., simulating virtual open-set classes that share spurious-discriminative features. In Particular, in Fig. 3, tortoise beetle and ladybird spider are the simulated virtual open-set classes that share spurious-discriminative features with the closed-set class ladybug. Through the above three questions, we can simulate the names of virtual open-set classes
that share spurious-discriminative features with a single closed-set class. To simulate the names of virtual open-set classes for all closed-set classes, we can simply ask the above three questions for each class in the closed set either sequentially, or concurrently via different ChatGPT instances.
In addition, to simulate a comprehensive list of virtual open-set classes to better cover the spurious-discriminative features, we aim to further enable ChatGPT to perform self-checking. In other words, we aim to guide ChatGPT to re-think if there are other spurious-discriminative features that are missed. To achieve this, for each closed-set class, after asking the above three questions, we expand the list of classes in question 1) with the obtained virtual open-set classes and re-ask ChatGPT the three questions. Note that by expanding the list with the obtained virtual open-set classes, we exclude the spurious-discriminative features that are already covered and thus force ChatGPT to look for spurious-discriminative features that are not discovered yet. This iteration of self-checking terminates when ChatGPT stops proposing new virtual open-set classes or a maximum number of cycle times is reached. After performing self-checking, inspired by [34] that all sorts of classes can appear in the real world, we also follow [34] to simulate virtual open-set samples that are less similar to closed-set samples (sharing less spurious-discriminative features) via asking ChatGPT: "Given a list of classes [classes], can you name classes that are not similar to them?" Note after expanding the list of closed-set classes with simulated virtual open-set classes through applying the above designs, those spurious-discriminative features that are originally discriminative among the closed-set classes would be "less discriminative" in the expanded list.
Generating diverse images and cross-assessing. Above we simulate the names of virtual open-set classes  $Y_{v\_o}$ . Here, inspired by that images can catch different details that can be complementary to the text domain [60], we aim to describe both the closed-set classes and the simulated virtual open-set classes in more detail by further generating diverse images for each class  $y \in Y_c \cup Y_{v\_o}$ . These generated images will be utilized in the inference process of LMC to reduce its reliance on spurious-discriminative features.
Specifically, for each class  $y \in Y_{c} \cup Y_{v\_o}$ , we generate diverse images for class  $y$  through the following two steps: 1) we first ask ChatGPT to generate diverse text descriptions as: "Can you write  $[K]$  diverse descriptions, each describing a different scene about the class  $[y]$ ?" where  $K$  is a hyperparameter representing the number of descriptions generated per class. 2) After that, leveraging the image-generative ability of DALL-E [47], we ask DALL-E to generate images based on these descriptions for each class as:
$$
i _ {y} ^ {k} = \operatorname {D A L L} - \mathrm {E} \left(d _ {y} ^ {k}\right), \text {w h e r e} k \in \{1, \dots , K \} \tag {1}
$$
where  $d_y^k$  denotes the  $k$ -th description generated by ChatGPT for class  $y$ , and  $i_y^k$  denotes the image generated by DALL-E based on  $d_y^k$ . While we can directly use the images generated above during inference, as shown in Fig. 5 which demonstrates a number of such images, we realize that some of such images may not hold a high enough quality to accurately represent their desired classes (e.g.,  $i_y^k$  may not represent class  $y$  accurately). Thus, directly using the above images can lead to suboptimal model performance during inference.
To address this problem, referring the images that cannot accurately represent their desired classes as the less accurate images, we aim to automatically detect these less accurate images, refine their corresponding descriptions, and regenerate them accordingly. To achieve this, inspired by that humans can refine their understanding based on others' feedback, here we want to investigate, can a large model also refine its output based on another large model's feedback? In particular, we propose a cyclic cross-assessing module, where CLIP acts as the "feedback provider", and provides ChatGPT with feedback about the less accurate images to help refine their corresponding descriptions.
Specifically, denote  $i_y^k$  an image generated for class  $y$ , and  $D$  the text descriptions "a photo of [class]" for all the classes (i.e.,  $Y_c \cup Y_{v_o}$ ). The cross-assessing module operates by iteratively conducting the following three steps: 1) Detection. To detect the less accurate images, we first leverage the strong image-text alignment ability of CLIP to align  $i_y^k$  with  $D$  as:
$$
p _ {\text {a s s e s s}} = \operatorname {s o f t m a x} \left(\operatorname {C L I P} _ {\text {v i s}} \left(i _ {y} ^ {k}\right) \left(\operatorname {C L I P} _ {\text {t e x t}} (D)\right) ^ {T}\right) \tag {2}
$$
where  $\mathrm{CLIP}_{vis}$  denotes the CLIP's visual encoder, and  $\mathrm{CLIP}_{text}$  denotes the CLIP's text encoder. The image  $i_y^k$  is determined as less accurate if compared to class  $y$ , it is more aligned towards another class  $u$  (i.e., instead of class  $y$ , another class  $u$  corresponds to the highest probability value in  $p_{assess}$ ). 2) Refinement. Next, if  $i_y^k$  is determined as a less accurate image, in order for its corresponding
description  $d_y^k$  to describe class  $y$  more properly, we ask ChatGPT to refine  $d_y^k$  based on feedback from CLIP as: "This description  $[d_y^k]$  seems more like class  $[u]$  to me. Can you refine it to enhance the characteristics of class  $[y]$ ?". 3) Regeneration. Lastly, the image  $i_y^k$  is regenerated through DALL-E based on the refined text description.
Through conducting the above three steps iteratively, during every iteration, we can leverage CLIP to provide specific feedback for ChatGPT so that ChatGPT can refine the descriptions more toward the direction of our desire. With these refined descriptions, images that can well represent each class will be generated. The above iteration terminates when all the generated images are most aligned with their desired classes or a maximum number of cycle times is reached. Note after reaching the maximum number of cycle times, we discard those images that are still less accurate.
3.2 Inference Process
Above we expand the list of closed-set classes with additional virtual open-set classes, and generate diverse images for each class in the expanded list. In this section, we discuss how our framework utilizes the expanded list and the generated images to make the spurious-discriminative features "less discriminative" during its inference process. Note our framework does not require any training process before its inference, which can facilitate its convenient usage in real-world applications.
Specifically, during inference, inspired by the strong image-text alignment ability of CLIP [45] and the strong image-image alignment ability of DINO [5], we propose to adopt both CLIP and DINO to perform open-set object recognition. Note CLIP and DINO are used in collaboration since they can complement each other well. To be more concrete, by aligning the testing image with the names of all the classes in the expanded list (i.e.,  $Y_{c} \cup Y_{v\_o}$ ), we first adopt CLIP to match the testing image with the overall concept of each class. Meanwhile, by leveraging DINO to align the testing image with the generated images of each class in image patch level, we better match the testing image with the detailed local features of each class.
Before entering the inference process, with the notice that certain CLIP and DINO features are used repeatedly across different testing images, we first pre-store these features to keep the efficiency of the inference process. Specifically, denoting  $f_{text}^{\mathrm{CLIP}} = \mathrm{CLIP}_{text}(D)$  the CLIP text feature of all classes, we first store  $f_{text}^{\mathrm{CLIP}}$ , which has already been computed in the above Eq. 2. Moreover, denoting DINO $_{vis}$  the DINO's encoder and  $I_y$  the set of images generated from class  $y$  through the cyclic cross-assessing module, we compute the DINO visual feature for class  $y$  as  $f_y^{\mathrm{DINO}} = \mathrm{DINO}_{vis}(I_y)$  and store  $f_y^{\mathrm{DINO}}$  for each class  $y \in Y_c \cup Y_{v\_ o}$ .
After pre-storing these features, we then introduce the inference process of the proposed LMC framework. Specifically, given a testing image  $i_{test}$ , utilizing pre-stored features, LMC first aligns  $i_{test}$  with the names of all the classes in  $Y_{c} \cup Y_{v\_o}$  through CLIP as:
$$
p _ {\mathrm {C L I P}} = \operatorname {s o f t m a x} \left(\operatorname {C L I P} _ {\text {v i s}} \left(i _ {\text {t e s t}}\right) \left(f _ {\text {t e x t}} ^ {\mathrm {C L I P}}\right) ^ {T}\right) \tag {3}
$$
where  $p_{\mathrm{CLIP}}$  denotes the softmax probability derived from CLIP. At the same time, LMC also aligns  $i_{test}$  with the generated images through DINO as: