SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p _ {\mathrm {D I N O}} = \operatorname {s o f t m a x} \left(\left\{l _ {\mathrm {D I N O}} ^ {y} \mid y \in Y _ {c} \cup Y _ {v _ {- o}} \right\}\right), \text {w h e r e} l _ {\mathrm {D I N O}} ^ {y} = \operatorname {a v e r a g e} \left(\mathrm {D I N O} _ {\text {v i s}} \left(i _ {\text {t e s t}}\right) \left(f _ {y} ^ {\mathrm {D I N O}}\right) ^ {T}\right) \tag {4}
$$
where  $l_{\mathrm{DINO}}^y$  is derived from aligning  $i_{test}$  with  $I_y$  through DINO, and  $p_{\mathrm{DINO}}$  denotes the softmax probability correspondingly calculated for all the classes in  $Y_c \cup Y_{v_o}$ . After deriving  $p_{\mathrm{CLIP}}$  and  $p_{\mathrm{DINO}}$ , LMC then incorporates them to determine whether  $i_{test}$  is from the closed-set classes or open-set classes as:
$$
p _ {i n c} = \alpha p _ {\text {C L I P}} + (1 - \alpha) p _ {\text {D I N O}}, \quad S = \max  _ {y \in Y _ {c}} \left(p _ {i n c} (y \mid i _ {t e s t})\right) \tag {5}
$$
where  $\alpha$  is a hyperparameter denoting the incorporation weight,  $p_{inc}$  denotes the softmax probability derived from incorporating  $p_{\mathrm{CLIP}}$  and  $p_{\mathrm{DINO}}$ ,  $p_{inc}(y|i_{test})$  denotes the probability value that class  $y$  corresponds to in  $p_{inc}$ , and  $S \in [0,1]$  is the closed-set score measured by our framework. Following [34], we represent  $S$  as the maximum probability value among the closed-set classes. By performing open-set object recognition in the above way, open-set images would no longer be easily misidentified into the closed-set classes due to the contained spurious-discriminative features. This is because, after expanding the list of closed-set classes with the virtual open-set classes, in the expanded list, besides a certain closed-set class, a spurious-discriminative feature can be held by some virtual open-set classes as well. Then, when an open-set testing image containing spurious-discriminative features is
Table 1: The AUROC results on the detection of closed-set and open-set samples. Results are averaged over five random dataset splits following [6, 56].
Method	Venue	CIFAR10	CIFAR+10	CIFAR+50	TinyImageNet
Methods that involve a training process
OSRCI [36]	ECCV 2018	69.9 ± 3.8	83.8 ± -	82.7 ± -	58.6 ± -
C2AE [39]	CVPR 2019	89.5 ± -	95.5 ± -	93.7 ± -	74.8 ± -
RPL [7]	ECCV 2020	90.1 ± -	97.6 ± -	96.8 ± -	80.9 ± -
OpenHybrid [68]	ECCV 2020	95.0 ± -	96.2 ± -	95.5 ± -	79.3 ± -
CVAECapOSR [16]	ICCV 2021	83.5 ± 2.3	88.8 ± 1.9	88.9 ± 1.7	71.5 ± 1.8
ARPL [6]	TPAMI 2021	90.1 ± 0.5	96.5 ± 0.6	94.3 ± 0.4	76.2 ± 0.5
ARPL+CS [6]	TPAMI 2021	91.0 ± 0.7	97.1 ± 0.3	95.1 ± 0.2	78.2 ± 1.3
PMAL [31]	AAAI 2022	95.1 ± -	97.8 ± -	96.9 ± -	83.1 ± -
ZOC [12]	AAAI 2022	93.0 ± 1.7	97.8 ± 0.6	97.6 ± 0.0	84.6 ± 1.0
MLS [56]	ICLR 2022	93.6 ± -	97.9 ± -	96.5 ± -	83.0 ± -
DIAS [34]	ECCV 2022	85.0 ± 2.2	92.0 ± 1.1	91.6 ± 0.7	73.1 ± 1.5
Class-inclusion [8]	ECCV 2022	94.8 ± -	96.1 ± -	95.7 ± -	78.5 ± -
ODL [30]	TPAMI 2022	85.7 ± 1.3	89.1 ± 1.4	88.3 ± 0.0	76.4 ± 1.7
ODL+ [30]	TPAMI 2022	88.5 ± 1.3	91.1 ± 0.8	90.6 ± 0.0	74.6 ± 0.8
CSSR [19]	TPAMI 2022	91.3 ± -	96.3 ± -	96.2 ± -	82.3 ± -
RCSSR [19]	TPAMI 2022	91.5 ± -	96.0 ± -	96.3 ± -	81.9 ± -
Methods that involve no extra training process
Softmax	93.7 ± 1.7	96.5 ± 0.6	95.1 ± 1.3	83.5 ± 2.7
Ours	96.6 ± 0.3	98.9 ± 0.7	98.5 ± 0.4	86.7 ± 1.4
Table 2: The OSCR results for open-set object recognition. Results are averaged over five random dataset splits following [6].
Method	Venue	CIFAR10	CIFAR+10	CIFAR+50	TinyImageNet
Methods that involve a training process
GCPL [62]	CVPR 2018	84.3 ± 1.7	91.0 ± 1.7	88.3 ± 1.1	59.3 ± 5.3
RPL [7]	ECCV 2020	85.2 ± 1.4	91.8 ± 1.2	89.6 ± 0.9	53.2 ± 4.6
ARPL [6]	TPAMI 2021	86.6 ± 1.4	93.5 ± 0.8	91.6 ± 0.4	62.3 ± 3.3
ARPL+CS [6]	TPAMI 2021	87.9 ± 1.5	94.7 ± 0.7	92.9 ± 0.3	65.9 ± 3.8
ODL [30]	TPAMI 2022	84.8 ± 1.4	92.5 ± 1.0	89.8 ± 0.7	64.3 ± 3.2
ODL+ [30]	TPAMI 2022	86.9 ± 1.5	93.2 ± 0.3	90.3 ± 0.2	59.2 ± 2.1
Methods that involve no extra training process
Softmax	90.1 ± 1.1	94.0 ± 1.4	92.8 ± 1.1	75.1 ± 5.0
Ours	93.6 ± 1.5	96.8 ± 0.7	96.4 ± 0.4	80.6 ± 3.4
aligned with the expanded list, it will tend to approach both the closed-set and virtual open-set classes that contain these features at the same time, rather than approaching only certain closed-set classes. Therefore, after softmax normalization, the closed-set score  $S$  of the open-set testing image will no longer be easily overestimated due to the spurious-discriminative features contained in the image.
To evaluate the effectiveness of our proposed framework LMC, we conduct experiments on four evaluation protocols including CIFAR10, CIFAR+10, CIFAR+50, and TinyImageNet. We conduct our experiments on an RTX 3090 GPU.
CIFAR10. CIFAR10 [24] is a ten-class object recognition dataset. On this dataset, following [6, 56], 6 closed-set classes and 4 open-set classes are randomly sampled for evaluation each time.
CIFAR+10 & CIFAR+50. Following [6, 56], we also evaluate our method on CIFAR+10 and CIFAR+50, which are extensions of the CIFAR10 evaluation protocol. Specifically, following [6, 56], for the CIFAR+N experiments, 4 classes are randomly sampled from CIFAR-10 as the closed-set classes and  $N$  non-overlapping classes are randomly sampled from CIFAR-100 [24] as the open-set classes for evaluation ( $N = 10$  for CIFAR+10 and  $N = 50$  for CIFAR+50).
TinyImageNet. TinyImageNet [9] is a subset of ImageNet and contains 200 classes. Following [6, 56], each time, 20 closed-set classes and 180 open-set classes are randomly sampled.
Evaluation metric. Following [6, 56], we use the following two metrics: AUROC and OSCR [10]. Among these two metrics, AUROC focuses on evaluating model performance on identifying open-set samples, and OSCR measures the trade-off between classification accuracy and open-set recognition performance. More details especially about the OSCR metric are in the supplementary.
In LMC, we integrate rich and distinct knowledge from off-the-shelf large models including ChatGPT, DALL-E, CLIP, and DINO in a complementary manner. Specifically, for ChatGPT, we use GPT-3.5-turbo. For CLIP, we use ViT-B/32 for its visual encoder, and the transformer architecture described in [46] for its text encoder. For DINO, we use the second version of DINO [37] with ViT-B/14. We set the maximum number of cycle times for ChatGPT's self-checking to 3. To generate diverse images for each class through the proposed cyclic cross-assessing module, we set the number of diverse detailed descriptions  $K$  generated for each class to 10, and the maximum number of cycle times for cyclic cross-assessing to 3. Besides, we set  $\alpha$  in incorporating  $p_{\mathrm{CLIP}}$  and  $p_{\mathrm{DINO}}$  to 0.6.
We report the AUROC results and the OSCR results respectively in Tab. 1 and Tab. 2. Specifically, besides comparing with existing open-set object recognition methods that assume no access to open-set classes, in these two tables, we also compare our framework with Softmax as a baseline that collaborates large models (CLIP and DINO) and involves no extra training process. Particularly, Softmax does not simulate any virtual open-set classes. It treats the ensemble of CLIP and DINO as a closed-set classifier and computes the maximum probability value of the classifier's output as its score  $S$  in the same way as Eq. 3 to Eq. 5 (more details about this baseline are in the supplementary). As shown, compared to state-of-the-art open-set recognition methods and the baseline Softmax, our framework achieves superior performance across different evaluation protocols, demonstrating the effectiveness of our framework.
In this section, we conduct extensive ablation experiments on the TinyImageNet evaluation protocol. In particular, in the experiments, we report the AUROC metric averaged over five dataset splits. More ablation studies such as experiments w.r.t. hyperparameters are in the supplementary.
Evaluation on the two alignments performed by LMC. In LMC, as shown in Fig. 2, during inference, we align the testing image with the images representing each class through DINO, as well as align the testing image with the name of each class through CLIP. Above, we already evaluate our framework as a whole. Here, we further assess the following two variants of our
Table 3: Evaluation on the two alignments performed by LMC.
Method	AUROC
w/o aligning with names	84.4
w/o aligning with images	82.1
LMC	86.7
framework, i.e., the variant (w/o aligning with names) that only aligns the testing image with the images representing each class, and the variant (w/o aligning with images) that only aligns the testing image with the name of each class. As shown in Tab. 3, ignoring either alignment would lead to performance drop compared to our framework. This demonstrates the effectiveness of our framework in concurrently aligning the testing image with the names and the generated images.
Evaluation on the two designs for class name simulation. To simulate names of virtual open-set classes in high quality, in our framework, we introduce two designs to formulate language commands for ChatGPT properly, i.e., guiding ChatGPT with intermediate reasoning and enabling ChatGPT to perform self-checking.
Table 4: Evaluation on the two designs for class name simulation.
Method	AUROC
w/o intermediate reasoning & self-checking	84.1
w/o intermediate reasoning	85.6
w/o self-checking	85.9
LMC	86.7
Here, to assess the effectiveness of the two designs introduced, we test three variants: 1) w/o intermediate reasoning & self-checking: this variant uses neither of the two designs and simulates names of virtual open-set classes for each closed-set class via directly asking ChatGPT the following question: "Given a list of classes [classes], can you name other classes that share visual features with [class] while these shared features are discriminative in the list?", where [class] represents a closed-set class and [classes]
represents the list of closed-set classes. 2) w/o intermediate reasoning: this variant enables ChatGPT to perform self-checking but does not guide ChatGPT with intermediate reasoning. Specifically, in this variant, after asking ChatGPT the question in variant 1), we expand [classes] in this question with the obtained virtual open-set classes and re-ask ChatGPT this question iteratively until ChatGPT stops proposing new virtual open-set classes or a maximum number of cycle times is reached. As with our framework, we set the maximum number of cycle times to 3 here as well. 3) w/o self-checking: this variant guides ChatGPT with intermediate reasoning but does not perform self-checking. Specifically, in this variant, we only ask the three questions proposed in Sec. 3.1 once but not in an iterative manner. As shown in Tab. 4, our proposed framework consistently outperforms all three variants. This demonstrates (1) the effectiveness of the intermediate reasoning design, which can lead ChatGPT to better understand the asked question; (2) the effectiveness of the self-checking design, which can lead ChatGPT to simulate a more comprehensive list of virtual open-set classes.
Impact of the cyclic cross-assessing module. To ensure that the images generated by DALL-E can well represent their desired classes, in our framework, we propose a cyclic cross-assessing module to refine the generated images iteratively. To validate this module, we consider three alternative ways to generate images: 1) w/o cross-assessing: this variant directly passes the generated images to the inference process of our framework without checking or refining them. 2) Check and discard: this variant
checks the generated images through CLIP and passes only those images that pass the checking (i.e., those images that are most aligned towards their desired classes) to the inference process of our framework. Note in this variant, those images that do not pass the checking are simply discarded and no refinement is involved. 3) Check and naively refine: this variant both checks and refines the generated images. However, in this variant, we directly ask ChatGPT to refine the descriptions of those incorrect images, and the feedback from CLIP (i.e., a target refinement direction) is not involved in the refinement. As shown in Tab. 5, the performance of all three variants are worse than our method. This shows the effectiveness of our proposed cyclic cross-assessing module, which can leverage CLIP to provide specific feedback for ChatGPT, so that ChatGPT can refine the descriptions more toward the direction of our desire.
Overlap between the virtual open-set classes and the open-set classes used for evaluation. In our method, we simulate virtual open-set classes to lead the framework to less rely on spurious-discriminative features. Here, we aim to verify that the effectiveness of our framework does not come from just matching the open-set classes used for evaluation. To achieve this, we evaluate it on two subsets
Table 6: Analysis on the overlap between the virtual open-set classes and the open-set classes used for evaluation.
Method	AUROC
Subset A	Subset B
Baseline(Softmax)	83.5	83.6
LMC	93.1	86.5
of the open-set classes used for evaluation. Specifically, the first subset (subset A) includes all those open-set classes that are contained in the list of simulated virtual open-set classes  $Y_{v\_o}$ , and the second subset (subset B) includes all the other open-set classes that are not contained in  $Y_{v\_o}$ . As shown in Tab. 6, compared to Softmax as a baseline, even for subset B, there is still a significant performance improvement, demonstrating the effectiveness of our method even when  $Y_{v\_o}$  and  $Y_o$  do not overlap at all. Moreover, we would like to point out that the virtual open-set classes and the open-set classes
Figure 4: Qualitative results on the TinyImageNet dataset. As shown, even when the testing image holds the spurious-discriminative feature ("with break" here), our framework can accurately identify if it is from closed-set or open-set classes. The above results are based on the fourth dataset split following [6]. More qualitative results are in the supplementary.
albatross (open-set)  
GT value of  $S:0$   
LMC's prediction of  $S:0.18$
goose (closed-set)  
GT value of  $S$ : 1  
LMC's prediction of  $S$ : 0.98
used for evaluation barely overlap. For example, for each data split of TinyImageNet, out of its 180 open-set classes used for evaluation, only no more than 6 classes (i.e., no more than  $3.4\%$  of classes) are within the list of virtual open-set classes. Yet, our method still achieves state-of-the-art performance. This further demonstrates that the effectiveness of our method does not naively come from matching the real open-set classes.
Qualitative results. Some qualitative results are shown in Fig. 4. As shown, our framework can perform open-set object recognition accurately even when the testing images contain spurious-discriminative features. This demonstrates that our framework can effectively reduce the reliance on spurious-discriminative features when performing open-set object recognition.
Visualization of images re-generated through the cross-assessing module. In Fig. 5, we visualize some images generated before and after passing through the proposed cross-assessing module. As shown, the images before regeneration cannot well represent their desired classes, whereas the images regenerated through the cross-assessing module can represent their desired classes more accurately. This demonstrates the effectiveness of the proposed cross-assessing module.
Desired class:	Academic gown	Christmas tree	Fox	Punching bag
Before regeneration through cross-assessing:
After regeneration through cross-assessing:
Figure 5: Visualization of the images generated before and after passing through the cross-assessing module. More visualizations are in the supplementary.
In this paper, we have proposed a novel open-set object recognition framework LMC, which collaborates different large models to recognize open-set objects. Specifically, in LMC, we utilize rich and distinct knowledge from different off-the-shelf large models to reduce the reliance of the proposed framework on spurious-discriminative features. Moreover, we also incorporate our framework with several designs to effectively extract implicit knowledge from large models. Our framework achieves state-of-the-art performance on four widely used evaluation protocols in a training-free manner. Besides, our method may potentially also (1) be applied in other related tasks such as confidence estimation [44, 43] and uncertainty estimation [25, 29]; (2) inspire new ways for follow-up research. Below, we list a few possible new ways that follow-up research work can further investigate: (1) how to collaborate large models with conventional open-set object recognition methods to further improve performance; (2) how to facilitate the usage of black-box large models in our framework through black-box optimization algorithms such as the CMA Evolution Strategy [52]. We leave these as our future works.
This work was supported by the National Research Foundation Singapore under the AI Singapore Programme (Award Number: AISG-100E-2023-121).
[1] Jon Almazán, Byungsoo Ko, Geonmo Gu, Diane Larlus, and Yannis Kalantidis. Granularity-aware adaptation for image retrieval over multiple tasks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XIV, pages 389–406. Springer, 2022. 2, 3
[2] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. 1
[3] Trevor Ashby, Braden K Webb, Gregory Knapp, Jackson Searle, and Nancy Fulda. Personalized quest and dialogue generation in role-playing games: A knowledge graph-and language model-based approach. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1-20, 2023. 2