OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Authors: Cong Hua
Qianqian Xu
Zhiyong Yang
Zitai Wang
Shilong Bao
Qingming Huang
Venue: ICML 2025 poster (2025)
Keywords: Open-world Prompt Tuning
OOD detection
Primary area: general_machine_learning

Abstract: Prompt tuning adapts Vision-Language Models like CLIP to open-world tasks with minimal training costs. In this direction, one typical paradigm evaluates model performance **separately** on known classes (*i.e.*, base domain) and unseen classes (*i.e.*, new domain). However, real-world scenarios require models to handle inputs **without prior domain knowledge**. This practical challenge has spurred the development of **open-world prompt tuning**, which demands a unified evaluation of two stages: 1) detecting whether an input belongs to the base or new domain (**P1**), and 2) classifying the sample into its correct class (**P2**). What's more, as domain distributions are generally unknown, a proper metric should be insensitive to varying base/new sample ratios (**P3**). However, we find that current metrics, including HM, overall accuracy, and AUROC, fail to satisfy these three properties simultaneously. To bridge this gap, we propose $\mathsf{OpenworldAUC}$, a unified metric that jointly assesses detection and classification through pairwise instance comparisons. To optimize $\mathsf{OpenworldAUC}$ effectively, we introduce **Gated Mixture-of-Prompts (GMoP)**, which employs domain-specific prompts and a gating mechanism to dynamically balance detection and classification. Theoretical guarantees ensure generalization of GMoP under practical conditions. Experiments on 15 benchmarks in open-world scenarios show GMoP achieves SOTA performance on $\mathsf{OpenworldAUC}$  and other metrics.

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Cong Hua" 12
Qianqian Xu
Zhiyong Yang²
Zitai Wang'
Shilong Bao
Qingming Huang 2.3
Abstract
Table 1. An overview of the existing metrics for OPT, A meaas
Prompt tuning adapts Vision-Language Mod- els like CLIP to open-world tasks with mini- mal training costs. In this direction, one typi- cal paradigm evaluates model performance sepa- rately known classes (i.e., base domain) and unseen classes (i.e,, new domain). However, real- world scenarios require models to handle inputs without prior domain knowledge. This practical challenge has spurred the development of open- world prompt tuning, which demands a unified evaluation of two stages: 1) detecting whether input belongs to the base or domain (P1), and 2) classifying the sample into its correct class (P2). What's more, as domain distributions are generally unknown, a proper metric should be in- sensitive to varying base/new sample ratios (P3). However, we find that current metrics, including HM. overall accuracy, and AUROC, fail satisfy these three properties simultaneously. To bridge this gap, we propose OpenworIdAUC, unified metric that jointly assesses detection and classi- fication through pairwise instance comparisons. To optimize OpenworldAUC effectively, we in- troduce Gated Mixture--o-Prrmpts (GMoP), which employs domain-specific prompts and a gating mechanism to dynamically balance de- tection and classification. Theoretical guaran- tees ensure generalization of GMoP under prac- tical conditions. Experiments on 15 bench- marks in open-world scenarios show GMoP achieves SOTA performance on OpenworldAUC and other metrics. We release the code
OverallAcc evaluates detection implicitly.
(PI Furs--taee
(P2)Second-stage clussification
(P3)Domain distribution incensitivity
Metrics
detection
HM
OverallAcc
AUROC
OpenvoridAUC
V
1. Introduction
Recent powerful Vision-Language Foundation Models (VLMs), such as CLIP (Radford et al., 2021), use natu- ral language to align visual concepts, enabling them to infer new samples. To better utilize such abilities with min- imal computational cost, Prompt Tuning (PT) has gained significant attention (Lester al., 2021; Zhou et al., 20226). PT allows the model adapt open-world scenarios with only a small set of learnable parameters for textual or visual prompts.
In this direction, common setting is base-to-new (Zhou et al., 2022b; Yao et al., 2024; Zhou et al., 2022a; Khat- tak et al., 2023b; Zhang et al., 2024), where the model trained on a set of known classes (i.e., base domain) and evaluated separately on the base domain and the unseen classes (i.e., new domain). This task implicitly assumes that the input samples are pre-labeled belonging to either the base or new domain. However, this assumption may not hold in practice. To address this limitation, (Zhou et al., 2024) adopt a divide-and-conquer strategy, inducing more practical setting named Open- Open-world Prompt Tuning (OPT). In this setting, the pipeline has two stages: the first stage performs base-to-new detection to determine whether an input belongs to the base new domain, and the second stage classifies the sample into its correct class.
In this complicated setting, how to evaluate model perfor- mance becomes challenging. We argue that an appropriate metric should comprehensively evaluate (P1) first-stage de- tection, (P2) second-stage classification, and also (P3) be insensitive to the domain distribution, i.e. the base/new ratio. Unfortunately, as summarized in Tab. 1, existing metrics do not satisfy the aforementioned properties simultancously, making them inconsistent with the actual model perfor- mance. Concretely, the harmonic mean (HM) of BaseAcc and NewAcc, a common metric for the base-to-new task,
"eey Lab of Intelligent Info. Processing, Inst. Com- put. Tech., CAS, Beijing, China "School of Comput, Sci. and Tech., UCAS, Beijing, China "Key Lab of Big Data Mining and Knowl. Manag., UCAS, Beijing, China. Correspondence to: Qianqian <xuqianqian eerigiiann@it....om ict, Qingming Huang <qmhuang @ uccas.ac.c>>.
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Leurablee prompts
Ficeycblee promppls
IPB
Learnable prounatt
B DI x
Learnable prompts
A photo of [base]
Iraa <a.
1 = (z,))
Pred
phoio of (pseudo base)
CF Toss
Now
Base
New
Maximize
Manual prompts
r
Pn
Reweighting Gate
h
A Apoooo (class]
New
OpenvoridAUC
a) Texining Dalasel Pscudo Partition
b) Base-to-New Detector
<) Classifier
d) Final Goal
Figure 1. OpenworldAUC and its optimization framework. a) We first perform pseudo partition on the training dataset to simulate new domain. b) Based on this partition, we calculate the AUROC-like ranking loss to optimize the detector. e) Then, to optimize base-domain classifier, the CE loss is calculated on the original training set. Herein, a gating mechanism selects the consecly-classified samples to calculate the aforementioned ranking loss. e) For new domain classifter, we adopt a fixed hand-crafted prompt to avoid the overfitting on the basc domain. d) Overall, each prompt fulfills its specific responsibility to jointly maximize the OpenworIdAUC metrie.
only evaluates second-stage performance but ignores de- tection performance (P1). The empirical results in (Zhou et al., 2024) and our theoretical analysis in Sec.3.1 both il- lustrate that models with the same HM can have significant performance gap in the OPT setting. To alleviate this issue, (Zhou et al., 2024) turn to calculate the accuracy defined on the entire class space, i.e., the mixture of base and new classes, denoted by OverallAcc. Compared with HM, this metric can assess detection to some extent by penalizing misaligned OOD/ID scores between domains. However, as shown in S.c.3.2, we find that OverallAcc is highly sensi- tive the domain distribution, violating (P3). To fix this issue, one naive option is AUROC (Fawcett, 2006; Yang & Ying, 2023; Bao et al., 2025), a popular metric for OOD detection (Hendrycks & Gimpel, 2017) due to its insensi- tivity to label distribution. However, AUROC ignores the classification performance, overlooking (P2). Hence, a natural question arises: Can we find a single metric that simultaneously satisfies three properties?
goal, the key challenge is that a single prompt is insuffi- cient to balance all three components in the OpenworldAUC objective since each component involves different inputs and sub-objectives. To address this issue, we propose a novel Gated Mfxture-of-Prompp (GMoP) approach. On the one hand, each prompt targets specific component. On the other hand, the gating mechanism selects the correctly- classified instances to optimize the detection performance via an AUROC-liKe ranking loss. In this way, we achieve a divide-and and-conquer optimization, which is exactly consis- tent with the OPT pipeline. Besides, we adopt the pseudo base-to-new partitions of the training set as proposed in (Zhou al., 2024) to calculate the ranking loss.
Last but not least, in Sec.5.2, we explore the generalization guarantee for our proposed learning framework, which, to the best of our knowledge, is rarely discussed in the PT community. Our theoretical results suggest that a training set with a moderate volume and a proper partition number can guarantee satisfactory generalization performance. Finally, in Sec.6, extensive empirical results fifteen benchmarks in the open-worke scenario demonstrate that our proposed method outperforms the state-of-the-ar art methods on both OpenworldAUC and other metrics.
To answer this question, in Sec.4, we establish novel metric named OpenworIdAUC, which evaluates bawe-to-new detection, base classification, and new classification in a unified manner. Specifically, this measure has a pairwise formulation, where each pair consists of base instance and a new instance For each pair, OpenworldAUC measures the joint probability that 1) the new instance has a higher OOD score than the base instance (P1), and 2) the two instances are correctly classified in their respective domains (P2). This ranking- based approach can naturally evaluate model performance under any domain distribution (P3). Further analysis theoretically shows that OpenworidAUC overcomes the limitations of the aforementioned metrics.
Overall, the contributions of this paper are as follows:
Novel metric. A novel metric named OpenworidAUC is proposed for the OPT task, which embraces base-to- new detection, base classification, and new classifica- tion unified way and is also insensitive towards varying domain distributions.
Learning framework. A learning framework named GMoP is proposed with theoretical guarantee, where multiple prompts jointly optimize OpenworldAUC via a and-cquu strategy.
In view of this, we design learning framework to maxi- mize OpenworIdAUC effectively in 1. To achieve this
2

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Empirical studies. Comprehensive empirical results fifteen benchmarks in the open-world scenario speak to the efficacy of our proposed method on both OpenworldAUUC and other metrics.
2024). Let BaseAcc and NewAcc represent the classifica- tion accuracy defined the base domain Dj and the new domain Da, respectively. Suppose 8, E E and 8n, E R are the logit scores of the base and new domain produced by and h, where C. and Cn are the numbers of classes in each domain. Then, BaseAcc and NewAcc are defined as follows:
2. Problem Formulation
This paper focuses on prompt tuning for open-world clas- sification. Let æ X' be the input sample and y E 3 := 0% U be the corresponding label, where X' is the input space; Vd, and Vn, are the label spaces of base classes and new classes, respectively. In open-world learning, the model is first trained on the base domain D. := X x Vi, and fur- ther evaluated on the overall domain D == Do U'n, where Dn := X' Vn denotes the new domain.
BaseAcc
:- No. (1E6.86)ES.
1
Σ
1Bm = arg max 801, y'E3%
NewAcc Nn
Σ
1ly arg max 8n];
y E'
(ES.
where N₂ and N₁ are the number of samples in S, and Sn.
respectively and [[] is the indicator function. Then, HM defined by the harmonic mean of BaseAcc and NewAcc:
To achieve this goal, Open-world Prompt Tuning (OPT) (Zhou al., 2024) fine-tunes the prompt of a foundation model, such as CLIP (Radford et al., 2021), on the base dataset S {(=l]) , Vb ) _ sampled from Dz, Specif- ically, let fo be the visual feature of the input image æ obtained by the frozen image encoder. For text representa- tion, class is first embedded through a prompt template P([CLASS:): 0) parameterized by the leanable token O. Then, the text feature ft, is obtained by feeding the i-th class prompt into the frozen text encoder. Finally, the poste- rior probability Ply i|az) is assumed be proportional to the cosine similarity between ft, and fe.
HM x BaseAcc NewAcc
(6)
BaseAcc + NewAcc
From this formulation, one can easily find that HM only evaluates the domain-specific classification performance in OPT. This limitation becomes evident in the following proposition, whose proof can be found in App.A.1.
Proposition 3.1. Given dataset S sampled from the overall domain D. for any (g. (g.h), h). one can always find worse-performing (g, hi in OPT that satisfies HM(g, h) HM( h).
During testing, OPT adopts a divide-and aeree-andd-ooooe conquer strategy by introducing an additional evaluation of detection, com- pared with the common PT task (Zhou et al., 2022b). In a nutshell, this task involves two stages: base-to-new detec- tion and domain-specifie classification. In the first stage, base-to-new detector X' - [0, 1] identifies whether the input sample belongs to the base or the new domain according to a threshold t, where a larger r(z) indicates a higher probability of the sample belonging to the base domain. In other words, this stage actually performs OOD detection with an OOD score defined by 1 r(x).. In the second stage, x is further sent its corresponding classifier, denoted by g and h for the base and new domain.
Remark. The proof follows this intuition: consider one base sample 2, and suppose that (g. h) predicts it correctly while satisfying maxy'e>, 8b maxy'e>, 8n, > -. We can construct (g.h) such that ğ remains correct on 24 but satis- fies maxy'ey, & maxy'e,, &, < 0. In this case, (g, h) and (ğ. h) share the same HM score, yet (ū,h) misidenti- fies more base sample new than hg in OPT, thereby matching the empirical observations in (Zhou et al., 2024).
From Prop.3.1, we can conclude that HM ignores the detec- tion performance (P1) and thus is not a proper choice for the OPT task.
3. Existing Metrics and Their Limitations
3.2. Overall Accuracy
In this section, we discuss the limitations of the existing metrics, including HM, OverallAce and AUROC from three key perspectives: comprehensively evaluating first-stage detection (P1), second-stage classification (P2), and being insensitive towards domain distribution (P3).
To alleviate this issue of ignoring (P1), the recent study DeCoOp (Zhou et al., 2024) turns to evaluate the accuracy performance directly defined on the overall domain, denoted by OverallAcc. This metric is defined as:
OverallAcc
1
Σ (a.y)ES
1ly = arg max so [s.). y'E)
3.1. Harmonic Mean
N
HM is popular metric in prompt tuning for the base-to- new task (Xian et al., 2017; Zhou et al., 2022b;a; Yao et al.,
where N is the number of samples in and denotes the concatenation operation.
3

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-worid Prompt Tuning
clear that AUROC ignores the classification performance in the second stage, overlooking (P2).
100
Acs
valuect)
Value(S)
80-
4. OpenworldAUC: A Novel OPT Metric
In this section, we first explore a naive solution by aggre- gating multiple metrics. Then, we develop a single met- ric named OpenworidAUC that can satisfy the aforemen- tioned three properties simultaneously.
10
2 0.7 0.5 0.3 0200 NewiBase Railo
10
2 0700503022
New Ratio
(a) DTD
(b) Flowers
Figure 2. The sensitive analysis of existing metric new/base ratio. OverallAcc sensitive to the domain distribution, while other metries remains stable with varying ratios of new samples.
4.1. A Naive Multiple Metries Aggregation
Since HM and AUROC are complementary and both insen- sitive to domain distributions, one potential solution is to aggregate them via linear combination. However, the fol- lowing proposition shows that this naive solution can induce another issue, with proof in App.A.2.
Compared with HM, OverallAcc considers detection to some extent. To be concrete, the prediction could be re- garded as correct only if the input has been identified in the correct domain. In other words, there exists implicit detector r(z) x maXy/e), 86 m.,, 8n. However, this metric becomes sensitive to the domain distribution, violating (P3). To illustrate this issue, we can rewrite the definition of OverallAcc as follows:
Proposition 4.1. Given a dataset S sampled from the over- all domain D. for any (r,g,h) h) under mild assumption AUROC(r) < 1, one can always find (ir, D. h) performs worse in OPT that satisfies:
Lin(AUROC(r). BaseAcc(g), NewAAcc) NewAcc(h))
OverallAcc IP, TPR, + P,, TPR,;
= Lii(AURRC())), BaseAcc(ā), NewAcc(h))
where TPR, and TPR,, are True Positive Rate of base and new domain, respectively, and Pb = N and Ir, = Na/N. From this formulation, one can find that OverallAcc can be dominated by the performance in the domain with more samples. As shown in 22g.2, OverallAcc increases significantly the number of base domain samples grows, while the other metrics remain stable. Since the domain distribution is generally unknown, a distributionn-sessitvve metric is improper for OPT.
where Lin(-) denotes the linear combination function.
Remark. The proof comes from the following intuition: given two base samples æ), and xi suppose that r and g are both wrong on z but both correct on æzf. We can construct (F,g) such that F misidentifies ai, but ğ makes correct prediction on x/. and vice versa for æ7. In this case, the linear combination remains the same, but (F, performs inferior to (r, (r.g) since more samples are misclassified.
3.3. AUROC
Another potential choice to evaluate the detection perfor- mance is AUROC (Fawcett, 2006), which is popular metric for OOD detection (Hendrycks& & Gimpel, 2017). To adapt this metric to the OPT task, we should consider the base-to- new detection as a binary classification problem. Specifi- cally, we first assign all base/new classes to one base/new super-class, respectively. Then, AUROC can be defined as the area under the curve of the True Positive Rate (TPR,.) and the False Positive Rate (FPR,s) of the new super-class, with a pairwise reformulation:
4.2. OpenworldAUC
To tailor this problem, we have to seek single measure, in which the prediction is regarded to be correct only when the model performs well in both stages. Following this idea, we first define the Miss Rate of the base class (MissRate,) and the Hit Rate of the new class (HitRate,,) in this pipeline:
> 2,30 # g(x))]] MissRates :=ED, (1((()) < + ((r(s))(ss))
HitRate, :EE,, (1 (x)) & f. yr. = h(xx,)]]
AUROC: J
TPRns (t) -d FPR
However, the two rates rely on the threshold i, which is hard to decide without prior knowledge of the new do- main. Inspired by AUROC, we calculate the area under the MissRate,,HlitRate,, curve, denoted as OpenworIdAUC:
ED (1(r(z)) > r(æn)]].
Benefiting from the pairwise ranking formulation, AUROC measures detection performance under any domain distri- bution (Yang& & Ying, 2023). As shown in 2 AUROC insensitive towards the base/new ratios. However, it is
OpenworIdAUC := J, HitRate, (f) dMissRates

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
At first glance, the integral formulation is hard compute. Fortunately, the following proposition can alleviate this is- sue, with proof in App.A.3.
parameterized by learnable tokens O:
min Es.. [1 * g 0)] + E.s., [1 o h( 0l]]
Proposition 4.2. Given pair (26: and (Ins Bu) sam- pled D, OpenworIdAUC equals to the joint possibility that 1) r ranks Ib higher than Vn- and 2) and h make correct predictions on Th and Fus respectively.
+ E fo. (r(m); 0) r(ar,: 0)) vxl
where tb == 1 g 0)] and Uy = Iv/n h( 0}]; fo.1 denotes the 0-1 loss. From this formula- tion, we identify two key challenges: (CI) Training a sin- gle prompt to balance the three components is inherently conflicting, as each component requires distinet inputs and targets on different sub-objectives. This can lead to mutual interference during optimization. (C2) New class samples are unavailable during training. Next, we elaborate on how to address these challenges.
Ep 1l6b - g(()))
1(((x)) > r(En))
1yyn = h(wx))
Base (P2)
Base-to-New (PI)
New (P2)
Benefiting from this concise formulation, one can easily find that OpenworldAUC satisfies (P1) and (P2). Further- more, as shown in 2 OpenworldAUC is insensitive to the domain distribution due to the ranking formulation, ful- filling (P3). To further highlight its advantages over existing metrics, we provide the following proposition, with proof in App.A.4.
Gated Mixture-of- Prrooppss. To address (C1), we pro- pose a novel Pexturre-of-rromtt approach with three spe- cialized prompts P(: 0.), P(:: 0g). P(: 0%) targeting for r: g, h, respectively. Among these, optimizing 0, is more challenging because the third term in the objective involves non-differentiable sample-selection operation Ub, Un. To tackle this issue, we use smooth and differentiable functions (b == (s (an;; 0y)) and yr, == (88% (x,,; 0x)) to approxi- mate the selection process, where a--) denotes the sigmoid function: si and s are the ground-trutt channels of sa and 8n. respectively. Here, the sigmoid function acts as a gate since correctly classified samples tend to have higher confidence scores in the ground-truth channel. This gate adaptively assigns the weights to each base-new pair based on the outputs of g, h to optimize r. encouraging correctly classified samples to be ranked correctly. Following this ap- proach, fgy On, 0, are merged through the gate mechanism into mixture 0mixi
Proposition 4.3. For any (r. h), we have:
Assume that OpenworldAUC equals u, a lower bound proportional to u is available for HitRaten for any domain distributions.
Constructing (F. g. h) as outlined in Prop.4.1 yields OpenworidAUC(f. ğ. h) < OpenworldAUC((: g, h).
These properties show that: 1) Compared with OverallAcc, optimizing OpenworldAUC can guarantee the model perfor- mance on the new domain under arbitrary base/new ratio. 2) OpenworldAUC is more consistent with the model perfor- mance than the naive aggregation of multiple metrics. These advantages encourage us to design an effective method to optimize OpenworldAUC in the next section.
Ma Dni 0mix) = pb Eng (r(ss; 0.) r(z,, 0.)) iP
5. The Optimization of OpenworldAUC
where mix (0,, 0g. 6x) represents all learnable tokens. Besides, following the surrogate loss framework (Gao & Zhou, 2015; Liu et al., 2020; Yang et al., 2022; 2023; Bao et al., 2022), we replace the non-differentiabbe 0-1 loss with a convex upper bound E(t). such as (tg = (1 - t)² for scores in [0, 1]. This smooth approximation enables gradient- optimization while preserving ranking se- mantics. Hence, our goal is then to solve the following problem:
Next, we explore how to optimize OpenworldAUC effec- tively, whose overall pipeline is shown in 1
5.1. Gated Mixture-of-Prompp for OpenworIdAUC Optimization
We start by reformulating OpenworldAUC maximization to an equivalent minimization problem via the following proposition, with proof provided in A.p.A.5. Proposition 5.1. Maximizing the OpenworIdAUC metric can be achieved by solving the following problem:
(OPP) min R(r, g. h) = Es, [fee (so( 0g), 3/)] Omi
+ Ess, (s 0n), vn Es [e(x,,, xx e mix)]
min ED [1 [yn * g(x)))] + 1 18m, # h((x)))
where we use the CE loss lar for (g, ht) and replace the non- differentiable lo,1 with the squared loss (tg(1) = (1 2)².
..
+ 1 1|m = g(x))) (x( < r(w,)) 1[y,, = hxx,))]]
Pseudo Nase-to-New Partition. To handle (C2). we fol- low the prior arts (Zhou et al., 2024) to perform pscudo base-to-new partition y (D). Di)) over the known class
Since the distribution D is unknown, we resort to minimiz- ing the empirical risk to tune the prompt P(: 0),, which is
5

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
S.. Specifically, Vi and Dn are pseudo base and new do- main which satisfy 1b U Vi = = and 5% n Dr 0. To reduce the partition bias, we estimate the expectation of the partition distribution by generating K pseudo partitions
Kabzo ICP 0.6
HilRaten
HitRate.
0.
57(4), k E [1, K], satisfying U )/(+) -%b Accordingly, we assign each partition with sub-detector r(*) 0r.k)) to
00
avoid conflicts. During testing, the highest score across all sub-detectors will be used as the final score r(z; 0,))
Rite
MissRate,
(a) SUN397
(b) Flowers102
Zero-shot New Domain Classifier. Last but not least, we observe that training new domain classifier on the base domain can lead to overfitting, reducing new class perfor- mance (Zhou et al., 2022b). To balance accuracy and effi- ciency, use a zero-shot CLIP model with fixed hand- crafted prompt 0 for the new domain classifier h.
Figure 3. The MissRate, curve SUN399 and Flow- ers102. Our method can outperform other competitors on the meaningful region with lower MissRate, and higher HitRate,.
with the empirical average from S, which can be minimized with a moderate N₂ due to the small parameter space of the prompt. Second, the error from finite pseudo base-to-nent partitions during detector optimization, which is reduced by increasing partition number K. Third, the error from a po- tential shift from the pscudo new domain to the underlying real new domain.
Overall, we have the following final objective, where S(K) denotes the dataset partition defined on X' x()(k) and miix (0,,1.-. dr 0g. 0%)
(m)))) min R' (r.gg. h) Es, (oo (56(26: Vb). 8'
+
È
SiN) (((,, ni mmix
6. Experiments
K
In this section, we describe details of the experiments and present our results. Due to space limitations, please refer to App.C and App. for more results about experiments.
5.2. Generalization Bound
In this part, we explore how well the proposed method can generalize to test data theoretically. Specifically, let
R(r, h) be the population risk of R(r, g, h) in (OPr). Our task is to bound the difference between R(r, g, h) and (r g. h). To this end, we present the following informal theorem, whose detailed proof is presented in App.B.
6.1. Experimental Setup
Task and Dataset Deseription. We evaluate three open- world tasks: open-world recognition, open-world do- main generalization and cross-dataset generalization. For recognition, models are trained on base classes and tested on the mixture of base and new classes within each dataset, with test sets resampled at varying base/new ratios to study im- balance. To examine domain-level imbalance, we resample the test sets with varying base/new ratios, which is different from class-level imbalance in long-tail settings (Wang et al., 2024; Zhao et al., 2024a;b; Wang et al., 2023). For domain generalization, models are trained on base classes of Ima- geNet and are tested on the full classes of ImageNet variants with domain shifts. For the practical cross-dataset setting, the prompts are tunned on the base domain in ImageNet dataset and tested on other full datasets. Experiments use 11 image recognition benchmarks, 4 ImageNet variants, and 10
Theorem 5.2. Let & and E' be the distributions over the expectation of y and D, respectively. Given the function space Hr and Hy. the following inequality holds for all r E Hr, S6 E Hy with high probability:
R(r,g,h) h) S R'(r,g,)) M) ED, (sn yn)
(1)
(2)
+ C. -.K 1/2 + No 1/2 + |E Eloo
(Co + C.
(3)
where C denotes the complexity of the hypothesis space; < denotes the asymptotic notation that omits undominated terms.
datasets (1:5 ratio) built via random sampling. Details are provided in App.C.2.
Theoretical insights. The bound in Thm.5.2 consists of the following terms: (1) Empirical Error: The empirical loss on the training set, optimized in (OPt.). (2) Error for New Classes: The expected error of zero-shot classifier h
Competitors. To demonstrate the effectiveness of our pro- posed method, we compare our method with 10 competi- tive SOTA competitors: a) Baseline, Zero-shot CLIP (Rad- ford et al., 2021); b) HM-oriented methods, including CoOp (Zhou et al., 2022b), Maple (Khattak et al., 2023a), KgCoOp (Yao et al., 2023), PromptSRC (Khattak et al., 2023b), DePT-Kg (Zhang et al., 2024) and TCP (Yao
in the new domain, which can be reduced with carefully- designed prompts or more powerful foundation models. (3) Stochastic Error including three parts: First, the data esti- mation error from approximating the training distribution D
6

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 2. The OpenworIdAUC empirical results on eleven benchmark open world prompt tuning recognition.
Method
IN
C101
Pets
Cars
F102
Food
AC
SUN
DTD
ES
UCF
Avg.
CLIP
47.31
82.31
76.17
43.43
48.51
75.09
7.23
42.52
25.22
28,01
50.37
47.83
CoOp 48.93
83.29
80.71
35.38
59.65
67.27
5.60
48.03
25.48
41.96
43,95
47.59
Maple
51.89
87.15
85.59
49.99
65.44
76.83
9.58
52.84
36.22
56.55
58.72
57.35
PromptSRC
52,44
86.74
86.10
50.90
69.36
77.33
11.40
54.19
40.30
52.56
58.94
58.21
LoCoOp
45.12
86.59
86.62
46.52
61.17
73.09
8.67
50.55
32.26
41.35
46.90
52.62
KgCoOp
51.45
86.63
86.80
49.40
62.96
76.96
8.18
52.09
34.87
39.16
57.29
55.07
DePT-Kg
51.35
92.74
87.81
55.24
69.46
79.45
12.71
56.42
37.56
44.90
61.38
59.00
Gallop
49.01
87.51
86.94
50.69
65.69
73.60
11,38
50.62
40.22
51.38
58,91
56.90
DeCoOp
51.98
92.72
88.72
53.59
70.28
80.67
8.17
57.00
37.07
46.66
59.57
58.77
TCP
51.34
88.65
85,50
53.18
69.20
77.27
10.72
54.86
37.92
55.89
63.39
58.90
Ours
52.64
92.81
89.77
55.31
72.79
81.25
11,42
58.54
40.37
53.09
62.39
60.94
Table 3. The OpenworidAUC empirical results on ten imbalance benchmark with different domain distributions. Fwd means base domain number is 5x larger than new domain, and Bwd means the opposite.
DTD
Food101
Flowers102
OxfordPets
SUN397
Method
Avg.
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
Fwd
Bwd
CLIP
25.71
25.74
76.11
75.76
48.97
45.37
82.82
80.30
42.39
42.68
54.59
CoOp
24.98
25.35
67.22
67.14
52.64
51.03
77.53
77.41
48.25
48.16
53.97
MaPLe
33.86
34.15
76.85
76.85
65.98
64.95
85.67
85.88
53.36
52.94
63.05
PromptSRC
40.51
39.40
77.51
77.25
70.08
69.44
85.47
86.10
54.43
$4.16
65.44
LoCoOp
32.34
31.62
73.23
72.96
61.25
60.67
86.29
86.32
51.09
50.68
60.65
KgCoOp
34.75
34,95
77.03
76.80
63.28
62.05
86.74
87.01
52.47
52.05
62.71
DePT-Kg
37.32
37.06
79.71
79.28
69.24
69.41
87,52
87.85
56.76
56.45
66.06
Gallop
41.25
40.91
73.61
73.88
66.22
65.86
86.65
86.30
51.14
50.89
63.67
DeCoOp
36.74
36.61
80.65
81.11
70.09
69.89
89.76
89.10
57.37
57.37
66.87
TCP
38.18
37.75
77.42
77.18
70.08
69.34
85.63
85.58
55.09
55.04
65.13
Ours
41.13
40.97
81.65
81.09
73.50
73.10
90.36
89.79
58.59
58,26
68.84
al., 2024); c) OOD-oriented methods, including Lo-
Table 4. The OpenworIdAUC empirical results on Image vari- ants benchmark for open world domain generalization task.
CoOp (Miyai et al., 2023) and Gallop (Lafon et al., 2024); d) OverallAcc-ooreetee algorithms, DeCoOp (Zhou et al., 2024). Detailed descriptions of each competitor are pro- vided in App.C.3
Source ImageNet
Target
Method
V2
S
R
A
CLIP
47.31
39.49
22.57
57.04
23.83
Implementation details. We adopt 16-shot prompt tuning setting following previous studies (Zhou al., 2022b; Yao al., 2024; Khattak et al., 2023b). To ensure fairness, we reimplement all competitor models our device using their open-source code. Results are reported as the average over 5 runs with different random seeds 1, 2, 3, 4, 5. The default backbone used is the publicly available VIT-B/16 of CLIP. For all competitors except DeCoOp, we use the maximum probability among the base domain as the base-to-new de- tection score The computation of the OpenworldAUC metric and the implementation of the corresponding AUC loss functions are based on the open-source library open- source library XCurve. More implementation details are deferred in the A.p.C.4 and D.3.
CoOp
48.93
39.88
21.69
56.58
24.22
Maple
51.89
42.44
23.79
59.71
25.10
PromptSRC
52.44
42.55
24.50
60.65
25.08
LoCoOp
45.12
38.45
20.88
57.31
22.98
KgCoOp
51.45
42.14
24.24
59.75
25.79
DePT-Kg
51.35
42.45
23.97
60.60
25.84
Gallop
49.01
39.86
21.57
56.78
22.28
DeCoOp
51.98
43.01
24.65
61.01
25.31
TCP
51.34
41.66
23.15
58.46
24.62
Ours
52.64
43.98
25.64
62.67
26.49
for the open-world recognition task. Our method out- performs existing approaches most datasets, achiev- ing average 2% improvement in OpenworidAUC with smaller parameter cost, shown in Fig.5. This highlights the effectiveness of the proposed optimization methods. To visualize the OpenworldAUC metric, we also plot the HaeRRat,,-HiRatee. curve shown in Fig.3, which demon-
6.2. Performance Comparison
Openworld Recognition. Tab. 2, Tab. 8, Tab. 9 com- pare the overall performance across eleven benchmarks
7

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
LoCoOp
DeCoOp
DeCodp
LoCoOp
DeOsOp
OpenworicAUG Sa
DeCoOpi
Ours
Dei
TGP
Map
AUCROC
AUCROC
Gullep
TCP
AUCROC
DePFikg
PromeptRC
Dullop
KgCoop
PrompesRic
PromuisR
Naple
LoCoOp
Maple
DePTiig
92
Gallop
KgCeOp
CoOp
KgCoOp
Dallop
KgCoOp
CoOp
TCP
Mapte
Go0p
78.0
77.5
HM
HM
HM
logt0(4param)
(a) Average in Tab.2
(b) 10owers102
(c) ImageNetR
(a) Average in Tab.2
Figure 4. Trade-off between the first-stage AUROC and the second-stag stage HM metrics. Our approach, located the upper right corner, shows better trade-off between strst detection and second- stage classification performance.
Figure 5. Tradeoff between formance and prompt complex- ity across different methods.
DeCoOp
93.0
OpenwortdAus
Ours
TCP
OpenworidAus
OpenworddAUC
K-
$2.5
$1
73.5
OpenworIdAUC
Shots
Lambda
Single W/o GMoP
(a) Effect of K
(b) Effect Shots
(c) Sensitivity Analysis of (d) Ablation study of Prompts
Figure 6. Sensitivity Analysis and Ablation Study. (a), (b), (c) and (d) performed Flowers 102, DTD, Caltech101 and ImageNet, respectively. More results are provided in App.D.7. A.8. A.p.D.10 10 and App.D.11.
strates that our optimization method can significantly outper- form other competitors on the region with MissRate, S a and HitRate, 2 3. This area is particularly meaningful, as it represents the task's 's key concern: skillful model should balance high HitRate with low MissRate, as emphasized in recent AUC literature (Yang et al., 2023).
robust towards the varying domain distribution, consistent with (P3).
Openworld Domain Generalization Tab, 4 compares the robustness performance of methods trained on ImageNet to various domain- shift datasets. We observe that our method consistently improves against all competitors in terms of OpenworIdAUC. Fig.4 (c), Tab. 10 and Fig.8 further demon- strate that our method can also achieve better performance trade-offs for stage-wise metrics this more generalizable task. Our fine-tuning strategy can further enhance the trans- ferability of the model the domain-shift scenarios, which is crucial for real-world applications.
Tradeoff between the AUROC and the HM metrics. As shown in Fig.4 (a)-(b) and Fig.7, methods targeting the HM metric often lack base-to-new detection abilities (P1), while OOD-oriented methods typically compromise classification performance (P2). These limitations align with the con- straints of the metrics, as discussed in Sec.3. Compared with optimization
these methods, our OpenworldAUC-orienne method can achieve better trade-off between the AUROC and HM, which further validates the comprehensivenees of the OpenworldAUC metric and the efficiency of our opti- mization method. This consistent with (P1) and (P2).
Cross Datss-Dtaset Generalization Tab. 13 compares the per- formance our method with existing competitors on the cross-dataset generalization task. The comprehensive re- sults of this cross-domain evaluation test the model's ability to handle both base and new categories across diverse visual domains, validating the robustness of our method.
Imbalance Setting for Recognition. Tab. 3 compares the overall OpenworldAUC performance across five datasets with different domain imbalance ratios. OpenworldAUC as a comprehensive and dissributionninsensitte metric shown in D.2, can effectively capture the performance of the model in the imbalance setting. Our method consistently outperforms existing approaches across all datasets, achiev- ing an average 2% improvement. This demonstrates that our OpenworldAUC-orrinnee optimization method is also
6.3. Ablation Study
Sensitivity Analysis of K. As shown in Fig.6 (a) and
10g. the performance monotonically increases with more base-to- new partitions (K) on Flowers 102 and SUN397, consistent with Thm. 5.2. In practice we set 3 for dffiieenc--eeformmnn balance.
8

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Effect of Different Shots. shown in Fig.6 (b) and Fig. I,, our method surpasses representive competitors across all shots (1/2/4/8/16) on DTD and OxfordPets, further demon- strating its effectiveness. Additionally, OpenworldAAUC im- proves as N increases, aligning with Thm. 5.2.
distillation from larger models, while (Zhang al., 2023; Khattak al., 2024) use LLMs to generate prompts with richer semantics.
While most methods focus on improving in-domain clas- sification, they overlook inter-domain misclassification risks-such as confusing base and new domain sam- hes-which is critical for open-world scenarios. This has led to growing interest in prompt tuning for OOD detec- tion. LoCoOp (Miyai et al., 2023) treats non-semantic re- gions (e.g., background) as new-domain signals and adds scoring constraints to the CoOp objective. Follow-up meth- ods introduce hierarchical (Lafon et al., 2024) and negative prompts (Zeng et al., 2024; Nie et al., 2024) to better balance accuracy and robustness.
Sensitivity Analysis of X. In the practical objective (mP) the second term serves as an AUC-style rank- ing loss. However, recent studies ( (Yuan et al., 2022; Yang et al., 2023; Han et al., 2024) show that optimizing AUC loss from scratch can harm feature representations. The cross-entropy regularization term À fee improves ranking performance while mitigating this issue, as illustrated in Fig.6(c) and Fig.12. Optimal results are usually achieved when Àe E 1..
Effect of Mixture-of- of-Proopts 6 (d) and Fig. show the effectivvness of the mixture-of prrmmppp strategy. This val- idates the challenge of optimizing OpenworldAÜC within a single prompt as discussed in Sec.5.1.
Recently, DeCoOp (Zhou et al., 2024) proposed the first open-world prompt tuning framework that jointly addresses OOD detection and balanced classification across domains. It combines an OOD detector for base-to-new separation with domain-specific classifiers to improve within-domain discriminability.
Ablation Study of the Gating Mechanism. We evaluate the gating mechanism in Tab. 17. Using a fixed 0-1 mask ("Ours 0-1 Gate") slightly outperforms removing the gate ("Ours w/o Gate") but underperforms the adaptive sigmoid gate, confirming the effectiivnesss of both sparse sample selection and gate approximation.
7.2. Opens-Set Recognition
A closely related topic to this paper is Open-Set Recognition (OSR). OSR a challenging and practical setting, where the model is required to detect open-set samples which do not come from the training and also correctly classify the close-set samples (Yue et al., 2021; Feng et al., 2022). Com- pared to the open-world learning discussed in this paper, OSR does not require precise classification of open-set sam- ples. In this direction, a variety of novel metrics (Yue et al., 2021; Kong & Ramanan, 2025; Scherreik & Rigling, 2016; Wang et al., 2022) have been proposed to evaluate OSR models, focusing on close-set classification and open-set OOD detection. However, these metrics have limitations when applied to our open-world setting because they cannot assess the classification performance of open-set samples.
7. Related Work
In this section, we briefly review the closely related work on few-shot prompt tuning and Open-world Learning.
7.1. Few-shot Prompt Tuning for VLM
CoOp (Zhou et al., 2022b) first introduced prompt tuning to adapt VLMs for downstream tasks, significantly improv- ing classification on base classes but showing poor gener- alization to new classes. To address this, early methods proposed more flexible prompt structures. For example, CoCoOp (Zhou et al., 2022a) generates input-conditional to- kens using lightweight networks, enabling dynamic prompts per instance. MaPLe (Khattak et al., 2023a) adopts a multi- modal joint learning strategy (Jiang et al.. 2025; Xu al., 2025). Another line of work (Yao et al., 2023; Bulat & Tz- imiropoulos, 2023; Zhu et al., 2023; Khattak et al., 2023b; Roy & Etemad, 2024: Zhang et al., 2024; Hua et al., 2024; Yao et al., 2024) enhances generalization by introducing task-agnostic objectives and preserving the zero-shot knowl- edge of VLMs. KgCoOp (Yao et al., 2023) and TCP (Yao et al., 2024) reduce the gap between learned and hand- crafted prompt features. Recent advances (Li et al., 2024; Mistretta et al., 2024; Wu et al., 2024; Zhang et al., 2023;
8. Conclusion
This paper explores Open-world Prompt Tuning for Vision- Language Models, involving base-to-new detection and domain-specifer classification. We argue that ideal metrics should consistently evaluate both stages and remain robust to domain distributions, yet existing metrics fall short. To bridge this gap, we propose OpenworIdAUC, a unified met- ric that simultaneously assesses detection and classification through pairwise sample comparison, thus being insensi- tive towards the varying domain distributions. In pursuit of this, our Gated Mixture-of-Prompe is proposed with theoretical guarantee where each prompt fulfills its specific responsibility to jointly maximize OpenworIdAUC. Exten- sive experiments speak to the effectiveness of our method.
Khattak et al., 2024) further boost performance by lever- aging external knowledge. For instance,(Li (Li et al., 2024;
Mistretta et al., 2024; Wu et al., 2024) apply knowledge
9

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Impact Statement
Fei-Fei, L., Fergus, R., and Perona, P. Learning generative visual models from few training examples: An incremen- tal bayesian approach tested 101 object categories. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 178-178, 2004.
We propose general evaluation metric for openworld recog- nition deal with the potential bias toward new domain samples. For fairness-sensitive real scenarios, might be helpful to improve fairess for groups with fewer occur- rences.
Feng, Y., Gao, Y., Zhao, X., Guo, Y., Bagewadi, N., Bui, N.-T., Dao, H., Gangisetty, S., Guan, R., Han, X., Hua, C., Hunakunti, C., Jiang, Y., Jiao, S., Ke, Y., Kuang, L.,, Liu, A., Nguyen, D.-H., Nguyen, H.-D., Nie, W., Pham, B.-D., Raikar, K., Tang, Q., Tran, M.-T., Wan, J., Yan, C., You, H., and Zhu, D. Shrec'22 22 track: Open-set 3d object retrieval. Computers and Graphics, 107:231-240, 2022.
Acknowledgements
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102000, in part by National Natural Science Foundation of China: 62236008, 62441232, U21B2038, U23B2051 62122075, 62206264 and 92370102, in part by Youth Innovation Pro- motion Association CAS, in part by the Strategic Priority Research Program of the Chinese Academy of Sciences, Grant 10,X0006,, in part by the Fundamental Re- search Funds for the Central Universities, in part by the China National Postdoctoral Program for Innovative Talents under Grant BX20240384, and in part by the Postdoctoral Fellowship Program of CPSF under Grant GZB20240729.
Gao, W. and Zhou, Z. On the consistency of auc pairwise op- timization. In International Joint Conference on Artificial Intelligence, pp. 939-945, 2015.
Han, B., Xu, Q. Yang, Z.,, Bao, S., Wen, P., Jiang, Y., and Huang, Q. Aucseg: Auc-oriented pixel-level long-tail semantic segmentation. In Annual Conference on Neural Information Processing Systems, pp. 126863-126907,, 2024.
References
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote. Sens., 12(7):2217-2226, 2019.
Bao, S., Xu, Q. Yang, Z., He, Y., Cao, X., and Huang, Q. collabora-
The minority matters: A diversity-promont tive metric learning algorithm. In Annual Conference on Neural Information Processing Systems, 2022.
Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and dustr--dit examples in neural networks. In International Conference on Learning Rep- resentations, pp. 1-12, 2017.
Bao, S., Xu, Q. Yang, Z., He, Y., Cao, X., and Huang, Q. Aucpro: Auc-oriented provable robustness learn- ing. IEEE Trans. Pattern Anal. Mach. Intell., 47(6):5779- 4596, 2025.
Bossard, L., Guillaumin, M., and Gool, L. V. Food- 101 mining discriminative components with random forests. In European Conference on Computer Vision, pp. 446- 461, 2014.
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F,, Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces robustness: A critical analysis of out-of-disttibuttin generalization. In IEEE/CVF International Conference on Computer Vision, pp. 8320-8322 8029, 202
Bulat, A. and Tzimiropoulos, G. Lasp: Text-to-text op- timization for language-aware soft prompting of vision language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23232-222222 -2224,3 2023.
Hendrycks, D.,, Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In IEEE/CV Conference on Computer Vision and Puttern Recognition, pp. 3606-3613, 2014.
Hua, C., Xu, Q., Bao, S., Yang, Z., and Huang, Q. Re- conboost: Boosting can achieve modality reconcilement. International Conference on Machine Learning, pp. 19573-19555 2024.
Deng, J., Dong, W., Socher, R., L. K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 248-255, 2009.
Jiang, Y., Hua, C., Feng, Y., and Gao, Y. Hierarchical set- to-set representation for 3-d cross-modal retrieval. IEFE Trans. Neural Networks Learn. Syst., 36(1):1302-1314, 2025.
Fawcett, T. An introduction to ROC analysis. Pattern Recognit. Lett. 27(8):661-8744 2006.
10

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Khattak, M. U., Rasheed, H. A., Maaz, M., Khan, S. H., and Khan, F. S. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 191 19112211 2023a.
Nie, J., Zhang, Y., Fang, Z., Liu, T., Han, B., and Tian, X. Out-of-distribution detection with negative prompts. In International Conference on Learning Representations, pp. 1-20, 2024.
Khattak, M. U., Wasim, T., Naseer, M., Khan, S., Yang, M.-H., and Khan, F. S. Self-regulating prompts: Founda- tional model adaptation without forgetting. In IEEE/CVF International Conference on Computer Vision, pp. 15190- 15200, 2023b.
Nilsback, M.-E. and Zisserman, A. Automated flower clas- sification over a large number of classes. In Indian Con- ference on Computer Vision, Graphics and Image Pro- cessing, pp. 722-729, 2008.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C. V. Cats and dogs. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3498-3505, 2012.
Khattak, M. U., Naeem, M. F., Naseer, M., Gool, L. V., and Tombari, F. Learning to prompt with text only supervi- sion for vision-language models. CoRR, abs/2401.022 102418, 2024.
Paszke, A., Gross, S., Chintala, S., Chanan, G., Yang. E., Vitoo,, Z., Lin, Z., Desmaison, A., Antiga, L., and Lerer, A. Automatic differentiation in pytorch. In Annual Con- ference on Neural Information Processing Systems Work- shop, 2017.
Kong, S. and Ramanan, D. Opengan: Open-set recognition via open data generation. IEEE Trans. Pattern Anal. Mach. Intell., (7(5) 32443 2025.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748- 8763, 2021.
Krause, J., Stark, M., Deng. J., and Fei-Fei, I., 3d object rep- resentations for fine-grained categorization. In IEEE/CVF International Conference on Computer Vision Workshop, pp. 554-561, 2013.
Lafon, M., Ramzi, E., Rambour, C., Audebert, N., and Thome, N. Gallop: Learning global and local prompts for vision-language models. In European Conference Computer Vision, pp. 264-282, 2024.
Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do ImageNet classifiers generalize to ImageNet? In Interna- tional Conference on Machine Learning, pp. p38...9--5.... 5389-5400, 2019.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for pawameter-effiieeeeeee efficcien prompt tuning. In Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, 3059, 2021.
Roy, S. and Etemad, A. Consistency-guided prompt learning for vision-language models. In International Conference on Learning Representations, pp. 1-14, 2024.
Li, Z., X., Fu, X., Zhang, X., Wang, W., Chen, S., and Yang. J. Promptkd: Unsupervised prompt distillation for vision-language models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26617- 26626, 2024.
Scherreik, M. D. and Rigling, B. D. Open set recognition for automatic target classification with rejection. IEEE Trans. Aerosp. Electron. Syst. 52(2):632-642, 2016.
Soomro, K., Zamir, A. R., and Shah, M. UCFI01: A dataset of 101 human actions classes from videos in the wild. CoRR, abs/1212.0002, 2012.
Liu, M., Yuan, Z., Ying, Y., and Yang. T. Stochastic auc maximization with deep neural networks. In International Conference on Learning Representations, pp. 1-24, 2020.
Wang. H., Ge, S., Lipton, Z. C., and Xing, E. P. Learning robust global representations by penalizing local predic- tive power. In Annual Conference on Neural Information 2019.
Maji, S., Rahtu, E., Kannala, J., Blaschko, M. B., and Vedaldi, A. Fine-grand treingg visual classification of aircraft. CoRR, abs/1306.511 306.5151,, 2013.
Processing Systems, pp.
Wang, P., Zhao, Z., Wen, H., Wang, F., Wang, B., Zhang, Q.. and Wang, Y. Llm-autoda: Large language model-drive automatic data augmentation for long-tailed problems. In Annual Conference on Neural Information Processing Systems, pp. 64915-64941, 2024.
Mistretta, M., Baldrati, A., Bertini, M., and Bagdanov, A. D. Improving zero generalization of learned prompts via unsupervised knowledge distillation. In European Conference on Computer Vision, pp. 1599447..1 2024.
Miyai, A., Yu, Q., Irie, G., and Aizawa, K. Locoop: Few- detection via prompt learning. In
Wang. Z., Xu, Q., Yang, Z., He, Y., Cao, X., and Huang, Q. Openauc: Towards auc-oriented open-set recognition. In Annual Conference on Neural Information Processing Systems, pp. 25033-25045, 2022.
shot nut-of--istribbu Annual Conference on Neural Information Processing Systems, pp. 0-290, 2023.

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-worid Prompt Tuning
Wang, Z., Xu, Q. Yang, Z., He, Y., Cao, X., and Huang, Q- A unified generalization analysis of re-weighting and logit-adjustment for imbalanced learning. In Annual Conference on Neural Information Processing Systems, 2023.
Yue, Z., Wang. T., Zhang, H., Sun, Q., and Hua, X.-S. Coun- terfactual zero-shot and open-set visual recognition. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15404-15414, 2021.
Zeng, F., Cheng, Z., Zhu, F., and Zhang, X. Enhancing out- lier knowledge for few-shot out-of-distribution detection with extensible local prompts. CoRR, 0as/2409.04796 2024.
Wu, G., Zhang, X., Li, Z.., Chen, Z.,, Liang, J., Yang, J., and Li, X. Cascade prompt learning for vision-language model adaptation. In European Conference on Computer Vision, pp. 304-321, 2024.
Zhang, J., Wu, S., Gao, L., Shen, H. T., and Song, Dept: Decoupled prompt tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12924- 12933, 2024.
Xian, Y., Schiele, B., and Akata, Z. Zero-shot learning- the good, the bad and the ugly. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3077- 3086, 2017.
Zhang, R., Hu, X., Li, B., Huang, S., Deng, H., Qiao, Y., Gao, P., and Li, H. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1521 1-15222, 2023,
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. Sun database: Large-scale scene recognition from abbey to 700. In IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp.
2010.
Xu, Y., Li, X., Yang, Y., Meng, D., Huang, R., and Wang, L. Carebench: A fine-grained benchmark for video caption- ing and retrieval, 2025.
Zhao, Z., Wang, P., Wen, H., Xu, W., Song, L., Zhang, Q.. and Wang, Y. Two fists, heart: Multi-objectinn op- timization based strategy fusion for long-tailed learning. In International Conference on Machine Learning, pp. 61071, 2024a.
Yang, and Ying, Y. AUC maximization the era of big data and AI: survey, ACM Comput. Surv, 55(8): 172:1 2023.
Zhao, Z., Wen, H., Wang, Z., Wang, P., Wang, F,, Lai, S., Zhang, Q. and Wang, Y. Breaking long-tailed learning bottlenecks: A controllable paradigm with hypernetwork- generated diverse experts. In Annual Conference on Neural Information Processing Systems, pp. 7493-7520, 2024b.
Yang, Z., Xu, Q., Bao, S., Cao, X., and Huang, Q. Learning with multiclass AUC: theory and algorithms. IEEE Trans. Pattern Anal. Mach. Intell., 44(11):7747-776,, 2022.
Yang, Z., Xu, Q.. Bao, S., He, Y., Cao, X., and Huang, Q. Optimizing two-way partial AUC with an end-to-end framework. IEEE Trans. Pattern Anal. Mach. Intell., 45 (8):10228 10246, 2023.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Condi- tional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16795-16804, 2022a.
Yang, Z., Xu, Q.. Wang, Z., Li, S., Han, B., Bao, S., Cao, X., and Huang, Q. Harnessing hierarchical label distri- bution variations in test agnostic long-tail recognition. In International Conference on Machine Learning, pp- 56624 566644, 2024.
Zhou, K., Yang, J., Loy, C. C., and Liu, Z. Learning to prompt for vision-language models. Int. J. Compul. Vis., 130(9) 2022b.
Zhou, Z., Yang, M., Shi. J.-X., Guo, L.-Z., and Li, Y.-F. Decoop: Robust prompt tuning with out-of -fi--ri-ittiitiionn detection. In International Conference on Machine Learn- ing, pp. 62161-62177 2024.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with knowledge-guiie IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0757-6767, 2023.
context optimization. In
Zhu, B., NIu, Y., Han, Y., Wu, Y., and Zhang, H. Prompt- aligned gradient for prompt tuning. In IEEE/CVF Interna- tional Conference on Computer Vision, pp. 15613-15623, 2023.
Yao, H., Zhang, R., and Xu, C. Tcp: Textual-based class- aware prompt tuning for visual-language model. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 23438-23448. 2024.
Yuan, Z. Guo, Z., Chawla, N., and Yang, T. Compositional training for end-to-end deep AUC maximization. In In- ternational Conference on Learning Representations, pp. 1-26, 2022.
12

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Contents
A. Proof for the Proposition
14
A.I. Proof for Proposition 3.I.
14
A.2. Proof for Proposition 4.1.
14
A.3. Proof for Proposition 4.2.
15
A.4. Proof for Proposition 4.3...
16
A.5. Proof for Proposition 5.1
18
B. Proof for the Generalization Bound
20
B.1. Proof Outline
20
B.2. Preliminary Lemma
23
B.3. Key Lemmas
24
B.4. Formal Proof
32
C. Additional Experimental Setup
34
C.I. Task Description
34
C.2. Datasets
34
C.3. Competitors
35
C.4. Implementation Details
36
C.5. Efficient Calculation OpenworldAUC
37
D. Additional Experimental Results
37
D.I. Additional Results for Openworld Recognition Task.
37
D.2. Additional Results for Openworld Domain Generalization Task.
40
D.3. Sensitivity Analysis of different metrics in varying domain distributions
40
D.4. Additional Results for Openworld Cross-dataset Generalization Task
D.5. Fine-grained analysis of OpenworIdAUC metric
41
D.6. Complexity analysis.
41
D.7. Effect of mutiple pseudo base-to-new Partitions...
43
D.8. Effect of Different Shots
43
D.9. Ablation Studies of the Gating Mechanism.
43
D.10. Effect of CE Regularization.
43
D.11. Ablation Studies of Mixture-of- Mixture-of-Promps
44
13

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
A. Proof for the Proposition
A.I. Proof for Proposition 3.1.
Restate of Proposition 3.1. Given dataset S sampled from the overall domain D, for any (g, h), one can always find a worse-performing (g. h) in OPT that satisfies HM(g. h) = HM(g.h) h).
Proof. Let 8, E R. and 8n E RC be the logits scores of the base and new domain produced by and h. Here, we only focus on differences in predictions between (g. h) and (g. h). Given one base sample (as. yb). (g. h) and (g. h) make the following predictions:
yb = g( maXy'ED% (x6) maxy' (1b)) >
yb = 5( maxy'Ess &n (m)) m 8
In this case, although 9 correctly classifies 2h, the joint model in OPT (ğ, h) nevertheless categorizes æ; into the new domain. This occurs because the class associated with the highest prediction probability for 24 resides in the new domain. Therefore, HM(g. h)- HM h) holds but (ğ. h) performs inferior to (g. h) in OPT setting.
In OPT task, the prediction could be regarded as correct only if the input has been identified into the correct domain. In other words, there exists an implicit detector defined as:
r ( x max (b(z) - max (x)
y'E
y'E
The OPT requires r(as) > O for base samples and r(æn) < 0 for new samples, From the definition of HM, one can easily find that HM lacks evaluations of this detector.
A.2. Proof for Proposition 4.1.
Restate of Proposition 4.1. Given a dataset sampled from the overall domain D, for any (rig, h) under mild assumption AUROC(r) 1, one can always find (F. g, h) performs worse in OPT that satisfies:
Lin(AUROC(r). (as)Acc(g), NewAcc(hi) = Lin(AUROC(F), BaseAcc(ā), NewAcc(h))
where Lin(-) denotes the linear combination function.
Proof. Similarly, only focus only differences in predictions between (r, h) and (F, h). Given two base-domain samples v.), (xī, v and two new-domain samples (Ir v.), (x vy),, suppose that (f. g. h) and (r, g, h) only make same predictions except the following cases:
.(xt). v and g(z)).0(a()) + vi
Mx*t), (a+h) # 2 and h (=x)), h(x2) v
r(x!) > r(x)) > r(x?)= > r(x))
F(zî) f(2{) > F(a)) > f(x+))
r(x)) f(xE), r(x))) = f(2.)) r( F(z)), r(x)) = =(xxh)
From this, observe that:
(a) BaseAcc(ğ) and NewAcc(h) = NewAcc(h) since (ğ. h) and (g, h) share the same predictions.
AUROC(r) AUROC(F)) since the ordering between base and new samples is also not changed.
14

---

OpenworidAUC:: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Consequently, have Lin Lin(AUROC(r)) BaseAc(g)) NewwAcc
Lin(AUROC(?), (isecc())) NewAcc(h)).
However, (f, g. h) performs worse than (r, g, ht) in the open-world prompt tuning task. Specifically,, if we select threshold t such that ((x))) > t > r(z)), then (r, ft) correctly predicts z and o while (i', g. h) misclassify all four samples. This shows that the linear aggregation of BaseAcc, NewAcc, and AUROC does not align with actual model performance.
A.3. Proof for Proposition 4.2.
Restate of Proposition 4.2. Given pair (æa; Ub) and (Ii, Vn) sampled from D, OpenworldAUC equals to the joint possibility that 1) ranks higher than In, and 2) g and h make correct predictions on Ey and In, respectively.
ED
1ym g(x))) Base (P2)
1 ((()))) > r(zn)) Un h(xx))
Base-to-New (PI)
New (P2)
Proof. We first review the definition MissRate, and HitRate, as:
MissRate, E
[1(())) 1(((x)) yo # g(x)))
(2bmt))
HitRate, :=
E (Zr J..
[1(((x))
S tt M(xx)]
Specifically, MissRate, measures the probability that (h,r) misclassifies the base samples and HitRate, is the probability that (g. r) makes correct predictions on the new samples.
MissRates (fr( I 2t] + P(r(s)) > t, vo + g(ø6)]
HitRate, P(r((,)) S t, yr h(xx)]
Thus, we can rewrite the OpenworldAUC as:
/
OpenworIdAUC: = J HitRate,, (t) dMissRates (t)
Pfr(sn) S 1n M(xx)) d((((((()))) S $4+ + P(r(1)) > 1,30 + g(()))}
P(r(In) 5 Un = h(xn)) d((((r(())) S +{+
P|r(xn) S tt = h(xr)] dP(r((s)) > 9, # g(xb))
(a)
For (a),
1=1
(a)
Plr(xn) S tt h(xx)] . dF(r(zs) S
= Ply,- = MxxM)) / PP(((N)) S fly = h(an)) dTP(r(as)) $
Plyn = h(x,)) (P(r(z0) > r(an))un = [(xxn)
P(r(x)) > r(an). Un M(xn))
For (6).
-
(b)
PP(((an) S t, h(xn))] dPpr(zs) > vo + g(x))]
1=0
Plyn M((r)))) PP(y # g(5h))
P(r(#n) fyy h(xx)) d(P(r(a))) fy9s * g(xr))
Flyn = k(xx)) ) Plys + ((x))
1 Pir(an) S t/yn = -(xx))] dPP(((a)) < flys + g(x)))
=-Ply, h(xw)) Ply * ((x)) PP(((1))) > r(wn)lys # g(xo). k(xx))
-PF((ax)) r(xx), + g(x)). h(*n)]
15

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning Thus,
OpenworIdAUC (a) + (6)
-(r((as) r(vn), yn M(x,)) P(((2))) r(arn), yb + g( yn h(x))) P(r(ws) > r r(un), - (((a), h(2n)]
IB
1/6 g(x6)] 1(r(2))) r(ann)
Un h(æn))
(26.86))-DD (ihn M8 Dn
Base
Base-to-New (P1)
New
A.4. Proof for Proposition 4.3.
Restate of Proposition 4.3. For any (r.g,h) hi), we have:
Assume that OpenworldAUC equals u, lower bound proportional is available for HitRate, for any domain distributions.
Constructing (f.g.h) as outlined in Prop.4.1 yields OpenworldAUC(F, g. h) OpenworidAUC(r..gth
Proof. We first demonstarte the first property that OpenworldAUC avoids domination by domains with larger sample sizes. To this end, we review that the OverallAcc is defined as:
EiErOOY TP: EiEroD. (TP, + FN,)
OverallAcc
P, TPR, + P,, TPR,,
Eier, TP: TPR, Σiex, + TPR,, Eier, (TP, + FN,)
Σiey. TP:
Nn No P, N
From this formulation, OverallAcc can be dominated by performance in the domain with more samples. To be specific, very low TPR,, with fewer new samples in prediction can still yield similar OverallAcc scores if TPR, compensates. In view of this, the OverallAcc metric is inconsistent with the actual performance of model in OPT setting. In contrast, our OpenworldAUC metric is ranking-based metric that is not affected by the sample size of each domain. Thus, this metric can effectively avoid the issue of domination by domains with larger sample sizes. To illustrate this, we show that the OpenworldAUC metric guarantees a lower bound for the HitRate of new domain.
According to the definition of OpenworldAUU, we have:
(EiE,, FN.) (ESev,
1 OpenworldAUUC
2
No Nn
where N₁ and N,, are the numbers of base samples and novel samples respectively. Then, given the definition of HitRate, and MissRates, we have:
(Eiey, FN.) No HitRate,
N2 ESEr. FN, Nn
MissRate,
Thus,
OpenworldAUC > MissRate, (1 - HitRate,,
We have:
OpenworldAUC MissRate,
HitRate, 2
16

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Therefore, when OpenworldAUC equals u and MissRate, a, we have HitRate, 2 1- under arbitrary base/new ratio. We conclude that proposed metric avoids the limitations highlighted in Prop. 4.1.
Escape from the limitations of Prop.4.1. This inconsistency arises because Lin(AUROC(r). BaseAcc(g). NewAcc(h)) evaluates performance in decoupling manner. In contrast, sample contributes to the OpenworidAUC score only when the detector correctly identifies it, and the domain classifier simultaneously classifies correctly.
To further examine this issue, we analyze the case where the Lin metric becomes inconsistent. We consider the following two cases. The difference between the two cases is marked in red.
Case 1: (P. and (r. It) make the predictions follows:
((rt) 2 and s(x)), + v²
h(rh), ((ah) + v and () h(x5)) + v
r(a)) r(a))) > r(aî) > r(2)))
F(t?) > f(a.)) > f(a)) > f(=+))
r(z)) F(a)), r(x)) = F(2.)) r(a)) = =(zz),, r(x2) = =(x%)
In this case, one can naturally conclude:
Lin(AUROC(r); BaseAcc(g), NewAcc(h)) = (AwN()())))
Next, we analyze this case our metric as following:
OpenworldAUC((,,o http://...nnnooooooooaoooc//
OpenworldAUC((f.g..
1lo(x)) wt] 1[r(xt)) r(x2)) 11h(x/)) v1l 1(]((t) vtl 1[F(x!)) F(x/)) 11(()) - vil
No . No
==1
>) [1((a))><<(d) 1[h(xz)) v - 1[r(x]) > r(at)) 1[h(x²)) =v'.
[[(((]))
No Nn
N,
[1(()) > r(x)) > r(aii]| [M(x)) = vill 2
No No
where No and Nn, denote the base samples and novel samples respectively.
The last equality holds only when there are no correctly classified novel samples between r(z() and r(æf). This condition no mild. Thus, we can conclude that OpenworldAUC(r, g, h) > OpenworldAUC(r, g, h) in this scenario.
Case 2: If (F, g. h) and (r,g. g. h) make the predictions as follows:
((xt)).(att tt and g ((())) + vē.
h(x)), (x + td and h(=:), h(x 24
r(a)) > r(a))) > r(a3)= > (2)))
F(z?) > f(x.) > f(a?)) > f(z))
r(a!) -(23), r(=))) F(x2), r(xi) F(x(), r(x2)) = F(x)))
The equality Lin(AUROC(r). BaseAcc NewAcc(h) (h)) Lin(BaseAcc(o), A still holds.
17

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Similarly, we analyze this case in our metric,
OpenworidAUC(r, g. h) OpenworidAUC(f..g.
No
[lo(x)) = vtl > r(2)]] 1[hxx))))))))))))))))))))))))1 = v2 1611-1066 1[(a)) = wt ([f(at) > f(())) 1((()))) = usll
N₁
N,
j=1
No
1[g(zz) vol > 1[b(x))) = - -17(a)) - vil 117(a))) > F( 11((02)) - vill
+ Nb . Nn
1(((x))) vbl 1()(x))) r( 11)(x²)) v
No. N
N.
11r(z)) r(12)) 11b(z/) = wijد - 1((xtt) F(x 1[1(x()) = vill
No
j=1
No.
[g((tt) vol (((z)))((()))) - 1[((TT) = vtl 1[((t)) > F(x2)]] No. N,
+ No
Nn
N. N,
[1(aa))> > r(=))))>(())) ( =(x2) 1[M(x)) vil]
No
No N, No
1lr(z)) vQll >()3) < <[13)]]
No N₂
t=1
1|((x)) > r(x,)) F(x)) < f( 1[((21)) = 2
No N₂
[[((z)) = u 1 > ((b)) < =((2))
1
+ No Nn
No N₂
No. N2
Finally, we have:
OpenworldAUC(r, g, h) - OpenworldAU((7.g.h))) 0 h
This further demonstrates that OpenworIdAUC with joint probability formulation is more consistent with the model performance than the naive aggregation of multiple metrics.
A.5. Proof for Proposition 5.1
Restate of Proposition 5.1. Maximizing the OpenworldAUC metric can be realized by solving the following minimization problem:
min v.g.h
Ep [y6 # g(xx)) [y # h(xx)) + 1 = g(x6)] 1[((xx)) S r(æn)) 1[ym = h(2,))]]
Proof. We first review the definition of OpenworIdAUC as:
Ep
[yo s(xra) [r(ar)) > r(xx))) 1 Jun h(an))
1.
In
According to the Truth Table in Tab.5, we have:
Ig I, In = -Iy + Ig -p In + I
Thus, maximizing Ig I, I, is equivalent to minimize -Ig + Iy II In + -I,- Therefore, the maximization of OpenworldAUC is equivalent to the following minimization problem:
min Ep [g/ * g(x6)) 1 + h(xx)) + 1ly = g(x0)] 1|r(xs) S r((w)) 1/yn k(x,))]] g.
18

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 5. Truth Table for the Ohjective Function.
Ig
I
Ip
1
Lg
1. lb
1
1 I,
In + -1n
1
1
0
0
1
0
1
0
1
1
0 1 1 1
O 1
1
I
1
I
0
0
1
This completes the proof.
19

---

OpenworidAUC:: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
B. Proof for the Generalization Bound
In this section, we present the complete proof of our theoretical results. Tab.6 summarizes the key notations used in proof. We begin with a proof sketch to give an intuitive overview. Next, we provide the key lemma and then present the detailed proof of the main results. Our primary theoretical results show how well the proposed method can generalize to test data.
B.1. Proof Outline
We first define the hypothesis space functions r and $b.
The Hypothesis Class In this paper, we consider the generalization ability base the prompt architecture. Our functions are chosen from the following hypothesis class H, and Hg.
H4 { m1)) (:: (r,k)) X |0. 1]. k E [1, K], Ar, represents learnable tokens in the prompt P(: 0r..)))
where 7:() (or; ar)) is the possibility that x belongs to the base domain. For the sake of simplicity, we denote the ensemble (x; (kEK) as r(z) and use r E H,, to express choosing one such collections out of H.,.
H ( 0,) :x' R 0g represents learnable tokens in the prompt P(: P(:0,))
Here sy))) abbreviates s5(2;; 0g) and use sy Hg to express choosing one out of the hypothesis class Hg.
The Norm of Hypothesis To measure the complexity of H, and Hy. define Here, we adopt the overall infinity norm (all classes and all input features X):
each hypothesis respectively.
|| sup |r(x)] =xx
|00ll :=
max JE(12, Ch
sup
1 s (x)
(x)
Here a1, (x:) is the logit score of the j-th channel for raw feature x.
In practical training, aim to solve the following OPhn problem:
1 (mPh)) mink'' (r, 9, h) = Es, (S6, ]b0] + K 1ps (rn(r((s)) r((x)) sp P.
where S(R) is sampled from the data distribution (R)) X ()()) under the k-tth base-to-new pseudo partition.
Theoretically, the generalization gap A should be measured by the expected risk on the joint distribution of overall label space y and the training data 2, expressed as:
R(r, g. h) = ED, (ss, ub]] + Eplys (rg()(())) r(xx))) spn] + ED,, (ee Un)]
Assume that the detection function base domain score function 8, are chosen from the hypothesis space Hy, H.g respectively. The generalization ability of the entire hypothesis set is often measured by the worst-case generalization gap A:
R(r.g.h) Rr, h.)]
sup 8. rEH.
However, this gap complex. Intuitively, it mainly comes from three parts:
A, 4 An.
where Ar, Ag. and D are the generalization gaps for the detector r, base-domain classifier g. and new-domain classifier h, respectively.
20

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 6. Some Important Notations Used in the Proof.
Notation
Description
Sigmoid score
vh (s)) (3) (xft))
The gate score of i-th base base-domain sample. The gate score of j-th new-domain sample.
p. = (s
(0) (xr)))
sup, ( (x:) E (ss, 8n The overall infinity norm on all input features X.
AUC-type Risks
(r (1r,2ti abr Loo
exg (r(p)) - r((2)))
(unoan) (en (r, 2b, an)1 The overall infinity norm all input features X.
È. (sn. Pb lag (r, Tn. m6) ypn Ng Σ Σ/n v (n((((1))) - r(x)))) s
Rgg(,g.h)
(sn-8a)-S..
The empirical loss training data under fixed pseudo partition )
(r,r,g,1)
E
(D Po log Rn, 2n) ipn. The expected loss under fixed pseudo partition }7.
Rich (r,
R (m ) 4Pb (ag (r, To) Pn- The empirical risk on training data under k-th pseudo partition ()(k)
(#n.8a) s!h))
Bio (r,
(w). ((t) Ph (rg (r, Pou. The expected on training data under k-th pseudo partition y)())
B5 [ig(r, 9, 1t]
Σx Rgas- The empirical average of Rgoa over mutiple partitions.
Ey (r, g, h)]
E Repco (r, h). The empirical average of Ryos (r, 9: h) over mutiple partitions.
Ey (r, g, h)
The expected Rp 9, h) over the meta distribution of pscudo partition distribution.
By RR(r, g. h)]
The expected (D(r, h)) over the meta distribution of real partition distribution.
Distribution
Generic definition
X
Input space.
y = Vb U D.
Overall label space.
V1.
Base label space.
Vn
New label space.
Da = X's x V1
Base domain data distribution.
S (1x), N
The base dataset sampled from D/ New domain data distribution.
D. A x D'n
S.
The new dataset sampled from D, Y' is the pseudo basc-to-new partition distribution.
9 Ji.
(00) (D(()) (9(0))
The k-thh base-to-new pscudo partition during the training stage. lt [1, K].
The number base-to-new pseudo partition during training stage.
((0))
The pseudo base domain in k-th base-to-new pscudo partition. The pseudo domain in k-th base-to-new pseudo partition.
()(A)
D (D,, D..
(x' Do. X x ). The data distribution under fixed base-to-new pseudo partition ))
$ = So, S.
S.. S, are sampled from D' and D... respectively.
((0) DUN ((0)
(x x 50)) X ):(0) The data distribution under the k-th base-to-new pseudo partition.
D ( ((K)
The pscudo base and new data distribution under the k-th base-lo-new pseudo partition.
8(h) ((R) ((k)
The dataset sampled from (*))
.On
S ((h)
The pseudo base and new dataset sampled from ()) (A)
21

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
To be specific, Ag is defined as:
A = sup S,H
ED, (Scc (b): Bss (oo wb)]
This error arises from approximating the expectation over the training distribution D using the empirical average from the training data S. Mcanwhile, this term meets the basic assumptions of standard generalization analysis techniques, where the loss function is represented as sum of independent terms. Consequently, bounding this term relatively straightforward.
A, is defined as:
D
sup rrE..
By [Rp] E3 [RS]
This formulation reveals two key challenges in bounding A,:
The empirical risk is defined on the 5 and S spaces, while the expected risk is defined over the Y and D spaces. This coupled discrepancy makes it difficult to directly bound the generalization gap for To address this issue, we decompose A, into three types of errors: partition-distribttion approximation error, empirical partitions estimation error, and data estimation error.
The standard generalization analysis techniques require the loss function to be expressed sum of independent terms. However, the AUC-type risk violates this assumption due to pairwise sample dependency. For instance, the optimization functions for the detector, (r, =g) and Exg (r, S Hf)), are interdependent any term is shared (e.g.. // zy or म = zi.). To address this issue, we use covering numbers and e-net arguments the subsequent proof to derive the generalization bound.
We decompose A, into three types of errors:
Partition-distribution Approximation Error: This error occurs because, the pseudo partition distribution ý is used to approximate the open-world real partition distribution V.
Ey [Rp] - Ry [Ro]]
sup
r.E.,
This error refers the discrepancy between the expectation of pseudo base-to-new partition distributions ) and the empirical averages over (M(),, k E [1, K].
Empirical Partitions Estimation Error:
BB [Ro] - BB [
sup H.
Data Estimation Error: This error stems from approximating the expectation over the training distribution D with the empirical average from the training data
K
sup EEH,
B Rpl Es
[
sup EEH,
R
Σ
Rga
A defined as:
A ED, [Lee (8 vn)]
Without prior knowledge the new domain, we cannot bound the expected error. We can reduce this error by carefully designing prompts or using powerful foundation models.
22

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
B.2. Preliminary Lemma
Definition 1 (Bounded Difference Property). Given group of independent random variables X1, X2, Xn where X, E X, Vt, X2 .x,) is satisfied with the bounded difference property, if there exists some non-negative constants c 02, Cn. such that:
sup
1 In) ,
f(11,' " 16--1,666" an)) S aa, 15 n.
(1)
Hereafter, if any function f holds the Bounded Difference Property, the following Mediarmid's inequality always satisfied.
Lemma B.1 (Mediarmid's Inequality). Assume we have independent random variables X1, X2....; Xn that all of them are chosen from the X. For a function f X IR Vt, 1 S n, if the following inequality holds:
sup
1f ; f( 10 Fall S co. vt
with x + x', then for all 0, we have
-26
Pils - B(f) 2 cl 2 cxp
E ot
Lemma B.2 (Hoeffding's Inequality). 2,, ... Zep are independent random variables such that Z₃ E (a, bj almost surely, then for any 2 0,
-2nt2
P
7
Σ
B[Z
21
2xxp
n
(a (a=b)² b)²
Definition 2 (e-Covering). Let (F,p) p) be (pseudo) metric space, and S F. (fi..... fk} } is said to be e-covering of g _ ==1 B(f e), i... Vg E G. 3i such that p(g. e.
Definition (Covering Number). According to the notations in Def.2, the covering number of g with radius e is defined as:
N(e:9.p) = min(n: Be - covering over G with size n}
Lemma B.3, The covering number of the hypothesis class H. has the following upper bound:
3r
log.N(c; HR,P) S dlog
where d is the dimension of embedding space.
Lemma B.4 (Union bound/Boole's inequality). Given the countable finite set of events E,, the probability that at least one event happens is less than or equal to the sum of all probabilities of the events happened individually, i.e.
P PUE
M P[E]]
(3)
Lemma B.5 (d-Lipschitz Continuous). Given set X and a function f X R2 if continuously differentiable on X such that, Vx, y X, the following condition holds with real constant 6::
ll(()) f(v)ll < 1| -
Thereafter, f is said to be d-Lipschitz continuous function.
Assumption 1. The fg- fh. and are Lipschitz continuous w.r.t. the input variable. Corollary B.6. The Sigmoid function is L-pissccitz continuous.
Corollary B.7. The Squared loss function is 2 2-Lipschitz continuous.
Lemma B.S. Let and y be postive and satisfy x + C. The maximum value of the function f(x, a +
23

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Lemma B.9. (Yang el al., 2024) The volume an c-dimensional probability simplex is cl. In other words, we have:
/
dx. 1 c
Ve
1 1.dx dx2
Eis
Proof. We proof it by induction.
Base Case, c=1. Obviously, we have 1/2 = fx dx 1.
0
Induction. Supposes that V (i - 1)!, we have:
/
V1
1 dæi dxz ....... dx,
Σj 21=1
1 dan drz
dri-1 dri
11 B1 1-2;
1-1
(1 33 1-1
duy duz
dui-1
da:
- ny
V/-1
(1 xz.) 16-11d -dds
(1/i)
Vi-1
= 1/1!
The proof is then completed by expanding the induction recursively.
B.3. Key Lemmas
Lemma B.10. When 9(-) is Lipschitz continuous, the following holds:
l1g(x) g( S sup // Ip ||.0 $11g.
+ = 1. where 1 "
Proof.
lØ(x) - g(t)|= = 1/" (Vg(To + (1 -)))) x - 33) dr
< #ex
sup
[|[ ||x 1
Corollary B.11. Specifically, when p 1 and g = 0c, we have
(lo(x) - (()))> sup V || All.o
Lemma B.12. The cross-entropy loss function (fe()))) v), where (f(x) E IRC) and C' is the number of classes, 2-Lip continuous w.E.t. the defined infinity norm I|
24

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Proof. According Corollary.B.11, if:
sup (a.
IV (f))) 2
(4)
Then we have:
(o v) 1f -fll
Hence, we only need to proof see this:
Plec (J(x), (f)
=1o- - 1 = v1
where
f((x)
σ¹
Σ 1i=1 exp (f)
Since we have:
sup (x.g).EFF
(f()) Elor
11j vl 2
This proof is completed since x, y, f are arbitrarily chosen.
Lemma B.13. Let {aa, Vs W be i.i.d samples from data distribution, the loss function of prediction is given by 24 v1) = (x((), 3i) [0, B) for scoring function E F. the loss function T lipschici continuous w.r.f. the defined infinity norm [[fll With high probability, the following inequality holds for all f E F:
2d V N
[ RiA S
log (3 D.rr.. NN
Proof. Based on the basic property of the covering number, we can construct a covering of J using a set of open balls B,,... ByM Each open ball Bj is centered at [j. where B, = (s F || Sillo S e/4Lc).
Accoring the lemma.B.4 the following inequality holds:
P
sup fEf
[[[]
26
P
sup SEB
$10
2
For one j. we have:
P
sup SEB,
E
P sup SEB,
[E[ - E
< 4
+P Ele,1 Elel
5] +P
sup SEB,
Ele
6 4
2.
Since f is L-lipschitz continuous w.z.. the defined infinity norm, we have:
| - Gl Le |lf -sll s Le
S 6
ALc 4"
Naturally, we have:
P sup SEB,
||[[|] - El2,l|
2 <
= 0. P
sup SEB,
E[e
E[e]]]
> =
4
According to the lemma. 2, have:
C..N
P
sup ʃEB,
E
Bi0 2
Bie,
B|e,|| 2 -7 5 exp
B2
25

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Thus,
P
sup fEF
Eld - [[0 2 2.N N (F.
:1111/0))
exp
N
ALC
2 B²
By futher choosing,
2d B V N
log (3-r.NN)
have:
d
2d V
4. Le
P
sup SEF
B
2 B
N
log (3.r r. N)
22
B. /B N log (3.r.
Finally, we conclude that, with high probability,
sup SEF
B[A - B10
B 2d N
log (3.r-N)
This completed the proof.
Lemma B.14. Assume that the gated weighting function sp.)) is Iuplipschitz continuous and the loss function is Le- Lipschitt continuous. Let be the generalization error between the empirical risk B& [R,p (r, and the pected risk Ey RR (r;g,h)]. Then by constructing o-covering {ri, ,NN. of H, with 6, and a-covering (86,1, 86,2, ... N,N,) Hg with eg. the following inequality holds:
sup (EBBmis)..sEB(oo..ss
Eg RR (r, - Bg [R (r,g, Wi]
Se
2 2P Es [R (1i, h)] BB [Ro (r,, gu, guo W]] S
Proof. Some simplified notations applicable only to this proof are given as follows:
(ss((x))) = Ppb ((s6,u(z))) = Go. (sx((1n) = PP
(rg(r(0)) r((x)) = l, log(1((10) - r((x)) =
We first turn to prove the following inequality,
E [RD (r. g. LE3 RRg (r. g, hi] - E [R In, ht] - -B3 [R (ri, Iur Jun
< Ba RB (r) - -E RR (ri, gu. Ai]]
+ |Eg RR (r, - -E, [R (11. h)] S
(+) follows the rule that je + yl S |a| + |y/ and ||,|| -Bll S y/. Thus, we only need to prove the following inequality,
B [RD (rig,1)]
Ey [Ro (ris Gu, A)] S
26

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
We have,
By [R (r,g,h)] -Bg R (ri: Ir. 1]]]
K
Rim (r) - Rom (ri, (ri, hi]
K
Ras (r,gh)) - RBc (T2, (rr 1t)l
max )(
max
pb L. pp 50 E. Enpm
Pm
(m ( (2n )-D(
f1000 1 1- pn ipb E. yn + pp pn
pnn - F0 E. pm pml
max
E
((R) (1 (())
(In (un M.. (
max
{2. Le 122 111 relloo + loo Lo |180 8b,ull20
(
Mn Dik)
2 Le . 1
G + l00 Li Cg 4
follows the fact that we perform K base-to-new pseudo partitions. (+ (***) follows the Lem. By further choosing e, 3.be by and Eg B. to we could construct the covering number N, and Ng:
(**) follows the rule that Jx + yl S |a|| + ly/-
N(e, 8 Le u2
Hy 11-11,0)
+ 4 20 Lo
N(eg 8. Le v2e + +..000 Iiw
t
Hy. l1
such that the following inequality holds:
Rgg [R (r, Bs RRD (ri, Sa: M)] S
This completes the proof.
Lemma B.15. Let (r, g, h) be the detector-classifer triplet and K be the number of pseudo base-to-new partitions. The following inequality holds given the definition of Use and l
2.. K
IP
Bg RB (r.g. hi] - -B [RD (r.g,ti)]
2 { < exep 2
2 ug. 82
Proof. Step 1: We first need demonstrate that the B3) RR (r, (r.gh)) satisfies the bounded difference property defined in
1ef.1. Let ) and y be two independent base-to-new pseudo partition distribution where exactly partition is different. The difference between ) and 3" could be caused by i-th pseudo partition process, denoted as ()))) and jo((i), respectively. Thus, we need to prove the following bound:
ES [R (r,g,)) Eg (r.g,)))))
sup m.E
Taking further step, have:
PS RD (r,g.h)] - E.g, [Ro (r. g.
sup ay rH.
1
sup rEH.
Rows (r.g. Ronc (r.g,
K
1
Roco (r,g,h) - Rono k)) (+)
u2 lo
sup H.H.
S K
K
27

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
holds because 0 sPo (r, Inz Fn)T 'nn u lo.. As result, the expected loss also satisfies Roc S v bx. This leads us to conclude that By [R (r, g, h)] satisfies the bounded difference property.
Step 2: Then, according to the I.m.B.I, we have:
e² K
P
Es [& (r.g.1) - -B [R (r.g.))] 2 2/ 2 exp
2 u f
This completes the proof.
Lemma B.16. Equipped with Lem.B.14 14 and Lem. for vr E Hy, 86 E Hg. the following inequation holds with high probability:
2d log (3 rr r. K)
B3 [R (r.g. h)] - -Bs [R.g (r.g.h)]] s lx V
K
Proof. First, need figure out the following probability:
P
sup E. H.E.
Bÿ [R (r. 1)] - -Bş [Ro (r. g.g hi] 20
P
sup
Es R& 10) -Es [Ro (r,g, 26
rE U Bo(reee)) A&E U (eb.uie))
N
AC.
s
P
sup
Eg [Ro (r. h)] - Rs [R. (r. g. mi] 20
u=1 1=1
N₂ A
Lem. Lem.B.14 B.14
P
Es [RD (ri, Fu. W]]
-E3 [R (r1, Iu. hi]
2
N
BB RR (ri. hi] -Es [Ro 9rt, hi] 2
Lem. B.15
e2.K
S NgN,2exp
(- v. 12
2.
According to Lem.B.14, we have
N₂ NN(er
Ho, 11-1)
8 Le v2 4 foc L,"
N₂ N(es
Hy. 11a))
8 Le ug lao Ln"
By further choosing
2d K
c= 22 loo
log (3.r. K)
further have:
2d log (3.r....) 52.
d
E& [R (r.g,)) 1)] E3 [Ro (r, g, hi] 2 v2 loo V
sup EEH.
K
2d 3 r K
where I' is universal constant depending on U locs Le, Lu, r. Therefore, arrive the conclusion that for Vss E Hy, r E Hr. the following inequation holds with high probability
2d log (3 r K)
B3 [RD (r.g. 1)] -B3 [R. (r. g. hi] u loo v
K
28

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-worid Prompt Tuning
Lemma B.17. Assume that the gated weighting function (p(:) is Ly-lisschitz continuous and the loss function é is Le- Lipschitz continuous. Let be the generalization error between Rg(r, 9, h) and B. B. g, h)). Then by constructing ,8b,Na) a-covering (2,,,2," ,TN,. of Hr with er and o-covering (8b,1, " ; } of Hg with Eg. the following inequality
holds:
P sup Rg(r,g,)) h) B|Rg(r, g. Mi] sc PP Ro((r 9u: h) B[Rg(re, gu, W)] $
Proof. Firstly, some simplified notations applicable only to this proof are given as follows:
((g((t)) = rb. (Son (x()) = V a(h(x4)) = v/
(rg()()))) r(x2)) = lis (xg(r1(8)) - m(x))) Eas
Given coverinng r2, F of H, and Eg covering (80,1, 86,,, ,UNN,, } of Hg. to prove the above lemma, we turn to prove the following inequality holds:
Ag(r; g,(r, - Elug (r, m)]] fg(rr, h) - (r B[Ag(r, g, h]] - EiigEEillTeeee Iu. hill s c
1 Rg(r, Rg(re, Iu. 1n) +
2
Similarly, we have:
No No Rg(r.g,h) - Rgg(re s No Nn i=1 j=1
lis 4/h - 56 Ris p/
100 les 1.. - vbb les : v/4l
x max (1)
max (s),ad) {108 lis . 4ph - S/g lag p/ + 10 les 1ph - 8b las ph
{2. 12. Le ||| ree relloo + too Doo Lo 11.80 - sb,allo0
max (ot
e 22 Lt o2 c lz Lu Cy S 4
where No and No are the number of pseudo base and new samples in S. (+) follows the fact that |ar yl follows the Lem.B.5.
S [ar] + lyyl- (+*)
By further choosing 6 B. the and Cg 8 8. f4 to we could construct the covering number NN and Ng as follows:
N, N.=N(G- 8 Le . us 4.0 L
Hr Hr) Pr), Pr ||re rl
N₂ -Nlee 8- Le v22 +4. 1o0 Ly
Hg. Hg pg) || sbll
such that the following inequality holds:
R - Ro(re,
Thus, we have:
Ro(r.g,)) Ro(re, Iu. bu: h)] -1Rg))))) 2
This completes the proof.
29

---

OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Lemma B.18. Let S and St be two independent datasets where exactly one instance is different. We conclude that Rg(r,g 9, h) satisfies the bounded difference property.
Proof. The following simplified notations are used in this proof:
(fa(z))) sb. a(s((Ts)) = Pb. a(86(Ta)) Eb
((h(12))) sh. ((8n(In)) - Pni a(8n(En)) Fnn
(r((n)) - r((x)) 1(,, shi, (ag(r((s)) - -r(x)) [(r(( Th. x/h) Lo (r(o)) - r(zn)) = E(r, x/; on); (r((r(2))- - -r(ān)) = E(r, . kn)
To prove this lemma, we need to seek the upper bound of
(r.gh))))) - R
sup H MEH.
We first recall that
N. N pb lag -((f))))x())))) ph
Rg(r.g,h)=
E Pb log Zn, Pp ( (n.ii..
No N,
Thus, the difference between S and S' could be caused by either the pseudo base domain sample (i.e. Ty and ïτ) or the pseudo new domain sample (i.e. on and Tx). Hence, we have the following two possible cases:
Case 1: Only base domain sample is different. Since xa and Ty are different in this case, have:
(r,gg. h) Rs(r.g.h)
sup rEE.,
N, Σ sh (rg(r((6)) - -r(xh)) 966 N,N,
N,
Σ
S/ Lag (r(ãs) r(x/h) .6
sup reH
NoN,
1=1
j=1
Nn
(0) 122 loo
Σ
1pt l(r, Ibs xn)) 46b p/, E(r, Cg, x4) 4b
sup NoN, on EEH. J=1
Nb
(+) holds because p/, l(r, nb, al)) yo [0, v. lool-
Case 2: Only new domain sample different. Since In and ā, are different in this case, similarly we have:
(r, Rs(r.gh)) s v2 loo sup *y HyE r. hEH,
Nn
Finally, taking two cases into account, we can conclude that (s(r. g,
h)) is satisfied with the bounded difference property.
Lemma B.19. Let No and Ñr be the number of pseudo base and new samples in dataset S. Equipped with the above Lem.B.l and B.m. 18, the following inequality holds given the definition of Uxo and Eoo:
( 2.. N
P ftgrri Iu. h) EE ( C) < 2exp
2
where
)=
1 -1
= 1000
r-2
+
N₂
N
30

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Proof. The proof could be easily achieved by applying Le top of L8 as:
2
P Rg(T Su. h) - B[Rg(T). hi] 2 S 2 exp
6
2
2. Σt= t=1
c
2
exp
od
2
N₂
N₂
Lemma B.20. Let Let ()) be the population risk of Rg(r, g.ngg,h, g... Then, equipped with Lem. B. B.17 and Lem. B. 18. Vr E He: Sb E Hg. the following inequation holds with high probability:
2d 2d - N
Ro(r.g, B(Rg(r,g s
N
Proof. First, need to figure out the following probability:
P
sup Nb EE..
(s((r,g,)) (r h)] 26
<
P
sup
- 26 26
re U [m]l
Br (r y unl
B, ())
N,
N₂
Ei 26
sup reB(m.c)).sBBBs
N,
Lem, Lem.B.11 B.17
Σ
2
S NgN,TPP
R Iur h) E[Rg(re, 2
Lem, B.18
.2.N
NrNNeexp
where Ny. Ny are the covering numbers the hypothesis space Hy, Hy. respectively.
Nr. = N(G 8. Le 12. + 4.4.g lo L,
6
Hr, Pr). = =Iim - -rll
Ng = N(e 8 L2.02 v + 4 4.0
c
Hg. Pg). Pg = ||s6. - 86 8b
Lo
By futher choosing,
d. log 3 m..r.
we have:
2d 2d log 3 Ñ
r
d
P Ro(r.g.)) -B(tr,,,))-/Rgrr....
2
52.
N
2d log
N
31

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning Similarly, we conclude that the following inequality holds with high probability,
2d log r- Ñ
-) B(Rg(r.g,)))) -B
$1
Vr E Hr, 8b E Hg
N
B.4. Formal Proof
We present an upper bound of the expected risk R(r, g, h) in (OP)). Note that our subsequent analysis is based on standard assumption that the base domain logit score function 8; and the detection function r are chosen from hypotheses class Hg and Hr respectively. These hypothesis classes are the specific types of frozen CLIP with learnable prompts.
Restate of Theorem 5.2. Let 3% be the base label space and Y be the overall label space, where the total number of classes is C. Suppose D represents the pseudo base-to-new partition distribution, and let & and E' denote the expectation distribution over y and V. Furthermore, consider a training dataset S. = {3ri 1i comprising i..dd. samples drawn from the data distribution D. Define R(;; g, h) as the population risk of R(r, g, h)) in OPo. For the loss function, let too = sup (w) (rg Tb, Zn)] bounds the squared loss, l,,00 = sup(x,y) Lee (5)(s))), y) bounds the cross-entropy loss and Uoo sup, (s) (M) (x:) bounds the sigmoid function. Assume that ((c; p) I
Then, for all Hy 86 E Hg.
with high probability, the following inequality holds:
v22 loo [|((P) (P)
R(h,g.r) & g') + ED, [lee(Sn(In)) Un)]
C!
2d
2d 2d 3 No
+ v2e
loo V K
log (3.r. r3.r K)
2 u2 loo
N
Proof. We first decompose the overall excess risk into the following three parts:
A sup EE
R(r,g,h) Rr,
Eg (RD(r.gh))) - Ee (
ED. lee (x6 v6)l Es, (s0( voll
sup a EEH,
+ sup e,EH,
A
ED, (eoe((n(())) Mn)]
According to our analysis the proof outline, analyzing A, and bounding Ag are relatively straightforward. We will first analyze A and Ay and then provide detailed analysis of A,.
As for A.. Without prior knowledge of the new domain, bounding the expected error An remains challenging. In practice, we can reduce this term by carefully designing hand-crafted prompts or using more powerful foundation models.
As for Ag. Accoring to the Lem.B. B.1. 12, the loss function foe is lipschitz continuous. By applying Lem.B.13, the following inequality holds with high probability:
2d
sup a,E H2
ED. (fee (80())) yb) Es, [f (Mo((a)))) wall < t0,00 V No
log (3 r..r... .r.r.M No)
32

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
As for A.. Directly bounding A, is difficult, we decompose the A, follows:
By [bd] E3 [
A
sup *o M,
sup M.
By [RD]
3 [Rb] + B3 [Rb] - Es [R,]] +Bs [Rp]]
Es [
By [RD] 1y [Rb]]
1 [Rb] Es [Rb]]
BB [Rtp] -E R
sup MEH,
sup a TEH,
sup
(a)
(0)
As for (a) Since Pb Lng (r, nn 2x)) yn is bounded by [0, 2 fool we have
S 2200 L00 /
sup m6 *E EEN
Ey [RDY)] - Es [RD]]
[E(P)) (P)) dP
Re
Lem B.9 2 loc [[E(P) (P)
where Sp-1 is the probabilistic simplex on R P is a C-dimensional probability allocation sampled either from & or E As for (b), by applying Lem.B.16, the following inequality holds with high probability:
Rs [R (r.g. 9, 1)] RR [Ro (r,g, m)] s v2 lo V
2d log (3.r r. K)
sup EEH.
K
As for (c), have:
K
K
sup M.EHg
B [R]] - E
[g
sup s, MEH.H
1
R
D(a)
R
K
K
k=
For on fixed {i(k), according to the Lem.B.20, the following inequality holds with high probability:
2d 2d 3 r- N
(g (t,g S
sup TEH.
N
where
Ñ 100 2-2
[y(1)) +
(R)
Due to (M) + N(00 = NN. we have:
N₂ Ñ Mmn 4 og Pg
Thus, for all (1(), l E [1, K]. with high probability, we further have:
2d log 3 r Nmin
2d log 3- r No
sup H EEH.
Rggo (r. h) - (r. <
22.12 loo
Nmin
N₂
Finally, by combining the bound for A.. Ay and An. we reach the conclusion of this theorem.
33

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
C. Additional Experimental Setup
C.1. Task Description
We perform empirical evaluations on Open-world recognition, Open-world domain generalization and Open-world cross-dataset generalization tasks:
Open-world recognition task. This setting involves dividing the class space of each dataset equally, with 50% of the classes designated as base classes and the remaining 50% as new classes, following (Zhou et al., 2022b). In the imbalance setting, we resample the test sets constructed with different base/new sample ratios. The model is trained on the base classes and evaluated on the mixture of base and new classes.
Open-world domain generalization task. The model is trained only on the base classes of ImageNet dataset and evaluated on the all classes of ImageNet variants datasets, each with additional different types of domain change.
Open cross-dataset generalization task. We train our prompt exclusively on the base domain of ImageNet and subsequently evaluate model performance on a combined test set containing both the original ImageNet-500 classes and new categories from seven external datasets: FGVC-Aircraft, Caltech-100 101, Stanford- Card, DTD, EuroSAT, SUN397, and UCF101. To ensure fair evaluation of open-world generalization, meticulously remove any categories from these external datasets that overlapped with the ImageNet-500 class space before evaluation.
C.2. Datasets
For open-world recognition task, we follow prior work (Zhou al.. 2022b;a; 2024) and conduct experiments on 11 benchmark datasets that span wide range of image recognition tasks. These datasets be categorized into five main types: 1) Generic object classification, which includes ImageNet (Deng et al., 2009) and Caltech101 (Fei-Fei et al., 2004); 2) Fine-grained classification, encompassing OxfordPets (Parkhi et al., 2012), StanfordCars (Krause et al., 2013), Flowers102 (Nilsback & Zisserman, 2008), Food101 (Bossard et al., 2014), and FGVCAircraft (Maji et al., 2013); 3) Scene recognition, represented by SUN397 (Xiao et al., 2010); 4) Action recognition, with UCF101 (Soomro et al., 2012); and 5) Specialized domains, such as texture classification with DTD (Cimpoi et al., 2014) and satellite image classification with EuroSAT (Helber al., 2019). For the open-world domain generalization task, we use ImageNet (Deng al., 2009) as the source domain and evaluate on its four variants: ImageNetV2 (Recht et al., 2019), ImageNet-Sketch (Wang et al.,
2019), ImageNet-A (Hendrycks et al., 2021b), and ImageNet-R (Hendrycks et al., 2021a). The details of each dataset shown in the following:
ImageNet. ImageNet, a widely recognized generic object classification dataset, comprises approximately 1.28 million training images and 50,000 test images across 1 1,000 object classes. Sourced from the web and organized using the WordNet hierarchy, it has become a benchmark for evaluating object recognition models.
Caltech101. The Caltech101 dataset, designed for general object classification, includes 101 object categories and a background class, featuring around 7,650 training and 3,300 test images. The images, collected at Caltech, present significant variation in scale, orientation, and lighting conditions.
OxfordPets. Focusing on fine-grained pet classification, OxfordPets contains 37 pet breed categories with nearly equal numbers of training (3,680) and test (3,669) images. The dataset provides not only breed annotations but also pixel-level segmentation masks, making it versatile for classification and segmentation tasks.
StanfordCars. StanfordCars targets fine-grained car model recognition with 196 classes distinguished by make, model, and year. It offers 8, 144 training images and 8,041 test images, sourced from various car-related platforms, capturing diverse vehicle angles and environments.
Flowers 02. Flowers 102 is a dataset of 102 flower species, designed for fine-grained classification tasks. With 6, 1,44 training and 1,020 test images, it presents visually intricate patterns, and challenging models to distinguish among visually similar flower categories.
Food101. 101 is tailored for fine-grained food classification, comprising 101 categories of dishes. The dataset includes 75,750 training images and 25,250 test images, offering challenges in recognizing overlapping ingredients and presentation styles. Images were curated from a range of food-related sources.
34

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
FGVCAircraft. FGVCAircraft specializes in fine-grained classification of aircraft, offering 100 categories that distinguish between models and manufacturers. It contains 6,667 training images and 3,333 test images, sourced from aviation databases, with focus on subtle visual differences in design.
SUN397. SUN397 is a comprehensive scene recognition dataset that encompasses 397 categories, covering diverse environments such as natural landscapes, indoor spaces, and urban areas. It includes approximately 50,000 training and 50,000 test images, offering rich resource for scene understanding.
UCF101 UCFI01 is a video dataset designed for action recognition tasks, featuring 101 action classes ranging from sports to daily activities. The dataset contains about 9,500 training clips and 3,700 test clips, collected from YouTube, with a variety of dynamic scenarios.
DTD(Describable Textures Dataset) The DTD dataset focuses texture classification, featuring 47 texture categories described using human-interpretabl attributes. It offers 3,760 training and 1,880 test images, providing a unique challenge in recognizing visually distinctive patterns from natural and artificial sources.
EuroSAT. EuroSAT is a satellite image dataset for land-use and land-cover classification, containing 10 classes such as agricultural areas, forests, and urban regions. Derived from Sentinel-2 satellite imagery, it includes 21,600 training and 5,400 test images, offering rich spatial and spectral diversity.
ImageNet-A ImageNet-A is a curated subset of ImageNet consisting of images that exhibit challenging, adversarial characteristics. These images are specifically selected to challenge existing models, often including unusual or hard-to-classify variations of the objects from ImageNet's standard classes.
ImageNet-R ImageNet-R is a variant of ImageNet that features images with artistic and stylized renditions of the original objects. These images have been altered to reflect different artistic interpretations, such as paintings, sketches, or computer-generated images. This dataset evaluates how well models generalize across diverse artistic representations of familiar objects.
ImageNetV2 ImageNetV2 is revised version of the original ImageNet dataset, designed to better represent real-world data distribution. It has slight variations in image selection and distribution. It serves as an alternative benchmark to assess the robustness of object recognition models under different data variations.
ImageNet-Sketch ImageNet-Sketch is variant of ImageNet, created by converting the original images into sketch-like representations. It consists of 50, 000 images, spanning the same 1, 000 classes the original ImageNet, but each image has been hand-drawn to emphasize the object's outline and basic features. This dataset challenges models to recognize objects in a more abstract form, testing their generalization ability across different visual representations.
We also construct 10 domain imbalance of 1 : datasets by randomly sampling DTD, Food101, Flowers 102, OxfordPets, and SUN397. Forward (Fwd) means base domain number 5 times the new domain number, while Backward (Bwd) means the opposite.
C.3. Competitors
To demonstrate the effectiveness of proposed method, we compare method with 10 competitive competitors: a) Baseline, Zero-shot CLIP (Radford al., 2021); b) HM-oriented methods, including CoOp (Zhou et al., 2022b), Maple (Khattak et al., 2023a), KgCoOp (Yao et al., 2023), PromptSRC (Khattak et al., 2023b), DePT-Kg (Zhang et al., 2024) and TCP (Yao et al., 2024); c) OOD-oriented methods, including LoCoOp (Miyai et al., 2023) and Gallop (Lafon et al., 2024); d) OverallAcc-oriented algorithms, DeCoOp (Zhou et al., 2024). The details of each competitor are provided in the following:
CoOp proposes learnable continuous prompts to replace manual text templates in CLIP-like vision-language models. Through gradient-based optimization of spec-spec textual context, it achieves strong few-shot performance but tends to overfit seen classes, compromising generalization unseen categories.
Maple couples vision- Languae prompts hierarchically, thus mitigates the limitations of single-brang branch adaptation and improves generaiization to unseen classes.
35

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
KgCoOp addresses catastrophic forgetting of general knowledge in prompt tuning by minimizing the diserepancy between learned prompts and fixed hand-crafted prompts. It enforces consistency via regularization loss, balancing base domain adaptation with CLIP's original zero-shot capabilities for improved classification performance new domain.
PromptSRC introduces a three-pronged regularization framework: (a) mutual agreement with frozen CLIP features, (b) self-ensemble of prompts across training phases, and (c) textual diversity augmentation. This approach reduces overfitting while preserving the model's generalization on both base and new classification tasks.
DePT decouples prompt optimization into distinct components, potentially separating base and new domain classifica- tion. We incorporate this method on top of KgCoOp to further improve the generalization performance, denoted as DePT-Kg.
TCP emphasizes class-aware prompt design using textual semantics, aligning prompts with fine-grained class contexts.
LoCoOp leverages CLIP's local features (e.g., background regions) pseudo- Ooodoo-OODDooo samples during training, and it optimizes text embeddings to separate in-distribution (ID) classes from OOD data, reducing nuisance signals in ID embeddings. However, this method sacrifices classification performance on base and new domain to some extent.
Gallop leverages both global and local visual representations. The key features of GalLoP are the strong discriminability of its local representations and its capacity produce diverse predictions from both local and global prompts. This method achieves better trade-off between domain classification and OOD detection.
DeCoOp firstly focuses on open-world prompt tuning, where models must classify mixed base/new classes without prior knowledge. It integrates OOD detection into prompt tuning via the DePT framework, using new-class detectors and sub-classifiers enhance base/new class discriminability.
To ensure fair comparison, use the same backbone architecture (ViT-B/16) for all competitors. We also adopt the same training strategy and hyperparameters according to their open-source code. The number of parameters for each method is shown in Tab.7.
Table The number of parameters for each method.
Method
CLIP
CoOp
Maple
PromptSRC
LoCoOp
KgCoOp
DePT-Kg
Gallop
DeCoOp
TCP
Ours
param
8192
3555072
46080
8192
8192
292914
606208
30720
331904
26624
For all competitors, the detection score is derived from the maximum probability over the base domain. In this task, the new domain class names are known during testing. Compared to out-of-distribution (OOD) detection-which operates without prior knowledge of new classes the base-to-new detection task is less challenging.
Algorithm 1: Base-to-new detection score calculation
Input: tmaage.feature
fext.feaaures. Number of base classes C. Output: Base-to-new detection score E [0, logits image features text. featurrs""
prob <- Softmax(logits, dim 1)
r' max prob|:.]]
15)5C,
return
C.4. Implementation Details
All models are implemented using PyTorch (Paszke et al., 2017). The length of the classifier prompt is set to 4. To ensure the parameter size of our method is comparable to recent state-of-the-art methods that incorporate additional structures or deep prompts, the detector prompt's length is set to 16, consistent with DeCoOp (Zhou et al., 2024). Results are reported as the average over 5 runs with different random seeds 1,2, 4, 5. We mention that we adopt a fixed hand crad-er prompt for the new domain classifier. To be specific, we use the prompt ensemble:
36

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
"itap.of.a.f. abbcd.hhoto.of.the..f aLorigami-0- a.phott..f..theeaargee.. O in video game. art of the ()- photo of the small ()"
in all datasets except for some fine-graine ane-grained datasets. For the challenging line-graing dataset, we use the prompt:
FGVCAireraft: "a photo of [CLASS], a type of aircraft
Food101: "a photo of [CLASS], a type of food"
Flowers 102: "a photo of [CLASS], type of flower"
to ensure the prompt is consistent with the dataset's characteristics.
C.5. Efficient Calculation of OpenworldAUC
During the test phase, all samples containing base and new samples are available. To calculate the OpenworIdAUC effiiciinttt..i we first mask each sample that has been misclassified and then calculate the AUROC on the classified samples through pairwise instance comparisons. To be specific, there are the following two steps.
Step 1: Mask misclassified samples.
E + max r(x6) r((a)
if h(x)) + V/ (f(a)))))))))))))))))())))) = yn
F(In)
minz.es; r(zn) - if ((x)) + yb g = y/b
f(zs)
r(())
where Sy, and S,, are the base and new domain test datasets.
Step 2: AUROC Calculation.
OpenworidAUC(rr,s/
Σ
[|w g(xx)) 1/r(w)) > r(xn) 1/yn = h(xr))]
Ng N
(ES
(Zn.Nn)ES,
Σ
[F((x)) > F(xw)) AUROC(F)
Ng No
D. Additional Experimental Results
D.I. Additional Results for Openworld Recognition Task
In this section, we present full results of open-world recognition task with standard deviation and also show stage-wise metrics (say AUROC and HM) in the Tab.8. Tab.9 and Tab.10. Fig.7 provides clearer visualization, showing that HM- oriented methods located on the right side of the image, implying higher HM metric values. However, these methods still have limitations in base-to-new detection performance, which consistent with the shortcoming of HM metric (P1). And the OOD-oriented method such LoCoOp may is located on the left upper of the image, meaning higher AUROC metric but sacrifices the classification performance to some extent. This limitation is consistent with the AUROC metric (P2). Compared with these methods, our OpenworldAUC-orieee the AUROC and HM, which further demonstrate the comprehensive of OpenworldAUC metric and the efficiency of our optimization method. This consistent with (P1) and (P2).
optimization method can achieve a better trade-off between
37

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table Quantitative comparisons on ImageNct, Caltech 101, OxfordPets, StanfordCars, Flowers 102 and the average of eleven datasets. The best and the runner-up method on each dataset are marked with blue and red, respectivcly.
Average
ImageNet
Method
HM
AUROC
OpenworIdAUC
HM
AUROC
OpenworIdAUC
CLIP
71.55
79.62
47.83
70.18+0.00
87.91+0.00
47.31
CoOp
70.65
84.27
47.59
71.19=0.51
95.15+0.22
48. 48.93+0.7 71
Maple
77.82
86.18
57.35
73.31=0.23
95.47+0.52
51.89+0.47
PromptSRC
78.83
86.20
58.21
73.59+0.36
95.83+0.08
52.44+0.53
LoCoOp
71.44
89.87
52.62
68.67+0.45
92.9620.31
45.1240.66
KgCoOp
76.31
84.23
55.07
73.00=0.08
95.3340.
51.4540.11
DePT-kg
78.37
87.37
59.00
73.13+0.14
93.38+0.62
51.35+0.2.
Gallop
77.75
85.81
56.90
70.83+0.14
95.93+0.15
49.01=0.16
DeCoOp
76.10
90.68
58.77
73.15=0.64
96.22+0.09
51.98+0.11
TCP
78.56
86.05
58.90
72.90+0.06
95.31+0.11
51.34+0.1 14
Ours
78.24
91.08
60.94
73.1920.15
96.9410.06
52.6420.16
Caltech101
OxfordPets
Method
HM
AUROC
OpenworldAUC
HM
AUROC
OpenworIdAUC
CLIP
95.67=0.00
89.44=0.00
82.31=0.00
93.36+0.00
85.09+0.00
76.17=0.00
CoOp
94.10=0.51
93.48=0.65
83.29=0.65
93.87+1.30
90.56+0.44
80.71+2.31
Maple
96.38+0.47
93.66=0.336
87.15=0.99
96.02=0.49
92.23+0.94
85,59+1.66
PrompiSRC
96.71+0.28
92.60=0.16
86.7420.35
96.43=0.17
91.83+0.44
86.10+0.42
LoCoOp
93.28+1.44
08.8990.39
86.59+2.65
93.8642.71
97.73+0.90
86.6245.57
KgCoOp
96.3720.19
92.9820.29
86.63+0.34
96.4240.26
92.47+0.46
86.80+0.43
DePT-kg
96.59+0.08
99.39=0.07
92.74=0.17
95.90=0.24
95 95.23++035
87.81+0.35
Gallop
95.7620.38
95.25=0.36
87.51
96.38=0.36
93.00=0.22
86.9420.60
DeCoOp
96.4520.07
99,48=0.07
92.72+0.10
94.83+0.65
98.21+0.42
88.72+1.21
TCP
96.50=0.12
92.34+2.85
88.6542.65
95.74+0.44
92.27+0.27
85.50#0.83
Ours
96.51
99.4940.04
92.81+0.12
95.5220.13
98.0340.92
89.7740.97
StanfordCars
Flowers102
Method
HM
AUROC
OpenworldAUC
HM
AUROC
OpenworIdAUC
CLIP
69.02+0.00
87.91=0.00
43.43+0.00
73.75+0.00
85,5620.00
48.51+0.00
CoOp
62.54+2.56
84.97=1.37
35.38+2.85
78.8623.50
92.2110.97
59.6511.40
Maple
73.6920.34
89.8420.55
49.9920.50
83.38+0.45
92.1720.52
65.44+0.33
PrompiSRC
74.52+0.65
89.15=0.15
50.90
85.55+0.58
93.38+0.22 20
$9.36+0.92
LoCoOp
68.66+1552
94.66=0.99
46.52+1.88 86
17.50=1.45
96.86+0047
61.17+2.07
KgCoOp
73.82+0.05
88.36=0.31
49.40=0.09
82.63+0.93
90.67+1.4
62.96+2.25
DePT-kg
76.8820.46
93.5720.85
55.2440.44
85.1040.40
92.83+0.75
69 69.46+1.00 1.22
Gallop
73.9120.69
89.57=0.56
50.6920.73
82.51+0.33
94.50+0.65
65.6940.30
DeCoOp
74.03+0.29
95.42+0.53
53.5920.47
84.15+0.45
96.8420.36
70.28:0.61
TCP
76.25+0.34
88.85=0.20
53.18:0.31
84.3421.97
94.05+0.26
69.20+0.35
Ours
75.32+0.17
95.58+0.27
55.31+0.32
85.7620.12
96.95+0.89
72.7940.60
38

---

OpenworidAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table Quantitative comparisons Food101, Fgve-aireraft, SUN397, DTD, EuroSAT and UCF101. The best and the runner-up method on each dataset are marked with blue and red, respectively.
Food101
FGVC.Aireraft
Method
HM
AUROC
OpenworIdAUC
HM
AUROC
OpenworIdAUC
CLIP
90.3+0.00
89.57=0.00
75,09=0.00
30.94=0.00
60.89+0.00
7.23+0.00
CoOp
85.26+1.2 .99
89.12=0.43
67.27=1.81
26.20+2.83
54.89+1.3
5.60=0.70
Maple
90.72+0.25
90.77=0.42
76.83=0.58
35.17=1.70
66.8243.18
9.58+0.46
PromptSRC
90.89:0.38
91.13=0.17
77.33=0.57
39.77=0.75
69.53+2.09
11.40=0.61
LoCoOp
86.41+1.73
95.9320.86
73.0943.09
31.0242.06
66.6 66.64+1.8 1.7
8.67+0.91
KgCoOp
90.8420.18
90.67=0.37
76.9620.24
34.08=0.51
56.01
8.18+0.22
DePT-kg
91.0620.15
94.5620.17
79.45=0.32
38.5421.04
74.47=0.77
12.7140.57
Gallop
88.96#0.48
90.48=0.24
73.60=0.92
40.32*1.16
18.83+1.36
11.38+0.42
DeCoOp
90.56+0.06
97.36=0.24
80.67=0.27
31.41=0.28
69.91+3.14
8.17+0.32
TCP
90.87+0.21
91.02=0.27
77.27=0.40
37.77=1.38
59.91+1177
10.72+0.73
Ours
90.7920.06
97.6320.08
81.2520.12
37.7820.61
71.23+2,27
11.4240.42
SUN3977
DTD
Method
HM
AUROC
OpenworldAUC
HM
AUROC
OpenworIdAUC
CLIP
72.37=0.00
77.39=0.00
42.52+0.00
56.62+0.00
64.69+0.00
25.22+0.00
CoOp
75.54=0.62
81.63=0.46
48.03=0.56
53.30#1.91
76.03+1.06
25.48+1.44 4.0
Maple
79.35+0.28
81.53=0.50
52.84=0.61 61
67.16=0.776
73.53+2.03
36.22+1.55
PrompiSRC
79.84:0.31
82.60=0.56
54.1920.67
69.60=0.82
77.19+1.2
40.30+0.94
LoCoOp
73.24+0.76
89.35=0.28
50.55=0.98
57.7343.43
81.14+2.01
32.2643.1
KgCoOp
78.6240.39
81.6820.41
52.09+0.72
65.5343.26
74.42+0.6
34.8743.00
DePT-kg
79,79+0.42
85.72=0.26
56,42+0.65
67.51+0.89
72.19+0.99 ,99
37.56+1.00 .05
Gallop
77.2620.49
82.5620.11
50.6220.70
70.14
76.85+1..1
40,22+0.79
DeCoOp
78.0820.11
89.9720.45
57.00=0.14
63.94+1.14
79.64+2.19
37.07+1.6
TCP
80.1340.18
83.03=0.16
54.86+0.36
66.36+0.65
78.4740.55
37.92+0.85
Ours
79.0420.10
90.70+0.48
58.5440.21
68.32+0.87
78.58+1.67
40.37+1.28
EuroSAT
UCF101
Method
HM
AUROC
OpenworldAUC
HM
AUROC
OpenworldAUC
CLIP
60.21+0.00
64.08z0.00
28.01=0.00
74.6620.00
83.2940.00
50.37+0.00
CoOp
68.07+3.93
82.87+3.88
41.9624.94
68. 18.2.52
86.02+0.41
43.95+2.94
Maple
80.15+5.84
86.04#5.21
56.55=10.18
80.71+1.18
85 85.8940.75
58.72+1.88
PrompiSRC
79.10=5.18
79.46+4.31
52.56+9.26
81.1620.85
85.46+0.66
58 58.94+1.36
LoCoOp
66.59+7.49
82.60=4.34
41.35+5.27
68.88=1.41
91.83+0.77
46.90 .29
KgCoOp
68.85+6.59
77.22+2.90
39.16+7.31
79.21+1.23
86.6740.88
57.29+1
DePT-kg
76.84+3.54
68.3342.69
44.90+1.96
80.74+1.02
91.4110.46
61.38+1.37
Gallop
80.06+5.90
78.28+3.13
51.3828.33
79.13+1.38
88 88.6440.52
58.91+1.8
DeCoOp
72.25+2.76
80.7013.44
46.66+1.81
78.25+0.40
93.73+0.78
59.57+0.72
TCP
80.13+4.117
83.29=2.60
55.89
83.1420.37
87.99+0.36
63.39+0.51
Ours
77.6640.22
83.72+2.68
53.09+1.36
80.76+0.32
93.07+1.15
62.3940.56
39

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
LeCoOp
DuOxOp
Gatlo
DeCcOp
DeCoOp
LeCoOp
DaCoOp
DefTAg
PrompisRc
LeCoOp
AUCROC
PrompiBRC
AUCROC
35.0
Coop
Hgpooop
Nepte
AUCROC
Oellog
AUCROC
GoOp
Prongtore
CoOp
DePT4g
Mapia Deff-igg
kgConp
Gallop
30
KgCodp
KgCoOp
96.0
CLP
CLIP
CLI
72.5
75.8
77.5
68
70
71
72
73
72
60
76
30
84
HM
HM
(a) Average
(b) ImageNet
(c) SUN397
(d) Flowers 102
DeCaOp
LoCoOp
ab
LoCoOp
Oura
Gollap
LoCoOp
DeDuOp
DePTTkg
DeCopp
AUCROC
AUCROC
AUCROC
33
AUCROC
LoCoOp
KgCodp
PromppeRR
CoOp
KgDoOp
32
CoOp
Map
32.5
ProngiBRC
a1
CLPP
Prompi5RC
CoOp
Prompesnc
TOP
92
Go0p
Pp
64
05
96
* 54
68
76
86
93
HM
HM
(c) Caltech101
(f) OxfordPets
StanfordCars
(h) Food101
Figure 7. Tradc-off between the first-stage AUROC metric and the second-stage HM metric on the Openworld Recognition Task.
D.2. Additional Results for Openworld Domain Generalization Task
In this section, we present comprehensive results with standard deviation and also provide stage-wise metrics. The Tab. and Fig.8 once again illustrate that optimizing our can more effectively balance the trade-off between the first-stage detection performance (P1) and the second-stage classification performance (P2). This further underscores the robustness and generalizability of across diverse domains, as well as the efficacy of our proposed approach in addressing the challenges of open-world domain generalization tasks.
DaPT-ig
Oun
LeCoOp
26
AUROC
PrompsRC
AUROG
AUROC
PrompaRR
DePT4g PrompsSRing
PromessRc
DuPT-ig
Maple
DePT.kg
80
CLIP
Mg
64
CoOp
Maple
CoOp
KgCpOp
CLP
KgCoOp-
88
CoOp
TCP
CLI
TCP
KgDoOp
CUP.
TTOP
CoOp
Oaltop
TOP
Napie
60
50
51
63
G4
49
50
52
70
70
HM
HM
HM
HM
(a) ImageNet Sketch
(b) ImageNetV2
(c) ImageNetA
(d) ImageNetR
Figure 8. Trade-off' between the first-stage AUROC metric and the second-stage HM metric on the Openworld domain generalization task.
D.3. Sensitivity Analysis of different metrics in varying domain distributions
Tab. 11 and Tab.12 show the OverallAcc can be dominated by the performance in the domain with more samples. To be specific, the varying new/base ratio is constructed by adjusting sample sizes: when the ratio exceeds (i.e., more new samples), the number of new samples is held constant while reducing base samples; conversely, when the ratio is below 1, base samples remain unchanged and new samples are decreased. Under this setup, when the new/base ratio larger than 1 meaning that there are more new samples, the OverallAcc metric is lower which is dominated by new domain. As the number of base domain samples grows, OverallAcc increases significantly the number of base domain samples grows, while our metric OpenworIdAUC and AUROC, HM remain stable. Since the prior knowledge about domain distribution is generally unknown, distribution--sessiiive metric is improper for OPT.
40

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 10. Quantitative comparisons on ImageNetv2, ImageNet-sketch, ImageNet-A and ImageNet-R. The hest and the runner-up method on each dataset are marked with blue and red, respectively.
ImageNetv2
ImageNet sketch
Method
HM
AUROC
OpenworidAUC
HM
AUROC
OpenworIdAUC
CLIP
64,49:0.00
93.04:0.00
39,49:0.00
49.65=0.00
87.12+0.00
22.57=0.00
CoOp
64.49:0.34
93.39=0.32
39.88=0.40
48.84=0.40
86.05+0.88
21 21.69+0.40
Maple
66.65+0.15
93.80=0.64
42.44=0.23
50.81=0.43
87.81+0.71
23.79+0.37
PromptSRC
66.57=0.48
94.20=0.16
42.55=0.58
51.24+0.54
89.23=0049
24.50+0.50
LoCoOp
62.37=0.78
95.49:0.16
38.45=0.90
46.67=0.50
91.71+0.20
20.88+0.47
KgCoOp
66,3940.24
93.47=0.14
42.14
51.31#0.17
87.51
24.24
DePT-kg
66.1010.21
94.21=0.83
42.45=0.39
50.00+0.41
87.84+1.68
23.9740.15
Gallop
64.1220.20
94.20=0.23
39.8620.28
48.4420.37
87.9340.48
21.57+0.35
DeCoOp
66.51#0.21
95.02=0.11
43.01=0.22
50.91=0.16
91. 02:00200
24.65+0.12
TCP
66.02+0.15
93.37=0.08
41.66+0.20
50.21=0.30
87 $7.02+0.25
23.15+0.31
Ours
67.0240.11
95.75=0.10
43.98:0.19
51.5660.14
91.67+0.21
25.6440.16
ImageNet_A
ImageNet.R
Method
HM
AUROC
OpenworIdAUC
HM
AUROC
OpenworidAUC
CLIP
50.61+0.00
87.23=0.00
23,8340.00
77.30=0.00
91.09+0.00
57.04+0.00
CoOp
51.10=0.45
88.09=0.77
24.22+0.43
77.08=0.68
90.66+0.88
56.58+1 .08
Maple
52.53+1.21
88.93=0.42
25.10=0.57
79.00=0.38
91 91.66+0.56
59.71#0.68
PromptSRC
51.64+0.95
89.27=0.70
25.08=0.91
79.21=0.59
93.16+0.44
60.65+0.95
LoCoOp
48.82+0.59
90.87=0.15 19
22.98=0.56
76.09=0.82
96.88=0,23
57.31
KgCoOp
52.58+0.50
88.22+0.36
25.79=0.49
78.87=0.08
92.01+0.
59.75+0.
DePT-kg
52.06+0.21
91.5110.44
25.84=0.40
78.91#0.12
93 93.88+0.11 14
60.60+0.15 12
Gallop
48.7140.53
88.9220.36
22.28=0.47
77.20=0.33
91.01+0.37
56.78+0.61
DeCoOp
52.21#0.33
90.15=0.23
25.31=0.22
78.31=0.22
96.71+0.18
61.01+0.11
TCP
51.35+0.37
88.01#0.24
24.62+0.35
78.18+0.17
91.31+0.27
58.4640.31
Ours
52.47:0.56
91.63=0.35
26.49=0.63
79.46+0.16
97.57=0.20
62.67+0.27
D.4. Additional Results for Openworld Cross-dataset Generalization Task
We extend our evaluation to investigate the generalization performance of our optimization framework in more challenging cross-dataset open-world scenarios. The comprehensive results of this cross-domain evaluation, which rigorously tests the model's ability to handle both known and new categories across diverse visual domains, are presented in the Tab.13. The experimental results further speak to the efficiency of our method.
D.5. Fine-grained analysis of OpenworIdAUC metric
The OpenworldAUCC comprehensively evaluates: 1) base-to-new detection 2) base classification 3) new classification. A further check
high OpenworldAUC indicates the model performs well three. To diagnose low OpenworldAUC,, three sub-metrics: AUROC, BaseAcc, Acc. To validate this, present fine-grained results on four datasets shown in the table below and Tab. 14, Fig.9, which reveal:
OOD-focused methods (Gallop) excel at BaseAcc-AUROC but struggle with new domain classification
Base-to-nes methods (DePT) show weaker detection performance
Our method achieves better OpenworldAUC, indicating improved trade-off's across three, which further validates the comprehensiveness of OpenworldAUC.
D.6. Complexity analysis
We have included the prompt complexity analysis in Fig.5 in the main text. To highlight the efficiency of our method, we present those numerical results in the 15b. along with additional results on inference speed in the Tab. 16. The Tab.15 compares average performance on three open-wone world tasks and learnable parameter counts across methods. Our method

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table Performance of TYP et al., 2024) on DTD dataset with respect to new/base ratios. clear that the Overall metric
sensitive to the new/base ratio.
The new/base ratio
BaseAcc
NewAcc
HM
AUROC
OverallAcc
OpenworidAUC
10
83.06
55.80
66.75
80.42
40.94
38.62
81.43
55.80
66.22
78.61
43.72
37.75
81.09
55,80
66.11
78.84
45,45
37.85
82.31
55.80
66.51
78.62
49.76
38.24
81.85
55.80
66.36
78,47
54.61
37.92
0.7
81.85
55.72
66.30
78.03
57.40
37.62
0.5
81.85
55.46
66.12
78.47
59.77
37.86
0.3
81.85
53.04
64.37
78.90
63.33
36.21
0.2
81.85
56.02
66.52
78.68
64.94
38.18
0.1
81.85
52.46
63.94
77.69
67.74
35.43
Mean
81.90
55.17
65.92
78.67
54.77
37.57
Variance 0.27
1.66
0.92
0.51
89.13
0.96
Table 12. Performance of TCP(Yao et al., 2024) on Flowers 102 datasct with respect to new/base ratios, It's clear that the Overall metric is sensitive the new/base ratio.
The new/base ratio
BaseAcc
NewAcc
HM
AUROC
OverallAcc
OpenworldAUC
10
96.15
75.76
84.75
92.84
72.77
67.78
97.47
75.76
85.25
94.00
74.25
69.34
96.17
75.76
84.75
93.53
75.18
68.39
97.67
75.76
85.33
94.40
77.59
69.67
97.19
75.76
85.15
94.05
80.97
69.19
0.7
97.19
75.86
85.21
94.09
83.16
69.34
0.5
97.19
75.47
84.96
94.17
84.81
69.01
0.3
97.19
76.07
85.34
94.67
87.71
69.97
0.2
97.19
76.26
85.46
94.39
89.31
70.08
0.1
97.19
75.52
85,00
94.78
91.33
69.59
Mean
97.06
75.80
85.12
94.09
81.71
69.24
Var.
0.25
0.05
0.06
0.32
43.75
0.50
outperforms SOTA methods on the average performance of these with a smaller parameter cost. While recent SOTA methods design deep prompt structures, we optimize multiple shallow prompts detector and classifiers. As shown in Tab. 16, while slightly slower than CoOp due to prompt mixing, our approach runs 34% faster than DeCoOp and matches DePT/Gallop speed, which maintains practical inference speeds.
42

---

OpenworidAUC:: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 13. The OpenworIdAUC empireal results on seven benchmark open-world eross dataset generalization.
Method
AC
C101
Cars
DTD
ES
SUN
UCF
Avg
CLIP
16.21
60.76
45.59
31.13
30.33
44.31
45.83
39.17
CoOp
12.06
62.50
44.67
26.75
25.70
44.72
45.38
37.40
MaPLe
16.36
66.29
47.34
31.89
31.56
48.72
48.74
41.56
PromptSRC
17.75
65.89
49.08
33,94
34.14
49.53
50.06
42.91
KgCoOp
16.15
64.84
47.75
31,46
33.39
48.34
49.40
41.62
DePT-Kg
17.47
66.54
49.98
33,96
35.17
49.38
49.62
43.16
DeCoOp
16.95
66.35
50.04
33,91
36.12
49.42
49.65
43.21
TCP
16.77
65.86
47.60
32.69
33.50
48.81
49.31
42.08
Ours
18.18
66.58
50.25
34.21
36.84
49.65
49.87
43.65
Table 14, Fine- ane-grained results in terms of BascAce, NewAce, HM, AUROC and OpenworIdAUC metrics on Flowers102 ImageNct,
ImageNet V2 and ImageNet-Sketch, comparing OOD-focused method Gallop, base-to-new method DePT, OPT method DeCoOp.
Flowers102
ImageNet
Method
BaseAcc
NewAce
AUROC
OpenworldAUC
BascAcc
NewAce
HM
AUROC
OpenworldAUC
DePT-Kg
97.68
75.38
85.09
92.83
69.46
76.75
69.85
73.14
93.38
$1.35
Gallop
98.60
70.94
82.51
94.50
65.69
79.25
64.02
70.83
95.93
49.01
DeCoOp
93.90
76.24
84.15
96.84
70.28
75.55
70.90
73.15
96.22
$1.98
Ours
96.53
77.16
85.76
96.95
72.79
76.13
70.46
73.19
96.94
$2.64
ImageNet V2
ImageNet-Sketch
Method
BaseAcc
NewAce
AUROC
OpenworldAUC
BascAcc
NewAcc
HM
AUROC
OpenworldAUC
DePT-Kg
70.25
62.42
66. 10
90.21
41.45
46.64
53.88
50.00
86.84
23.37
Gallop
73.20
57.04
64.12
94.20
39.86
49.79
47.16
48.44
87.93
21.57
DeCoOp
70.55
62,91
66.51
95.02
43.01
48.23
53.90
50.91
91.02
24.65
Ours
71.04
63.60
67.02
95.75
43,98
48.77
54.69
51,56
91.67
25.64
D.7. Effect of mutiple pseudo base-to-new Partitions.
Fig. 10 illustrates the effect of using different numbers of base-to-new partitions (K) on the Flowers and SUN397 datasets. The results show that OpenworldAUC increases monotonically as more class partitions are performed. This observation agrees with Thm. 5.2. To purse the tradeoff between the efficiency and the efficacy, choose K 3 in main experiments.
D.8. Effect of Different Shots
Fig. l and 6 illustrate the performance comparison between different shot settings (1-shot, 2-shot, 4-shot, 8-shot and 16-shot) between the proposed method and representative competitors the DTD and OxfordPets datasets. The results show that our method consistently outperforms existing representative approaches across all shot settings, further demonstrating its effectiveness. Additionally, OpenworlAAUC improves as N increases, aligning with our theoretical findings.
D.9. Ablation Studies of the Gating Mechanism
We further explore the effectiveness of the gating mechanism in the Tab. 17. Replacing the sigmoid-weighted gate with fixed 0-1 mask ("Ours 0-1 Gate") slightly improves over removing the gate entirely ("Ours w/o Gate") but under-performs compared to the adaptive sigmoid gate. It validates the effectiveness of sparse sample selection mechanism and the gate approximation mechanism.
D.10. Effect of CE Regularization
In the practical optimization objective (OP the second term is actually an AUC-like ranking loss. However, recent AUC-based optimization studies (Yuan et al., 2022; Yang et al., 2023) show that maximizing the AUC loss from scratch can degrade feature representations. Therefore, we add a cross-entropy regularization term (Xeee) improve discriminability
43

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
BaseAcc
DePT-kg
BaseAcc
DePT-kg
Gallop
Gallop
DeCoOp
DeCoOp
Ours
Ours
NewAcc
OpenworIdAUC
NewAcc
OpenworidAUC
HM
AUROC
HM
AUROC
(a) Flowers 102
(b) ImageNet
BaseAcc
DePT-kg
BaseAcc
DePT-kg
Gallop
Gallop
DeCoOp
DeCoOp
Ours
Ours
NewAcc
OpenworidAUC
NewAcc
OpenworidAAUC
HM
AUROC
HM
AUROC
(c) ImageNet V2
(d) ImageNet Sketch
Figure Fine-grained results on four datasets, Flowers 102, ImageNet, ImageNet ImageNet Sketch, comparing three competitive methods and ours. Our method achieves better OpenworIdAUC, indicating improved trade-off's across three sub-metrics, which further validates the comprehensiveness of OpenworIdAUC.
following (Yang et al., 2023). To assess the impact of X, we carry out sensitivity tests by altering X's value. We present the results for Caltech101 101 and Food101 in the Fig. 12 and Fig.6. A proper E [1/2, 1] can effectively improve the ranking performance.
D.11. A Allation Studies of Mixture-of-Prompe
In order to show the effectiveness of our proposed mixture-of-prompts, we compare its performance with the following two variants of MoP:
w/o Zero Shot new domain classifier h abbreviated as w/o ZS h. This variant of our method replaces the handcrafted prompt for the new domain classifier with tuned prompt based on the base domain.
Single Prompt. This is variant of our method where we only use one prompt to balance three component in R(r, g, h).
The empircal results on ImageNet, StanfordCars, DTD, Flowers 102, SUN397 and UCF101 datasets are provided in Fig.6 and Fig. 13. From these results, we can see that: 1) Traing a new domain classifier on the base domain can lead to overfitting, compromising generalization on unseen new domains. Therefore, w/o ZS h could not outperform our mixture-of oottourp-o--ropppss....o prompts. 2) Single prompt performs significantly worse than mixture prompts. This validates the challenge of optimizing the OpenworIdAUC within single prompt. This strengthens the effectiveness of our proposed mixture-of- -rommpp scheme.
44

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 15. The comparisons of average performance open- open-world tasks and learnable parameter counts across methods.
Method
Recognition task
Imbalanced recognition task
Domain adaption
#param
CLIP 47.83
54.59
47.31
CoOp 47.59
53.97
48.93
8.2k
Maple
57.35
63.05
51.89
3555.
PromptSRC
58.21
65.44
52.44
46.1k
LoCoOp
52.62
60.65
45.12
8.2k
KgCoOp
55.07
62.71
51.45
8.2k
DePT-Kg
59.00
66.06
51.35
292.9k
Gallop
56.90
63.67
49.01
606.2k
DeCoOp
58.77
66.87
51.98
30.7K
TCP
58.90
65.13
51.34
331.9k
Ours
60.94
68.84
52.64
26.6k
Table 16. Average per-sample inference time comparison across ten datasets (ImageNet excluded) in the open-world recognition task.
CoOp
DePT
Gallop
DeCoOp
Ours
0.001 17S
0.00167S
0.00148S
0.00273S
0.00180S
6
K4
K
3-
2
71.5
72.0
72.5
73.0
73.5
74.0
56
57
58
59
OpenworidAUC
OpenworldAUC
(a) Flowers102
(b) SUN397
Figure 10. The Effeet Using Different Number of basc-to-new Partitions (K).
DeCoOp
OpenworldAUGy (%)
39
TGP
88
35
86
31
1
2
16
16
Shots
Shots
(a)
DTD
(b) OxfordPets
Figure 11. Comparison of few-shot learning with 1, 16-shot samples DTD and OxfordPets.
45

---

OpenworIdAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Table 17. The ablation study of the gating mechanism.
Method
Avg.
IN
C101
Pets
Cars
F102
Food
AC
SUN
DTD
ES
UCF
Ours w/o Gate
60.29
52.49
92.56
89.47
55.06
72.59
79.30
10.97
56.96
40.63
51.27
61.85
Ours 0-1 Gate
60.65
52.61
92.77
89.50
55.20
72.71
79.92
11.08
57.13
40.72
52.78
62.75
Ours Sigmoid Gate
60.94
52.64
92.81
89.77
55.31
72.79
81.25
11.42
58.54
40.37
53.09
62.39
OpenvortdAUcco (Ha
92.75-
OpenwortdAUc
02.50
92.25
92.00
0.3
0.5
0.7
1.5
0
0.3
0.5
0.7
1.5
Lambda
Lambda
(a) Caltech101
(b) Food101
Figure 12. The Sensitive Analysis of A.
75
56
40
70-
35
65
60
48
50
20
single prompt 25
GMoP
single prompt
GMoP
single prompt Z5
GMoP
(a) StanfordCars
(b) DTD
(c) Flowers
60
60
52
50
50
48
single prompt
single prompt
single prompt
(d) ImageNet
(e) SUN397
(1) UCF101
Figure 13. The ablation study of mixture-of-promple
46


Reviews:
[{"review_id": "ha3dJipWhd", "reviewer": "Reviewer_fECn", "summary": "Since existing evaluation metrics cannot comprehensively assess performance in open-world prompt tuning, this paper proposes a unified evaluation metric called OpenworldAUC. This metric not only measures the detection capability of base/new samples (P1) and classification accuracy (P2), but also ensures robustness against changes in the proportion of base to new samples. Additionally, a multi-prompt combination method based on a gating mechanism is proposed to optimize the OpenworldAUC metric.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments! Due to space constraints, we include full tables and figures https://anonymous.4open.science/r/R-4D06. References to Tab.X, Fig.X  correspond to those provided in this link.\n\n> Q1: OpenworldAUC for fine-grained model capability analysis\n\nThe OpenworldAUC comprehensively evaluates: 1) base-to-new detection 2) base classification 3) new classification. A high OpenworldAUC indicates the model performs well on three. To diagnose low OpenworldAUC, we can further check three sub-metrics: AUROC, BaseAcc, NewAcc. To validate this, we present fine-grained results on four datasets shown in the table below and Tab.1, Fig.1, which reveal:\n- OOD-focused methods (Gallop) excel at BaseAcc-AUROC but struggle with new domain classification\n- Base-to-new methods (DePT) show weaker detection performance\n- Our method achieves better OpenworldAUC, indicating improved trade-offs across three, which further validates the comprehensiveness of OpenworldAUC.\n|F102|Baseacc|Newacc|AUROC|OpenworldAUC|\n|-|-|-|-|-|\n|DePT|97.68|75.38|92.83|69.46|\n|Gallop|**98.60**|70.94|94.50|65.69|\n|DeCoOp|93.90|76.24|96.84|70.28|\n|Ours|96.53|**77.16**|**96.95**|**72.79**|\n\n> Q2: How to ensure the quality of prompts\n\nThe prompts of base classification and base-to-new detection can be **automatically optimized** via our carefully designed loss function, which is a paradigm widely validated in existing prompt tuning research. Additionally, the prompt of new classification is hand-crafted to alleviate overfitting on the base training set. Unlike the naive prompt template \"a photo of {}\", in our initial submission, we choose a more informative prompt template based on the base domain, further ensuring the new-domain classification performance, shown in the table below and Tab.2.\n|Prompt for ImageNet|NewAcc|OpenworldAUC|\n|-|-|-|\n|a photo of {}|68.12|51.43|\n|Prompt ensemble:a {} in a video game.art of the {}.a photo of the small {}...|70.46|52.64|\n\n> Q3:  Computational complexity of GMoP\n\nIn fact, we have included the prompt complexity analysis in Fig.5 in the initial version. To highlight the efficiency of our method, we present those numerical results in the table below and Tab.3, along with additional results on inference speed.\n\n- Our method outperforms SOTA methods on the average performance of three open-world tasks with a smaller parameter cost. While recent SOTA methods design **deep** prompt structures, we optimize **multiple shallow** prompts in detector and classifiers.\n\n||Recognition task|Domain adaption|#param(k)|\n|-|-|-|-|\n|CoOp|47.59|48.93|8.2|\n|Gallop|56.90|49.01|606.2|\n|Maple|57.35|51.89|3555.1|\n|DeCoOp|58.77|51.98|30.7|\n|DePT|59.00|51.35|292.9|\n|Ours|**60.94**|**52.64**|26.6|\n\n- We measure inference speed by comparing average processing times per sample across ten datasets, testing representative methods and ours. While slightly slower than CoOp due to prompt mixing, our approach runs 34% faster than DeCoOp and matches DePT/Gallop in speed(S)\n|CoOp|DePT|Gallop|DeCoOp|Ours|\n|-|-|-|-|-|\n|0.00117|0.00167|0.00148|0.00273|0.00180|\n\n>Q4: The choice of multiple loss function weights across different datasets\n\nIt's a pity that our method is not fully understood. In fact, our optimization framework involves only one loss weight λ of the CE regularization added to the AUC loss, which has been discussed in App.E.6 in the initial version. Sensitivity tests across four datasets show consistent performance when λ∈[1/2,1], with SOTA results in this range, shown in the table below and Tab.4\n||C101|SUN|\n|-|-|-|\n|DePT|92.74|56.42|\n|DeCoOp|92.72|57.00|\n|λ=1/4|92.64|57.59|\n|λ=1/2|92.96|58.92|\n|λ=3/4|92.88|58.86|\n|λ=1|92.81|58.54|\n\n> Q5: The generalization of pseudo base-new partition\n\nThe foundation model itself has strong base-to-new generalization ability. Our task is to enhance such ability by leveraging base training data. To this end,  we adopt the following partition strategy to simulate new class detection.\n\n- We perform **multiple** (K) base-to-new partitions to ensure statistical stability of the new class simulation \n\n- We ensure K pseudo base classes **fully cover** the base class\n- Thm.5.2 suggests increasing K reduces such approximation error. Experiments on two datasets confirm this shown in the table below and Tab.5. We usually set K=3 to balance performance and efficiency\n|SUN397|AUROC|OpenworldAUC|\n|-|-|-|\n|K=1|84.45|53.16|\n|K=2|87.57|56.14|\n|K=3|90.70|58.54|\n|K=4|91.02|58.63|\n|K=5|91.25|58.71|\n\n> Q6: Imbalanced sampling of base/new data may affect detection optimization\n\nTest Imbalance: Since true new-class data is unavailable during training, test imbalance doesn't impact optimization\n\nPseudo Imbalance: We sample pseudo base/new pairs from true base for AUC loss training. AUC's distribution insensitivity ensures robustness to pseudo base-to-new imbalance, validated by stable performance with varying ratios on DTD\n|b/n ratio|AUROC|OpenworldAUC|\n|-|-|-|\n|2:1|78.58|40.37|\n|3:1|78.30|40.32|\n|5:1|78.45|40.39|", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and/or evaluation criteria are sensible for the problem at hand.", "theoretical_claims": "Yes, I checked the correctness of all proofs for the theoretical claims, and they are correct.", "experimental_designs_or_analyses": "Yes, I checked the soundness and validity of all experimental designs and analyses, and they are appropriate and well-executed.", "relation_to_broader_scientific_literature": "The key contributions of this paper are closely related to the broader scientific literature on evaluation metrics for open-world recognition tasks. Existing metrics, such as HM, Overall Accuracy, and AUROC, have been widely used but suffer from specific limitations. These metrics fail to simultaneously address three critical requirements: (P1) distinguishing between base and new classes, (P2) ensuring correct classification, and (P3) adapting to varying class distributions. Building on these prior findings, the authors propose a novel evaluation metric, OPEB, which aims to address these limitations and provide a more comprehensive framework for evaluating open-world recognition systems.", "essential_references_not_discussed": "No, the paper cites all essential references necessary to understand its key contributions. The authors have adequately discussed prior related findings and provided a comprehensive context for their work.", "other_strengths_and_weaknesses": "Strengths:\nInnovation: This paper proposes a new evaluation metric, OpenworldAUC, to overcome the limitations of existing evaluation methods.\nMethod effectiveness: The GMoP method introduced in this paper optimizes detection and classification through multi-prompt combinations, achieving outstanding performance on multiple state-of-the-art tasks.\nTheoretical support: Through theoretical reasoning, the paper demonstrates that OpenworldAUC offers more stable evaluations and that the GMoP training objective exhibits strong generalization.\nExperimental comprehensiveness: The results from testing on 15 datasets and various ablation experiments validate the robustness of this approach.\n\nWeaknesses:\nData distribution sensitivity: Imbalanced sampling of base/new data may affect detection optimization.  \nLimited generalization ability: Unseen new class data may lead to overfitting on the base class, impacting detector performance.  \nHigh optimization difficulty: Manual adjustment of loss weights is required, and the approach is sensitive to hyperparameters.  \nHigh computational complexity: Multiple prompts are still needed during inference, making it unsuitable for low-compute devices.", "other_comments_or_suggestions": "No.", "questions_for_authors": "Based on the \"Weaknesses\" section, I have the following concerns:\n1. Establishing a fair and objective metric can reflect the true performance of a model and reveal its current limitations. However, can the proposed OpenworldAUC in this paper provide a fine-grained analysis of the model's various capabilities and effectively uncover its limitations?\n2. The quality of the prompt determines the model's classification performance. If the prompt is poorly designed, classification performance may degrade. How does the author ensure the quality of the prompts?\n3. The GMoP method still requires multiple prompts during the inference stage. Does this introduce additional computational and storage overhead?\n4. GMoP requires manual tuning of multiple loss function weights. How does the author ensure the optimal weights across different datasets?\n5. Since the \"new\" class is unknown during training, the paper simulates this through a pseudo base-to-new partitioning approach. How does the author ensure the accuracy of this partitioning method?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hJAcR330Mw", "reviewer": "Reviewer_PcZP", "summary": "This paper addresses the open-world prompt tuning problem and uncovers the fundamental limitations of current evaluation metrics in this field. To tackle this challenge, the authors propose a novel, unified metric, OpenWorldAUC, which jointly evaluates the model’s detection and classification performance without requiring prior domain knowledge. On top of this, a Gated Mixture-of-Prompts (GMoPs) approach is introduced to optimize OpenWorldAUC directly. Both theoretical analyses and empirical studies consistently demonstrate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments, and we would like to make the following response.\n\n> **Q1:** The necessity of introducing the Gated Mixture-of-Prompts remains unclear. A more detailed explanation is needed to justify this design choice.\n\nThe three components $g$, $h$, and $r$ have **conflicting** objectives: $g$ classifies base samples (its objective function relies on base samples), $h$ classifies new samples (its objective function uses new samples), and $r$ ranks base-new pairs (its objective function uses base-new sample pairs). Using a single prompt leads to **mutual interference**. To address this, our design assigns distinct prompts to each component, separating their optimization processes. A gate mechanism, which uses sigmoid-weighted confidence scores, adaptively combines their outputs. This ensures that the $r$-prompt focuses on ranking correctly classified pairs. By doing so, we prevent conflicts and allow each prompt to specialize for its task. Empirical results from six datasets confirm the difficulty of optimizing OpenworldAUC with a single prompt and demonstrate the effectiveness of our mixture-of-prompts strategy.\n\n||ImageNet|SUN397|DTD|Cars|UCF101|Flowers102|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Single Prompt|50.75|48.47|23.59|47.03|49.88|50.79|\n|GMoP|52.64|58.54|40.19|55.31|62.39|72.79|\n\n> **Q2:** How is ( OP_0 ) derived from Proposition 5.1? Additionally, why is it valid to approximate the 0-1 loss using the square loss? I believe the authors should provide a clearer explanation, along with relevant references, to aid readers who may be less familiar with this topic.\n\n($OP_0$) is derived by replacing the population risk in Proposition 5.1 with its **empirical approximation** (since $\\mathcal{D}$ is unknown). Besides, following the framework of surrogate loss, we replace the non-differential 0-1 loss with a convex loss function $\\ell$, such that $\\ell(t)$ is an upper bound of $\\ell_{0,1}(t)$. Note that if the scores live in $[0,1]$, standard loss functions such as $\\ell_{sq}(t) = (1-t)^2$ often satisfy this constraint. This smooth approximation enables gradient-based optimization while preserving ranking semantics. The details can also be found in the recent survey [1,2,3].\n\n> **Q3:** The motivation behind the zero-shot new domain classifier is not well articulated. A more thorough discussion would help clarify its role in the overall framework.\n\nThank you for your constructive suggestion!  The motivation behind the zero-shot new domain classifier falls into the following two aspects.\n\n**Balance efficiency and accuracy:** Recent prompt-tuning methods show that learnable prompts, optimized on the base domain with only CE loss, may hurt new-domain classification performance compared to zero-shot CLIP with fixed prompts. The results on the NewAcc of the Zeroshot CLIP and the PT baseline CoOp confirm this point, as shown in the table below.\n\n|  | ImageNet | SUN397 |  DTD  | Cars | UCF101 | Oxford_flowers |\n| :-----------: | :------: | :----: | :---: | :----------: | :----: | :------------: |\n| Zeroshot-CLIP | **68.10** | **75.62** | **60.51** |    **75.02** | **78.64** |     **77.19**  |\n|   CoOp    |  67.03   | 71.28  | 40.19 |    56.37     | 52.96  |     66.57      |\n\nTo improve generalization, many recent methods focus on **maintaining alignment** (loss design) with zero-shot CLIP while using **structured prompts** (structure design) to preserve zero-shot knowledge. These methods slightly outperform zero-shot CLIP in term of new domain accuracy but introduce additional computational and storage overhead. To purse a tradeoff between the, we just design a hand-crafted prompt.\n\n**Decoupling Classification in different domains:** Since the base-to-new detector effectively distinguishes base and new samples in open-world scenarios, we can **decouple** the learnable prompt $\\theta_g$ (for base-domain classification) and the fixed prompt  $\\theta_{h}^*$(for new-domain classification) and achieve promising performance. In other words, using different prompt parameters for classification task in different domains is feasible and effective in practical scenarios.\n\nTo further validate the effectiveness of zero-shot new domain classifier, we replace $\\theta_{h}^*$ with $\\theta_g$ and observe the overall OpenworldAUC performance drops, confirming the necessity of this decoupling and the effectiveness of the zero-shot new domain classifier.\n\n||ImageNet|SUN397|DTD|StanfordCars|UCF101|Oxford_flowers|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Ours w/o ZS h|51.78|55.27|34.77|48.54|53.55|62.76|\n|Ours|**52.64**|**58.54**|**40.19**|**55.31**|**62.39**|**72.79**|\n\n> [1] Auc maximization in the era of big data and ai: A survey. ACMComputing Surveys (CSUR), 2022.\n>\n> [2] On the consistency of AUC pairwise optimization. IJCAI, 2015.\n>\n> [3] Stochastic auc maximization with deep neural networks. ICLR,2019.", "claims_and_evidence": "The claims made in the work are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "This study presents a well-motivated and rigorous framework to open-world prompt tuning.", "theoretical_claims": "I have carefully checked the proofs of theoretical arguments including Prop.4.2, Prop.4.3, Prop.5.1 and Thm.5.2, and confirmed their correctness.", "experimental_designs_or_analyses": "Yes. The experiment designs of this paper follow the standard evaluation setups in this area and the authors compare the proposed method/metric with 10 recent state-of-the-art methods. Overall, the empirical studies are convincing and comprehensive.", "relation_to_broader_scientific_literature": "This paper borrows the ideas from open-world recognition, prompt tuning, and mixture-of-experts models, contributing a novel perspective on evaluation and optimization in open-world prompt learning. The proposed method has broad application potential in various AI-driven scenarios.", "essential_references_not_discussed": "No. The literature included in this paper is satisfactory.", "other_strengths_and_weaknesses": "Novel Open-world metric for reliable evaluations: Existing methods assess model performance separately on base and unseen domains, resulting in inconsistent evaluations. This work provides a systematic analysis to reveal the limitations of existing metrics and proposes a novel metric, OpenworldAUC, to address these shortcomings. Furthermore, an end-to-end algorithm is developed that enables the model to learn directly from OpenworldAUC. This approach establishes a more comprehensive and reliable evaluation framework, thereby facilitating more effective model optimization in open-world scenarios.\n\nSufficient theoretical guarantee: This paper conducts theoretical analysis of the proposed learning framework, which provides an interesting perspective to understand the work mechanism for OpenworldAUC. The results show that optimizing OpenworldAUC could lead to a satisfactory generalization performance. \n\nComprehensive experiments: The authors perform extensive experiments to showcase the advantages of the proposed method, including 11 benchmarks and 10 competitors. Additionally, ablation studies are presented to compare the effectiveness of OpenWorldAUC against previous counterpart metrics. Empirical results consistently show that maximizing OpenWorldAUC leads to superior performance in both open-world recognition and open-world domain generalization tasks.\n\n\nHowever, I have the following minor concerns:  \n\n1. The necessity of introducing the Gated Mixture-of-Prompts remains unclear. Why are the inputs to each component distinct? A more detailed explanation is needed to justify this design choice.  \n\n2. How is \\( OP_0 \\) derived from Proposition 5.1? Additionally, why is it valid to approximate the 0-1 loss using the square loss? I believe the authors should provide a clearer explanation, along with relevant references, to aid readers who may be less familiar with this topic.  \n\n3. The motivation behind the zero-shot new domain classifier is not well articulated. Why is \\theta_{h}^* necessary, and what is the underlying intuition for its inclusion? A more thorough discussion would help clarify its role in the overall framework.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please refer to the Weaknesses part.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uQAe2xS09Q", "reviewer": "Reviewer_Df2f", "summary": "This paper explore a new evaluation metric openworldauc for the practical open-world prompt tuning (OPT) task, which jointly measure the inter-domain detection and intra-domain classification performance and remains insensitive towards the varying data distributions. Further, the mixture-of-prompt learning framework GMoP is proposed to optimize openworldauc during training. Besides, generalization analysis are conduced to support the method theoretically. Comprehensive experiments on various benchmark show the  advantages of the proposed metric and the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:\n\n> **Q1:** More detailed explanation of the challenges associated with generalization analysis.\n\n- Our main theoretical findings focus on how well our optimization method generalizes to **unseen** test data distributions with **new classes**. According to standard learning theory, we measure generalization error by comparing: (a) the expected error across the joint distribution of the real new domain $\\mathcal{Y}$ and test data $\\mathcal{D}$, and (b) the empirical average across the pseudo new domain $\\hat{\\mathcal{Y}}$ and training data $\\mathcal{S}$. To construct the pseudo new domain, we perform $K$ pseudo base-to-new partitions $\\hat{\\mathcal{Y}}^{(k)}, k \\in [1, K]$, which incorporates a hierarchical sampling approach, class sampling, and data sampling. This leads to a **hierarchical** structure of stochastic errors, presenting a significant challenge in our theoretical analysis.\n- The another major challenge arises from the AUC-type risk $\\ell_{sq}(r(x_b;\\theta_r) - r(x_n;\\theta_r))$ in optimizing the detector $r$. The standard generalization analysis techniques, such as Rademacher complexity-based theoretical arguments[1,2], require the loss function to be expressed as the sum of independent terms. Unfortunately, the pairwise AUC-type risk can not satisfy this assumption. For instance, the optimization functions for the detector, $\\ell_{sq}(r(x_b^i;\\theta_r) - r(x_n^j;\\theta_r))$ and $\\ell_{sq}(r(\\tilde{x}_b^{i};\\theta_r) - r(\\tilde{x}_n^j;\\theta_r))$, are interdependent if any term is shared (e.g., $\\tilde{x}_n^j = x_n^j$ or $\\tilde{x}_b^i = x_b^i$). To this end, in this study, we use **covering numbers** and **$\\epsilon$-net arguments** [3] in the subsequent proof to derive the generalization bound.\n\n> **Q2:** The authors should elaborate further on the calculation of the base-domain confidence score  $r$.\n\nFollowing the setting in [4], the new domain class names are known during testing in the OPT task and the base-domain confidence score is derived from the maximum probability over the base domain. A high value of $r$ means the high probability of the sample belonging to the base domain. Given the `image_features` [1,512] and `text_features` [512,C] where 512 means  the dimension of the latent feature and $C$ is the number of all classes, we calculate the  `prob = Softmax(image_features·text_features,dim=1)` and then base domain confidence score can be obtained via  $\\max_{1\\le j\\le C_b}\\textbf{prob}[:,j]$, where $C_b$ is the number of base classes.\n\n> **Q3:** The ablation studies of  the gating mechanism. \n\nFollowing your suggestion, we further explore the effectiveness of the gating mechanism. Replacing the sigmoid-weighted gate with a fixed 0-1 mask (\"Ours 0-1 Gate\") slightly improves over removing the gate entirely (\"Ours w/o Gate\") but under-performs compared to the adaptive sigmoid gate. It validates the effectiveness of sparse sample selection mechanism and the gate approximation mechanism.\n\n| Method | Avg.  | IN      | C101      | Pets  | Cars  | F102  | Food  | AC    | SUN   | DTD   | ES    | UCF   |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n| Ours w/o Gate     | 60.29     | 52.49     | 92.56     | 89.47 | 55.06 | 72.59 | 79.3  | 10.97 | 56.96 | 40.63 | 51.27 | 61.85 |\n| Ours 0-1 Gate     | 60.65     | 52.61     | 92.77     | 89.50 | 55.20 | 72.71 | 79.92 | 11.08 | 57.13 | **40.72** | 52.78 | 62.75 |\n| Ours Sigmoid Gate | **60.94** | **52.64** | **92.81** | **89.77** | **55.31** | **72.79** | **81.25** | **11.42** | **58.54** | 40.37 | **53.09** | **62.39** |\n\n> [1]Rademacher and Gaussian Complexities: Risk bounds and structural results. COLT, vol. 2111, pp. 224–240, 2001.\n>\n> [2]Foundations of Machine Learning. MIT Press, 2012. \n>\n> [3]Probability in Banach Spaces: Isoperimetry and processes. 1991.\n>\n> [4]DECOOP: Robust Prompt Tuning with Out-of-Distribution Detection. ICML 2024.", "claims_and_evidence": "The submission provides well-substantiated claims supported by both theoretical and empirical evidence. \n\n- The paper provides sufficient arguments to analyze the limitations of existing metrics and demonstrates the superior properties of the proposed OpenworldAUC metric from both theoretical and empirical perspectives.\n- The paper also demonstrates the effectiveness of the proposed GMoP method through a novel generalization bound and comprehensive empirical studies.", "methods_and_evaluation_criteria": "The proposed evaluation metric and the proposed learning framework make sense for the open-world learning at hand. To be specific,\n\n- Theoretical results show that the proposed openworldauc is consistent with the goal of OPT while escape from all identified limitations. In light of this, this metric can evaluate models more robustly and guide model optimization more effectively, thereby providing valuable insights to the open-world learning community and advancing the field. \n\n- To purist of this, the proposed mixture-of-prompt framework  co-optimizes the prompts for the detector and classifier using a sparse gating mechanism. This innovative approach offers new insights and design principles for optimizing models in open-world scenarios.", "theoretical_claims": "I generally review the correctness of the proofs for several theoretical claims within the manuscript, which include:\n\n- The proofs related to metric analysis, corresponding to Proposition 4.1, Proposition 4.2, and Proposition 4.3.\n- The proof for Proposition 5.1.\n- The proof related to the generalization bound.\n\nOverall, the theoretical analysis is both correct and intuitively sound. In particular, the proof of the generalization bound demonstrates ingenuity, as the authors decompose the complex generalization gap into several parts and analyze the source of each error term, resulting in an informative and illustrative generalization bound.\n\nIn the context of the generalization bound for detector optimization, the authors use covering number and ϵ*ϵ*-net arguments to derive the generalization bound. My minor concern is the challenge of this analysis. The authors should provide more explanation for why traditional Rademacher Complexity-based theoretical analysis cannot be directly applied here.", "experimental_designs_or_analyses": "I review the experimental setup, results, and analysis in both the main text and appendix. In my view, the experimental setup in this paper is well-justified and the results are convincing. Notably, the authors first incorporate the additional experiments on distribution imbalance compared to previous research.  The experiments further validate the comprehensiveness of OpenworldAUC and its insensitivity to imbalanced data distributions. The lightweight learning framework outperforms competitors in OpenworldAUC and achieves a superior balance between the detection metric AUC and the classification metric HM, confirming its effectiveness.\n\nHowever, some details regarding the calculation of the detection score  $r(\\cdot)$  in the main text could be improved. The authors mention in the appendix that the calculation of  $r(\\cdot)$ can utilize new class names in this experimental setting for all competitors. I recommend that the authors emphasize this implementation detail in the main text.", "relation_to_broader_scientific_literature": "Towards the practical and challenging Openworld prompt tuning (OPT) task, prior methods adopt the HM metric , AUROC metric and the Overall-accuracy metric to evaluate the model, here, the authors argue that these metrics suffer from three types of limitations. To this end, this paper proposed the novel metric openworldauc and the corresponding empirical learning framework, which are the key contributions to the openworld learning and prompt tuning communities.", "essential_references_not_discussed": "The paper covers most of the essential related works in the field.", "other_strengths_and_weaknesses": "The strengths of this paper are as follows:\n\n- This paper provides a systematic analysis of the limitations of existing metrics for the OPT task. The proposed OpenworldAUC metric, which features a concise formulation, effectively addresses these limitations and demonstrates desirable properties from both theoretical and empirical perspectives.\n- The paper introduces a novel and unified learning framework designed to optimize the OpenworldAUC metric. This framework dynamically balances multiple prompts targeting specific goals through a sparse gating mechanism. To support this approach, the authors derive a novel and informative bound, a contribution rarely explored in prior literature.\n- The empirical results are convincing. They highlight the limitations of the existing accuracy metric and underscore the advantages of OpenworldAUC. Additionally, the results validate the effectiveness of the proposed learning framework.\n\nMy minor concerns are as follows:\n\n- The authors should provide a more detailed explanation of the challenges associated with generalization analysis in the main text.\n- The authors should elaborate further on the calculation of the base-domain confidence score \\( $r$​ \\).\n- The authors should supplement the ablation studies by investigating the performance of the gating mechanism. Specifically, it would be valuable to examine how the model performs when the gating mechanism is removed and replaced with a simple binary 0-1 mask for selecting correctly classified samples.\n\nOverall, I hold a clear positive view of the proposed novel metric and the corresponding empirical learning framework. I believe this paper makes significant contributions to the open-world learning and prompt-tuning community and meets the standards of ICML.", "other_comments_or_suggestions": "Please see the weakness part above.", "questions_for_authors": "Please see the weakness part above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DgaIKhISL2", "reviewer": "Reviewer_uVwC", "summary": "The paper introduces OpenworldAUC, a unified evaluation metric for open-world prompt\ntuning (OPT) that jointly assesses base-to-new detection (P1), domain-specific classification\n(P2), and insensitivity to domain distribution (P3). To optimize OpenworldAUC, the authors\npropose Gated Mixture-of-Prompts (GMoP), which employs domain-specific prompts and a\ngating mechanism to balance detection and classification. Theoretical guarantees for\ngeneralization are provided, and experiments on 15 benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your time and effort in providing us with such constructive comments. We would like to respond to them as follows:\n\n> Q1: Generalization to truly unseen domains, e.g., cross-dataset evaluation.\n\nFollowing your insightful suggestion, we extend our evaluation to investigate the generalization performance of our optimization framework in more challenging cross-dataset open-world scenarios. The experimental settings are listed as follows:\n\n- We train our prompt exclusively on the ImageNet-500 base domain and subsequently evaluate model performance on a combined test set containing both the original ImageNet-500 classes and new categories from seven external datasets: `FGVC-Aircraft`, `Caltech-101`, `Stanford-Cars`, `DTD`, `EuroSAT`,  `SUN397`, and `UCF101`.\n- To ensure a fair evaluation of open-world generalization, we meticulously remove any categories from these external datasets that overlapped with the ImageNet-500 class space before evaluation.\n\nThe comprehensive results of this cross-domain evaluation, which rigorously tests the model's ability to handle both known and new categories across diverse visual domains, are presented in table below. The experimental results further speak to the efficiency of our method.\n\n||AC|C101|Cars|DTD|ES|SUN|UCF|Avg|\n|-|-|-|-|-|-|-|-|-|\n|CLIP|16.21|60.76|45.59|31.13|30.33|44.31|45.83|39.17|\n|CoOp|12.06|62.50|44.67|26.75|25.70|44.72|45.38|37.40|\n|MaPLe|16.36|66.29|47.34|31.89|31.56|48.72|48.74|41.56|\n|PromptSRC|17.75|65.89|49.08|33.94|34.14|49.53|**50.06**|42.91|\n|KgCoOp|16.15|64.84|47.75|31.46|33.39|48.34|49.40|41.62|\n|DePT-Kg|17.47|66.54|49.98|33.96|35.17|49.38|49.62|43.16|\n|DeCoOp|16.95|66.35|50.04|33.91|36.12|49.42|49.65|43.21|\n|TCP|16.77|65.86|47.60|32.69|33.50|48.81|49.31|42.08|\n|Ours|**18.18**|**66.58**|**50.25**|**34.21**|**36.84**|**49.65**|49.87|**43.65**|\n\n> Q2: The trade-offs between **performance** and **complexity** of mixture-of-prompts.\n\nThank you for your constructive suggestion! In fact, we have included the prompt complexity analysis in Fig.5 in the initial submission. To highlight the efficiency of our method, we present those numerical results in the table below, along with additional results on inference speed.\n\n- The table compares average performance on three open-world tasks and learnable parameter counts (#param) across methods\n\n|Method|Recognition task | Imbalance recognition|Domain adaption | #param |\n|-|-|-|-|-|\n|CLIP|47.83|54.59|47.31|0|\n|CoOp|47.59|53.97|48.93|8.2k|\n|Maple|57.35|63.05|51.89|3555.1K|\n|PromptSRC|58.21|65.44|52.44|46.1k|\n|DePT|59.00|66.06|51.35|292.9k|\n|Gallop|56.90|63.67|49.01|606.2k|\n|DeCoOp|58.77|66.87|51.98|30.7K|\n|TCP|58.90|65.13|51.34|331.9k|\n|Ours|**60.94**|**68.84**|**52.64**|26.6k|\n\n- The table presents a comparison of per-sample inference time, averaged across ten datasets\n\n|CoOp|DePT|Gallop|DeCoOp|Ours|\n|-|-|-|-|-|\n|0.00117S|0.00167S|0.00148S|0.00273S|0.00180S|\n\nAccording to these empirical results, we answer the questions raised by the reviewer:\n\n- Our method outperforms SOTA methods on the average performance of three open-world tasks with **a smaller parameter cost**. While recent SOTA methods design **deep** prompt structures, we optimize **multiple shallow** prompts in the detector and classifiers.\n\n- While slightly slower than CoOp due to prompt mixing, our approach runs 34% faster than DeCoOp and matches DePT/Gallop in speed, which maintains practical inference speeds.\n- **The performance gain outweighs complexity**. Our method outperforms SOTA methods by **1.94% –13.55%** across tasks while using **≤ 9% parameters** of methods like Maple or DePT. The moderate speed trade-off (slightly slower than CoOp) is justified by significant performance improvements.\n\n> Q3: Fairness-sensitive real-world scenarios application of the proposed method\n\nThe **cross-dataset task** mentioned above involves a significant **imbalance between the base and new domains**, as shown in the table below. The base domain includes 500 ImageNet categories with 25,000 testing samples.\n\n|New domain|#categories|class-imbalance|#samples|sample-imbalance|\n|-|-|-|-|-|\n|StanfordCars|196|2.6|8041|3.1|\n|UCF101|101|5.0|3783|6.6|\n|Caltech101|84|6.0|2135|11.7|\n|DTD|47|10.6|1692|14.8|\n|EuroSAT|10|50.0|8100|3.1|\n|Sun397|392|1.3|19600|1.3|\n|FGVC_aircraft|100|5.0|3333|7.5|\n\nAccuracy-driven metrics often favor the majority domain of ImageNet, creating fairness risks in scenarios with extreme data imbalances (e.g., 15× sample or 50× class ratios). These imbalances mirror real-world cases where critical minority samples are suppressed by dominant base classes. OpenworldAUC overcomes this bias by fairly evaluating all classes through pairwise ranking. Unlike traditional metrics that amplify imbalance-related errors, OpenworldAUC improves reliability in critical applications.", "claims_and_evidence": "The authors provide a detailed analysis of the limitations of existing metrics, such as HM,\nOverallAcc, and AUROC, and demonstrate how OpenworldAUC overcomes these limitations.\nThe effectiveness of the proposed GMoP framework is supported by extensive experimental\nresults on multiple benchmarks.", "methods_and_evaluation_criteria": "Methods: OpenworldAUC: The pairwise formulation (jointly ranking detection and\nclassification correctness) is novel and aligns well with OPT requirements. The use of\ndomain-specific prompts and gating mechanisms is logical for balancing conflicting sub\u0002objectives. The pseudo partition strategy and zero-shot classifier for new domains address\npractical constraints (unseen classes).\nEvaluation Criteria: Benchmarks cover diverse scenarios (recognition, domain generalization,\nimbalance), and metrics include OpenworldAUC, HM, AUROC, and OverallAcc. However,\ncomputational efficiency is not discussed.", "theoretical_claims": "The paper presents theoretical guarantees for the generalization of the GMoP framework.\nWhile the proofs are not provided in the main text, the authors reference the appendix for\ndetailed proofs.", "experimental_designs_or_analyses": "The experimental designs and analyses are well-structured and valid. The authors evaluate\nthe proposed method on a diverse set of benchmarks, including open-world recognition and\ndomain generalization tasks. The experiments are designed to test the method's\nperformance under various conditions, such as different domain distributions and imbalance\nsettings.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\nThe paper is well-structured and clearly presents the problem, the proposed solution, and\nthe experimental results.\nWeaknesses:\nWhile the paper discusses the potential impact on fairness-sensitive real scenarios, it lacks\nspecific examples or case studies demonstrating the application of the proposed method in\nreal-world settings.", "other_comments_or_suggestions": "The implementation details section could benefit from more specific information about the\ntraining strategy and hyperparameters used for each competitor.", "questions_for_authors": "How does the pseudo partition strategy generalize to truly unseen domains (e.g., cross\u0002dataset evaluation)?\n\nWhile the paper discusses the potential impact on fairness-sensitive real-world scenarios, it lacks specific examples or case studies demonstrating the application of the proposed method in real-world settings. Could the authors provide concrete examples or case studies that illustrate how the proposed method can be applied in such scenarios?\n\nThe Gated Mixture-of-Prompts (GMoP) framework introduces multiple prompts and a gating mechanism. Is this approach practical in terms of computational cost and implementation complexity? \n\nHow does the performance gain from GMoP compare to the increased complexity? \n\nGiven that the framework may be too complex for real-world applications, especially in scenarios with limited computational resources, could the authors provide a detailed analysis of the trade-offs between performance and complexity?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: A new unified evaluation metric, OpenWorldAUC, is proposed for open-world prompt tuning, which combines detection ability, classification accuracy and robustness to varying proportions of base and new samples. The paper received three detailed reviews and an author rebuttal was submitted. The ability of the proposed metric to yield fine-grained insights is questioned by fECn, which the author rebuttal suggests can be addressed by consideration of other traditional metrics in conjunction with the proposed one, which is a reasonable approach. PcZP is strongly supportive of acceptance based on evaluation of technical contributions, theoretical justifications and comprehensive experimentation. Clarification questions on the motivations for design choices like the gated mixture of prompts and the zero-shot domain classifier are well-addressed by the rebuttal. Df2f also recommends acceptance based on the method and analysis, with the further ablation study of the gating mechanism satisfactorily provided by the rebuttal. In balance, the ACs agree with the reviewer majority that the paper brings new insights and the proposed new metric has been sufficiently justified with theoretical and empirical analysis to indicate potential broad usage in future works. So, the paper is recommended for acceptance at ICML. The authors are encouraged to include the further clarifications and experiments from the rebuttal in the final version of the paper.
Decision: Accept (poster)