Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning
Authors: Xiaomeng Fan
Yuchuan Mao
Zhi Gao
Yuwei Wu
Jin Chen
Yunde Jia
Venue: NeurIPS 2025 poster ()
Keywords: Open-Vocabulary Learning; Distribution Estimation;  Open Environments; Prompt Learning
Primary area: deep_learning
TL;DR: In this paper, we study how to learn beyond the seen classes in open environments. We derive distribution estimation theorems that prove the distribution can be estimated by generating unseen-class data, with an upper bound on the estimation error.

Abstract: Open-vocabulary learning requires modeling the data distribution in open environments, which consists of both seen-class and unseen-class data.
  Existing methods estimate the distribution in open environments using seen-class data, where the absence of unseen classes makes the estimation error inherently unidentifiable.
  Intuitively, learning beyond the seen classes is crucial for distribution estimation to  bound the estimation error.
  We theoretically demonstrate that the distribution can be effectively estimated by generating unseen-class data, through which the estimation error is upper-bounded.
 Building on this theoretical insight, we propose a novel open-vocabulary learning method,  which generates unseen-class data for estimating the distribution in open environments.
The method consists of a class-domain-wise data generation pipeline and a distribution alignment algorithm.
The data generation pipeline generates unseen-class data under the guidance of  a hierarchical semantic tree and domain information inferred from the seen-class data, facilitating accurate distribution estimation.
With the generated data, the distribution alignment algorithm estimates and maximizes the posterior probability to enhance generalization in open-vocabulary learning.
Extensive experiments on 11 datasets demonstrate that our method outperforms baseline approaches by up to 14%, highlighting its effectiveness and superiority.

Beyond the Seen: Bounded Distribution Estimation for Open-Vocabulary Learning
Xiaomeng Fan Yuchuan Mao 1+ Zhi Gao 1
Yuwei Wu Jin Chen' Yunde J
Beijing Key Laboratory of Intelligent Information Technology.
School of Computer Science Technology, Beijing Institute of Technology GGunngdong Lahoratory of Machine Perception and Intelligent Computing,
Shenzhen MSU-BIT University
https://githuu: wtw.ttuu..ooBBBoodd-oeeoo -eeee/Beeodd------------lllllll
Abstract
Open-vocabulary learning requires modeling the data distribution open environ- ments, which consists of both seen-class and unseen-class data. Existing methods estimate the distribution in open environments using seen-class data, where the absence of unseen classes makes the estimation error inherently unidentifiable. Intuitively, learning beyond the seen classes is crucial for distribution estimation to bound the estimation error. We theoretically demonstrate that the distribution can be effectively estimated by generating unscen-clat data, through which the estimation error is upper-bounded. Building on this theoretical insight, we pro- pose a novel open- vocabulary learning method, which generates unseen-class data for estimating the distribution in open environments. The method consists of mass-domain--wse data generation pipeline and a distribution alignment algorithm. The data generation pipeline generates unseen-class data under the guidance of hierarchical semantic tree and domain information inferred from the seen-class data, facilitating accurate distribution estimation. With the generated data, the distribution alignment algorithm estimates and maximizes the posterior probability enhance generalization in open-vocabulary learning. Extensive experiments on 11 datasets demonstrate that our method outperforms baseline approaches by up to 14%, highlighting its effectiveness and superiority.
Introduction
Open-vocabulary learning, increasingly prominent task in computer vision, aims to recognize objects for both seen and unseen classes open environments [65] 49 80). Effectively modeling the data distribution in open environments requires capturing both seen-class and unscen-class distributions. However, existing methods estimate open-environment distributions only based on 24 25), and the absence of unseen classes makes it challenging obtain accurate distribution estimation in open environments.
seen-class data 48
78 77
In this paper, we study how to learn beyond the seen classes in open environments. We derive distribution estimation theorems that prove the distribution can estimated by generating unseen- class data, with upper bound on the estimation error. Furthermore, these theorems reveal that narrowing the distribution gap between seen-class and generated unseen-class data tightens this upper bound, leading to more precise estimation. Motivated by these theoretical insights, it is desirable to generate unseen-class data that closely aligns with the seen-class data distribution, enabling a
Equual contribution. Conresoonding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

more accurate distribution estimation in open environments. To this end. we propose a novel open- vocabulary learning method, which generates unseen-class data for distribution estimation in open environments. The method is composed of class-domain-wise data generation pipeline and distribution alignment algorithm.
The Hlass-domain-wisg data generation pipeline consists of three key components: hierarchy- guided unseen class predictor, caption-based domain information generator, and text-to-image model. These components collaboratively generate unseen-class data while minimizing distribution differences from the seen-class data. Specifically, the hierarchy-guidde unseen class predictor unseen class predictor leverages a hierarchical semantic tree, constructed from seen classes (leaf nodes) and their superclasses (parent nodes). This tree is expanded with candidate unseen classes sourced from WordNet [14] or large language models (LLMs) [60] [1]. The predictor selects the most relevant unseen class by identifying the nearest leaf node to ensure minimal distribution distance to seen classes. The caption- based domain information generator extracts domain attributes, such as styles and backgrounds, from the seen-class data via image captioning. The text-to-image model (e.g., Stable Diffusion (50)) is utilized to generate unseen-class data, with the guidance of the generated domain information and predicted unseen classes. Supported by our theoretical guarantees, this process significantly reduces the distribution gap, enabling accurate distribution estimation in open environments.
With the generated data, the distribution alignment algorithm is proposed to estimate and maximize the posterior probability of model outputs in open environments. We derive an evidence lower bound (ELBO) of the logarithmic posterior, which can be approximated as the expectation of logarithmic posterior on seen-class data minus the Kullback-Leibler (KI.) divergence between the distributions of seen-class and generated unseen-class data. Hence, we employ a KL--based loss to minimize the distribution gap between the seen-class and generated unseen-class data. However, due to inherent variations in mini-batches, enforcing strict alignment in every iteration introduces misalignment, thereby degrading learning performance. To address this, we propose a sparse loss computation strategy that accumulates output distributions across iterations and then minimizes the alignment loss periodically. This approach effectively mitigates misalignment while ensuring that the posterior probability is maximized, thereby enhancing the generalization capability in open environments.
We evaluate the proposed method open environments across two settings: sase-to-baseebase-ttt new, and cross-datases using 11 image recognition datasets. Our method consistently outperforms the baseline on all datasets across two settings, demonstrating its effectiveness. Notably, on the ET dataset, our method achieves significant improvements of 14% and 9.48% in the base-to-new and cross-dataset settings, respectively, highlighting its effectiveness and superiority. Furthermore, ablation studies confirm the impact of the proposed dlass-domainwwiss data generation pipeline and the distribution alignment algorithm. The results reveal that reducing the distribution distance between seen-class and generated unseen-class data indeed enhances performance in open environments, confirming the critical role of the proposed method. The contributions can be summarized as:
We present a theoretical analysis that demonstrates the distributions in open environments can be effectively approximated by generating unseen-class data, with an upper bound on estimation error.
We propose a novel ooca-vocay learning method, which introduces a Mlas-domain-wise data generation pipeline to generate unscen-class data and a distribution alignment algorithm to accurately estimate and utilize the distribution in open environments for enhancing performance in open environments.
2
Related Work
2.1 Open- Voce-Vocaaulary Learning
Existing upen-vocabular learning methods can be categorized into pre-training and prompt learning.
Pre-training methods, such as CLIP [48] and ALIGN (22]. train vision- aision-language models (VLMs) on
large-scale image-text pairs (e.g., 400M to 1B) to learn rich multi-modal representations. Recent
efforts focus on scaling datasets (e.g.. Datacomp [15], LAION-5B (56) or improving training
strategies (e.g., caption diversity enhancement R4,, fine-grained semantic alignment (36), masked cross modsl learning [58), scalable training optimization (64), multimodal guidance alignment (31] )

---

However, these methods often require retraining from scratch, which is -ssuureeinttensve in terms of time, data, and annotations.
Prompt learning methods address the retraining issue by introducing learnable prompt tokens at the in- these methods adapted for vision-language models (VLMs). CoOp pioneers continuous prompt optimization in the language branch [78), while CoCoOp improves generalization by generating conditional prompts based on visual features [77].
put [17]. Initially successful in NLP tasks [32
35 40
VPT extends this to the visual branch by optimizing visual prompt tokens 23 Recent advance-
ments include multi-modee prompt fusion 26. 74 70, regularization techniques 77
33] 27
and leveraging local VLM features to enhance performance [30] 59 6 These methods focus on estimating the open-environment distribution using seen-class data without theoretical guarantees for upper- bounddb estimation error. In contrast to existing methods that rely solely on seen-class
data, we explore learning beyond the seen by generating unseen-class data for accurate and bounded distribution estimation in open environments.
2.2 Learning from Synthetic Data
Synthetic data enhances performance across computer vision tasks like object detection
+
semantic segmentation [ 52), autonomous driving [1) and robotics [42
73
Recent text-to-
image models, powered by diffusion techniques, generate high-quality images from text 54
Existing methods combine descriptive prompts and classes to create synthetic images 12
which,
when paired with real data, improve tasks like image classification [1] 18 38
55
62 object
detection 7 (8 76],, and semantic segmentation [16 28 46 66 67 71). Unlike these methods that
focus on generation of seen classes, our method generates images for unseen classes to precisely
estimate the open-environment distribution.
3 Preliminaries
3.1 CLIP
We implement our method on CLIP
48 that consists of an image encoder f+, (-) and text encoder {1. By). Given an image the image encoder embeds æ and adds a learnable class token obtain the visual feature. Then, the text encoder projects the
(+,(+), (+), where parameters are +
corresponding class label wrapped within text template to get the textual feature. Given image
and ground-truth ÿ from all classes, CLIP computes the posterior probability as
p a. D) exp(sim(04,, (7), fa, (x)))T)
(1)
Σ. exp(sim (g+. (y:), fø. (x)/T)²
where sim(..) .) is the cosine similarity and F is temperature parameter. In this paper, we utilize the prompt learning to optimize CLIP. We add learnable language and visual prompts given 01 and va to the textual and visual inputs, respectively. Prompts v = [ v2] are optimized with the loss as
exp(sim(fp. TTT)
LCE llo Σ, exp(simm )/T)
(2)
where F = ((u1, æ)), and Opp. = Io. (102.
3.2 vocablocayylary Learning
Open- Open-vocabulary learning requires recognizing objects of unseen classes and seen classes in open environments. We denote the data in open environments as D., which consists of image-label pairs/ {x }.)). Accordingly, D. can be divided into two disjoint datasets, i.e., the scen-class dataset D. = {I y,)} } and the unseen-class dataset D., = {( vn) The corresponding label sets denoted as Y { Y,
{y.}, Y. ),.), which satisfy that Y. n Y. 0 and Y, U YY,,Y. Y... A unique aspect of open-vor gpen-vocabblary recognition tasks is the inclusion of language vocabulary knowledge encoded in a large vocabulary space, such as the description of textual classes.
Open-vocabulary learning tasks aim to maximize the model outputs posterior p b). Intu- itively, the posterior distribution p( x., 4) is strong related to p( p(7)e [x., 2) and pl T R., 2). Existing
3

---

methods tend to utilize seen-class data to model pl }) which is further to estimate p 4). Ignoring p(u n.,, +) results in significant estimation errors that cannot be guaranteed be bounded, further impacting the generalization in open environments.
Theoretical Analysis
To facilitate open- -occbuacyyy learning, we explore learning beyond the seen classes for accurate distribution estimation in open environments. In this section, we demonstrate that the distribution in open environments can be estimated by generating unseen-class data G2 = { (I ya) The label set
of Gu is denoted as Y. (ye) that satisfies Y. n Y, 0 and Y. Y. Y.. Our theoretical
analysis is conducted from two perspectives, i.e,, the joint probability distribution and the posterior probability distribution.
The joint probability distribution in open environments is denoted as p(æ., v.). Due to YenY, = and Y, " U.Y. = YY.. p(x., yoo can be modeled
pla Ho) = p(w,, yy) + p( Mu):
where p( y,) and p v.) denote the joint probability distribution of seen classes and unseen classes, and p(æs, yy) can be directly modeled from seen-class data. We propose that be estimated by generating unseen-class data G_ = {(x.. }e)}, where this estimation error is upper bounded, shown in Theorem
p( p( V.)) can
Theorem 1.
With probability least 8, we have the following,
d(p(æte b.lkee.y...oom P(.u, Bw)) $ dplla.. We), p(a.p, Hz)) +
+
In
(4)
2m
2m
where d(.,,) denotes the distribution distance, and m. denotes the size of seen-class dataset.
This theorem demonstrates that the distance between the joint distributions of unseen-class and gener- ated unseen-class data has an upper bound. In Theorem [ we can observe that as the distance between the joint distributions of generated unseen-class data and seen-class data dpp(æe: ye), p(æ.. y y,)) decreases, the upper bound in Eq. 4 decreases. This indicates that we can narrow the gap between the joint distributions of generated unseen-class data and unseen-class data by reducing the distance between the joint distributions of generated unseen-class data and seen-class data. Theorem D holds for any distribution distance (e,g., KL divergence, total variation distance, or other forms).
In terms of posterior probability distribution, we demonstrate that the estimation eтror between P(ū Le, 4) and p(ğ uu, Φ) upper bounded, which is presented in Theorem 2 Without loss of generality, we define the distribution distance as KL divergence for analysis.
Theorem 2. Given the predicted classes Y = {-e}- Suppose that predicted class Y have any nonzero probability p(Y.). )). With probability at least 1 - å over the m instances of generated
unseen-class data (we. De)}, we have that
Dxa (p(gle 4 1p( 2))
In MY..) Iu
In
Y Y
(5)
2m
Discussion. p(Y.) ) the probability assigned to Y. under the distribution over Y Since Y. C Y a.. .)Y.) > 0 always holds without any constraint on Y..
Theorem demonstrate that the estimation of the posterior probability distribution in open environ- In
ments has upper bound. From Theorem 2 we can observe that lim p + 0, which
2m
indicates that increasing the amount of samples m can decrease the approximation error. As the probability p(Y.) and Y. increase, the approximation error bound decreases, which conforms to common sense. The step-by-step derivations are presented in the appendix.
From Theorems I and 2 we can conclude that the distance between the distributions of generated unseen-class data and unseen-class data in open environments has an upper bound. This provides the theoretical guarantee for the open-environment distribution estimation.

---

Clas-Domaln-Wic Data Generation Pipeline
Generated Unseeu- Classs Data
Semsntic
Goldtias
Vensheac
WonlNet or L.Ms
Teuch
Salmou
Saleor
Grand
Texls
Text to- Model
Hierarchy-Guided Unseen Class Predictor
eyg pready
VTMc
plrk wwwerrcliigggaeee..ooo allge.
TLMc
cale youkeo intorent hexay of
andfee
Caption
Figure 1: Formulation of Mlass-Domain- Wise Data Generation Pipeline
5
Method
Based the theoretical analysis, propose novel open-vocabulary learning method that includes class-domain- wise data generation pipeline to generate unseen-class data and a distribution alignment algorithm to estimate and utilize the estimated distribution for generalization.
5.1
Class-Domain-Wire Wise Data Generation Pipeline
Inspired by Theorem that indicates estimation error is related to the distribution distance between seen-class data and generated unseen-class data, our goal is to generate unseen-class data aligned with the seen-class data distribution. Our pipeline includes a hierarchy-guide identify classes close seen-class data, a caption-base domain information of scen-class data, and text-to-image model. These components work together to generate unseen-class data that align with the seen-class data distribution. The overall pipeline is shown in Figure II
unseen class predictor to domaín information generator to extract
5.1.1 Hierarchy-Guided Unseen Class Predictor
In open environments, the semantic structure of classes exhibits prominent hierarchical nature, akin to the label space ImageNet [10], which is also organized hierarchically. Motivated by this observation, the unseen class predictor leverages the semantic structure of seen classes to predict unseen classes.
We first construct a semantic tree for the seen classes Y s> where Y, are the leaf nodes and parent nodes represent superclasses derived from WordNet [14] or large language models (LLMs) 60 61
If WordNet contains the classes, superclasses are their hypernyms; otherwise, LLMs are queried. To predict unseen classes, we expand the semantic tree by adding missing hyponyms of the superclasses via WordNet LLMs. These hyponyms, representing sibling nodes of the seen classes, serve as potential unseen class candidates. For each candidate, we compute its cosine similarity with seen classes using textual embeddings from text encoder g+,() (-), and select the top Ko closest candidates as predicted unseen classes.
5.1.2 Caption-Based Domain Information Generator
Domain information (such as image style and scene information) plays important roles in the alignment of generated unseen-class data and seen-class data. We aim to capture the domain information from seen-class data by utilizing Vision-Language Models (VLMs) [9) In doing so, we have to encounter two main issues. Firstly, the hallucination originated from VLMs may result in unmatched domain information. Secondly, domain information extracted from slen-class data may be limited, undermining the diversity of generated images.
To solve these issues, we utilize VL.Ms generate class-specific captions for each classes, and then calculate the similarity between these descriptions and the images of corresponding classes. By
5

---

selecting top K results with the highest similarity, we ensure that the generated textual descriptions closely align with the image content, thereby effectively reducing hallucination issues. Then, the selected captions are summarized as top K₂ class-specific domain information by using LLMss
5.1.3 Text-to-Image Model
The text-to-image model [ST] is utilized to generate unseen-class data, with the guidance of the predicted unseen classes and the corresponding class-specific domain information.
Because the unseen classes close to classes and domain information aligns with seen-class data, we narrow the distribution gap between the generated unseen-class data and the seen-class data.
5.2 Distribution Alignment
After generating unseen-class data G.. = (c)): we estimate and maximize the posterior probability in open environments, thereby enhancing generalization. The posterior probability distribution of model outputs in open environments satisfies that
p T Co. 4 x p(ğ æu. 2 ))p(p/. 4)
(6)
The Evidence Lower Bound (ELBO) of the logarithmic posterior probability can be derived
log p( Co. 4 2 [loog p 2) Dxa æ.. 4) ) au, (.. 2)),
(7)
where the proof is presented in the appendix. Due to the absence of (æ., yu), we leverage the generated unseen-class data (xe. y.) to estimate ELBO, i.e.,
Bllog x.., 4) (xa(p(y/... () 2)).
(8)
As Theorem 2 suggests that the KL divergence between p(y|
x.. : 3) and p xe: 2) is upper- bounded, this estimation in Eq. (8) is reasonable and practically effective. Thus, log p 2) in Eq. (54) can be maximized by minimizing - (( x,, +)] and (p(g)/z. + ||ppp// xe, T)).
To this end, we design distribution alignment algorithm that adopts prompt learning to optimize CLIP Specifically, we minimize the LcE(-) on D, Eq. ( to minimize -Ellog x,, $) To minimize DRI. (p((pp() la.,, 4b) (p)(pr Fe, 4)), we introduce KL-based loss for distribution alignment, which is formulated as
LKI. DKL. [P(ū y ab, + (P(glar $,, v))-
By minimizing fCE(.) and (.). p(ū( o,, b) is maximized.
The proposed loss LKL. (-) introduces additional challenges, ie., the data from unseen and seen classes in the mini-batch may differ significantly, and the loss could forcefully align their output distributions despite their large inherent differences. This misalignment between the distributions can, turn, compromise the learning performance of both the base and unseen classes. To address the issue, we propose a sparse loss computation strategy that accumulates output distributions of seen-class data across iterations and then minimizes the alignment loss periodically. During each iteration, we save the output distributions the seen-class data. For each batch of unseen-class data, we compute the similarity between the saved output distributions of seen-class data and that of the unseen-clate data. Alignment is then performed on the top K3 most similar distributions, which helps alleviate the misalignment problem by ensuring a more accurate and consistent alignment across batches.
In order to decrease the distribution distance between the generated data and real data in the feature spaces, we introduce a Maximum Mean Discrepancy (MMD) loss. Specifically, we first generate some extra seen-class data G. {(x'.. .. y,)} using the same generation pipeline as presented in Section 5.1 Then the MMD loss is formulated as
LAMD Σ K(x), xb) +
Σ
K(x', x36)
K(a,, x',),
(10)
Imm
where n is batch size. K(x,y) (( = loss, the distance of feature spaces between generated data and real data is decreased, further improving alignment. We also employ the sparse loss computation strategy for the MMD loss.
2
represents Gaussian kernel. By minimizing the MMD
Overall, the loss function for updating parameters is
Liotal LCE aLKL + 3LMMD.
(11)

---

where a and | are parameparameters. By minimizing Lioat the posterior probability in open environ- ments p(ğ z. 2) is maximized, thereby improving capability open environments. This algorithm is summarized in appendix.
Table 1:
Results in base-to- to-new/base-to- base generalization setting. We bold the best results and underline the second-best results. H denotes the harmonic mean of performance on base and new.
CLIP 48
CoOp
CoCoOp
DePT
TCP
CuTCP
DeKg [37
PromptSRC (baseline)
Ours
Gain A
Dataset
72
21
Base
69.34
82.69
80.47
85.19
13
84,21
84.96
84.26
86.40
+2,14
Average
New
74.22
63.22
71.69
76.
75.36
76.
76.38
76.10
80.52
+4.42
H
71.70
71.66
75.83
80.43
79.51
79.95
80.44
79.97
83.36
+3.39
Base
72.43
76.47
75.98
78.20
77.27
77.73
77.40
77.60
77.91
+0.31
ImageNet
New
68.14
67.88
70.43
70.27
69.87
70.50
69
70.73
70,74
+0.01
H
70.22
71.92
73.10
74.02
73.38
73.94
73.07
74.01
74.15
+0.14
Base
96.84
98.00
97.96
98.57
98.23
98.47
98.64
98.10
98,97
+0.87
Caltech
New
94.00
89.81
93.81
94.10
94.67
95.27
95.20
94.03
95.85
+1.82
H
95.40
93.73
95.84
96.28
96.42
96.84
96.89
96.02
97.38
+1.36
Base
91.17
93.67
95.20
95.43
94.67
95.07
94.47
95.33
96.01
+0.68
Pets
New
97.26
95.29
97.69
97.33
97.20
97.83
97.76
97.30
98,27
+0.97
H
94.12
94.47
96.43
96.37
95.92
96.43
96.09
96.30
97.12
+0.82
Base
63.37
78.12
70.49
80,80
80.23
78.27
82,93
+4.66
Cars
New
74.89
60.40
73.59
75.00
74.13
74.27
74.75
74.97
80.81
+5.84
H
68.65
68.13
72.01
77.79
77.32
77.13
77.83
76.58
81.86
+5.28
Base
72.08
97.60
98.40
97.73
98.10
98.58
98.07
98.77
+0.70
Flowers
New
77.80
59.67
71.75
77.10
75.57
75.58
75.18
76.50
80.92
+4,42
H
74.83
74.06
81.71
86.46
85.23
85.38
85.30
85.95
88.96
+3.01
Base
90.10
88.33
90.70
90.87
90.57
90.47
90.73
90.67
91.39
+0.72
Food
New
91.22
82.26
91.29
91.57
91.37
91.77
91.55
91.53
92.99
+1.46
H
90.66
85.19
90.99
91.22
90.97
91.11
91.14
91.10
92.18
+1.08
Base
27.19
40.44
33.41
45.70
41.97
42.43
45.20
42.73
48.98
+6.25
Aircraft
New
36.29
22.30
23.71
36.73
34.43
36.37
35.09
37.87
44.03
+6.16
H
31.09
28.75
27.74
40.73
37.83
39.17
39.51
40.15
46.37
+6.22
Base
69.36
80.60
79.74
83.27
82.63
83.00
82.52
82.67
83.64
+0.97
SUN
New
75.35
65.89
76.86 86
78.97
78.20
78.23
78.30
78.47
80.15
+1.68
H
72.23
72.51
78.27
81,06
80.35
80.55
80.35
80.52
81.86
+1.34
Base
53.24
79.44
77.01
84.80
82.77
83,00
83.80
83.37
85.53
+2.16
DTD
New
59.90
41,18
$6.00
61.20
58.07
59.40
$9.66
62.97
71.50
+8.53
H
56.37
54.24
64.85
71.09
68.25
69.24
69.70
71.75
77.89
+6.14
Base
56.48
92.19
87.49
93.23
91.63
90.87
94.02
92.90
97.17
+4.27
FurSAT
New
64.05
54.74
60.04
77.90
74.73
77.13
81.69
73.90
87.90
+14.00
H 60.03
68.69
71.21
84.88
82.32
83.44
87.42
82.32
92.30
+9.98
Base
70.53
84.69
82.33
87.73
87.13
86.87
88.06
87.10
89.14
+2.04
UCF
New
77.50
56.05
73.45
77.70
80.77
80.80
81.77
78.80
82.53
+3.73
H
73.85
67.46
77.64
82.46
83.83
83.72
84.80 80
82.74
85.71
+2.97
Experiments
6.1 Experiment Settings
We evaluate our method on open- vrcc-vocabulary benchmarks with 11 image recognition datasets, follow- ing the setting of the baseline method PromptSRC [27].
Datasets. We evaluate the proposed method on 11 image recognition datasets: ImageNet [10], Cal-
tech101 (Caltech) [13), OxfordPets (Pets) [45]. StanfordCars (Cars) 29), Flowers (Flowers)
4
Food101 (Food) (5). FGVCAircraft (Aircraft) [41), SUN397 (SUN) 69] UCF101 (UCF) 57
DTD 9 and AT [9]
Benchmark Settings. We evaluate method open- learning benchmarks.

---

Table 2: Results of our method and state-of-the-art methods for cross-dataset evaluttion.
Source
Target
ImageNet Caltech Pets
Cars
Flowers Food Aircraft SUN DTD EuroSAT UCF
Average
CoOp
71.51
93.70
89.14 64,51
68.71
85.30
18.47
64.15 41.92
46.39
66.55
63.88
CoCoOp
71.02
94.43
90.14 65.32
71.88
86.06
22.94
67.36 45.73
45.37
68.21
65.74
ASPrompt
71.05
94.57
90.79 66.90
72.30
86.17
25.16
67.32
47.35
50.25
69.52
67.03
PromptSRC
71.27
93.60
90.25 65.70
70.25
86.15
23.90
67.10
46.87
45.50
68.75
65.81
Ours
71.22
93.87
90.46 67.36
72.88
86.61
25.14
67.68
48.27
54.98
69.44
67.68
Gain A
-0.05
+0.27
+0.21 +1.66
+2.63
+0.46
+1.24
+0.58
+1.40
+9.48
+0.72
+1.87
(a) DTD (Class Quantity)
(b) OxfordPets (Class Quantity)
(c) UCF101 (Class Quantity)
Scureer
(d) DTD (Data Quantity)
(e) OxfordPets (Data Quantity)
(1 UCF101 (Data Quantity)
Figure 2: Ablation Studies on Quantity of Predicted Unseen Classes and Generated Images.
Base-to-basee tae-tto-bbaseBas-tto-neww generalization. We equally split each dataset into base and new classes. The model is trained on base classes and evaluated on both base classes (base-to-base) and new classes (base-to-new) across all 11 datasets.
Cross-dataset evaluation. To evaluate our method on the unseen classes under different domain environments, we adopt cross-dataset setting, where we train a model on ImageNet (source domain) and evaluate the model on the other 10 datasets (target domains) without any fine-tuning.
we use a few-shot training Implementation details. Following the setting of PromptSRC 27
strategy in all experiments at 16 shots which are randomly sampled for each class. We apply prompt learning on pretrained ViT-B/16 based CLIP model and report results averaged over 3 runs. Other details such hyper-parameters are provided in appendix.
6.2 Kass-to-BaseeBasstto--Wee Generalization
We compare method with other prompt learning methods in Table [ Results show that method significantly improves the performance of PromptSRC open environments for both base- to-base and base-to-new settings, demonstrating its effectiveness in open environments. Notably, new classes, our method achieves up to a 14% performance improvement, indicating its ability to effectively estimate the distribution of unseen classes.
also compare our method with other state-of-the-art methods: CLIP 48)
CoOp 78. Co-
CoOp 777. DePT 75). TCP (72), CuTCP 211, and DeKg [37]. Results show that our method

---

Table 3: Ablation studies class quality and data quality. "Acc" denotes the accuracy. "Dis" denotes the distribution distance the generated unseen-class data and seen-class data.
Ours
Class Quality
Data Quality
Dataset
LowSim
w/o Tree
Picture
Photo
Image
Acc
Dis
Acc f
Dis
Acc f
Dis
Acc
Dis
Acc
Dis
Acc
Dis
Base
98.97
98.26
97.93
98.26
98.19
98.06
Caltech
New
95.85
9.99
94.32
10.49
93.67
13.16
94.21
11.84
94.11
11.95
93.78
12.12
97.38
96.25
95.75
96.19
96.11
95.87
Base
96.01
95.85
95.00
95.69
95.43
95.27
Pets
New
98.27
7.58
97.60
8.19
96.81
11.71
97.04
9.14
96.98
9.21
96.92 10.27
H
97.12
96.72
95.90
96,36
96.20
96.09
Base
82.93
78.94
77.86
77.99
78.71
78.24
Cars
New
80.81
8.92
75.29
9.33
73.36
13.78
74.75
10.65
75.04
10.08
74.92 10.48
H
81.86
77.07
75.54
76.33
76.83
76.54
Base
98.77
98.29
97.15
97.82
97.91
97.63
Flowers
New
80.92
6.29
77.52
6.88
75.04
10.29
76.88
7.99
77.09
7.84
76. 16.17 8.22
88.96
86.68
84.67
86.09
86.26
85.57
Table 4: Ablation study on distribution alignment. "w/o da" denotes prompt learning without distribution alignment.
Caltech
Pets
Cars
Flowers
Aircraft
DTD
EuroSAT
UCF
w/o Ours w/o da Ours w/o da Ours w/o da Ours w/o da Ours w/o Ours w/o da Ours w/o Ours
Basel
97.42 94.10 96.01 74.14 82.93 96.49 98.77 39.80 48.98 75.81 85.53 93.57 97.17 85.16 89.14
New
90.07 95.85 88.65 97.65 71.63 80.81 71.63 80.92 23.39 44.03 53.50 71.50 64.13 87.90 62.03 82.53
H
93.60 97.38 91.29 96.82 72.86 81.86 82.22 88.96 29.47 46.37 62.73 77.89 76.10 92.30 71.78 85.71
achieves the best performance, demonstrating its superiority in open environments, especially its generalization ability on unseen classes.
6.3 Cross-Dataset Evaluation
The comparison between our method and other prompt learning methods is presented in Table Compared to PromptSRC, our method achieves the comparable performance on the source dataset. On the target datasets, our method significantly outperforms PromptSRC with a notable improvement of 9.48% on the Eurosat dataset, demonstrating that our method can improve the generalization on unseen classes across different domains, effectively working in open gen-vocabulary learning task.
6.4 Ablation Studies
6.4.1
Effectiveness of Class-Domain-We Wise Data Generation Pipeline
We evaluate the effectiveness of two components with respect this pipline, ie., hierarchy-guided unseen class predictor and caption-based domain information generator.
Hierarchy-Guuiee Unseen Class Predictor. We evaluate the effectiveness by adjusting the quantity and quality predicted unseen classes. As to the quantity, we randomly select Sale X N classes from the N predicted unseen classes for training, where Sdl 1 denotes sampling ratio. We conduct experiments on three datasets (DTD, Flowers 102, UCF101), and we set Sala as 0.1 to 0.9 in increments of 0.1 for each dataset. As shown in Figures 2a 2b and 2c we observe that increases of quantity lead to better performance, especially for new classes, which is consistent with Theorem 2
As to the quality, we introduce "LowSim" that chooses candidate classes with lowest cosine similarity and "w/o Tree" that directly L.LMs to query unseen classes. We calculate the distribution distance between the generated unscen-class data and the seen-class data. As shown in Table 3 our method

---

can significantly reduce the distribution distance and improve alignment with seen-class data. Results also reveal that alignment improves recognition, which is consistent with Theorem I
Saption-Based Domain Information Generator. We evaluate its effectiveness of by adjusting the quantity and quality of generated images. As the quantity, we randomly select Simag X M images of each predicted unseen classes for training, where Sels < denotes sampling ratio and M denotes the amount of data for each classes. Figures 2d 2e and 2f show that increase of quantity leads to better performance, especially for new classes, which is consistent with Theorem 2
As to the quality, modify the prompt template for data generation, without the domain information inferred from the generator. We inject "Picture", "Photo" and "Image" into stable diffusion model to generate images, which are denoted the the prompt template "A picture of a [class)". "A photo of (class)", and "An image of (class]". respectively. Results are shown in Table We observe that our method can significantly reduce the distribution distance and improve alignment with seen-class data. Results also reveal that this alignment improves recognition, which is consistent with Theorem |
To further evaluate the generated image quality, computed CLIPScore [20]. CLIPScores on UCFI01, DTD, SUN397, Caltech 101, OxfordPets, and StanfordCars are 0.43, 0.42, 0.43, 0.43, 0.44, and 0,42, respectively. Results show that these images exhibit high semantic quality (CLIPScore >033 is considered high quality 66 54). We further conducted a user study and a GPT-4-based evaluation, where both human annotators and GPT-4 independently rated 200 randomly selected images on a 1-5 scale (5 is the highest quality). The resulting average scores of 4.67 (human) and 4.59 (GPT-4) confirm the high quality of the generated images.
6.4.2 Effectiveness of Distribution Alignment
We evaluate the effectiveness of the distribution alignment algorithm by directly using the generated images for prompt learning during training without the distribution alignment algorithm, denoted by "w/o pda". Results are shown in Table 5 which indicate that the proposed algorithm can improve the model performance in open environments by aligning the output distributions of model between generated data and real data.
6.5 Efficiency Analysis
We take Eurosat an example for analysis. The training time and memory of our method requires at most 16. 14 GB and 739.2 s. The baseline requires 12 GB and 101.25 s. The added time and memory mainly come from data generation. At inference time, no extra parameters or computation are introduced. Thus, our runtime (3.8 s) and memory (1.84 GB) are identical to the baseline.
Conclusion and Discussion
In this paper, we have investigated learning beyond the seen for bounded distribution estimation in the open-vocabulary task. We have demonstrated the distribution in open environments can be estimated by generating unseen-class data with upper-bounded estimation error, as evidenced by the constructed theoretical analysis. The proposed open- vocabulary learning method consists of dlass-domainwwin data generation pipeline and distribution alignment algorithm. The data generation pipeline generates unseen-class data via the introduced unseen class predictor and domain information generator, enabling accurate distribution estimation. With the generated data, the proposed distribution alignment algorithm can effectively estimate and maximize the posterior probability in open environments for improving generalization in open environments. Experiments on 11 datasets demonstrate that our method can generate unseen-class data for accurate distribution estimation, leading to consistent improvements in generalization across diverse open environments.
The generated data exists biases from the utilized LLMs and wordnet. the future, we plan to design collaboration strategy that leverages diverse pretrained models and agreement based selection to reduce dependence and bias on any single model. Moreover, to extend our method to support truly unknown classes, such newly cartoon characters, we plan to integrate RAG mechanisms that dynamically retrieve emerging classes from external sources to enhance the ability of the model to generate up-to-date images, enabling the model to adapt in real time to new concepts.
a
10

---

Acknowledgements. This work was supported by the Shenzhen Science and Technology Program under Grant No. JCY1202412221100440066 the Natural Science Foundation of Shenzhen under Grant No. JCY12002000774220000M the Natural Science Foundation of China (NSFC) under Grants No. 62406009, No. 62172041 and No. 62176021.
References
[1] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes. International Journal of Computer Vision, 126:961-972, 2018.
[2] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David Fleet. Synthetic data from diffusion models improves imagenet classification. Transactions Machine Learning Research.
[3] Eslam Mohamed Bakr, Pengzhan Sun, Xiaoqian Shen, Faizan Farooq Khan, Li Erran Li, and Mohamed Elhoseiny. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20041-2080 20053, 2023.
[4] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. htlps://cdn. openai. com/papers/dalle-3. pdf, 2(3):8, 2023.
[5] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-minin discriminative components with random forests. In European Conference on Computer Vision, pages 446-461, 2014.
[6] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. arXiv preprint arXiv:2210.01253, 2022.
[7] Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing Hong, Zhenguo Li, and Dit- Yan Yeung. Geodiffusion: Text-prompted geometric control for object detection data generation. arXiv preprint arXiv:2336. 04607, 2023.
[8] Yuhua Chen, Wen Li, Xiaoran Chen, and Luc Van Gool. Learning semantic segmentation from synthetic data: A geometrically guided input-output adaptation approach. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1841-1850, 2019.
[9] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2014.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[11] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang. Joseph Gonzalez, and Trevor Darrell. Diversify your vision datasets with automatic diffusion-based augmentation. Advances in neural information processing systems, 36:79024-79034 2023.
[12] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian. Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7382-7392, 2024.
[13] Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 178-178,1 2004.
[14] Christiane Fellbaum. Wordnet. WordNet An Electronic Lexical Database, page 69, 1998.
[15] Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.

---

[16] Rui Gong, Martin Danelljan, Han Sun, Julio Delgado Mangas, and Luc Van Gool. Prompt- ing diffusion representations for cross-domain semantic segmentation. arXiv preprint arXiv:2307.021338 2023.
[17] Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin. Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision-language foundation models. arXiv preprint arXiv: 12900, 2023.
[18) Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and XIAOJUAN QI. Is synthetic data from generative models ready for image recognition? In The Eleventh International Conference on Learning Representations.
[19) Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, pages 2217-2226, 2019.
[20] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514-7528, 2021.
[21] Min Huang, Chen Yang, and Xiaoyan Yu. Cutep: Custom text generation- class-aware prompt tuning for visual-language models. Scientific Reports, page 2681, 2025.
[22] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- His-H Sung. Zhen Li, and Tom Duerig. Scaling up visual and vision- vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 1904-4916
PMLR, 2021.
[23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709-727. Springer, 2022.
[24] Yangbangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Dm2c: Deep mixed-modal clustering. Advances in Neural Information Processing Systems, 32, 2019.
[25] Yangbangyan Jiang, Qianqian Xu, Yunrui Zhao, Zhiyong Yang. Peisong Wen, Xiaochun Cao. and Qingming Huang. Positive-unlabeled learning with label distribution aligmment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):15345-1555 2023.
[26] Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fa- had Shahbaz Khan. Maple: Multi- modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19113-19122, 2023.
[27] Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz. Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15100- 2023.
[28] Neehar Kondapaneni, Markus Marks, Manuel Knott, Rogério Guimaraes, and Pietro Perona. Text-image alignment for diffusion-based perception. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 13883-13893, 2024.
[29] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop, pages 554-561, 2013.
[30] Marc Lafon, Elias Ramzi, Clément Rambour, Nicolas Audebert, and Nicolas Thome. Gallop: Learning global and local prompts for vasion-language models. arXiv preprint 2rXiv:2407.01400 2024.
[31] Samuel Lavoie, Polina Kirichenko, Mark Ibrahim, Mido Assran, Andrew Gordon Wilson, Aaron Courville, and Nicolas Ballas. Modeling caption diversity in contrastive vision-language pretraining. In Forty-first International Conference on Machine Learning.
[32] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficiee prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2021.
[33] Juncheng Li, Minghe Gao, Longhui Wei, Siliang Tang, Wenqiao Zhang, Mengze Li, Wei
Ji, Qi Tian, Tat-Seng Chua, and Yueting Zhuang. Gradient-regulated meta-prompt learning
12

---

for generalizable vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2551-2562, 2023.
[34] Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35:7290-7303, 2022.
[35] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582 4597, 2021.
[36] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23390-23400, 2023.
[37] Yilun Li, Miaomiao Cheng, Xu Han, and Wei Song. Divergence-enhanced knowledge-guine guided context optimization for visual visual-language prompt tuning. In The Thirteenth International Conference on Learning Representations, 2025.
[38] Zheng Li, Yuxuan Li, Penghai Zhao, Renjie Song, Xiang Li, and Jian Yang. Is synthetic data from diffusion models ready for knowledge distillation? arXiv preprint 2rX.::2305.12954 2023.
[39] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892-3444 34916, 2023.
[40] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 5:208-215, 2024.
[41] S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aireraft. Technical report, 2013.
[42] Arthur Moreau, Nathan Piasco, Dzmitry Tsishkou, Bogdan Stanciuleseu, and Arnaud de La Fortelle. Lens: Localization enhanced by nerf synthesis. In Conference on Robot Learning, pages 1347-1356. PMIR, 2022.
[43] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics & Image Processing, pages 722-729, 2008.
[44] Jinyoung Park, Juyeon Ko, and Hyunwoo Kim. Prompt learning via meta-regullarization In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 26940-26950, 2024.
[45] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3498 3008-3505,200 2012.
[46] Duo Peng, Ping Hu, Qiuhong Ke, and Jun Liu. Diffusion-based image translation with la- bel guidance for domain adaptive semantic segmentation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 808-820, 2023.
[47] Xingchao Peng, Baochen Sun, Karim Ali, and Kate Saenko. Learning deep object detectors from 3d models. In Proceedings of the IEEE international conference on computer vision, pages 1278-1286 2015.
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[49] Shuhuai Ren, Aston Zhang, Yi Zhu, Shuai Zhang, Shuai Zheng, Mu Li, Alexander J Smola, and Xu Sun. Prompt pre-trat pre-training with -tenty-thuuaadd classes for open-vocabulary visual recognition. Advances in Neural Information Processing Systems, 16:12559-11255
2023. [50] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
13

---

[51] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10600 June 2022.
[52] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 3234-3243, 2016.
[53] Artem Rozantsev, Vincent Lepetit, and Pascal Fua. On rendering synthetic images for training an object detector. Computer Vision and Image Understanding, 137:24-37, 2015.
[54] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[55] Mert Bülent Sarıyiidız, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pages 8011 8011-8021, 2023.
[56] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25299 2022.
[57] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 1r1iv:1212.0402, 2012.
[58] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
[59] Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. Advances Neural Information Processing Systems, 35:30569-30582. 2022.
[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2...ii:2302.19977... 2023.
[61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint 2wxXii:230700288: 2023.
[62] Brandon Trabucco, Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv:2302.07944 2023.
[63] Lifu Wang, Daqing Liu, Xinchen Liu, and Xiaodong He. Scaling down text encoders of text- to-image diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 18424-18433, 2025.
[64] Longhui Wei, Lingxi Xie, Wengang Zhou, Houqiang Li, and Qi Tian. Mvp: Multimodality- guided visual pre-training. In European conference on computer vision, pages 337-353. Springer, 2022.
[65] Jianzong WU, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang, Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2024.
[66] Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations using diffusion models. Advances in Neural Information Processing Systems, 36:54683-54695, 2023.
[67] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 2206-1217, 2023.
14

---

[68] Zhenyu Wu, Lin Wang, Wei Wang, Tengfei Shi, Chenglizhao Chen, Aimin Hao, and Shuo Li. Synthetic data supervised salient object detection. In Proceedings of the 30th ACM International Conference Multimedia, pages 5557-5565, 2022.
[69] Jianxiong Xiao, James Hays, Krista A Ehinger. Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey 7oo. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[70] Zehao Xiao, Jiayi Shen. Mohammad Mahdi Derakhshani. Shengcai Liao, and Cees GM Snock. Any-shift prompting for generalization over distributions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13849-13860, 2024.
[71] Lihe Yang. Xiaogang Xu, Bingyi Kang, Yinghuan Shi, and Hengshuang Zhao. Freemask: Synthetic images with dense annotations make stronger segmentation models. Advances in Neural Information Processing Systems, 36, 2024.
[72] Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for visual-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23438 23448, 2024.
[73] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Tsung- Yi Lin, Alberto Rodriguez, and Phillip Isola. Nerf-superviiion: Learning dense object descriptors from neural radiance fields. In 2022 international conference on robotics and automation (ICRA), pages 6496-6503, IEEE, 2022.
[74] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv:2210.07225, 2022.
[75] Zhang, Shihan Wu, Lianli Gao, Heng Tao Shen, and Jingkuan Song. Dept: Decoupled prompt tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12924-12933, 2024.
[76] Manlin Zhang, Jie Wu, Yuxi Ren, Ming Li, Jie Qin, Xuefeng Xiao, Wei Liu, Rui Wang, Min Zheng, and Andy J Ma. Diffusionengine: Diffusion model is scalable data engine for object detection, arXiv preprint arXiv:2309.03893 2023.
[77] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- lisiun-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages (6816-16825, 2022.
[78] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9)22337-2348, 2022.
[79] Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15659-15669, 2023.
[80] Zhen Zhu, Yiming Gong, and Derek Hoiem. Anytime continual learning for open vocabulary classification. In European Conference on Computer Vision, pages 269-285. Springer, 2025.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction include the claims made in the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
Specific details are provided in supplementary materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide the implementation details in Section and supplementary materi-
als.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification: We do not provide open access to the data and code at this time, but can publishpart of them the rebuttal stage if the reviewers need it. The complete data and code willbe published after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ce)
stsiii//guidess/oeeuubbmimii oosssssoisssiooooooooooroo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips.c//pppp
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the implementation details see Section and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We follow existing work in the areas we work in and do not provide statistical significance for fair comparisons.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the computer resources in supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Our work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of our work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper contains no such risky models or data.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We've cited the original paper of the code and model we used.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We will provide open access to part the new assets at the rebuttal stage the reviewers need it. The complete assets will be published after the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
The LLM is used only for writing and editing.
Guidelines.
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: for what should or should not be described.
22

---

A Proof of Theorem 3
In the manuscript, we present Theorem 3 to demonstrate that the estimation error of p( yu) is upper bounded. Here we provide further derivations of Theorem 3f the manuscripts. To this end, We first review the defined setting in open-vocabulary learning task, present some lemmas, and their proofs, which are based on PAC- Cay-saye Theorems [?, ?].
Sefting. The data in open environments can be denoted D. {D,, Du). which consists of image-label pairs (( y = { ys), (I.., v,)),. We assume a predicted unseen-clase data distribution E, where (a ye) has the probability Ej. Similarly, the distributions of training data, unseen data and data in open environments are denoted as S, U and O respectively.
Lemma 1. With probability at least 1 over the training dataset of size m, we have the following. 4m 8
Σ Se (2m-
(12)
Proof. By the Chernoff bound we have P(y 2) S 2e 2ma" We now consider the density func- tion f(7) maximizing Jo e(2m-1ly? 1)p f(y)dy subject to the constraint that f f(7)dy S 2e 2e-2mat 2ma²
The maximum occurs when have Ja f(7)dy 2e 2e- =m=x which is realized when f(7)
Smye 2may" So we have the following.
Es (2m-1)
(2m-1)) f())dy
(2m 2mm? dy
(13)
x db
4m,
which suffices to the following.
Vi Es [(2m- ((2m-1)77] 4m.
(14)
So we have the following.
Es Es (2m- 1h1 < 4m.
(15)
By applying Markov's inequality on Eq.
[5] we get the following.
Σ 2m 1)2
4m
P
< 8,
(16)
which suffices to lemma
To prove lemma 6 we we consider selecting a training data distribution S and predicted unseen-class data distribution E. Lemma implies that with probability least 1 1-6 we have the following.
1)7
4yn.
(17)
8
So to prove lemma it now suffices to show that Eq. all distributions E such that di E, S) < 2m.
17 plus In S 2m implies the following for
In
E. S. + In }/2 + 5 In m +8 2
E CENSEEN
(18)
2m
To prove Eq. (18) for given E select 7/s as maximize the quantity Σ Erys subject the
constraint Eq. (17. Using Lagrange multipliers we set the gradient the constraint to be equal to a multiplier À times the gradient of the objective function.
2(2m (2m- S. AE
(19)
23

---

4m Eq. 18 is trivially true if E, > 0 but S; = 0 for some i. So we can assume without loss of generality
that S whenever E, 0. This allows the above to be rewritten follows.
XE
2(2m 7) (2m 1n
(20)
S;
Note that 2(2m 1)ye(2mm11)""
1) is an unbounded monotonically increasing function of y. We now define (A) be the unique non non-negative value satisfying the following.
AE S:
2(2m - ())
(21)
Now note that Σ ( is an unbounded monotonically increasing function of X. We now define \ be the unique nonnegative value such that we have the following.
Σ (.c
4m 8
(22)
Note that (0 0 and Σ -1)22m11)4210 1)4,(() A,(A*) > 0 for E; > 0.
< 8 So we must have X 0 and hence
Li
Lemma 2. For any 7 satisfying Eq.
17 we have the following.
Σ EonSE E,,,(A))
(23)
Proof. Consider the following four situations:
(1) 3i < 0
Σ Ery can be increased by replacing " with -y for y 0. Hence assume without loss of generality that 7/4 2 O.
(2) S.e (2m-))) 1)m? <
8
Σ Em can be increased by raising N. with E, > 0. Hence we can assume without loss of generality
that Σ (2m 1)7;
(3) 3i E, = 0.7 >
Σ E,ys can be increased by setting = 0 with E, = 0 while raising with E, >> 0. Hence we can assume without loss of generality that 7/2 = 0 whenever E, = 0.
(4) Jj k E,
> 0, Ex 0,
E, Tye 1nf
8
T 127-
Es
Eq. 19 is trivially true if E; 0 and 71 So we can rewrite Eq. (19) as follows.
S 22m 1) (2m 1)7
X;
(24)
E,
From Eq. 24 we can get the following.
S 2(2m 1) (mm
St 2(2m (2m-22111122 d
>;
1)1
(25)
E;
E
For X; > 0, Eq. 19 can be rewritten as follows.
S 2(2m E,
1777
(26)
A
So we have the following.
S 2(2m 1)n2 (2m (2mm 1)7?
E
Em
(27)
X;
24

---

Let fx;) be the function mapping to Σ 8: 2(2m - 1)zzelmm11
S
(())) = 4m - -1) Σ
Ize (2mm
(28)
For 2, = Fk, (xz) < f' (*x).
Σ Er can be increased by increasing " and decreasing " while holding Σ (eeeemmmm1))) constant. Hence we can assume without loss of generality that there exists a value X such that for all indices
i with E; > O we have 2(2m 1) (2m- 1yr? XEE; which implies 7/s = 4,()). We also have
S:
4m 3 which implies X = X So we have 7 = = which implies the
result.
Now proving lemma ! we have the following
suffices to bound Σ E,A,(A*). Eq. 21) implics that for À >> 1 and E; 2 S
AE In
(A ~ V
S.
(29)
2m 1
This approximate relationship is made more precise in the following two lemmas. Lemma 3. For 2 S; 0, E, S; and e, have the following.
In AE
(A -
S.
(30)
2m-1 1m-11
Proof. Let g(r) be the function mapping to 2(2m 1)mee By definition, 4x( satisfies (A), = AEE Note that for 2 0 we have that g(x:) is monotonically increasing function.
9.
Hence for x 2 0 and g(x) 2 aE we must have 4x(A) S x. Under the assumptions of lemma we
S,
have In AE
5, 2 1 which implies the following.
g(1
In XE: S
2 (2m - 1)ln
ΛE XE,
2
XE;
(31)
2m -
S: S:
S:
Lemma 4. For 2 S, > 0, E, S and e, we have the following.
In dE
Inm
In AE
S.
(32)
2m 1
Proof. By an argument similar to that in the proof of lemma particular we have the following. show that g(x) < AE. S.
show that Ax(A) 2 x it suffices to
In XE + 1 Iim In In AE 2
9(
2m -
XE;
ΛE;
2v
(2m- 1)(In
In In In
2)
S,
S
XE;
S
vm In AE,
c²
5.
(33)
XE;
XE;
< 4
In
S
S
v mm
AE 8,
X
1
S:
|n
XE
e²
S.
λE
<
S:
25

---

Lemma 5, For ((E,)) & 2m and In 3 S 2m, we have Xa S 64c 64 9/2
Proof. Let h(x) be the function mapping x to Σ (2m)))))))) The quantity X defined by h(X*) 4m Since h is monotonically increasing function, h(x)) 2 8
2m implies X S x. Lemma
implies that for 2 e we have the following.
1(xx) Σ S S S,e(mm-1)AA(0 1)A)(a)
Σ
S TE
S
VT In
E e2
E
6² m
E
S;
(34)
2² m
Σ
E,
In E.
e² mm B) Inx)
e² (2 + lnx)
Now inserting 61e2ma/2
we get the following.
64e2m 5/2
64m²
h1
8
8 (2m +
5 In m
In }/2 + 8)
2
64m²
(35)
82m + 5
2m + 8m)
4m
8
Lemma 6. Without loss of generality, we define the distance between distributions d(P, Q) P, Q, for analysis. For In S 2m we have that with probability 1 1 5 over the training dataset size m the following holds for all distributions satisfying d(E, S) S 2m.
In
E
In 1 + Immm + 8
d(E ((E,) < Σ E.
(36)
2m
Proof. Note that d(E,O) - -(E,)) = Σ Eln
E
S Σ Esy, where 7i abbreviates
S.
In E, O, S;
In E: I, the lemma can be viewed as an upper bound of Σ E,y.
From the above-mentionel lemmas, we have that
d 0) - d) S Σ Era
Σ
E,A,(X*
m
E,D.(
(37)
In
64c2 64 m/2
Σ E. VEN
2m
1
$ Σ
E
In
E S.
+
In 3 + 5
2
In m + 8
2m
26

---

Lemma 7. Denote the dl(,:) as the distribution distance. With probability at least ó, we have the following,
d(p(xe: p(as: y.))
dpp(ao: Vol,p(aes Uo)) I (p((re: Vel. plrs: Ua)) +
2m
(38)
+ In m +
2m
where denotes the size of training dataset.
Proof. By applying Jensen's inequality on lemma
we can get Theorem
In E.
In
Inm + 8
((E,0)=d(E,)) Σ E.
2
2m
d(E,S) + In
+ 5 Ilmm+
S sd(E,S) +1
(39)
2m
()
In + In +
S (dEES))
2m
2m
1
From Lemma 7 we can obviously observe that the distribution distance of the generated unseen-class has an upper bound, which indicates that the rationality of generating
data and the open-environme unseen-class data for distribution estimation in open environments. Obviously, we also can observe that this upper bound is strongly related class data and the seen-class data. The conclusion of Lemma motivates us to construct the proposed open- vecebuacyy method. From Lemma 7 we can directly obtain Theorem 3
distribution distance between the generated unseen-
7 is same with Theorem 3 This also
Theorem 3. Denote the d(.,-) as the distribution distance. With probability at least 1 - 5, we have the following,
d(p(xe, (e).p((xs.) yz))
d(plEu. Mn). pl), s
2m
(40)
+ In m +
2m 1
where denotes the size of training dataset.
B Proof of Theorem
In the manuscripts, we present Theorem 4 to demonstrate that the estimation error of p(g x D) is upper bounded. Here provide specific derivations of Theorem 4
Theorem 4. Given the predicted classes Y = =v))- Suppose that predicted class Y have any nonzero probability p(Y c)- With probability at least 1 8 over the m instances of generated
unseen-class data { (xe. V. we have that
Daa DRa (p( me, + Ip(7)au, T))
In p(Y.)
+ In
In
Y-
(41)
2m
Y
Proof. We denote p"()) p(ūle,, Φ). E Y u.. For analysis, we define P(Y,)= = Σ p" (r))), and
VEY.
we define the conditional distribution as P. Py)) p(p 4)
(' P.Y.
E Y... Pa(y) satisfies
27

---

that Pe(y) = 1. We also denote that p(y) = p( c. Φ). In this way, the KL divergence yEY.
between ((y) and p²(1) (z) is computed
p(y) In ply)
(xl((())ll"()))
Σ DEY p0
(42)
can be formulated as pr(y)
ply)
i(y) p(y) Pe(u)
p(y)
(43)
p" (y) Pe(y) pr
PP(y)) P(Y.)'
and thus
In p(y)
In p(y) P.(y)
In P(Y.).
(44)
(y)
By substituting Eq. 44) into Eq 42 we have that
Σ p(y)
In p(y)
- In).)
Pε(
yEY.
Σ p(y) p(y) Σ by
(45)
Pe VEY.
VEY.
(pxa - In P(Y)))
By the Chernoff bound, for the generated unseen-elass data {(Tte ye)h we have that
P [ (p(y) P. 2 8| S 2mm²
(46)
From the union bound, and the classes are countable, we have that
PΣY.: e: DRL (xL(())))))())))
S Σ
2 4).
(47)
Y.
Assign the probability p(Y,) for Y., and thus Σr, p(Y x) = 1. In this way, we have that
2mn²
Р[3.. DKr (p( 2 8) <
2mt2 Σ .(
(48)
p(Y.)
Y
Y.
Letting e- 22m²² p(Y))()))))))))))) we have that
PΣY Dxa 2 2 Σ () 8.
(49)
where is computed as
In
In 8
p(u.)
(50)
2m
In this way, we can derive that
In p(u:)
In
P DEL (1(l)))))P)) 2
< 8,
(51)
2m
Therefore, with probability least
5, have that
(xL((()))IP(t)) <
P(D2)
(52)
2m
We substitute Eq. (52) into Eq. 45
With probability at least have that
x.. DKa. (p(v æ., + ()]pp( æ.., 2)
In MV.. + In
In
Y
(53)
2m
Y
28

---

C Proof of ELBO in Eq. (7) in the Manuscripts
Proposition The Evidence Lower Bound (ELBO) of the logarithmic posterior probability can be derived as
log m. +) Fl 4)
(a((pl] m, {) +)).
(54)
Proof. The probability p æo) can modeled as a function f(y). Therefore, the logarithmic probability log p(y) computed
Dog(yy = log (v) = log (f ((0)0(0-1)) y)dd
(55)
where (-) is a Dirac function, and ÿ is a intermediate variable. The last equality holds since the proposition of the Dirac function. We introduce variational distribution q Then, log f(y holds that
log f(y) = log (ʃ f(i))(( vdd = log (ʃ (I())
(1)) 5( ydd (())
(56)
(ʃ (()) (0)6(( y) dÿ
= log
4
From the Jensen inequality, we can derive that
/ ((ū)) log f(ū)8(ū y) dÿ (g(()) log f y) ((y)
log f f(y)
(57)
(
To guarantee the boundedness of the ELBO, we ignore the Dirac function. Eq. (57 can be further modeled as
log f(y) 2 ( [log f3)) - (g [log q(v)]-
(58)
Then, substituting f(y) = logp(( m. +) and ((y) = p(ū is: b) we have that
log [... 2) Ep(gl= = ) log po x.,. 4) + Eptole, +) [log p Φ)
Ep .) ()) pp 4) [log pl a +))
log p(ū Tu. +
(59)
Eplgle. 3
log x... 4 + Epcglo,, +)
p x. +)
Eppgj,,
The probability term p(y can be ignored because it appears as a constant term in the variational lower bound (ELBO). It depends only on the model parameters & and independent of the input data æ. As such, it does not affect the gradient computation or the update of model parameters $. Since this term does not contribute to the optimization process, it can safely omitted, simplifying the derivation. In many works, derivation of ELBO commonly omit such constant terms, as they do not affect the optimization objective and can be safely ignored to simplify the computation [?, ?]. Therefore, we model Eq. (59) as
log p(p Ro. $) > E [logplyla. +) (Ku ( }))lp(p(a., $)).
(60)
D Details and Illustration of Class-Domain-Wing Wise Data Generation Pipeline
D.1 Specific Details Hierarchy-Guided Unseen Class Predictor
The hierarchy-guides unseen class predictor identifies the potential unseen classes, which are close to training classes. This achieved by constructing hierarchical semantic tree, where leaf nodes represent training classes and parent nodes represent their superclasses. The tree is expanded by adding leaf nodes of the candidate unseen classes sourced from WordNet or LLMs.. As illustrated in Figure I given the training classes "Goldfish," "Tench," " and "Ray." hierarchical semantic tree is constructed where these classes are set as leaf nodes and their superclass "Fish" set parent node. After LLMs queried, "Salmon" is added leaf node of the candidate unseen class under
29

---

Tench
Shark
minimum
distance
Salmon
Ray
Goldfish
Figure 3: Distance between Classes
the "Fish" superclass. To select the closest candidate unseen class, the predictor computes the cosine similarity between the textual embeddings of candidate unseen classes and given training classes from the text encoder of pretrained CLIP. The top K closest candidates are chosen the identified potential unseen classes. Take the classes shown in Figure as as an example, the cosine similarity between the textual embeddings of "Tench" and candidate unseen classes "Salmon", "Shark" is computed. The candidate unseen class with the highest similarity "Salmon" is chosen as a predicted unseen class.
D.2 Specific Details of Caption- Domain Information Generator
The caption-based domain information generator extracts contextual attributes, such as styles and backgrounds, from the training data to ensure the generated unseen data align with the visual characteristics of training data. This achieved by generating class-specific captions for each training class using VLMs. The generator then computes the similarity between these captions and the corresponding data, selecting the top K, captions with the highest similarity to mitigate hallucination issues. These selected captions are further summarized into top K2 class-specific domain information using L.LMs. Finally, the predicted unseen classes and the summarized domain information are combined into textual prompts, which guide the data generation process using text-to- text-to-image model such as Stable Diffusion. In Figure I for instance, captions for training images of "Tench" are first generated using VLMs. The generator then calculates the similarity between these captions and the corresponding images of "Tench". The top captions with the highest similarity are selected, which deseribe the domain information such as "holded by man", "golden-colored" and "prominent eye". These captions are summarized into top class caass-ppecffic domain information using LLMs, which is presented in the caption template in the red box. Finally, the predicted unseen class "Salmon" is inserted into the template to create an image caption, which is then used as input for Stable Diffusion generate images of "Salmon".
D.3 Prompt Templates
In the manuscripts, we propose to utilize LLMSs and VLMs to identify potential unseen classes and extract domain information of training data. Here provide the utilized 3 prompt templates.
30

---

As to the unseen class predictor, we aim to query the Hypernym of training classes by the following prompt. We first construct the in-context examples for accurate results, as shown in Template !
Template 1. (class) denotes the training class.
Q: What is the Hypernym category of [class 1? A: [class2) is the Hypernym category of [class]].
The prompt is given in Template 2
Template 2. felass) denotes the training class. Q: What is the Hypernym category of [class3]?
Then, with the generated Hypernym, we leverage LLMs to identify potential unseen classes us- ing LLMs, where the in-context examples and prompts are shown in template and template 4 respectively.
Template 3. fclass) denotes the training class. Q: What is the Hyponym category of [classl]? [class2) the Hyponym category of [classl). ).
Template 4. felass) denotes the training class. Q: What is the Hyponym category of [class3]?
We leverage template 5 generate class-specific captions for each training class using VLMs. Template 5. (class) denotes the training class.
user prompt:
This an image of fclass). Summarize the main style, scene, and key elements of this image one sentence.
We leverage template tt summarize captions into class-specific domain information using LLMs Template 6. (class) denotes the predicted unseen class.
system prompt:
As a caption summarizer, your task is to transform the provided captions from their original category new specified category and condense them into concise set of 3 distinct one-sentence captions. Make sure the new captions maintain coherence with the original style but reflect the characteristics of the new target category. Each caption must capture a unique artistic style or visual theme. Only generate the transformed me-sentence captions-no introductions, explanations, or comments. The output should strictly follow this format:
1. Capption 1]
[Caption 2]
3. |Caption 3]
user prompt: Transform and condense the following captions into 3 new one-sentence captions describing (class), each focusing on a distinct artistic style or visual theme.
E
Distribution Alignment
In this paper, adopt prompt learning method to optimize the pretrained model. We propose distribution alignment algorithm which aligns the output distributions of model on seen-class data and generated unscen-class data to maximize the logarithmic posterior probability in open environments. The proposed algorithm is summarized in Algorithm
F Experiment Details in Manuscripts
In this section, we provide more specific details of experiments in the manuscripts. Specifically, we present the specific implementation details, details and extra analysis ablation studies of the manuscripts.
31

---

Algorithm Distribution Alignment Algorithm
Input: Parameters $ Data Dx, Gus Gs, Epoch E, batch-size B Output: Optimized Prompts Umaa iter
Initialize: 0, Vo. S 4
while e _ E do
2: for = 1, 2, .... /D.//B F do
3:
Compute the posterior probability of model output in the current batch of the seen-class dataset D
4:
Accumulate the output posterior probability for distribution alignment into S.
5:
i % 8 = 0 then
6:
SKI= ().
7:
for j = 1,2, /.ll/BB do Compute KL. divergence between the accumulated posterior probability on the seen-class data and the mini-batch of generated unseen-class data du DxL/p(g/x, 4, 2.v)lll f.v)lpp@ v. Update the set SkL.append(da)) end for
8:
9:
10:
11:
Compute LXL based on top K3 smallest in the set SKL LKL Elopks du-
Smmd = 0.
13:
for m = 1, /9.l/B do
Compute MMD loss mandd based on the generated unseen-class data and the seen-class data on the current batch, and save them into Smand end for
16:
Compute LMND based on top K3 smallest lmmdd LMMD = 1 Kg Erapss mmmd- Compute total loss Lestal LCR + aLLKı. + BLMMD Backward and update the prompt o using LCr and Liotal- Clear saved data S = (
17:
19:
else
21:
Compute Lce the mini-batch of seen-class dataset D) Backward and update the prompt o using LCE.
22:
23:
end if
24:
end for
25:
end while
26:
return The updated prompts v.
F1 Implementation details
generalization, train each model for 20 epochs using 4 token prompts in the first 9 transformer layers on both visual and text branch. For cross-datase, evaluation, we train the source model for 4 epochs using prompts in the first 3 transformer layers both visual and text branch. Prompts are randomly initialized with normal distribution except the text prompts of the first layer which initialized with the word embeddings of photo of a". The SGD optimizer is adopted, and the learning rate is set as 0.0025. Hyperparameters for the class-domain data generation pipeline and distribution alignment are determined empirically. Specifically, We a 3 = 1, Ko as 1, Kj as 8, K2 as and K3 as 1. The corresponding hyperparameters are fixed across all datasets and benchmarks.
For www-bassebbeeeeee
For LLMs and VLMs, we use Doubao-pro-1 128k to identifies the potential unseen classes, use LLa LLaVA-v1.6- 2.-.---Vinna-113 13B [39] generate class- mass--peeific captions for each training class, use Llama- 03.1-Instruct. [6]] to summarize captions into class-specific domain information, and use Stable Diffusion v2.1 50 the text-to-image model generate unseen-elass data.
Experiments are performed on an NVIDIA A40 GPU, with at most hours 20 GPU memory required to complete training across 11 datasets.
32

---

Table 5: Ablation study sparse loss computation strategy. "w/o spa" denotes distribution alignment algorithm without sparse loss computation strategy.
Caltech
Pets
Cars
Flowers
Food
Aircraft
DTD
FaroSAT
UCF
|w/l
Ours w/o spa Ours w/o spa Ours w/o spa Ours w/o Ours w/o spa Ours wio spa
Ours w/o spu Ours w/o Ours
Basel
98.52
98.97
93.36
96.01
81,88
82.93
96.68
98,77 91.14
91.39
46.40
48.98
82.41
85.53
95.95
97.17
87.07
89.14
New
94.21
95.85
97.65
79.48
78.09
92.56
92.99
44.03
61.72
71.50
76.92
87,90
79.61 82.53
H
96.32
97.78
92.33
96.82
80,66
81.86
86,39 88.96 91,84
92.18
42,80
46.37
70.58
77.89
85,39
92.30
83,17
85.71
F.2 Details and Extra Analysis of Ablation Study
In this subsection, we present the details in ablation studies in hierarchy-guided unseen class predictor and caption-based domain information generator. Then, we present the extra analysis in ablation studies of the manuscripts.
F.2.1 Details of Hierarchy-Guided Unseen Class Predictor
To evaluate the effectiveness of hierarchy-guided unseen class predictor, we first investigate how the quantity of predicted unseen classes impacts the model performance in open environments. Specifically, we introduce class sampling ratio Sets to control the quantity of predicted unseen classes used for training. Assume that we initially generate N unseen classes for a given dataset, with each class containing M images, we randomly select Scle x N classes from the predicted N unseen classes and then train the model using 8,1 X N x M images of these classes, discarding the remaining classes and their images.
Next, investigate how the quality of predicted unseen classes impacts the model performance in open environments. "Low Similarity" denotes that when predicting unseen classes, we compute the cosine similarity textual seen classes for each candidate class and choose the one with the lowest cosine similarity as the predicted unseen class. "w/oTree" denotes that instead of constructing hierarchical semantic tree to predict unseen classes, we directly ask LLMs to provide a predicted unseen class corresponding to the given base classes.
F.2.2 Details of Caption-Baze Based Domain Information Generator
To evaluate the effectiveness of caption-based domain information generator, we first investigate how the quantity of generated unseen-class images impacts the model performance in open environments. Similarly, we introduce image sampling ratio Sing to control the number of generated images of predicted unseen classes used for training. For given dataset, we initially predict N unseen classes with each class containing M generated images. Based on the image sampling ratio Simg, randomly select 8img X M images from each predicted unseen class. The selected x 8img X M images from N unseen classes are then used for training, while the remaining images are discarded.
Next, we investigate how the quality of generated images from predicted unseen classes impacts the model performance in open environments. We modify the prompts used in our unseen image generator control the quality of image generation. In this work, we use the prompts "A picture of category ", "A photo of category ", and "An image of a category as templates for the Stable Diffusion model when generating images of unseen classes, respectively.
F.3
Extra Analysis
The ablation studies provide a comprehensive analysis of the relationship between distribution distance and accuracy, which aligns with the theoretical analysis. Recall that Theorem shows that the estimation error between the unscen-class data distribution and the generated unseen-class data distribution is upper-bounded, and reducing the distribution gap between generated unseen-class data and seen-class data tightens this bound, thereby improving model performance in open environments. Experimental results from the ablation studies validate this theoretical claim. Specifically, as the distribution distance decreases, accuracy consistently improves across various datasets, particularly for new classes. This confirms that reducing the distribution gap between generated unseen-class and seen-class data leads to more accurate estimation of unseen-class data distribution, enhancing the model's 's generalization ability in vocn-vocabulary learning tasks.
33

---

G Robust Analysis
G.1 Hyperparameter Analysis
We add experiments analyze the sensitivity of hyperparameters Ko, K1, K2, K3 in the data generation pipeline and a, 8 in the loss function. Specifically, we conduct experiments on the EuroSAT dataset by adjusting these hyperparameters. Results Kg, K1, K2, K3 are shown in respectively. Results of a, 3 are presented in Table [1] Results show that the performance remains relatively stable with varying hyperparameter values, indicating that the method is only minimally sensitive to hyperparameter variation.
Table
8 9
Table 6: Ko Analysis
Ko =
Ko = 5
Ko = 10
Ko 20
Base
97.17
96.64
96.43
96.24
New
87.90
87.07
85.85
85.84
H
92.30
91.61
90.83
90.57
Table 7: K2 Analysis
K1 8
K1 = 6
K1 = 4
K1 2
Base
97.17
97.02
96.95
96.83
New
87.90
86.31
86.0
85.85
H
92.30
91.35
91.15
91.01
Table 8: K2 Analysis
K2 3
K2 2
K2
Base
97.17
96.92
96.79
New
87.90
86.59
85.31
H
92.30
91.47
90.69
Table
K3 Analysis
K2
K2 2
K₂ 3
Base
97.17
96.33
96.60
New
87.90
85.36
86.31
H
92.30
90.51
91.16
Table 10: a/3 analysis (a + = 1)
Base
New
H
0.3
0.7
96.88
86. 15
91.20
0.4
0.6
96.86
86.26
91.25
0.5
0.5
97.17
87.90
92.30
0.6
0.4
96.81
87.06
91.68
G.2 Robustness to Long-tailed Setting and Noise
Robustness long-tailed setting. We construct long-tail distribution setting by removing a portion of samples from selected classes and conduct comparative experiments (with the number of samples per class being 16, 16, 16, 10, and 2, respectively).
this setting, we evaluate performance of our current top alignment strategy against sample duplication strategy designed for long-tail settings. The results in Table IT demonstrate that in the long-tailed setting, the sample duplication strategy can improve performance.
34

---

Table 11: Experiments on long-tailed benchmark
Our method
Our method
(with top Kg alignment) (with top Kg alignment + sample duplication)
Base New
95.79
97.00
85.39
87.39
H
90.29
91.94
Table 12: Robustness evaluation the noisy semantic tree
Correct Superclasses
Wrong Superclasses
PromptSRC (Baseline)
Base
97.17
96.17
92.90
New
87.90
86.49
73.90
H
92.30
91.07
82.32
Robustness to noise. To verify the robustness, we conduct an experiment where we select incorrect superclasses on the EuroSAT dataset. Results in Table show that the performance slightly degrades, demonstrating the robustness of our method.
H Extra Ablation Studies
In this section, we present extra ablation studies for validating the effectiveness of sparse loss computation strategy in distribution alignment algorithm. We demonstrate the effectiveness of the sparse loss computation strategy by conducting experiments on the distribution alignment algorithm without it, denoted as 'w/o spa'. The results, shown in Table 13 reveal that the sparse loss computation strategy significantly improves performance, particularly on the new classes. Notably, on the Pets, Cars, DTD, and ET datasets, the strategy achieves improvements of 6.32%, 9.1 9.18%, 9. 9.78%, and 10.98% on the new classes compared to wwo spa'. These results further confirm the effectiveness of proposed strategy.
I Visualization
In this section, we visualize the unseen-class images generated to demonstrate the effectiveness. of the proposed class-domain-we
data generation pipeline. We compare the proposed method with three prompt templates for the text-to-image model mentioned in the ablation studies, namely, 'A picture of class", photo of class', and An image of a class'. We use the images generated based on the 1altech101 dataset for analysis.
We use the caption- based domain information generator to capture the class-specific domain informa- tion of seen-class data. This domain information is then used to generate the corresponding seen-class data via a text-to- to--age model. For visualization, we adopt the seen classes 'motorbike' and 'barrel'. As shown in Figures 4and compared to the three commonly used prompt templates, the generated seen-class data from the proposed pipeline better align with the seen-class data terms of both style and scene information. This demonstrates that our pipeline is more effective at capturing the domain information of seen-class images for data generation.
Regarding the generation of unseen-class data, we use the hierarchy-based unseen class predictor to infer that the unseen classes 'car' and 'drum' are closest to 'motorbike' and 'barrel', respectively. The captured class-specific domain information and inferred unseen classes are then used to generate the unseen-class images via a text-to-image model. We compare the generated unseen-class data from our pipeline with data generated using the three commonly used prompt templates. As shown in Figures and [5] the generated unseen-class data align better with the seen-class data. For example, in Figure 4 the car generated by our pipeline reflects the style of the seen-class data, and the realistic scene depicted in the generated images mirrors the scene the seen-class data. These results further demonstrate that our pipeline effectively captures the domain information of seen-class images, and generated unseen-class images align closely with seen-class data, confirming the effectiveness of the proposed pipeline.
35

---

Table 13: Ablation studies on class quality and data quality. "Ace" denotes the accuracy. "Dis" denotes the distribution distance of the generated unseen-class data and seen-class data.
Ours
Class Quality
Data Quality
Dataset
Acc
Dis 4
LowSim
w/o Tree
Picture
Photo
Image
Ace f
Dis
Ace f
Dis
Ace f
Dis
Acc 1
Dis
Acc
Dis
Base
98.97
98.26
97.93
98.26
98.19
98.06
Caltech
New
95.85
9.99
94.32
10.49
93.67
13.16
94.21
11.84
94.11
11.95
93.78
12.12
H
97.38
96.25
95.75
96.19
96.11
95.87
Base
96.01
95.85
95,00
95.69
95.43
95.27
Pets
New
98.27
7.58
97.60
8.19
96.81
11.71
97.04
9.14
96.98
9.21
96.92
10.27
97.12
96.72
95.90
96.36
96.20
96.09
Base
82.93
78.94
77.86
77.99
78.71
78.24
Cars
New
80.81
8,92
75.29
9.33
73.36
13.78
74.75
10.65
75.04
10.08
74.92 10.48
81.86
77.07
75.54
76.33
76.83
76.54
Base
98.77
98.29
97.15
97.82
97.91
97.63
Flowers
New
80.92
6.29
77.52
6.88
75.04
10.29
76.88
7.99
77.09
7.84
76.17
8.22
H
88.96
86.68
84.67
86.09
86.26
85.57
Base
91.39
90.90
90.45
90.56
90.60
90.81
Food
New
92.99
9.01
92.25
9.49
91.79
13.80
91.83
11.24
91.97
11.10
92.02 10.94
92.18
91.57
91.12
91.19
91.28
91.41
Base
48.98
43.88
42.56
42.98
42.80
43.64
Aircraft
New
44.03
8.63
37.97
8.89
34.85
15.79
36.23
12.86
35.51
13.64
36.89
12.04
46.37
40.71
38.32
39.32
38.82
39.98
Base
85.53
83.91
81.48
83.22
82.87
83.57
DTD
New
71.50
7.67
65.22
8.01
57.73
8.84
63.77
8.38
62.32
8.73
63.89
8.20
H 77.89
73.39
67.58
72.21
71.14
72.41
Base 97.17
94.71
91.05
91.83
92.12
91.62
EuroSAT
New
11.48 92.30
87.90
80.49
11.71
65.28
12.39
68.08
11.88
71.67
11.82
67.41
12.07
87.02
76.04
78.19
80.62
77.67
Base
89.14
86.97
85.88
86.25
86.66
86.14
UCF
New
82.53
11.70
77.99
12.29
76.53
13.39
77.45
12.70
77.88
12.67
77.29
13.06
85.71
82.23
80.94
81.61
82.04
81.47
36

---

Seen-Class Data
Generated Seen-Class Data
Generated Seen-C lass Data from Our Pipeline
Template 'picture'
Template 'photo'
Template 'image'
Generated Seen- Data from Three Templates
enerated Unseen-Class Data
G Generated Inseen-Class Data from Our Pipeline
Template 'picture'
Template 'photo'
Template 'image'
Generated Cassen-Class Data from Three Templates
Figure 4: Comparison between the images generated with class-domain wise data generation pipeline and three prompt templates mentioned in ablation studies. The seen class is 'motorbike' and the inferred unseen class is 'car'.
37

---

Seen-Class Data
Generated Seen-Class Data
Generated Seen-Class Data from Our Pipeline
Template 'picture' Generated Seen-Class Data from Three Templates
Template 'photo'
Template 'image'
G enerated Unseen-Class Data
Generated Cassen-Cllass Data from Our Pipeline
Template 'picture'
Template 'photo'
Template 'image'
G enerated Unseen-Class Data from Three Templates
Figure 5: Comparison between the images generated with class-domain-weer wise data generation pipeline and three prompt templates mentioned in ablation studies. The seen class is 'barrel' and the inferred unseen class is 'drumm.
38


Reviews:
[{"review_id": "PW8eXBxZRI", "reviewer": "Reviewer_BCAC", "summary": "This paper introduces a novel framework for open-vocabulary learning by addressing the challenge of distribution estimation in open environments. Specifically, the method consists of two main components: (1) a class-domain-wise data generation pipeline that generates unseen-class data under the guidance of a hierarchical semantic tree and domain information inferred from the seen-class data, and (2) a distribution alignment algorithm that estimates and maximizes the posterior probability to enhance generalization in open-vocabulary learning. Experiments on benchmark datasets show significant improvements over existing methods.", "questions": "1. Alignment is performed on the top K3 most similar distributions. Would such a setting ignore the long-tail distribution categories?\n2. How to handle the noisy semantic tree and noisy generated noisy images?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper is well-structured, logically organized and easy to follow. It shows comprehensive analyses of the motivation of the studied problem and the proposed method.\n\n2. The paper provides theoretical results and proofs, which enhances the depth of understanding of the algorithm's performance.\n\n3. Extensive experiments on various datasets are conducted to prove the effectiveness of the proposed approach. The empirical results demonstrate superior or comparable performance to the mentioned methods.\n\nWeaknesses:\n\n1.The corresponding hyperparameters are fixed across all datasets and benchmarks. It is suggested that the authors briefly discuss parameter sensitivity.", "was_revised": true, "final_justification": "The authors have addressed my previous concerns. I keep my score.", "rebuttal": "Thanks for your insightful and thorough review. We will address your concerns one by one.\n\n> **W1:** The corresponding hyperparameters are fixed across all datasets and benchmarks. It is suggested that the authors briefly discuss parameter sensitivity.\n\n**R:** Thanks for your suggestions. We add experiments to analyze the sensitivity of hyperparameters $K_0, K_1, K_2, K_3$ in the data generation pipeline and $\\alpha, \\beta$ in the loss function. Specifically, we conduct experiments on the EuroSAT dataset by adjusting these hyperparameters. Results presented in Table 1 show that the performance remains relatively stable with varying hyperparameter values, indicating that the method is only minimally sensitive to hyperparameter variation. \n\n\n **Table 1(a): $K_0$ analysis**\n|   | $K_0 = 1$ | $K_0 = 5$ | $K_0 = 10$ | $K_0 = 20$ | \n|:-:| :----: | :--------: | :--------: | :--------: | \n|Base|   97.17 | 96.64    | 96.43 | 96.24 |\n|New | 87.90  |  87.07    | 85.85 | 85.84| \n|H    | 92.30  | 91.61   | 90.83 | 90.57 |\n\n \n**Table 1(b): $K_1$ analysis**\n|   | $K_1 = 8$ | $K_1 = 6$ | $K_1 = 4$ |  $K_1 = 2$ | \n|:-:| :----: | :--------: | :--------: | :--------: |\n|Base|   97.17 | 97.02    | 96.95 | 96.83 |\n|New | 87.90  |  86.31    | 86.0 | 85.85 | \n|H    | 92.30  | 91.35    | 91.15 | 91.01 |\n\n\n\n \n**Table 1\\(c\\): $K_2$ analysis**\n|   | $K_2 = 3$ | $K_2 = 2$ | $K_2 = 1$ | \n|:-:| :----: | :--------: | :--------: |\n|Base|   97.17 | 96.92    | 96.79 |\n|New | 87.90  |   86.59    | 85.31 |\n|H    | 92.30  | 91.47   | 90.69 |\n\n\n\n**Table 1(d): $K_3$ analysis**\n|   | $K_3 = 1$ | $K_3 = 2$ | $K_3 = 3$ | \n|:-:| :----: | :--------: | :--------: |\n|Base|   97.17 | 96.33    | 96.60 |\n|New | 87.90  | 85.36   | 86.31 |\n|H    | 92.30  | 90.51    | 91.16 |\n  \n**Table 1(e): $\\alpha$/$\\beta$ analysis $(\\alpha + \\beta = 1)$**\n| α | β | Base | New | H |\n|----|----|------|------|------|\n| 0.3 | 0.7 | 96.88 | 86.15 | 91.20 |\n| 0.4 | 0.6 | 96.86 | 86.26 | 91.25 |\n| 0.5 | 0.5 | 97.17 | 87.90 | 92.30 |\n| 0.6 | 0.4 | 96.81 | 87.06 | 91.68 |\n\n\n> **Q1:** Alignment is performed on the top K3 most similar distributions. Would such a setting ignore the long-tail distribution categories?\n\n**R:** Following the baseline and most prompt learning methods, our experiments focus on settings with balanced data distributions, where the long-tail issue does not arise. Under these conditions, performing alignment on the top $K_3$ most similar distributions is reasonable and effective.\n\nThanks for your suggestions. To further investigate, we construct a long-tail distribution setting by removing a portion of samples from selected classes and conduct comparative experiments (with the number of samples per class being 16, 16, 16, 10, and 2, respectively). In this setting, we evaluate the performance of our current top $K_3$ alignment strategy against a sample duplication strategy designed for long-tail settings. The results in Table 2 demonstrate that in the long-tailed setting, the sample duplication strategy can improve performance.\n\n**Table 2: Experiments on long-tailed benchmark**\n  |   | Our method (with top $K_3$ alignment) | Our method (with top $K_3$ alignment + sample duplication) |\n  |:-:| :-------------------------------: | :------------------------------------------------------: |\n  |Base| 95.79 | 97.0 |\n  |New | 85.39 | 87.39 |\n  |H    | 90.29 | 91.94 |\n\n>**Q2:** How to handle the noisy semantic tree and noisy generated noisy images?\n\n**R:** We separately analyze the robustness of our method to noisy semantic trees and noisy generated images. Below are details.\n\n\n- Noisy semantic trees. The hierarchy-guided unseen class predictor can mitigate the effect of noisy semantic trees through a cosine similarity-based candidate selection strategy, which effectively filters out irrelevant or overly distant candidate unseen classes. \n\n\tTo verify the robustness, we conduct an experiment where we select incorrect superclasses on the EuroSAT dataset. Results in Table 3 show that the performance slightly degrades, demonstrating the robustness of our method.\n\n\t**Table 3: Robustness evaluation to the noisy semantic tree**\n\t|   | Correct Superclasses | Wrong Superclasses | PromptSRC (Baseline) |\n\t|:-:| :-----------: | :-----------: | :-----------: |\n\t|Base|   97.17    | 96.17       | 92.90 |\n\t|  New | 87.90    | 86.49       | 73.90 |\n\t|  H | 92.30      | 91.07       | 82.32 |\n\n- Noisy generated images. The caption-based domain information generator can mitigate the effect of noisy generated images by extracting diverse and class-relevant domain attributes (e.g., styles, scenes) from seen-class data to guide image generation more accurately.\n \n\tTo evaluate the quality of the generated images, we compute CLIPScore on six datasets, as shown in Table 4. All results exceed the threshold of 0.35, which indicates high semantic quality [1,2]. We further conduct a user and GPT-based evaluation by rating 200 randomly sampled images on a 1–5 scale, where 5 denotes the highest quality. The average scores are 4.67 from human annotators and 4.59 from GPT-4. These results confirm the high visual and semantic quality of the generated images.\n \n\t**Table 4: CLIPScore of generated images**\n\t| UCF101  | DTD | SUN397 | Caltech101 | OxfordPets | StanfordCars\n\t|:-:| :-----------: | :-----------: | :-----------: | :-----------: | :-----------: |\n\t|0.43|   0.42    | 0.43       | 0.43 | 0.44 | 0.42 |\n\n\t\n- Furthermore, the proposed sparse loss computation strategy (Sec. 5.2) also contributes to handling both types of noise by selectively aligning the most relevant distributions, thereby mitigating the effect of noise during training.\n\t\n[Reference]\n\n[1] Wang L, Liu D, Liu X, et al. Scaling Down Text Encoders of Text-to-Image Diffusion Models[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 18424-18433.\n\n[2] Bakr E M, Sun P, Shen X, et al. Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 20041-20053."}, {"review_id": "Ygjihge402", "reviewer": "Reviewer_YUQg", "summary": "This work proposes a novel approach to the problem setting of open-vocabulary learning. The key idea is to generate unseen-class data for distribution estimation in open environments. The work provides the theoretical bounds that indicate that the gap between the joint distributions of generated unseen-class data and unseen-class data can be narrowed down by reducing the distance between the joint distributions of generated unseen-class data and seen-class data. Empirically, this work demonstrates state-of-the-art performance on 11 datasets, outperforming prompt-based baselines, especially in base-to-new and cross-dataset generalization.", "questions": "1. The main concern is according to Weakness 1. The assumption (a) is commonly used in the area of open-vocabulary learning, so it is fair that this paper also uses this assumption. Under this assumption, it is important to show that there is a potential to extend the method to the truly unknown classes. However, in the paper, even if the authors claim that they will extend our analysis and method to hyperbolic spaces to capture the complex geometric properties, it is hard to see why it is possible to extend the current method. For example, for the truly unknown classes, how can the LLMs and WordNet predict them? Moreover, even if a new word is predicted, how can the text-to-image model generate the synthetic data, or can we still trust the synthetic data?\n2. Another concern is according to Weakness 1, but assumption (b). In general, the basic assumption of open-vocabulary learning is that the model has never seen the visual examples of the unseen classes during training. However, the pre-trained text-to-image models likely saw examples of unseen classes during training. Is this a kind of data leakage risk?\n\nPersonally, I appreciate that this work shows the theoretical bounds of the method. However, the main concerns make me worry about the feasibility of applying this method in a real open-world environment. If the author addresses my concern, I will improve the score to 5.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This work shows the theoretical bounds of estimation errors in joint and posterior distributions when using generated unseen-class data, which provides theoretical support to the method.\n2. The method demonstrates strong empirical results across 11 diverse datasets. In addition, this work also conducted sufficient ablation studies.\n\nWeaknesses:\n1. The proposed method is based on two very strong assumptions: (a) The unseen classes are semantically close to the seen ones so that the WordNet or LLMs can predict the name of the unseen classes. (b) The images of the predicted unseen classes can be generated by the pretrained text-to-image model.\n2. There is no discussion of computational efficiency in the paper. Since the method depends on other models like some LLMs and Stable Diffusion, the computational cost can also be much higher.", "was_revised": true, "final_justification": "My main concern is the data leakage problem. The authors address my main concern by explaining the problem setting using the previous works and showing the results of the EuroSAT dataset, where the class overlap is 0%.", "rebuttal": "Thanks for your insightful and thorough review. We will address your concerns one by one.\n\n>**W1:** There is no discussion of computational efficiency in the paper. Since the method depends on other models like some LLMs and Stable Diffusion, the computational cost can also be much higher.\n\n**R:** Thank you for the suggestion. We provide more details regarding the computational efficiency of our method, which will be added to the revised version.\n\nThe training memory and training time of the component of our method on RTX 4090 (using the Eurosat dataset as an example) are as follows:\n- VLMs (LLaVA-1.5-7B) for generating image captions: 15.65 GB GPU memory and 121.8 seconds for 80 images;\n- LLMs (Llama-3.1-8B-Instruct) for extracting domain information: 16.14 GB GPU memory and 11.2 seconds for 5 prompts;\n- Stable Diffusion for generating images: 7.13 GB GPU memory and 480 seconds for 240 images;\n- Training for models: 6.20 GB GPU memory and 126.2 seconds.\n\nSince these steps run sequentially without needing to load all into memory, the overall method requires at most 16.14 GB and 739.2 seconds. The baseline (PromptSRC) requires 6.12 GB and 101.25 seconds. The added time and memory mainly come from data generation.\n\nAt inference time, no extra parameters or computation are introduced. Thus, our runtime (3.8 seconds) and memory (1.84 GB) are identical to the baseline. In summary, although our method requires additional computation time and memory usage, the performance improvement of up to 14% makes this cost worthwhile.\n\n>**Q1:** The main concern is according to Weakness 1. The assumption (a) is commonly used in the area of open-vocabulary learning, so it is fair that this paper also uses this assumption. Under this assumption, it is important to show that there is a potential to extend the method to the truly unknown classes. However, in the paper, even if the authors claim that they will extend our analysis and method to hyperbolic spaces to capture the complex geometric properties, it is hard to see why it is possible to extend the current method. For example, for the truly unknown classes, how can the LLMs and WordNet predict them? Moreover, even if a new word is predicted, how can the text-to-image model generate the synthetic data, or can we still trust the synthetic data?\n\n**R:** Our future work to explore hyperbolic spaces is aimed at addressing the issue of inconsistency between the geometric structure of data and the Euclidean spaces used in models, and leveraging the hyperbolic manifolds to obtain the more robust data representation. This future work is not intended to break assumption (a) (the unseen classes are semantically close to the seen ones so that the WordNet or LLMs can predict the name of the unseen classes) to extend to truly unknown classes.\n\nIn fact, compared to existing methods, our method exhibits greater potential to handle truly unknown classes. From the conducted experiments, we observe that \n1. In cross-dataset experiments, seen classes from ImageNet are semantically far from target classes such as those in EuroSAT or DTD. Our method still achieves 9.48% and 1.4% improvements, respectively. \n2. In base-to-new experiments, on EuroSAT, we surprisingly find that our predicted unseen classes are entirely different from the test unseen classes, with no overlap between them. Our method still achieves a 14% improvement over the baseline. \n\nThese improvements stem from that our method can provide accurate and effective distribution estimation in open environments, without a strong constraint that the unseen classes are semantically close to seen classes.\n\nIn future work, to extend our method to support truly unknown classes, such as newly cartoon characters and emerging vehicle models, we plan to integrate Retrieval-Augmented Generation (RAG) mechanisms that dynamically retrieve emerging classes from external sources, enabling the prediction pipeline to go beyond static knowledge in WordNet and LLM. Then, we plan to retrieve relevant images from the Internet as in-context examples to enhance the ability of the model to generate up-to-date images, enabling the model to adapt in real time to new concepts.\n    \n>**Q2**: Another concern is according to Weakness 1, but assumption (b). In general, the basic assumption of open-vocabulary learning is that the model has never seen the visual examples of the unseen classes during training. However, the pre-trained text-to-image models likely saw examples of unseen classes during training. Is this a kind of data leakage risk?\n\n**R:** We consider that there is no risk of data leakage in our method. The key reason is that, based on related literature, the core constraint in open-vocabulary recognition is that the model should not have access to labels or explicit clues for unseen classes during training [1, 2], rather than \"the model has never seen the visual examples of the unseen classes during training\". For example, the work [3] explicitly explores ways to utilize pretrained VLMs to obtain image-level clues for unseen classes; some methods [4,5,6] also leverage large-scale additional image data to tune CLIP for improvement in open-vocabulary tasks, which inevitably introduces overlap with categories in the test set. Especially, the work [4] conducts prompt learning on real images of 21,841 classes from ImageNet21K, and we find that the class overlap rate between the training set and the test set reaches up to 23%. These observations collectively suggest that the core constraint in open-vocabulary learning lies in the absence of category-level clues about unseen classes during training, rather than a strict prohibition of any form of exposure to their visual instances.\n\n\nThe difference between our method and the above methods is that we utilize synthetic images whose categories are predicted by LLMs or WordNet, whereas prior work relies on real images collected manually. In our method, while predicted classes may match the test classes, we find that the class overlap rate across datasets is at most 6% and at least 0%, being significantly lower than that in previous work (23%) [4]. Apart from this, our method indeed does not use any images or information from the test set. Therefore, we believe that there is no risk of data leakage in our method.\n\n\nMoreover, we added an additional analysis in our experiments. On the EuroSAT dataset, where the class overlap is 0% (i.e., no predicted unseen classes match the test classes), our method still achieves up to a 14% improvement. Additionally, we conduct an experiment by removing all overlapping test classes. Evaluation on the remaining test classes shows that our method continues to perform strongly, achieving up to an 8.29% improvement on the DTD dataset. These confirm that the improvements of our method are not attributed to the class overlap, which the reviewers refer to as \"data leakage\".\n\n\n[Reference]\n\n[1] Ma Z, Zhang S, Wei L, et al. Ovmr: Open-vocabulary recognition with multi-modal references[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 16571-16581.\n\n[2] He S, Guo T, Dai T, et al. Open-vocabulary multi-label classification via multi-modal knowledge transfer[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(1): 808-816.\n\n[3] Zang Y, Li W, Zhou K, et al. Open-vocabulary detr with conditional matching[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 106-122.\n\n[4] Ren S, Zhang A, Zhu Y, et al. Prompt pre-training with twenty-thousand classes for open-vocabulary visual recognition[J]. Advances in Neural Information Processing Systems, 2023, 36: 12569-12588.\n\n[5] Xu Y, Zhang M, Fu C, et al. Multi-modal queried object detection in the wild[J]. Advances in Neural Information Processing Systems, 2023, 36: 4452-4469.\n\n[6] Kaul P, Xie W, Zisserman A. Multi-modal classifiers for open-vocabulary object detection[C]//International Conference on Machine Learning. PMLR, 2023: 15946-15969."}, {"review_id": "nmQK1kUhop", "reviewer": "Reviewer_2wK7", "summary": "This work proposes a method for estimating the distribution of unseen classes in the field of open-vocabulary learning, by generating of samples from unseen classes. It also provides a theoretical analysis of the estimation error bounds. Based on this theory, the paper propose an open-vocabulary learning method that constructs unseen-class samples to maximize the posterior probability, achieving significant performance improvements.", "questions": "(1) Include the supplementary materials and adjust the content between the main text and the supplementary materials accordingly.\n\n(2) In Section 5.1 \"Class-Domain-Wise Data Generation Pipeline,\" the data generation process control could be made more rigorous.\n\n(3) The sparse loss computation strategy is well-designed, but it could be explored in greater depth.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\nThe theoretical analysis is solid, the method is cleverly designed, and the results are remarkable. The experiments are thorough, and the paper is logically rigorous.\n\nWeaknesses:\n\n(1) The supplementary materials need to be included, as too many important details have been omitted.\n\n(2) The paper lacks a framework diagram and sample visualizations, making it difficult for readers to understand.", "was_revised": true, "final_justification": "I maintain my previous rating.\n\nI have read the author rebuttal and found that it addressed my requests. I am also satisfied with the authors’ responses in the discussions with other reviewers. Considering these points, I have no remaining concerns and therefore keep my current rating.", "rebuttal": "Thanks for your insightful and thorough review. We will address your concerns one by one.\n\n>**W1:** The supplementary materials need to be included, as too many important details have been omitted. The paper lacks a framework diagram and sample visualizations, making it difficult for readers to understand.\n>**Q1:** Include the supplementary materials and adjust the content between the main text and the supplementary materials accordingly.\n\n**R:** Thanks for your suggestions. We will add the necessary important details in the supplementary materials to the revised version, including the framework diagram and sample visualizations.\n\nSpecifically, we will add the formulation of the class-domain-wise data generation pipeline (shown in Figure 1 of the supplementary materials) to Section 5. The sample introductions and specific details of the proposed method presented in Sections 4.1 and 4.2 of the supplementary materials will be merged into Section 5.1 of the revised version. Moreover, we will add the visualization of extracted domain information and generated unseen-class data (shown in Section 8 in the supplementary materials) to Section 6 of the revised version.\n\nTo accommodate these changes within the page limit, we will reduce the content related to the ablation studies (Section 6.4). Some detailed descriptions and results will be moved to the supplementary materials accordingly.\n\n>**Q2:** In Section 5.1 \"Class-Domain-Wise Data Generation Pipeline,\" the data generation process control could be made more rigorous.\n\n**R:** The rigorous control over the data generation process is indeed essential to ensure the quality and alignment of the synthetic unseen-class data, further leading to accurate distribution estimation. \n\nTo this end, our current pipeline already incorporates multiple design elements aimed at minimizing noise and improving distribution alignment. The unseen class predictor utilizes a cosine similarity-based candidate selection strategy to effectively filter irrelevant or overly distant candidate unseen classes. The caption-based domain information generator extracts style and scene attributes from seen-class data to guide image generation more accurately. Moreover, we use multiple class-specific captions to extract diverse domain information, thereby enhancing the diversity of generated images and reducing unnecessary semantic noise.\n\nThanks for your suggestions. In future work, we plan to enhance the pipeline by constructing confidence thresholds and entailment constraints on semantic tree construction for robust unseen class predictions, and introducing CLIP-based filtering to discard generated images with low semantic consistency. \n\n>**Q3:** The sparse loss computation strategy is well-designed, but it could be explored in greater depth.\n\n**R:** Thank you for the positive feedback on the proposed sparse loss computation strategy. We design it to alleviate distribution misalignment caused by per-batch variability. To further explore and validate its effectiveness, we conduct additional experiments, including ablations and hyperparameter analysis. Specifically, we compared our method, which aligns the distributions of stored seen-class batches with those of the top-$K_3$ most similar unseen-class batches, with two alternative strategies:\n1. Random batch matching: seen-class distributions are aligned with randomly sampled unseen-class batches;\n2. Least-similar batch matching: alignment is performed with the top-$K_3$ unseen-class batches that are least similar (i.e., worst-case alignment).\n\nThe results shown in Table 1(a) clearly demonstrate that our method significantly outperforms both alternatives, confirming the effectiveness of the proposed sparse loss computation strategy.\n\n\n**Table 1(a): Alignment strategy analysis**\n|   | Our method | Random batch matching | Least-similar batch matching | \n|:-:| :----: | :--------: | :--------: |\n|Base|   97.17 |  96.43   | 96.02 |\n|New | 87.90  |  84.69   | 82.87 |\n|H    | 92.30  | 90.18    | 88.96 |\n\n\nAdditionally, we conduct a hyperparameter study on the value of $K_3$ (the number of most similar unseen-class distributions used for alignment). Experimental results presented in Table 1(b) show that performance remains robust as $K_3$ varies.\n\n**Table 1(b): K_3 Analysis**\n|   | $K_3 = 1$ | $K_3 = 2$ | $K_3 = 3$ | \n|:-:| :----: | :--------: | :--------: |\n|Base|   97.17 | 96.33    | 96.60 |\n|New | 87.90  | 85.36   | 86.31 |\n|H    | 92.30  | 90.51    | 91.16 |\n  \n \nIn future work, we aim to explore more advanced variants of this strategy, including adaptive memory weighting (e.g., attention mechanisms) and class-aware matching. These directions could further improve the robustness and generalization of distribution alignment in open environments."}, {"review_id": "HBbzD8soAV", "reviewer": "Reviewer_gXSZ", "summary": "The paper tackles the open-vocabulary learning challenge by generating synthetic data for unseen classes to address the distribution estimation problem. Theoretical analysis shows that this approach can bound the estimation error (Theorems 1 and 2). The method includes a data generation pipeline using semantic trees and domain info to guide image synthesis, and a distribution alignment algorithm with KL-divergence and MMD losses.  Experiments on 11 datasets demonstrate SOTA performance. Ablations highlight the importance of each component.", "questions": "1)\tData Generation Quality: How does the quality of generated unseen-class data impact the overall performance of the method? Could the authors provide more details on the quality metrics used for evaluating the generated data?\n2)\tScalability to Large-Scale Unseen Classes: The class predictor uses top-K_0 candidates from a hierarchical tree. How does performance degrade as K_0 increases (e.g., 100+ classes)? Provide metrics (e.g., distribution distance, accuracy) for larger K_0 in the rebuttal.\n3)\tHow robust is the distribution alignment algorithm to noise in the generated data? Could the authors discuss potential improvements to enhance the robustness of the method?", "limitations": "Limitations are discussed (Section 8): The Euclidean feature space may not capture complex data geometries. Future work will explore hyperbolic spaces. However:\n- Computational Costs: Add inference time/memory metrics (e.g., vs. PromptSRC).\n- No discussion of biases in generated data (e.g., via WordNet hierarchies). \nFurther discussion is needed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1)\tThe paper introduces a new approach to open-vocabulary learning by generating unseen-class data, which is a unique combination of data generation and distribution estimation.\n2)\tThe theoretical analysis provides a solid foundation for the proposed method, and the experimental results demonstrate significant improvements over existing methods.\n3)\tThe method addresses a critical challenge in open-vocabulary learning, offering substantial improvements in generalization to unseen classes, which is highly impactful for the research community.\n4)\tThe paper is well-written, with clear problem formulation (Section 3), method description (Section 5), and experimental analysis (Section 6). Theorems are intuitively interpreted.\n\nWeaknesses:\n1)\tData Generation Limitations: The quality of generated unseen-class data may be limited by the text-to-image model and the domain information extracted, potentially affecting the overall performance.\n2)\tGeneralization Scope: Experiments focus on image recognition; validation on more complex tasks (e.g., object detection) is needed.\n3)\tComputational Cost: The pipeline (LLM queries, Stable Diffusion) incurs high inference overhead. Efficiency analysis (e.g., latency vs. performance) is omitted.\n4)\tTheoretical Assumptions: Theorem 2 assumes p(Y_e)>0, which may not hold for highly dissimilar unseen classes. Sensitivity to this assumption is untested.", "was_revised": true, "final_justification": "The authors have addressed my previous concerns. I raised my score.", "rebuttal": "Thanks for your insightful and thorough review. We will address your concerns one by one.\n\n>**W1:** Data Generation Limitations: The quality of generated unseen-class data may be limited by the text-to-image model and the domain information extracted, potentially affecting the overall performance. \n**Q1:** Data Generation Quality: How does the quality of generated unseen-class data impact the overall performance of the method? Could the authors provide more details on the quality metrics used for evaluating the generated data?\n\n**R:** The quality of generated unseen-class data affects overall performance, as validated in Section 6.4.1. The results (Table 3 in the manuscript) show that higher data quality leads to improved overall performance. The quality metric is the KL divergence between feature distributions of generated unseen-class data and seen-class training data, extracted by a non-tuned CLIP. A smaller divergence indicates higher quality. As shown in Table 3, our method achieves the best generation quality and performance compared to using plain prompts in text-to-image models.\n\nThe effectiveness of our method stems from the proposed caption-based domain information generator, which generates high-confidence and diverse domain prompts. With such carefully constructed prompts, current text-to-image models can generate high-quality images [1,2]. To further evaluate the generated image quality, we computed CLIPScore [3], as shown in Table 1. Results show that these images exhibit high semantic quality (CLIPScore >0.35 is considered high quality [4,5,6]). We further conducted a user study and a GPT-4-based evaluation, where both human annotators and GPT-4 independently rated 200 randomly selected images on a 1–5 scale (5 is the highest quality). The resulting average scores of 4.67 (human) and 4.59 (GPT-4) confirm the high quality of the generated images.\n\n**Table 1: CLIPScore of generated images**\n| UCF101  | DTD | SUN397 | Caltech101 | OxfordPets | StanfordCars\n|:-:| :-: | :-: | :-: | :-: | :-: |\n|0.43|   0.42    | 0.43       | 0.43 | 0.44 | 0.42 |\n\n[1] ImagineFSL: Self-Supervised Pretraining Matters on Imagined Base Set for VLM-based Few-shot Learning. CVPR2025.\n\n[2] IS SYNTHETIC DATA FROM GENERATIVE MODELS READY FOR IMAGE RECOGNITION? ICLR2023.\n\n[3] CLIPScore: A Reference-free Evaluation Metric for Image Captioning. EMNLP2021.\n\n[4] Scaling Down Text Encoders of Text-to-Image Diffusion Models. CVPR2025.\n\n[5]  Hrs-bench: Holistic, reliable and scalable benchmark for text-to-image models. ICCV2023.\n\n[6] Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS2022.\n\n\n>**W2:** Generalization Scope: Experiments focus on image recognition; validation on more complex tasks (e.g., object detection) is needed.\n\n**R:**  We focus on open-vocabulary distribution estimation, with extensive experiments on 11 recognition datasets across base-to-new and cross-dataset settings, which sufficiently validate the effectiveness of our method. As our method builds on prompt learning, we follow prior works [1–3] and evaluate primarily on image recognition tasks. The proposed method investigates general principles of distribution estimation, and is agnostic to the specific downstream vision task. We appreciate this suggestion and plan to leverage large pretrained models (e.g., SAM or Grounding-DINO) to provide pseudo annotations and adapt our alignment algorithm to object detection and segmentation tasks accordingly.\n\n[1] Learning to prompt for vision-language models. IJCV2022.\n\n[2] Self-regulating prompts: Foundational model adaptation without forgetting. ICCV2023.\n\n[3] Divergence-enhanced knowledge-guided context optimization for visual-language prompt tuning. ICLR2025.\n\n>**W3:** Computational Cost: The pipeline (LLM queries, Stable Diffusion) incurs high inference overhead. Efficiency analysis (e.g., latency vs. performance) is omitted. \n>**L1:** Computational Costs: Add inference time/memory metrics (e.g., vs. PromptSRC).\n\n**R:** Thank you for the suggestion. We provide more details regarding the computational efficiency of our method, which will be added to the revised version.\n\nThe training memory and training time of the component of our method on RTX 4090 (using the Eurosat dataset as an example) are as follows:\n- VLMs (LLaVA-1.5-7B) for generating image captions: 15.65 GB GPU memory and 121.8 seconds for 80 images;\n- LLMs (Llama-3.1-8B-Instruct) for extracting domain information: 16.14 GB GPU memory and 11.2 seconds for 5 prompts;\n- Stable Diffusion for generating images: 7.13 GB GPU memory and 480 seconds for 240 images;\n- Training for models: 6.20 GB GPU memory and 126.2 seconds.\n\nSince these steps run sequentially without needing to load all into memory, the overall method requires at most 16.14 GB and 739.2 seconds. The baseline (PromptSRC) requires 6.12 GB and 101.25 seconds. The added time and memory mainly come from data generation.\n\nAt inference time, no extra parameters or computation are introduced. Thus, our runtime (3.8 seconds) and memory (1.84 GB) are identical to the baseline. In summary, although our method requires additional computation time and memory usage, the performance improvement of up to 14% makes this cost worthwhile.\n\n>**W4:** Theoretical Assumptions: Theorem 2 assumes p(Y_e)>0, which may not hold for highly dissimilar unseen classes. Sensitivity to this assumption is untested.\n\n**R:** In practice, $P(Y_{e}) > 0$ always holds without any constraint on $Y_e$, as analyzed below. We first clarify the definitions and relationships of label sets in open environments. In the open-vocabulary setting, by definition, the open-environment label set $Y_{o}$ is the union of seen-class label set $Y_{s}$ and **unseen-class label set $Y_{u}$**, which is mathematically represented as\n$$\nY_{o} = Y_{s} \\cup Y_{u}, Y_{s} \\cap Y_{u} =  \\emptyset.\n$$This implies that any class in open environments outside the seen class set $Y_{s}$ is considered an unseen class.  Our **predicted unseen-class label set $Y_{e}$** satisfies  $Y_{e} \\cap Y_{s} = \\emptyset$ and thus $Y_{e} \\subset Y_{u}$. The **test  unseen-class label set $Y_{t}$** also satisfies that $Y_{t} \\cap Y_{s} = \\emptyset$ and thus $Y_{t} \\subset Y_{u}$. We clarify that in Theorem 2, **$P(Y_{e})$ is defined as the probability assigned to the set $Y_{e}$ under the distribution over the set $Y_{u}$.** Since $Y_{e} \\subset Y_{u}$, it naturally follows that $P(Y_{e}) > 0$. Moreover, even if $Y_{e} \\cap Y_{t} = \\emptyset$, indicating that $Y_{e}$ may be highly dissimilar to $Y_{t}$,  it still holds that $P(Y_{e}) > 0$. As a result, the assumption is not sensitive, even for highly dissimilar unseen classes. Thank you for pointing this out. We will add a related clarification to the revised version.\n\n>**Q2:** Scalability to Large-Scale Unseen Classes: The class predictor uses top-K_0 candidates from a hierarchical tree. How does performance degrade as K_0 increases (e.g., 100+ classes)? Provide metrics (e.g., distribution distance, accuracy) for larger K_0 in the rebuttal.\n\n**R:** We set $K_0 = 1$ in our main experiments. We analyze the scalability of our method with larger $K_0$  by setting $K_0 = 5, 10, 20$, as shown in Table 2. Results show that our method remains robust across different $K_0$. This robustness stems from our sparse loss computation strategy, which aligns the distribution of seen-class batches with the semantically similar unseen-class distribution, reducing misalignment and exhibiting robustness as $K_0$ increases.\n\n **Table 2: $K_0$ analysis**\n|    | $K_0 = 1$ | $K_0 = 5$ | $K_0 = 10$ | $K_0 = 20$ | \n|:-:| :-: | :-: | :-: | :-: | \n|Base|   97.17 | 96.64    | 96.43 | 96.24 |\n|New | 87.90  |  87.07    | 85.85 | 85.84| \n|H    | 92.30  | 91.61   | 90.83 | 90.57 |\n\n>**Q3:** How robust is the distribution alignment algorithm to noise in the generated data? Could the authors discuss potential improvements to enhance the robustness of the method?\n\n**R:** Our distribution alignment algorithm is robust to noise in the generated data due to its sparse loss computation strategy. Instead of aligning seen classes with all generated batches, it selectively aligns seen-class distributions with the semantically similar generated unseen-class batch, avoiding noisy alignments and even filtering out low-quality samples, thereby improving robustness. To verify this robustness, we conduct an experiment using incorrect superclasses. Results in Table 3 show that the performance only slightly degraded. This demonstrates the robustness of the algorithm to the noise in generated images.\n\n**Table 3: Robustness evaluation to the noisy semantic tree**\n|   | Correct Superclasses | Wrong Superclasses | PromptSRC (Baseline) |\n|:-:| :-: | :-: | :-: |\n|Base|   97.17    | 96.17       | 92.90 |\n|  New | 87.90    | 86.49       | 73.90 |\n|  H | 92.30      | 91.07       | 82.32 |\n\nA potential direction is to incorporate confidence-aware weighting, down-weighting generated samples with low confidence or high uncertainty during alignment. Another is to iteratively refine the generation and alignment stages, enabling the model to improve data quality based on alignment feedback.\n\n>**L2:** No discussion of biases in generated data (e.g., via WordNet hierarchies). Further discussion is needed.\n\n**R:** Thanks for pointing it out. While biases may exist in WordNet or the training data of pretrained language models, these biases do not undermine our core contributions, as our method is empirically validated to be robust to such imperfections.\n\nTo mitigate this issue, we plan to design a multi-expert collaboration strategy that leverages diverse pretrained models and agreement-based selection to reduce dependence and bias on any single model. This discussion will be added to the revised manuscript."}]

Author rebuttal:

Meta-review: The paper addresses open-vocabulary learning: models are required to generalise beyond seen classes. The paper approximates unseen-class distributions with synthetic samples and computes an upper bound on the "estimation error". The authors propose a class-domain-wise data generation pipeline and a distribution alignment algorithm. Across 11 datasets, the method achieves significant improvements over the state-of-the-art prompt-based baselines.

Strengths. The paper provides theoretical rationale supporting the superiority of the proposed approach (Theorems 1 and 2). Introduces the idea of generating unseen-class data guided by semantic hierarchy and domain information, not just prompts. The idea is compared against suitable baselines on 11 datasets, with both base-to-new and cross-dataset settings. Ample ablations (class quality, data quality, alignment strategies). Addresses noise in semantic hierarchies and generated data, shows insensitivity to hyperparameters, and evaluates long-tailed distributions. Shows consistent improvements over baselines, with especially large margins in challenging datasets (EuroSAT, DTD, Aircraft).

Weaknesses. The work relies on the assumption that unseen classes are semantically close to seen ones and assumes text-to-image models can generate reasonable unseen-class data. The pipeline involves LLMs and diffusion models; efficiency analysis was absent in the original submission. Evaluations are restricted to recognition tasks; object detection and segmentation remain untested.

Rebuttal & discussion. The authors successfully addressed remaining concerns for the reviewers. All reviewers expressed satisfaction post-rebuttal, with improved or maintained positive scores.

Final recommendation: Accept as poster. Key reasons: good problem, novel method, good results.
Decision: Accept (poster)