ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning
Authors: Zihan Ye
Shreyank N Gowda
Shiming Chen
Xiaowei Huang
Haotian Xu
Fahad Shahbaz Khan
Yaochu Jin
Kaizhu Huang
Xiaobo Jin
Venue: ICLR 2025 Poster (2025)
Keywords: Zero-shot Learning
Generative Model
Diffusion Mechanism
Effective Learning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We find, quantify and empirically prove a spurious visual-semantic correlation problem amplified by fewer training samples, and we propose a novel data-effective framework ZeroDiff to keep a robust performance under even 10% training set.

Abstract: Zero-shot Learning (ZSL) aims to enable classifiers to identify unseen classes. This is typically achieved by generating visual features for unseen classes based on learned visual-semantic correlations from seen classes. However, most current generative approaches heavily rely on having a sufficient number of samples from seen classes. Our study reveals that a scarcity of seen class samples results in a marked decrease in performance across many generative ZSL techniques.  We argue, quantify, and empirically demonstrate that this decline is largely attributable to spurious visual-semantic correlations. To address this issue, we introduce ZeroDiff, an innovative generative framework for ZSL that incorporates diffusion mechanisms and contrastive representations to enhance visual-semantic correlations. ZeroDiff comprises three key components: (1) Diffusion augmentation, which naturally transforms limited data into an expanded set of noised data to mitigate generative model overfitting; (2) Supervised-contrastive (SC)-based representations that dynamically characterize each limited sample to support visual feature generation; and (3) Multiple feature discriminators employing a Wasserstein-distance-based mutual learning approach, evaluating generated features from various perspectives, including pre-defined semantics, SC-based representations, and the diffusion process. Extensive experiments on three popular ZSL benchmarks demonstrate that ZeroDiff not only achieves significant improvements over existing ZSL methods but also maintains robust performance even with scarce training data.  Our codes are available at https://github.com/FouriYe/ZeroDiff_ICLR25.

Published as a conference paper at ICLR 2025
ZERODIFF: SOLIDIFIED VISUAL-SEMANTIC CORRELATION IN ZERO-SHOT LEARNING
Zihan Ye!.6, Shreyank N Gowda', Shiming Chen', Xiaowei Huang",
Haotian Xu Fahad Shahbaz Khan Yaochu Jin Kaizhu Huang'. Xiaobo Jin" 'ii' an Jiaotong-Liverpood University, UUniversity of Liverpool, UUniversity of Nottingham, * *Mohamed bin Zayed University of Artificial Intelligence, $ Linkköping University, "Westlake University, Duke Kunshan University zithye@cutlook.com
ABSTRACT
Zero-shot Learning (ZSL) aims to enable classifiers to identify unseen classes. This is typically achieved by generating visual features for unseen classes based on learned visual-semantic correlations from seen classes, However, most current generative approaches heavily rely on having sufficient number of samples from seen classes. Our study reveals that a scarcity of seen class samples results in a marked decrease in performance across many generative ZSL techniques. We argue, quantify, and empirically demonstrate that this decline is largely attributable to spu- rious visual-semantic correlations. To address this issue, we introduce ZifoDiff, an innovative generative framework for 7SI. that incorporates diffusion mechanisms and contrastive representations to enhance visual-semantic correlations. ZeroDiff (I) Diffusion augmentation, which naturally transforms limited data into an expanded set of noised data to mitigate generative model overfitting; (2) Supervised-contras
comprises three key components:
(SC)-based representations that dy- namically characterize each limited sample to support visual feature generation; and (3) Multiple feature discriminators employing
mutual learning approach, evaluating generated features from various perspectives, including pre-defined semantics, SC-based representations, and the diffusion pro- cess. Extensive experiments on three popular ZSL benchmarks demonstrate that ZeroDiff not only achieves significant improvements over existing ZSL methods but also maintains robust performance even with scarce training data. Our codes are available at https://github.cc http:/ihhhuhcc/oooouieeeee wewo///orrie//eerDiiii ICLR25.
INTRODUCTION
Machine learning models have achieved remarkable success in data-intensive applications, largely due to the availability of abundant labeled samples. However, collecting extensive labeled datasets often cone-consuming and expensive, making it unrealistic to assume access to substantial volumes of labeled data. To improve data efficiency for new classes, zero-shot learning (ZSL) (Xian et al., 2018a) offers promising solution by transferring knowledge from seen classes to unseen classes through the use of pre-defined class semantic knowledge, such as attributes (Chao et al., 2016) and text-based representations (Zhu et al., 2018; Chen & Yeh, 2021). Generative ZSL methods synthesize features for unseen classes by establishing visual-semantic correlations from seen classes, leading to excellent performance. These methods typically leverage some form of adaptations of generative adversarial networks (GANs) (Han et al., 2021; Gowda, 2023; Hou et al., 2024; Gu et al., 2024) to aid the feature generation.
Despite rapid advances in ZSL, it is generally presumed that there is substantial number of samples available for each seen class. When labeled data for each seen class limited, it remains uncertain whether ZSL can still perform effectively. This oversight becomes evident as few ZSL methods consider the scenario of limited samples during training (Verma al., 2020; Gao al., 2022;
1 Corresponniig author.

---

Published as a conference paper at ICLR 2025
Reil seen class Sautares
lott net trining set
seen class Soatares treening
Teainung seets cless corréation
Mintaslad acarii
Class fix
'Fary'
Single discriminatoe
real
103%
Leamed
Conneptddag iznges
"Paws"
Pro detizad sersnnte
Leared fooe 30% leassng and
Spurious visiou-semantic correlation
Validating seen class commlation Mismmited somontics
Clean real feature desribution
"Red"
Lucnal 10% set
fake
Compepaniing
Singla discriminanoe
Generated (éeture (unseen class)
snago
Tre-daltnd mesedds
Imitted data
semantics
Single-view discriminating
Fails gradually
(a) Standard GAN-based ZSL approach
sesn class Statures treining
Toaniing snen class, cormdation
Multpple distriminatios different views
Rail seen class Sautames
Sowart
Difiiaoos
Class for:
'Redd
Learasl frocs triinng
bott not in triniig set
'Tury"
Mutad
real
Infiste
Compprrding
SC lased ropresentatiess
Substantial
Leereed trazing
vislon-emaaticc corrulation
Validaling seen class comelation
Clesn real visual
Suwarr
Duffuson
Elaos
Leaned froor 10%
festure ditributionn
Whike
Muzal
real
Cunsepadiig tmages
Infinita motel foetew
Pano
Generated festure (unsees, class)
SC-awedd improveations
Multiple discriminatoes from differnt views
Diffuslon augmentation
Dynamic
M3 Multiview discriminating
Succeeds consistently
(b) Our proposed ZeroDiff
Figure 1:
Core idea of our ZifoDiff. (a) Standard GAN-based ZSL approaches suffer from (1) Over- fitting to limited data; (2) Mismatched static pre-defined semantics; (3) Single-view discriminating. Finally, fewer samples in the training set lead to more spurious vision-semantic correlations and feature generation fails gradually. (b) In contrast, the proposed ZeroDiff overcomes these shortcom- infinite features; (2) Dynamic SC-based representations; (3)
ings using: (1) rnffuuiion-augmente Multi-view discriminating, Finally, ZiroDiff learns substantial vision-semantic correlation and keeps a robust performance with even 10% training set.
Tang et al., 2024). Through empirical investigations, we observe that most existing generative ZSL methods suffer from performance degradation and collapsed generation modes as the number of training samples is gradually reduced. As illustrated in the t-SNE plot in Fig. 1 (a), f-VAEGAN (Xian et al., 2019) fails to synthesize unseen classes effectively when the size of the training set diminishes.
Through further empirical analysis, we find that the degradation might be caused by spurious visual-semantic correlation learned from a limited number of seen samples. As illustrated in Fig. 1 (a), we re-split the real seen class samples into two groups: one for training and the other for validating the correlation. We then feed the visual features of the split samples into the GAN discriminator to obtain critic scores (i.e., the output of the discriminator), which indicate whether the discriminator perceives the input features real fake with respect to the class semantics. Next, calculate the difference in critic scores as a measure to determine whether the learned visual-semantic correlation is spurious substantial. Our observations reveal that the number of training samples decreases, the critic score difference becomes increasingly larger, suggesting that the generative models perceive the validating seen class samples as progressively more 'fake.' This phenomenon indicates that limited number of training samples amplifies the spurious visual-semantic correlation. Further details can be found in Fig. 2.
To strengthen the vision-semantic correlation under conditions of limited seen class samples, we propose novel ZSL framework: ZeroDiff. As shown in Fig. (b), our approach motivated by three key aspects: (1) Diffusson augmentation: Limited training samples can be easily memorized by models. We incorporate the diffusion mechanism (Song et al., 2020a; Wang et al., 2023a) into our method, which allows single clean sample to be augmented into an infinite number of noised samples by varying the noise-to-data ratios. (2) Dynamic semanties: Predefined semantics are static, meaning that class semantics remain the same across different instances. However, each limited sample may only represent part of the predefined semantics. For example, in the AWA2 dataset, all images of the 'fox' class are labeled with the semantics 'red' and 'white'. even though this is inaccurate for white foxes. To address this, we revisit the classical Supervised Contrastive (SC) loss (Khosla et al., 2020) and suggest that SC-based representations can generate instance-level semantics for every sample,
2

---

Published as a conference paper at ICLR 2025
enhancing the generation of visual features. (3) Multi-view discriminating: We combine three types of discriminators to assess the authenticity of generated features from different perspectives: predefined semanties, the diffusion process, and SC-based representations. To integrate knowledge from all discriminators, we propose a mutual learning loss based on Wasserstein distance, further reinforcing substantial correlations.
To summarize, our contributions are as follows:
We reveal and quantify the spurious visual-semantic correlation problem, and empirically demonstrate that the problem would be amplified by fewer training samples.
We propose a novel generative ZSL framework, ZeroDiff, which advances more efficient ZSL in scenarios with limited seen class samples. It strengthens the through diffusion augmentation, dynamic representation of limited samples, and multi-view discrimination.
correlation
We introduce a new protocol to evaluate generative ZSL methods under varying data conditions. Experimental results show that ZeroDiff outperforms various generative models across different amounts of training samples.
RELATED WORK
2.1 ZERO-SHIOT LEARNING
ZSL is research area focused on class-level generalizability (Xian et al., 2018a). The primary approaches to ZSL can be categorized into embedding and generative methods. Embedding meth- ods (Akata et al., 2015; Yang et al., 2016; Ding et al., 2017; Chen et al., 2022b; Ye et al., 2023; Zhang et al., 2025) learn a direct mapping from visual to semantic spaces or vice versa. For example, TransZero++ (Chen et al., 2022a) presents cross-modal transformer-be based architecture to ZSL. However, when training samples are limited, embedding methods often underperform compared to generative methods on smaller datasets(Chen al., 2022b; Ye al., 2023).
In contrast, generative ZSL. methods use various generative models (Peng et al., 2025) to synthesize visual features for unseen classes and then train final classifier for these classes. This paper distinguishes itself from previous works in four key aspects: (1) Recent generative ZSL methods have also identified incorrect visual-semantic correlations (Ye al., 2021; Chen al., 2024b), but they lack in-depth quantitative analysis. In contrast, we verify this issue using discriminator scores and demonstrate that the problem exacerbated by a limited number of training samples. (2) Some ZSL works combine GANs and VAEs to address the well-known mode collapse issue (Luo & Yang, 2024), such as F-VAEGAN (Xian et al., 2019), 9), TF- VAEGAN (Narayan et al., 2020), and Bi- VAEGAN (Wang show that VAEGAN-basee methods still experience mode collapse when
al., 2023c). However, the training set is reduced. As a solution, we propose integrating the diffusion mechanism (Ho et al., 2020; Han et al., 2024) mitigate this problem. (3) To address the limited discriminative power of predefined semantics, recent works propose dynamically updating these semantics, like DSP (Chen et al., 2023) and VADS (Hou et al., 2024). Our approach revisits the classical SC learning (Khosla et al., 2020) and argues that SC-based representations can serve as a new source for instance-level semantics due their high intra-class variation (Islam et al., 2021). (4) Recent studies (Clark & Jaini, 2024; Li et al., 2023; Rombach et al., 2022) have shown that specific large-scale diffusion models, like Stable Diffusion, also possess zero-shot classification abilities. However, they do not strictly ensure that unseen classes are excluded from training and rely on large-scale models with huge parameters and extensive training sets. contrast, our interpretation of 'diffusion' stays true to its core principle: a generative paradigm that learns data distributions by denoising noised data. More importantly, our method does not violate the ZSL. premise (Xian et al., 2018a).
2.2 DATA-EFFICIENT GENERATIVE MODELS
Exploring data-efficient generative models is crucial, as the success of generative methods often depends on collecting a vast amount of diverse training samples, which is both costly and challeng- ing (Webster et al., 2019; Saha et al., 2022). In previous works, DiffAugment (Zhao et al., 2020) introduced differentiable augmentation for GANs and successfully trained with only 10% of the data. Ady (Chen et al., 2021a) demonstrated that a specific GAN architecture could reduce the amount
3

---

Published as a conference paper at ICLR 2025
16
125
14
10% tmiining sot
1.0
12
10
30% triining set
8
0.50
025
100% takiniing set
0.00
-0.25
50
100
150
200
250
50
100
150
200
250
300
0
50
100
150
250 Ppach
200
300
350 400
Rpach
Ponch
(a) AWA2
(b) CUB
(c) SUN
Figure 2: The Ando -poch curve for the classical E-VEGAA (Xian et al., 2018b). Larger Aads indicates Dado thinks real testing seen examples are more fake, i.e., learns more spurious visual- semantic correlation.
of required training data using the lottery ticket hypothesis. PatchDiff (Wang et al., 2023a) developed new conditional score function at the patch level. Denoising Diffusion GAN (DDGAN) (Xiao et al., 2022) suggested that combining diffusion models with GANs could reduce overfitting, though without sufficient empirical validation. Generative ZSL methods can also be generative models, as they eliminate the need to collect data or train models for unseen classes. This paper contributes in three key ways: (1) Unlike existing data-efficient generative models, take it step further by reducing the need for unseen class data collection and requiring fewer examples from seen classes. (2) We propose quantitative metrics assess overfitting visual-semantic correlation. (3) We further integrate GANs and diffusion models using Wasserstein-distance-l mutual learning approach distill knowledge across multiple discriminators.
form of data-efficient
3 METHODOLOGY
3.1 NOTATIONS
In the ZSL setting, there are two disjoint label sets: a seen set y used for training and an unseen set )" used for testing, where y n = 0. The training dataset is denoted Der {(x*, , v a²) x" E xx,y" E y², a E ) where X², AA" and ya represent the image, semantic, and label spaces for the seen classes. The goal of ZSL classify unseen images in the testing dataset D D {xx", """,,,"" aa") xx E 2"" E y" a" e A"), i.e. fest X + y" In the Generalized ZSL (GZSL) task, images from seen classes must also be classified during testing. Therefore, portion of the seen class samples is reserved for testing, denoted as Drc.s, other words, the testing dataset becomes D Déc,s D", and the goal
use the training dataset D to create classifier that can
becomes fazst X U X - 1'..
3.2 SPURIOUS VISION-SEMAN SEMANTIC CORRELATION
Given image x" from the training set D existing GAN-based ZSL works a feature extractor Ft pre-trained by Ctoss-Entropy (CE) loss (Eq. 17) to extract its visual features vå = (x*). Then, feature generator takes class semantics a and latent variables inputs synthesize class-specific sample features V Gadv(a", z): and a discriminator Dade used to distinguish real features vे from fake features vg by predicting the Wasserstein distance W adv between them according to predefined semantics a". Specifically, take the baseline f-VAEGAN as an example, its
D adtv maximizes the following loss Ladvi
Ladv W Wade >xgpado Eopade
(1)
E.Dado W E.Dado (vū, a²]] [")]-B[[aada (Vg, a"),,
(2)
Lgpado E[([ V Dadio (V ( a²) a²) 1
where V/ = av + (1 a)vz with a ~ UO, 1), and Agpado is coefficient of the gradient penalty term Lypado (Gulrajani al., 2017) that aims to stabilize the training of GANs.
4

---

Published as a conference paper at ICLR 2025
@a) Diffusion-based feature generalor
Samantic label
Adversaria looming
① Mutual leaming
(DRG)
Lade
Latent vanable
Diffusion
a
Dads
Visual festure
Feature Adiscriminator
W-distance
Image Feature extractor (frozen)
Real noised visual features
DRG
Wadt
Lass
(b) Diffusion-baser (DRG)
Dovt
mm
Ditfusion
discriminatoo
used train p:: used to test G
(d) Denoksing learinng
W-distance
Fake clean con. rep.
DRG
(frozem)
Tep
Dynamically represent
Drep
Rep.
Image
Rep. extractor (frozem)
rep.
96
Representation keaming
discriminator
W-distanoe Wyep
Figure 3: Training pipeline of our DFG. C represents the concatenation operation. Given frozen extractors Fx and F. we take them to extract clean visual features Vn and contrastive representations the diffusion forward chain (Eq. 5) obtain real noised visual features V,-1 1--1 and Vi- Next, G' in DFG denoises/generates a fake clean feature Vg. conditioned by the concatenation of the semantic label a, latent variable z, diffusion time noised feature Vv. and SC-based representation To. The fake clean feature is evaluated from three different learning perspectives: adversarial learning (Does match predefined semantics?), denoising learning (oes match diffusson processes?), and representation learning (Does it match contrastive representations?). Finally, we present the mutual learning loss Cyu to integrate knowledge of all discriminators.
No. Then,
Since higher critic score means that discriminators consider the input data more 'real', we take the critic score difference between real training features v and testing seen class features Vo te, to indicate whether the learned vision-semantic correlation is spurious or substantial, i.e.
Aadv (vo. Vo (e Dade (Vā a²) Dadv (vie a (e.)).
Taking this metric, we display the results of F-VAAG in Fig. 2. increases and the difference in critic score becomes more significant on smaller training sets, showing that the learned generative ZSL methods could drop when the training set shrinks.
We can find Audo continuously
correlation is spurious; this explains why the performances of the
3.3 ZIRODIFF
To solidify the visual-semantie correlation, we propose our ZeroDiff based on our three key insights. The entire training and testing algorithms can be found in Appendix A. We now gradually introduce our proposed components that are motivated by our three insights. These components are integrated into the baseline f-VAEGAN resulting in the proposed ZeroDiff.
3.3.1 DIFFUSION AUGMENTATION
Discriminator Overfitting Following traditional GAN-based methods, we also adapt Dado and the traditional adversarial loss (Eq. 1) determine whether the generated features align with the predefined semantics, shown in Fig. 3(c). However, one of the causes of spurious visual-semantic correlation that limited training sets are memorized by Dadv- To this end, the first key insight is to leverage the diffusion mechanism to augment the limited training set into infinite noised data to mitigate the overfitting. Specifically, we propose our Diffusion- based Feature Generator (DFG) G. as shown in Fig. (a).
Diffusion-basee based Generating Given an image x from the training set D we use F and F to extract its clean visual features v F (x*) and clean contrastive representation rg F (x*)

---

Published as a conference paper at ICLR 2025
(The Fie and rú are introduced in the next section). Following previous diffusion models (Ho et al., 2020; Xiao et al., 2022), we apply the diffusion noising process to generate infinite data at various noise levels, from weak to strong, according to the diffusion forward chain:
((vierlv)) = a(viilv)-
(5)
12
- 11 10l1111-v_11. 1,
(6)
where Be pre-defined variance schedule. When t T, the noised feature v; become fully Gaussian noise. We set the maximum diffusion time T and randomly sample diffusion time We denote the noised visual features as vê-1 and vē at the diffusion times / --1
t ~~ UI,..... and t, respectively. Next, model the denoising process, concatenate the noised feature vä. diffusion time t. predefined class semantics a", latent variable z, and contrastive representation ro as the input of G, i.e. Vū = G(a", rū, t, vi, z). After denoising/generaiing a clean feature vū. is evaluated from three aspects: adversarial learning, denoising learning, and representation learning.
Diffusion-basse based Diseriminating To evaluate whether the denoised/generated clean features align with the diffusion processes, we design the diffusion discriminator Darry as illustrated in Fig. 3(d). Paifs needs to approximate the true denoising distribution q(v;- |v?).. To this end, we posterior- sample the real noised visual feature v 1 as well as the fake noised visual feature v
V ~ Vo vi) = NV (6),Br))
Rr vū) var-1
- -ar(1-a-11) v
Vi 1- at
(8)
at.
where 3, -
I-ā, 3, and ā. IT;=1 Bz). Then, train Dais to learn the Wasserstein
distance between them:
Cass = Waiss - Agods Cypdkss>
(9)
Waiss E[D ais (V:-1 Fo. f)] -E D[Dass (V.-1 Vt, t)],
(10)
Cypdiff = B[[[V Dairs Part::= vi, rō, a² f)12 - -)
(11)
where V av;- (1 (1-a)vf_, with (0, 1). Our G minimizes Londifs, which equates to minimizing the learned divergence per denoising step:
(12)
3.3.2 SC-BASED REPRESENTATIONS
Static Class-level Semantics Another cause of spurious visual-semane correlation the use of class-level semantics. In other words, existing semantic labels class-level, meaning all instances within a class share the same semantic label. This can result in mismatched correlations between instances and their semantics, as each limited sample may only represent a subset of the predefined semantics. For example, as shown in Fig. 1, all images in the "fox" class are labeled with the semanties "red" and "white," " even though white foxes are not "red." Such mismatches further amplify spurious visual-semantie correlations.
SC-based Representations To address the static class semantic problem, we revisit the SC loss (Khosla et al., 2020) and point out that SC-based representations could be used to represent instance-level semantics. Previous work (Islam et al., 2021) showed that SC-based representations have larger inter-class variation than those of CE-based features. This indicates that SC-based representations mirror the characteristics for every instance within classes. Our empirical study in Appendix A.4 also verifies this point (Fig. 6 and Fig. 7). For example, the different sub-classes of fox, e.g., white fox, red fox and grey fox, are clustered in the contrastive space as shown in Fig. 7. Thus, except fine-tuning the CE-based extractor Fee, we also fine-tune F.c. with the SC loss (Eq. 18), and fix it as F to extract contrastive representations re = Fx (x*). The extracted r is taken as the input to G for instance-level semantics, as shown in Fig. 3(b).

---

Published as a conference paper at ICLR 2025
Representation Discriminating As shown in Fig. 3(e), the representation discriminator Drep is responsible for distinguishing features via the contrastive representation view. It operates in a similar manner to Dudus but with the pre-defined semantics replaced by the contrastive representation:
Crep = Wrep Agprep
(13)
where Wrep and Cerrep is in Eq. 19 and Eq. 20.
Unseen Representation Generating At the testing stage, cannot get real unseen class represen- tations ro and feed to G. Thus, we train another representation generator DRG R to learning the mapping between instance-level SC representations to from class-level semantic labels The DRG training algorithm is similar to DFG, provided in Alg. 2.
3.3.3 MUTUAL-LEARNED DISCRIMINATORS
As shown in Fig. 3(),, since our three discriminators evaluate features different ways, they have different criteria for judging them. If we can enable them to learn mutually, we can obtain stronger discriminators, resulting better guidance for the generator. For example, the objectives of Dady and Daiss are two very similar but distinct tasks: one separates clean features and the other separates noised features. Clearly, separating noised features a harder task because with more diffusion steps, less information remains. Thus, if we can distill the knowledge from Dade to Da (. the separation ability on noised features and use the stronger Daiff to improve denoising. In contrast, distilling the knowledge from Daits to Dade could prevent Dadu from memorizing training samples. To this end, we propose the daaserrtein-distance based distillation loss:
enhance
Lmu = x? * (l[Wasff Wadellz + |[asf W Wreplli) I[Wodo W Wree 11,
(14)
where Ke is the Noise-to-Daie (N2D) ratio, represented 1- v -I;=1(1 By), and y is smoothing factor. As increases, Hi also increases, making it harder to distinguish between fake and real noised features, and Wasf/ provides less guidance for W Wep and Wadv. Therefore, introduce smoothing factor 7 2 0 to control the strength of discriminator alignment.
3.4
OVERALL OPTIMIZATION AND ZSL INFERENCE
The ZeroDiff model alternately trains G and three discriminators Dudus Dasff, and Drep optimize the following objective function:
min
max
(Cado + Laifs + Crep Amufmu))
(15)
G Dade: Pass
where Amn is the hyper-parameter related the mutual learning. After completing the training, we freeze them which are denoted as R² and G*.
For the final ZSL inference, the test images X are projected to the visual feature space V by the frozen F Then, we adopt the frozen R to sample contrastive representations and feed them into the frozen G synthesize visual features x6 of unseen classes (generating Negn samples for each unseen class). Next, we train the final classifier Fest, i.e. X - V" + y
4 EXPERIMENTS
4.1 DATASET
We conduct experiments on three popular ZSL benchmarks: AWA2 (Xian et al., 2018a), CUB (Welin- der et al., 2010) and SUN (Patterson & Hays, 2012). AWA2 consists of 65 animal classes with 85-D attributes and includes 37,322 images. CUB is bird dataset containing 11,788 images of 200 bird species. SUN has 14,340 images of 645 seen and 72 unseen classes of scenes. We follow the commonly used setting (Xian al., 20) to divide the seen and unseen classes. We report the average per-class Top-1 accuracy of unseen classes for ZSL. For GZSL, we evaluate the Top-1 accuracy on seen classes (S) and unseen classes (U)), and report their harmonic mean H 2xSXU'
S4U
4.2 IMPLEMENTATION DETAILS
The imlementation deatils provided in Appendix A.2.
7

---

Published as a conference paper at ICLR 2025
Table 1: Comparisons with the state-of-the-arts. U, S, and represent the top-1 accuracy (%) of unseen classes, seen classes, and their harmonic mean, respectively. The best and second-best results are marked in Red and Blue, respectively. i denoted the results using our fine-tune features, while using other fine-tune features. The upper group indicates embedding methods and the lower group is for generative methods.
7SL. AWA2 CUB
GZSL
Method
Venue
Backbone
SUN
AWA2
CUB
SUN
T1
TI
H
S
CLIP
ICML.21
ViT
55.2
54.8
55.0
CoOp*
IJCV22
ViT
49.2
63.8
55.6
ICIS
ICCV23
Res101
64.6
60.6
51.8
35.6
93.3
51.6
45.8
73.7
56.5
45.2
25.6
32.7
ReZSL
TIP23
Res101
70.9
80.9
63.0
63.8
85.6
73.1
72.8
74.8
73.8
47.4
34.8
40.1
PSVMA
CVPR23
VIT
73.6
77.3
75.4
70.1
77.8
73.8
61.7
45.3
52.3
ZSLVIT
CVPR24
VIT
70.7
78.9
68.3
66.1
84.6
74.2
69.4
78.2
73.6
45.9
48.4
47.3
F-CLSWGAN
CVPR18
Res101
75.9
84.5
75.5
65.1
68.9 66.9
76.4
83.3
79.7
63.8
55.7
59.5
-AVEGAN!
CVPR19
Res101
75.8
85.1
75.4
67.3 65.6 66.4 77.4
83.5 80.3
63.6
54.1
58.4
TFVAEGAN
ECCV20
Res101
72.4
85.8
74.1
54.2 89.6 67.5 79.0
83.3 81.1
51.8
53.8
52.8
SDVAE!
ICCV21
Res101
69.3
85.1
77.0
57.0 72.3 63.8 81.6 74.2 77.7
62.3
56.9
59.5
CEGAN
CVPR21
Res101
72.8
84.6
74.1
57.1
89.0
69.6 78.9 80.9 79.9
58.1
57.4
57.8
DFCAFlow
TCSVT23
Res101
74.4
83.9
77.2
67.6 81.0
73.7
77.3 82.9 80.0
63.0
59.6
61.2
S
ICCV23
VIT
55.3
58.9 57.1
DSP
ICML23
Res101
60.0 86.0
70.7
51.4
63.8 56.9
48.3
43.0 45.5
DML
ACCV24
Res101
62.2
82.3 70.9
57.1
81.6 67.2
39.6
52.7
45.9
VADSS
CVPR24
VIT
82.5
86.8
76.3
75.4
83.6
79.3
74.1
74.6
74.3
64.6
49.0
55.7
ZeroDift
Ours
Res101
87.3
87.5
77.3
74.7
89.3
81.4
80.0
83.2
81.6
63.0
56.9
59.8
4.3 COMPARISON WITH STATE-OF-TIE--NE
We select representative generative ZSL methods across different generative model types. These methods include: 1. GAN-based methods: f-CLSWGAN (Xian et al., 2018b) and CEGAN (Han et al., 2021); 2. VAE-based methods: SDVAE (Chen et al., 2021b); 3. VAEGAN-based methods: f- f-VAEGAN (Xian et al., 2019), TFVAEGAN (Narayan et al., 2020), DSP (Chen et al., 2023), DML Zhang et al. (2024) and VADS Hou et al. (2024); 4. Flow-based method: DFCAFlow (Su et al., 2023). We also provide comparisons with embedding bbased SOTAS: ICIS (Christensen et al., 2023), ReZSL (Ye al., 2023), PSVMA (Liu al., 2023) and ZSLViT (Chen et al., 2024a). Besides, large-scale vision-language models, i.e. CLIP (Radford et al., 2021), CoOp (Zhou et al., 2022) and SHIP (Wang et al., 2023b), have shown significant potential for dataset-level ZSL ability. But they do not follow the strict class splitting. Nonetheless, we still include them.
The results are presented in Table 1. For ZSL, compared to all embedding and generative methods, our ZifoDiff consistently achieves the best performance of 87.3%, 87.5% and 77.3% AWA2, CUB and SUN datasets, respectively. Our ZifoDiff obtains significant performance boost compared to other generative methods, i.e., by 3. 3.9% and 0.7% on AWA2 and CUB, respectively. For GZSL, our ZeroDiff performs even better at AWA2 and CUB, Our H performances are 79.5% and 81.6%, whilst the second best are only 73. 7% and 81. 1%. Notably, our ZeroDiff also significantly outperforms the large-scale vision- language based methods (e.g. CLIP, CoOp and SHIP).
4.4 LIMITED TRAINING DATA
To investigate the performance of generative approaches under limited training data, we randomly keep training samples of each seen class by different ratios. As the remaining ratio decreases, 7SI. becomes more challenging since over-fitting the training samples becomes much easier. For fairness, we decrease the number of synthesized unseen features proportionally to avoid class imbalance in GZSL. The results are reported in Table 2.
The following observations can be made: First, GAN-based approaches, f-CLSWGAN and CEGAN, deteriorate severely when only limited training data available. This indicates that GANs fragile under limited data conditions. Second, f- E-VAEGAN combines VAE and GAN to alleviate the mode collapse of GAN. However, when the training set is limited, the mode collapse problem still remains,
8

---

Published as a conference paper at ICLR 2025
Table Comparison on limited training data. We evaluate generative methods with 30% and 10% training samples. 7'1 represents the top-1 accuracy (%) of unseen classes in ZSL. In GZSL, U, S, and represent the top-1 accuracy (%) of unseen classes, seen classes, and their harmonic mean, respectively. The best and second-best results are marked in Red and Blue, respectively.
AWA2
CUB
SUN
Method
30%D
10%D
30%D
10%D
30%D"
10%D
H
T1
T1
H
T1
F-CLSWGAN (Xian al., 20)
68.9
57.8
54.0
35.7
82.1
75.3
75.1
66.8
73.4
51.5
66.9
29.3
F-VEGAAN (Xian al., 2019)
81.2
64.9
73.1
54.4
84.5
78.3
75.0
72.0
49.0
58.4
27.8
CEGAN (Han et al., 2021)
72.2
70.4
69.0
66.3
83.6
77.6
81.3
74.8
65.9
45.6
DFCAFlow (Su et al., 2023)
74.5
72.6
77.9
70.7
82.7
77.2
80.3
74.1
74.2
55.8
70.2
31.3
ZeroDiff (Ours)
84.9
80.2
83.3
77.0
85.5
78.7
82.9
76.1
75.4
51.3
68.1
33.3
which sometimes performs even worse than the methods only using GAN. For example, with 30%
Der of SUN, f-VAEGAN achieves 72.0% TI performance but the vanilla f-CLSWGAN achieves 73.4% T1. Third, with the introduction of the diffusion mechanism and instance-level representations, our ZifoDiff achieves the best performance in most cases. This demonstrates the effectiveness of our method in improving data efficiency.
We also provide qualitative comparison between the baseline f-VAEGAN and our ZiroDiff using t-SNE visualization of the real and synthesized sample features in Fig. 11. The visualization shows that the number of training samples decreases, f-VAEGAN gradually fails to generate unseen classes, while our ZeroDiff maintains a highly robust ability to generate them.
4.5 ABLATION STUDY
We conduct the component effectiveness study in Sec. 4.5.1, hyper-parameter sensitivity See, 4.5.2. Besides, we also provide the ablation study about DFG inputs in Appendix. A.5.
Table Ablation study of our ZeroDiff. and denote yes and no. O denotes using corresponding versions for DRG.
Component
AWA2
CUB
SUN
ID
G
R
D
Dast
Drep
fym
T1
H
T1
H
T1
H
79.9
67.0
83.7
78.5
74.7
57.8
b
/
X
X
X
82.3
72.9
84.1
79.5
75.1
58.7
V
X
X
82.7
73.3
84.7
80.9
75.4
59.0
d
X
V
C
O
XXXXX
X
83.8
67.1
78.3
66.8
71.6
47.6
e
V
V
X
79.3
72.8
83.4
78.5
73.6
56.2
f
S
V
X
80.3
69.0
85.8
80.5
73.5
56.4
g
V
82.9
73.4
84.8
79.1
74.4
57.0
V
X
V
85.2
78.3
87.2
81.2
76.8
59.4
84.6
77.5
84.8
79.3
76.8
59.2
j
87.3
81.4
87.5
81.6
77.3
59.8
4.5.1 COMPONENT EFFECTIVENESS
We examine the effect of the proposed components: G, R, Dadv. Daifs- Drep- and L The results are reported Table 3. We observe that Daiss and Lmu consistently improve the performance across the three datasets. Another observation is that, in many cases, G benefits from the inclusion of R. although R does not perform better than G alone. It also evidences our motivation that SC-based representations could capture instance-level semantics support feature generation.

---

Published as a conference paper at ICLR 2025
100%
30%
10%.0 D
50
100
150
200
230
300
0
90
100
150
200
250
300
epoch
epoch
epoch
(a) Oadv cjoch curve
0.4
0.2
0.0
bulff
0.0
0.0
-04
004
1300d
spoch
Aui spucc-
Figure 4: The effect of L. to Andv (Eq. 4) and Days (Eq. 16) on AWA2. (a) indicates that our fymu mitigates the overfitting of Dadv in the training set. (b) shows that the distinguishing ability of Dais is enhanced by our L.n--
4.5.2 HYPER- EPPER-PRRAMETER SENSITIVITY
We also provide the hyper- perammeree sensitivity analysis for the smoothing factor y (Eq. 14), the loss weight for Amu (Eq. 15) and the number of generated samples per unseen class Neyn The results are shown in Fig. 9 and Fig. 10. From Fig. (a), we can find our method performs robust for the change of y, and with the N2D reweight, it is consistently better than without N2D reweight. We conclude that we set Ama as 5, and set y as 1.5 on AWA2 to achieve good performance. From Fig. 10, the accuracy for unseen class increases when the number of synthesized samples increases and when Nagn is large enough, the performances become stable. This result demonstrates that the features synthesized by our method effectively mitigate the issue of missing data for unseen classes,
4.6 MUTUAL LEARNING EFFECTIVENESS
To further verify the effect of our Lya. we designed an additional experiment to show the change in critic score. First, as shown in Fig. 4 (a). we display the curve of Dady (Eq. 4). With L.us Dads decreases significantly compared to the counterpart without Lmx in CUB and SUN, suggesting that knowledge from Dais helps Dado reduces over-fitting the training set. Second, conversely, the knowledge from Dade can also benefit Daifs. As shown in Fig. 4 (b), we display the critic score difference of Dosf between noised real training features Ve and noised fake training features V., i.c.
(Viss(Ve, vi) Dasff (V&, V++1, ro, a, f) Daiff Vt+1, ru, a, t).
(16)
After adding Lmus Adsfs increases significantly. This indicates that the distinguishing ability of Dass is enhanced by the knowledge from D
CONCLUSION
In this paper, we investigate the zero-shot learning (ZSL) problem under varying amounts of training data, revealing that the issue of spurious visual-semantic correlation is exacerbated when training samples are scarce. To enhance visual-semantic correlation, we introduce ZeroDiff, which incor- porates three key components: (1) a diffusion forward chain to augment the limited training set; (2) SC-based representations to effectively represent each limited sample; and (3) mutually learned discriminators to validate generated features from multiple perspectives, including predefined seman- tics, contrastive representations, and diffusion processes. Experiments conducted on three popular datasets demonstrate the data efficiency of our method. Our ZeroDiff significantly enhances zero-shot capacity, performing well with both abundant and limited training samples.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by National Natural Science Foundation of China under No. 9237011 and 62376113 113, Research Development Fund with No. RDF-22-01-02 and No. RDF-TP-0019 and National Natural Science Foundation of China under Grant U1804159.
REFERENCES
Zeynep Akata, Scott Reed, Daniel Walter, Honglak Lee, and Bernt Schiele. Evaluation of output embeddings for fine-grained image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2927-2936, 2015.
Wei-Lun Chao, Soravit Changpinyo, Boqing Gong, and Fei Sha. An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In European conference on computer vision, pp. 22-68. Springer, 2016.
Shiming Chen, Ziming Hong, Wenjin Hou, Guo-Sen Xie, Yibing Song, Jian Zhao, Xinge You, Shuicheng Yan, and Ling Shao. Transzero++: Cross surribute-gided transformer for zero-shot learning. IEEE transactions pattern analysis and machine intelligence, 2022a.
Shiming Chen, Ziming Hong, Yang Liu, Guo-sen Xie, Baigui Sun, Hao Li, Qinmu Peng, Ke Lu, and Xinge You. Transzero: Attribute-guidn guided transformer for zero-shot learning. In AAAI, 2022b.
Shiming Chen, Wenjin Hou, Ziming Hong, Xiaohan Ding, Yibing Song, Xinge You, Tongliang Liu, and Kun Zhang. Evolving semantic prototype improves generative zero-shot learning. In Proceed- ings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 461 1-4622. PMLR, 23-29 Jul 2023.
Shiming Chen, Wenjin Hou, Salman Khan, and Fahad Shahbaz Khan. Progressive semantic-guided vi- sion transformer for zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 229944,,,, 2024a.
Shuhuang Chen, Dingjie Fu, Shiming Chen, Wenjin Hou, Xinge You, al. Causal semantic correlation for zero-shot learning. In ACM Multimedia 2024, 2024b.
Tianlong Chen, Yu Cheng, Zhe Gan, Jingjing Liu, and Zhangyang Wang. Data-efficient gan training beyond (just) augmentations: A lottery ticket perspective. In Advances Neural Information Processing Systems, volume 34, pp. 20941-20955, 2021a.
Yan-He Chen and Mei-Chen Yeh. Text-enhanced ciribute-bssed attention for generalized zero- shot fine-graine image classification. In Proceedings of the 2021 International Conference on Multimedia Retrieval, pp. 447-450, 2021.
Zhi Chen, Yadan Luo, Ruihong Qiu, Sen Wang. Zi Huang, Jingjing Li, and Zheng Zhang. Semantics disentangling for generalized zero-shot learning. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 8712-8720. 2021b.
Anders Christensen, Massimiliano Mancini, A. Sophia Koepke, Ole Winther, and Zeynep Akata. Image-free classifier injection for zero-shot classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19072-19081. October 2023.
Kevin Clark and Priyank Jaini. Text-to-image diffusion models are zero shot classifiers. Advances in Neural Information Processing Systems, 36, 2024.
Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 715-724, 2021.
Jicquan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, and Jiaya Jia. Generalized parametric contrastive learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
11

---

Published as a conference paper at ICLR 2025
Zhengming Ding, Ming Shao, and Yun Fu. Low-rank embedded ensemble semantic dictionary for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattrr recognition, pp. 2050-2055 2017.
Rui Gao, Fan Wan, Daniel Organisciak, Jiyao Pu, Junyan Wang, Haoran Duan, Peng Zhang, Xingsong Hou, and Yang Long. Absolute zero-shot learning, arXiv preprint 1xXiv:2202.11311 2022.
Shreyank N Gowda. Synthetic sample selection for generalized zero-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 58-67, 2023.
Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, and Sijie Zhu. Multi-reward as condition for instruction-basedd image editing. CoRR, abs/2411 2024. doi: 10.48550/ ARX0V.2411.04713. URL https::t////. /oii org/10. 11t111..455/////// att.///////v..a/a 2411. 04713.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans, Advances in neural information processing systems, 30, 2017.
Xu Han, Linghao Jin, Xiaofeng Liu, and Paul Pu Liang. Progressive compossiiityyy text-to-image generative models, 2024. URL https: /arxiii www.oorggabb//1/1//e// 1671
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive embedding for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2371-2381 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Wenjin Hou, Shiming Chen, Shuhuang Chen, Ziming Hong, Yan Wang, Xuetao Feng, Salman Khan, Fahad Shahbaz Khan, and Xinge You. Visual- augmented dynamic semantic prototype for generative zero-shot learning. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 23627-23637, June 2024.
Ashraful Islam, Chun-Fu Richard Chen, Rameswar Panda, Leonid Karlinsky, Richard Radke, and Rogerio Feris. A broad study on the transferability of visual representations with contrastive learning. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2021.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Improving contrastive learning on imbalanced data via open-world sampling. Advances Neural Information Processing Systems, 34:5997-6009, 2021.
Prannay Khosla, Piotr Teterwak, Chen Wang. Aaron Samn,, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:1 8661-18673, 2020.
Alexander Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your diffusion model is secretly zero-shot classifier. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023.
Man Liu, Feng Li, Chunjie Zhang, Yunchao Wei, Huihui Bai, and Yao Zhao. Progressive semantic- visual mutual adaption for generalized zero-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15337-15346 2023.
Yixin Luo and Zhouwang Yang. Dyngan:
Solving mode collapse in gans with dynamic clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Sanath Narayan, Akshita Gupta, Fahad Shahbaz Khan, Cees GM Snoek, and Ling Shao. Latent embedding feedback and discriminative features for zero-shot classification. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XXII 16, pp. 495. Springer, 2020.
12

---

Published as a conference paper at ICLR 2025
Genevieve Patterson and James Hays. Sun attribute database: Discovering, annotating, and recogniz- ing scene attributes. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2751-2758. IEEE, 2012.
Long Peng, Yang Cao, Yuejin Sun, and Yang Wang. Lightweight adaptive feature de-drifting for compressed image classification. IEEE Transactions on Multimedia, 2024.
Long Peng, Wenbo Li, Renjing Pei, Jingjing Ren, Jiaqi Xu, Yang Wang, Yang Cao, and Zheng- Jun Zha. Towards realistic data generation for real-world super-resolution. In The Thirteenth International Conference on Learning Representations, 2025. URL
net/ met/fozue http:/ooumm---OCKCBBUOUCCCC
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Scott Reed, Zeynep Akata, Honglak Lee, and Bemt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 49-58, 2016b.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Oindrila Saha, Zezhou Cheng, and Subhransu Maji. Ganorcon: Are generative models useful for few-shot segmentation? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9991-10000, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song, Jascha Sohl-Dickstein Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020b.
Hongzu Su, Jingjing Li, Ke Lu, Lei Zhu, and Heng Tao Shen. Dual-aligned feature confusion alleviation for generalized zero-shot learning. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
Bowen Tang, Jing Zhang, Long Yan, Qian Yu, Lu Sheng, and Dong Xu. Data-free generalized zero-shot learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 5108-5117, 2024.
Vinay Kumar Verma, Dhanajit Brahma, and Piyush Rai. Meta-learning for generalized zero-shot learning. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 6062- 6069, 2020.
Zhendong Wang, Yifan Jiang, Huangjie Zheng, Peihao Wang, Pengcheng He, Zhangyang " "Atlas" Wang, Weizhu Chen, and Mingyuan Zhou. Patch diffusion: Faster and more data-cfficient training of diffusion models. In Advances in Neural Information Processing Systems, volume 36, pp. 72137-72154, 2023a.
Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, and Tieniu Tan. Improving zero-shot generalization for clip with synthesized prompts. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3032-3042, 2023b.
Zhicai Wang, Yanbin Hao, Tingting Mu, Ouxiang Li, Shuo Wang. and Xiangnan He. Bi-directional distribution alignment for transductive zero-shot learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19893-19909 2023c.
13

---

Published as a conference paper at ICLR 2025
Ryan Webster, Julien Rabin, Loic Simon, and Frédéric Jurie. Detecting overfitting of deep generative networks via latent recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11273-11282, 2019.
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-ucsd birds 200. 2010.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-2265 2018a.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5542-5551, 2018b.
Yonggin Xian, Saurabh Sharma, Bernt Schiele, and Zeynep Akata. f-vaegan-d2: feature generating framework for any-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10275-10284, 2019.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion GANs. In International Conference Learning Representations, 2022.
Yang Yang, Yadan Luo, Weilun Chen, Fumin Shen, Jie Shao, and Heng Tao Shen. Zero-shot hashing
via transferring supervised knowledge. In Proceedings of the 24th ACM international conference Multimedia, pp. 1 1286-1295 2016.
Zihan Ye, Fuyuan Hu, Fan Lyu, Linyan Li, and Kaizhu Huang. Disentangling cemantic-to-visut confusion for zero-shot learning. IEEE Transactions on Multimedia, 2021.
Zihan Ye, Guanyu Yang, Xiaobo Jin, Youfa Liu, and Kaizhu Huang. Rebalanced zero-shot learning.
IEEE Transactions on Image Processing, 2023.
Chong Zhang, Mingyu Jin, Qinkai Yu, Haochen Xue, Shreyank N Gowda, and Xiaobo Jin. Bridging the projection gap: Overcoming projection bias through parameterized distance learning. In Proceedings of the Asian Conference on Computer Vision, pp. 3327-3343, 2024.
Rui Zhang, Ruixu Geng, Yadong Li, Ruiyuan Song, Hanqin Gong, Dongheng Zhang, Yang Hu, and Yan Chen. RFMamba: Frequency-avee awaree state space model for RF-based human-centric perception. In The Thirteenth International Conference on Learning Representations, 2025. URL fettps://openrewee
net / forum? 1ed
Shengyu Zhao, Zhijian Liu, Ji Lin, Jun- Yan Zhu, and Song Han. Differentiable augmentation for data-efficient gan training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 7559-7570. Curran Associates, Inc., 2020.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2343 2022.
Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6908-691 177 2022.
Yizhe Zhu, Mohamed Elhoseiny, Bingchen Liu, Xi Peng, and Ahmed Elgammal. A generative adversarial approach for zero-shot learning from noisy texts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1004-1013. 2018.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
We present additionally (1) the detailed training and testing algorithm: (2) the detailed implemen- tation details; (3) the detailed comparison of CE-based features and SC-based representations: (4) experimental results hyper- parameter sensitivity; (5) visualization for generated and real features.
A.1 FULL ALGORITHM
We include here pseudo-code for training algorithms (Alg.1 and Alg.2) and testing algorithm (Alg. 3),
We also provide details all loss functions here:
Fine-tuning stage:
For the feature extractor Fee:
D
Loz Σ
y; logg(gV))
(17)
For the representation extractor Fsc:
exp(hT hh///)+)7)
Lsc - log
(18)
exp(hTh+/7) Σx 1=
exp(hThç/7
where h+, h- K and > 0 are the positive example (have the same class label with h), negative example (have a different class label to h), the number of h in the batch, temperature parameter, respectively.
DFG training: We supply the Wrep and Lyprep here.
Wrep = EE(tg)) (Dep (Vo rf)l - -Epo(v)) (Drp()
(19)
Cgpeep =
E a(v8), [[I] Drep (V r0)122 J. pG (V()6)
(20)
DRG training: Similar to DFG training, our DRG F' generates fake clean representation iń + R(2, a", rf41,t) t) by the guidance from an adversarial discriminator D'ato and diffusion discriminator
Dasrs to guide SC-based representation generation. The adversarial learning loss DRG is
Lado
W adv
Nypado Cappaoo
(21)
W edv
B[D'de ado (rū, a²)] - -E(D' adv (Fo, a²)],
(22)
gapads [( (fg )|1 - 1)
(23)
And the denoising learning loss of DRG is
Eass Wa - S
(24)
W diss = E[D's (x a., t)] EE E[D'dsfy ri a²,
(25)
Copdisss = E((IIV (f a", ,012 1)
(26)
The param-eass X Xypadr and Xypdiff set 10.
15

---

Published as a conference paper at ICLR 2025
Algorithm I Fine-Tuning of ZefoDiff
Input:
a' E Training data D {(xx, a² ,²,"") ,²,
, e e)),, the iteration number for fine-tuning
Ny.-
[Fine-tuning with CE and SC]
Define F.r and Fats pretrained feature extractor and feature classifier.
Define F.. and Frro as pretrained representation extractor and contrastive projector. for N, do.
Draw batch of samples a=",""," from D
Extract feature v" +- Fee (x").
Classify feature T" f- (v)).
Update For and Fets with Loe by Eq. 17.
Extract representation r² <- (x=("")
Project representation into contrastive space h <- (rro(r").
Update F.e and H.. with Lsc by Eq. 18.
end for
Freeze Fee and Fee as Fee and Fic.
Output: Fü, and F
Algorithm 2 Generator Training of ZeroDiff
Training data D = {(x" a" v'))x' E X a² A", y E VV}. }. the iteration number for generator training N,, the iteration number for discriminator in step Ndis-
Input:
[Training DRG]
Define R as diffusion-based representation generator. Define D'av and Dasss clean representation discriminator and noised representation discriminator. for Ng do
Draw batch of samples (x", ya" from D
for Nas. do
Extract clean contrastive representation rš (x" Sample t ~ (O, -- 1).
Add noise into clean representation at t + rē and rf+1 by Eq. Generate fake clean representation F0 +- _((2 a", rf-1 (0,
Renoise fake representation iद by Eq. 7.
Update D'av and D'ssss by minimizing Eq. 21 and Eq. 24.
end for Update R by maximizing Ciuto and C'asss.
end for
Freeze R as R..
(Training DFG]
Define G as diffusion-bante feature generator.
Define Dadv. Daif and Drep as discriminators for clean feature, noised feature and contrastive represen- tation aspects.
for N₂ do
Draw batch of samples (x", a,,""" from Drr.
Nais do
Extract clean representation ră <- Fix(x") and feature vū +- Fx"))
Sample & UO...... T 1).
Noising feature at è and t + v and visa by Eq. S.
Generate fake clean feature Vă <- G(z, a" rō. vet1. 1), NOO,
Renoise fake feature Vi by Eq. 7.
Update Dadu, Daify and Drep by minimizing Eq. 1, Eq. 9, Eq. 13 and Eq. 14.
end for Update G by maximizing Eq. 1, Eq. 9 and Eq. 13.
for
Freeze G as G
Output: Gh and R
16

---

Published as a conference paper at ICLR 2025
winged
2:11
3.11
0.07
5:11
6:20
1.00
0.31
10
Greebeax
6.71
200
AB9
0.5
6:7
6:21
8.1
100
6.12
8/6
OAT
CA
5.11
House Sparrow
6:0
381
6M
610
535
6:72
800
L80
033
805
0.
0.79
130
600
150
0.0
6M
CC3
Henslow Sporms
379
0.79
6:30
302
643
660
552
AA1
0.5
205
AAT
Art Tec
1.0
BAY
020
0.42
6A0
5.6
Lostsiana Woterthmueh
Seyoris
Chestnut Warkler
[e) Pre defined semantic
(b) VADS
[d] Ours SC basod represontation
Figure 5: Heatmap comparison of the semantic prototype similarity among (a) pre-defined seman- tic (Reed et al., 2016a), ) (b) dynamic VADS (Hou al., 2024), and (c) our proposed SC-based representation. We randomly select 8 classes on CUB. Our method improves semantic prototypes to distinguish between categories, e.g., the similarities marked by the red dashed line.
Algorithm ZSL Testing of ZifoDiff
a" E A y" E }" trained DFG F" and DRG R", the iteration number for pscudo training Nte. the generation number per class Neyn.
Input: Testing datasct D = {(xx a" aa"."" "x" E x²",a""
[Testing DFG and DRG]
for every unseen class c" do
Define 5" as copying e" Nagn times and a" as class semantics. Generate fake clean representation F0 R2 (2, a" rū, 1), Generate fake clean feature vi G [z, Fo, v 1), N/ ~N(0,)
end for
Construct a new dataset De {(06, 76, a" }²
Define Fist as the final ZSL classifier.
for Nee do
Draw batch of samples (Vö, Fū, "") from D'
Classify pseudo sample 3"' +- (Va
Update Fist with Lo by Eq. 17.
end for
Freeze Fist as Fisi-
for every real x" do
Extract clean contrastive representation ră +- (x"( and clean feature vž + (v")..
Classify real sample 8" <- FFFaavv ro).
end for
Output: 7SI. classification results )}
A.2 IMPLEMENTATION DETAILS
For visual features, we extract 2,048-D features for all datasets using ResNet-101 (He et al., 2016) pre-trained with CE ImageNet-1K (Deng al., 2009). For contrastive representations, ResNett-101 pre-trained with PaCo (Cui et al., 2021) on ImageNet- K. For semantic labels, we use attribute vectors for AWA2 and 1 1,024-D attributes extracted from textual descriptions (Reed et al., 2016b) for CUB and SUN. We use Adam to optimize all networks with an initial learning rate of 0.0005. For all datasets, Agpadus Agpdiff, f. and >aprep are fixed 10. Following DDGAN (Xiao et al., 2022), the number of diffusion steps T is set to 4, and we use the discretization of the continuous natinnuous-time extension, known as the Variance Preserving (VP) SDE (Song al., 2020b) to compute 3 in Eq. 5. All experiments are conducted in Quadro RTX 8000.
adopt
A.3 VADS v.s. SC-BASED REPRESENTATIONS
We find that our SC- based representations exhibit better class discriminability compared to pre-defined semantics and the VADS method, which dynamically updates class-level semantics to instance-level
17

---

Published as a conference paper at ICLR 2025
-50
-50
-100
-100
-100
-50
50
100
-100
-50
50
100
CE beard fistuuss
beard rpprsenatiion
Figure 6: Comparison CS-based features and SC-based representations with t-SNE visualization for all classes on AWA2. We can find SC-based representations have larger intra-class variation than CE-based features.
80
Fox, 10373
fox 10238
60
Fox. 10080
40
fox
fox 10067
20
fox 1011
10573
Ioy, 10307
Scx, 10601
400
Fox, 10400
-30
fox 10012
20
60
-35
(e) CE-based leatures for the class 'fox'
(b) SC-based representation for class "fox"
Figure 7: Comparison CS-based features and SC-based representations with t-SNE visualization for the class 'fox' on AWA2. We can find all instances of fox are clustered in a group in CE-based space while different sub-classes (i.e. red fox, white fox and grey fox) are gathered in different groups in SC-based space.
semantics. While VADS can also extract instance-level semantics, it occasionally sacrifices class discriminability.
Specifically, shown in Fig. 5, randomly select 8 classes from the CUB dataset and visualize the heatmap of class prototype similarities. For the three class similarities marked by the red dash, the similarities from pre-defined class-level semanties are (0.84, 0.82. 0.85), while the similarities produced by VADS remain very similar: (0.85, 0.85, 0.86). In contrast, our SC-based representations make these three hard classes more distinctive, with class similarities of (0.78, 0.89, 0.83).
A.4 CE-BASED FEATURES V.S. SC-BASED REPRESENTATIONS
We provide more detailed comparison between CE-based features and SC-based representations. First, we use i-SNE to visualize the CE-based features and SC-based representations in Fig. 6. We can find every classes cluster more tightly in CE-based space than in SC-based space. It verifies previous work claims SC-based representations have larger intra-class variation. also means that SC-based representations contain more intra-clase uncertainty. To further verify this point, we visualize the class 'fox' in Fig. 7. We can find that all instances are grouped to single cluster in CE-based space while different sub-classes of fox (i.e. red fox, white fox and grey fox) could be separately grouped
18

---

Published as a conference paper at ICLR 2025
Cross-entuupy-Be Festure
snnarvisdd-ortzssddiii wwseraadiiieeeesddddoeoo Represantaiion
7SL.T1
GZSL.
ZSLTI
C2SL
Accuncy (K)
Aeet 0 (8)
60
50
40
40
30
20
10
SCLSWCAN
FVAEGAN AWA2
TF VADSAN
FCLSWGAN
IVARGAN TF VADGAN
FULSWWAN
VADLAN
CUB
SUN
Figure 8: Comparison of performance between CE-based features and SC-based representations. We selected three typical generative ZSL methods to evaluate these two fine- tin-tuning losses. We found that better than SC most cases, except for the 7SL T1 accuracy in AWA2.
in SC-based space. In other word, SC-based representations could reflect the characteristics every instances better than CE-barc lE-based features. Thus, we use SC- SC-based representations as new source of semantics.
Second, our study also challenge the assumption claimed in previous ZSL that SC-based representa- tions are more discriminative than CE-bbaeed features (Han al., 2021). Recent influential works have shown that SC-based representations are more sensitive to data imbalance (Cui et al., 2021; Jiang al., 2021; Zhu et al., 2022; Cui al., 2023; Peng et al., 2024). Fewer training samples lead to poorer SC-trained representations. Thus, we also use classical generative ZSL approaches to evaluate the ZSL performance of SC-based representations and CE-based features. We found that performance is generally worse significantly compared to that of traditional CE. The results can be found in Fig. 8.
In short, SC-based representations has better intra-class variations while CE-based features more discriminative. Thus, directly using SC-hased representations to generative 7SL. methods not better than CE-based features, but using SC-based representations semantics is eligible.
A.5 ABLATION STUDY FOR INPUT DFG
Table 4: Ablation study of the input of our DFG. v and denote yes and
DFG inputs
AWA2
CUB
SUN
ID
ro
Vt
t
T1
H
T1
H
T1
a
X
X
77.0
69.8
85.5
80.0
76.8
58.3
b
X
V
X
X
80.0
75.7
85.1
79.9
73.1
55.9
c
V
X
X
84.3
76.0
86.9
80.6
77.0
58.5
d
V
V
V
84.2
78.1
85.5
80.3
75.0
57.6
c
V
V
87.3
81.4
87.5
81.6
77.3
59.8
We conduct an ablation study on the inputs of our DFG, as shown in Table 4. The results demonstrate that all inputs are necessary, with the diffusion time being particularly critical. Specifically: (1) IDs a, and с confirm that the class-level semantic label and instance-level SC representations (2) IDs с and show that using only v, does not allow the method to fully benefit from reconstructing Vo from v., highlighting the importance of the diffusion time t. (3) IDs d and illustrate that inputting both and t results in significant improvements, demonstrating the effectiveness our diffusion augmentation.
To consistently improve performance across the three datasets.
A.6 HYPER-PARE PARAMETER SENSIBILITY
In our ZifoDiff, the main hyper- parametrrss include the loss weight Ann (Eq. 15), the smoothing
factor y that controls the smoothing of the noise-to-data ratio (Eq. 14) and the number of synthesized samples for each unseen class nyyn The sensitivity analysis of Am. and ry is illustrated Fig. (a) and (b). The sensitivity varies across different datasets. From Fig. 9 (a), can find method
19

---

Published as a conference paper at ICLR 2025
grtn 5.00 Pan
Arcamay 28g
Aetunn (8g)
Acsunoy
GAL
8
78
25
25
AWA2
CUB
AWN2
CUB
(b)
Figure 9: pprameer Sensitivity. (a) The effect weight factor Amu (Eq. 15). (b) The effect of smoothing factor for N2D reweight x7 (Eq.14). The shaded area indicates the performance improvement compared to hyper-parametee set as O.
84
Accuracy (%
Aonuray (9%)
Accurcyy (%)
CZSL U
80
G2SL
70
68
76
s2
1200
2100
3000
1000
7290
8400
(000
10800
12000
-4
400
720
990
1200
1000
1980
1800
2060
2000
100
200
300
500
(00
R2
(00
800
1000
N
Mayn
Noyn
(a) AWA2
(b) CUB
(c) SUN
Figure 10: The effect of the number of synthetic samples Nayn-
perform robust for the change of y, and with N2D reweight consistently better than the counterpart that without N2D reweight. We conclude that we set Xmn as 5, and set y 1.5 on AWA2 to unseen class increases with the number of
achieve good performance. From Fig- 10, the accuracy synthesized samples is up and when Neyn is large enough, the performances become stable. This result demonstrates that the features synthesized by our method effectively mitigate the issue of missing data for unseen classes.
A.7 TSNE VISUALIZATION
We use t-SNE visualization for the real and synthesized sample features to provide a qualitative comparison between the baseline f-VAEGAN and our ZeroDiff in Fig. 11. is clear that ZeroDiff maintains robust generation for unseen classes with scarce seen class samples.
20

---

Published as a conference paper at ICLR 2025
100% training samples
30% training samples
10% training samples
f-VAEGAN
ZeroDiff
Figure 11: Qualitative evaluation with t-SNE visualization on AWA2. We randomly selected 100 generated samples per class and all real samples for 10 unseen classes. We use different colors to denote classes, and use and x to denote the real and synthesized sample features, respectively (Best Viewed in Color).
21


Reviews:
[{"review_id": "TZ57ToivvT", "reviewer": "Reviewer_FpaJ", "summary": "This paper proposes to exploit diffusion mechanism to enhance the generative models in zero-shot learning. Specifically, existing generative zero-shot learning methods heavily relies on a large quantity of seen class samples, and the performance of such methods degrades with insufficient samples. To address this problem, the proposed method augments the training set by generating more seen-class and unseen-class samples with diffusion mechanism. With more available samples (either real or generated), the proposed method surpasses existing zero-shot learning methods in both general and sample-limited situations.\n\nBesides, the proposed mutual learning mechanism helps learning better discriminators in the generative process, which in turn improves the generators and classifiers, also contributing to the better performance of the proposed framework.", "questions": "(1)\tMore experiments in the ablation study should be conducted to clearly show the contribution of the additional contrastive feature. Specifically, it would be better to see the results of G+R+D_{adv}, G+D_{adv}+D_{diff}+L_{mu}, and etc..\n\n(2)\tIn recent years, GPT-series models exhibit amazing performances on zero-shot learning tasks, sometimes even surpasses the models specifically designed for ZSL tasks. It would be interesting to discuss the pros and cons of specifically designed ZSL models compared to such large multi-modal models (LMMs). For example, what are the advantages of specifically designed ZSL models? Are they still necessary in the existence of such large multi-modal models? If so, in which scenarios could the ZSL models perform better than the LMMs? Is it possible to combine both ZSL models and LMMs to achieve even better performances?\n\nIt would be sufficient to compare the proposed method with LMMs on a small portion of test samples (e.g. 100 images per dataset), and the results and discussions could be included in a separate new section. I believe these experiments and discussions would surely make the paper more insightful.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(1)\tThe proposed method is reasonable and technically solid. To be specific, although addressing the limited data issue by generating more data is quite straightforward, the details of the structure are still novel and effective. For example, the mutual learning mechanism of the discriminators and the incorporation of the diffusion module are well designed.\n(2)\tThe proposed method is effective. As shown in the experiments, the proposed method can achieve better performances than existing methods in most general zero-shot learning situations, and the proposed method consistently surpasses existing generative ZSL methods in data-limited situations. Such results validate the effectiveness of the proposed method.\n(3)\tThe authors provided some visualization results of the learned features, which could provide us with some insights into the possible future direction for further improving ZSL methods.", "weaknesses": "(1)\tThe main manuscript is quite confusing without the appendix. For example, how to finetune the feature extractors, and the details of training and testing are not clearly presented in the main manuscript, making it somehow hard to understand the proposed method when reading the main paper. It would be better if the authors could briefly explain and highlight such key details in the main paper as well as explain them in detail in the appendix.\n(2)\tIt would be better to conduct more experiments in the ablation study part (Table 3). Since the proposed method adopts two kinds of visual features instead of one kind of feature as in most existing ZSL methods, it would be necessary to show how the additional feature contribute to the final performance. That is to say, it would be better to show the performances of G+R+D_{adv}, G+D_{adv}+D_{diff}+L_{mu}, etc..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHwqA5SdQh", "reviewer": "Reviewer_Kqoz", "summary": "This paper proposes ZeroDiff, a generative framework for zero-shot learning (ZSL). It includes three key components: 1) diffusion augmentation, which transforms limited data into an expanded set of noised data to mitigate generative model overfitting; 2) supervised-contrastive-based representation, which dynamically characterizes semantics of individual samples; and 3) multiple feature discriminators, which evaluate generated features from various perspectives.", "questions": "- How to ensure stable training of such a complex pipeline?\n- What are the motivations of the design choices of the key modules?\n- How to effectively evaluate the design choices of each module?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The analysis of the performance degradation of ZSL due to a spurious visual-semantic correlation learned from a limited number of seen samples is inspiring.\n- The proposed diffusion augmentation and dynamic semantics methods are interesting.", "weaknesses": "- Identify and highlight the 1-2 most critical components that provide the key insights. The proposed pipeline is quite complex. It might be hard to tune the whole model. Moreover, it is also hard to know what is the key insight of the proposed method, since there are too many components.\n- Provide more motivations and justifications on the design choices of the key components. For example, 1) a clear explanation of the complementary benefits of using both CE and SC loss-based features should be provided. It claims that the SC loss-based representation contains more instance-level semantics. Then why both CE loss-based features and SC loss-based features are used jointly, rather than simply use SC loss-based representation? 2) The theoretical reasoning behind how the denoising discriminator alleviates instance overfitting is required. 3) A detailed justification for each element in the concatenated input to the DFG, explaining how each contributes to the model's performance. In other word, what is the motivation of taking the concatenation of the semantic label a, latent variable z, diffusion time t, noised feature vt, and SC-based representation r0 as condition of DFG? It would be better to give more insightful and in-depth analysis to these design choices, rather than simply verifying by experiments.\n- More detailed ablation study would be beneficial to show the effectiveness of the proposed method. Since the proposed method contains many components, the ablation study may not sufficient. For example, though table 3 gives the ablation study of each component, it is still unclear the effectiveness of the detailed design choice within each component, such as the condition of DFG module.\n- Provide a more detailed caption for Fig. 4 that explains the key differences between (a), (b), and (c). It is hard to see the key difference among Fig. 4 (a), (b), and (c). It would be better to give more explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4azmzv7DIH", "reviewer": "Reviewer_ZpvL", "summary": "The paper proposes a novel diffusion-based generative framework ZeroDiff for ZSL. Starting with a very motivating indicator, the paper claims that current generative methods learn spurious visual-semantic relationships when training data is insufficient, resulting in the failure of generated features. The comparison results well validate the effectiveness of the proposed method.", "questions": "Please see the weakness above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper proposes a novel diffusion-based generative method for ZSL.\n2. The experiments are comprehensive.", "weaknesses": "1. The paper claims that generative-based methods learn spurious visual-semantic relationships when training data is insufficient. Is the conclusion applicable to non-generative ZSL methods?\n2. As shown in Fig.2, \\delta_{adv} increases over the three datasets as the training progresses. However, the paper aims to learn a model with low \\delta_{adv} values. Does that mean we are getting a worse model as the training continues? It isn't very clear. The authors need to clarify the relationship between increasing \\delta_{adv} values and model performance, and explain how this relates to the goal of learning substantial visual-semantic correlations. It would be helpful to explore what an ideal \\delta_{adv} curve should look like.\n3. In L262, the authors fine-tune the backbone model by the SC loss. However, the baseline methods do not fine-tune the feature extractor. Therefore, the comparison seems to be unfair. The authors should either apply the same fine-tuning to the baseline methods, or provide results for their method without fine-tuning, to ensure a fair comparison. Additionally, it would be helpful to discuss the impact of this fine-tuning on the overall performance.\n4. The whole training process is divided into three stages, which makes it very complicated and hard to read. In addition, the author uses too much information as input for G in L301. Is this reasonable and necessary? The authors need to discuss potential ways to simplify the approach or explain why each component is necessary for the method's performance. It would be helpful to conduct an ablation study that shows the impact of each input on the final performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)