ZeroDiff: Solidified Visual-semantic Correlation in Zero-Shot Learning

## Abstract

Abstract Zero-Shot Learning (ZSL) aims to enable classifiers to identify unseen classes by enhancing data efficiency at the class level. This is achieved by generating image features from pre-defined semantics of unseen classes. However, most current approaches heavily depend on the number of samples from seen classes, i.e. they do not consider instance-level effectiveness. In this paper, we demonstrate that limited seen examples generally result in deteriorated performance of generative models. To overcome these challenges, we propose ZeroDiff, a Diffusion-based Generative ZSL model. This unified framework incorporates diffusion models to improve data efficiency at both the class and instance levels. Specifically, for instance-level effectiveness, ZeroDiff utilizes a forward diffusion chain to transform limited data into an expanded set of noised data. For class-level effectiveness, we design a two-branch generation structure that consists of a Diffusion-based Feature Generator (DFG) and a Diffusion-based Representation Generator (DRG). DFG focuses on learning and sampling the distribution of cross-entropy-based features, whilst DRG learns the supervised contrastive-based representation to boost the zero-shot capabilities of DFG. Additionally, we employ three discriminators to evaluate generated features from various aspects and introduce a Wasserstein-distance-based mutual learning loss to transfer knowledge among discriminators, thereby enhancing guidance for generation. Demonstrated through extensive experiments on three popular ZSL benchmarks, our ZeroDiff not only achieves significant improvements over existing ZSL methods but also maintains robust performance even with scarce training data. Code will be released upon acceptance.

###### keywords:

## 1 Introduction

Figure: Figure 1: Core idea of our ZeroDiff. (a) Standard GAN-based ZSL approaches suffer two problems in ZSL: (1) their performance drops when training data are decreased; and (2) they use incomplete pre-defined semantics to critic visual features. In contrast, our ZeroDiff leverages the diffusion noise process to obtain infinite noised real features for instance-level effectiveness; another diffusion-based representation generator is also used to get potential semantics for class-level effectiveness.
Refer to caption: /html/2406.02929/assets/x1.png

Machine learning models have achieved remarkable success in data-intensive applications, largely due to the availability of a huge number of labeled samples. However, as collecting such extensive labeled datasets is usually time-consuming and expensive, it is often unrealistic to assume that a large amount of labeled data is available. To address this limitation, researchers try to develop models which generalize well even in case of a limited number of samples. Among them, zero-shot learning (ZSL) [chen2021hsva] stands out by separating categories into seen and unseen categories. During training, only image samples of the seen categories are accessible, while those of the unseen categories are reserved for testing. Generalized ZSL (GZSL) [chao2016empirical] reflects a more realistic scenario, where both seen and unseen categories are included in the testing phase.

To bridge the gap between seen and unseen classes, researchers have explored leveraging predefined semantic knowledge about classes such as attributes [chao2016empirical], text2vec [zhu2018generative, chen2021text], and DNA [badirli2021fine]. ZSL methods mainly follow two paths: embedding methods [xu2020attribute, chen2022transzero++, ye2023rebalanced] that learn direct mappings between visual and semantic modalities, and generative methods that use generative models such as GANs [xian2018feature, li2021generalized, ye2021disentangling] and VAEs [chen2021semantics, chen2021hsva] to synthesize data for unseen classes. Currently, ZSL methods with generative models have become a popular approach.

Despite rapid advances in ZSL, a common assumption is that a sufficient number of samples per *seen* class are available.
In case of limited labeled data from each seen class, it remains uncertain if ZSL may still work, i.e. instance-level data effectiveness is not guaranteed.
This oversight becomes apparent as few ZSL methods consider the case of limited or no samples during training [verma2020meta, gao2022absolute, tang2024data]. For example, a data-free ZSL method leverages the knowledge of CLIP [tang2024data] to demonstrate the potential of training without any real seen class data. However, this approach does not conform to the universal evaluation protocols [xian2018zero], as the training of CLIP did not avoid using unseen classes in ZSL. In our experiments, we observe that most existing generative ZSL methods suffer from performance degradation and collapsed patterns when the number of training samples is gradually reduced. As shown in the t-SNE plot of Fig.[1](#S1.F1) (a), the classic method f-VAEGAN  [xian2019fvaegan] drops the GZSL accuracy by 26.4% on 10% of the training data, compared to using all the training data. More results can be seen in Sec. [4.4](#S4.SS4).

To further address the data requirement challenges at both class and instance levels, we propose a diffusion-based generative ZSL model (ZeroDiff), which combines a diffusion model to generate more samples to address the identified limitations. As shown in Fig. [1](#S1.F1) (b), our approach draws inspiration from the potential of diffusion models to improve instance effectiveness through a diffusion forward chain. Specifically, the original finite data is perturbed with different noise-to-data ratios, resulting in an infinite set of perturbed data points. In addition, to improve class effectiveness, we introduce a two-branch generation structure: diffusion-based feature generation (DFG) and diffusion-based representation generation (DRG) to distinguish it from traditional GAN-based methods. DFG is specifically designed to denoise visual features, transforming them from noise-enhanced to sharp states using predefined and potential semantics. These potential semantics come from frozen DRGs, which can capture a wider range of undefined semantics as they are trained on contrastive representations. In addition, we combine three types of discriminators to evaluate the authenticity of examples from various perspectives: predefined semantics, diffusion process, and contrastive representation. To promote mutual learning among all discriminators, we introduce a knowledge distillation loss based on Wasserstein distance, which further improves the model effectiveness.

Our contributions are multifaceted:

- •
ZeroDiff paves the way for class- and instance-efficient ZSL via a novel diffusion generation framework. To the best of our knowledge, ZeroDiff is not only the first to apply generative diffusion models for zero-shot image classification but also the first to explore mutual learning of discriminators.
- •
We propose a novel mechanism to address instance-level overfitting via a forward diffusion chain in the diffusion model, where noise is mixed with clean samples in different proportions to avoid memorizing the training samples.
- •
We enhance the effectiveness of categories via a two-branch generation paradigm, which generates contrastive representations as potential semantics to help feature generation, thereby improving class-level effectiveness.
- •
We establish a new performance protocol to evaluate ZSL methods across varying data amounts, where experimental results demonstrate the superiority of ZeroDiff in both ZSL and GZSL settings.

## 2 Related Work

### 2.1 Zero-shot Learning

Zero-Shot Learning (ZSL) is an area of research focusing on class-level generalizability [xian2018zero]. The primary approaches to ZSL can be categorized into embedding and generative methods. Embedding methods [akata2015evaluation, yang2016zero, ding2017low, yang2016zero, chen2022transzero, ye2023rebalanced] learn a direct mapping from visual to semantic spaces or vice versa.
TransZero++ [chen2022transzero++] presents a transformer-based architecture into ZSL and introduced mutual learning [zhang2018deep] to align the classification results of semantic and visual space.
In case of a limited number of training samples, embedding methods often under-perform generative methods on smaller datasets [chen2022transzero, ye2023rebalanced].

In contrast, generative ZSL methods employ a variety of generative models to synthesize images of unseen classes and then train a final classifier for these classes. There are also certain drawbacks for these methods. For example, GAN-based approaches frequently suffer from poor mode coverage [nowozin2016f, martin2017wasserstein, zhao2018bias], while VAE-based methods are plagued by low sample quality [verma2018generalized]. To address these issues, some ZSL methods combine GANs and VAEs, such as f-VAEGAN [xian2019fvaegan] and TF-VAEGAN [narayan2020latent]. Furthermore, to compensate for the lack of discriminative power in predefined semantics, some generative ZSL approaches incorporate contrastive learning or disentangled learning to uncover potential semantics, as seen in CEGAN [han2021contrastive] and SDVAE [chen2021semantics]. However, these potential semantics are not directly used to generate images of unseen classes; they are only utilized for classifying test samples at test time.

### 2.2 Diffusion Models

Diffusion models [sohl2015deep, ho2020denoising, song2020denoising, song2020score, xiao2022tackling] have a forward diffusion chain and a reverse denoising process.
The forward diffusion chain gradually adds noise to the clean data $\mathbf{x}_{0}\sim q(\mathbf{x}_{0})$ in $T$ steps:

$$ $q(\mathbf{x}_{1:T}|\mathbf{x}_{0})=\prod_{t\geq 1}q(\mathbf{x}_{t}|\mathbf{x}_{t-1}),$ (1) $$

$$ $q(\mathbf{x}_{t}|\mathbf{x}_{t-1})=\mathcal{N}(\mathbf{x}_{t};\sqrt{1-\beta_{t}}\mathbf{x}_{t-1},\beta_{t}\bm{I}),$ (2) $$

where $\beta_{t}$ is a pre-defined variance schedule.
When $t=T$, the noised data $\mathbf{x}_{t}$ becomes fully Gaussian noise.
The reverse process leverages a denoising model $G$ to remove noise from $\mathbf{x}_{t}$.
The classical DDPM [ho2020denoising] designs three modes: (1) predicting noise added at $t$ step, (2) predicting $\mathbf{x}_{t-1}$, and (3) predicting clean data $\mathbf{x}_{0}$ (also called directly denoising), i.e.,
$p_{G}(\mathbf{x}_{0}|\mathbf{x}_{t})=p_{\mathbf{x}_{0}\sim G(\mathbf{x}_{t},t)}(\mathbf{x}_{0}|\mathbf{x}_{t})=\mathcal{N}(\mathbf{x}_{0};\bm{\mu}_{\theta},\bm{\sigma}^{2}_{t}\bm{I})$.
The third mode is also adopted in Denoising Diffusion GAN (DDGAN) [xiao2022tackling] that unifies diffusion model and GAN by utilizing a discriminator to implicitly learn the denoising goal that maximizes the likelihood $p_{G}(\mathbf{x}_{0})=\int p_{G}(\mathbf{x}_{0:T})d\mathbf{x}_{1:T}$.

Large-scale Diffusion Models: Recent works [clark2024text, li2023your] have shown that large-scale diffusion models, such as the Latent Diffusion Model (also known as Stable Diffusion) [rombach2022high], have the ability to perform zero-shot classification. However, these studies do not strictly ensure that unseen classes are not used for model training. In these large-scale diffusion studies, the zero-shot ability is more task-oriented. They primarily focus on the performance across various agnostic tasks and datasets. The assumption is that with sufficiently large training samples, good performance can be achieved for any task or dataset. In contrast, our work aims to improve effectiveness on small training sets. The concept of a “diffusion model” differs in this context. In prior works, a “diffusion model” refers to one or several specific pre-trained diffusion models. In this paper, our interpretation of a “diffusion model” is closer to its fundamental nature: a type of generative paradigm that learns the data distribution through denoising noised data. Our training strategy incorporates ideas from diffusion, but crucially, it *does not* violate the zero-shot premise.

We demonstrate the dramatic impact of the number of training samples to generative ZSL approaches.
Additionally, our ZeroDiff introduces three innovations:
(1) An integration of diffusion models to improve instance-level effectiveness.
(2) A two-branch generation paradigm to improve class-level effectiveness.
(3) A Wasserstein-distances-based loss to distill knowledge from multiple discriminators.

## 3 ZeroDiff

Figure: Figure 2: The training pipeline of our DFG (the DRG training is similar) is as follows. ⓒ represents the concatenation operation. Given frozen extractors $F^{*}_{ce}$ and $F^{*}_{sc}$ (Sec. [3.2](#S3.SS2)), we use them to extract clean visual features $\mathbf{v}_{0}$ and contrastive representations $\mathbf{r}_{0}$. Then, we use the diffusion forward chain Eq. [1](#S2.E1) to obtain real noised visual features $\mathbf{v}_{t-1}$ and $\mathbf{v}_{t}$. Next, $G$ in DFG denoises and generates a fake clean feature $\tilde{\mathbf{v}}_{0}$, conditioned by the concatenation of the semantic label $\mathbf{a}$, latent variable $\mathbf{z}$, diffusion time $t$, and noised feature $\mathbf{v}_{t}$. The fake clean feature is evaluated from three different learning perspectives: adversarial learning (Does it match predefined semantics?), denoising learning (Does it match diffusion processes?), and representation learning (Does it match contrastive representations?). Finally, we present the mutual learning loss $\mathcal{L}_{mu}$ to integrate the three discriminators.
Refer to caption: /html/2406.02929/assets/x2.png

Our training algorithm contains three stages: (1) Fine-tuning stage by cross-entropy loss (CE) and supervised contrastive (SC) loss to extract visual features and contrastive representations; (2) Diffusion-based Representation Generator (DRG) training stage that models the contrastive representation distribution to support feature generation; (3) Diffusion-based Feature Generator (DFG) training stage that generates features conditioned by pre-defined semantics and contrastive representation. After training DFG and DRG, we use them to generate pseudo samples of unseen classes, and then utilize these generated samples to train our final ZSL/GZSL classifier.
The whole training and testing algorithms can be found in the appendix.

### 3.1 Preliminary

In the ZSL setting, suppose there are two disjoint label sets: a seen set $\mathcal{Y}^{s}$ used for training and an unseen set $\mathcal{Y}^{u}$ used for testing, where $\mathcal{Y}^{s}\cap\mathcal{Y}^{u}=\emptyset$. We denote the training dataset as $\mathcal{D}_{tr}=\{(x^{s},y^{s},a^{s})\mid x^{s}\in\mathcal{X}^{s},y^{s}\in\mathcal{Y}^{s},a^{s}\in\mathcal{A}^{s}\}$, where $\mathcal{X}^{s}$, $\mathcal{A}^{s}$, and $\mathcal{Y}^{s}$ represent the image, semantic, and label spaces corresponding to the seen classes. The goal of ZSL is to use the training dataset $\mathcal{D}_{tr}$ to create a classifier that can classify unseen images in the testing dataset $\mathcal{D}^{u}_{te}=\{(x^{u},y^{u},a^{u})\mid x^{u}\in\mathcal{X}^{u},y^{u}\in\mathcal{Y}^{u},a^{u}\in\mathcal{A}^{u}\}$, i.e., $f_{zsl}:\mathcal{X}^{u}\to\mathcal{Y}^{u}$. In GZSL, we also need to classify images of seen classes during the testing stage. In other words, the testing dataset becomes $\mathcal{D}_{te}=\mathcal{D}^{s}_{te}\cup\mathcal{D}^{u}_{te}$, and the goal becomes $f_{gzsl}:\mathcal{X}^{s}\cup\mathcal{X}^{u}\to\mathcal{Y}^{s}\cup\mathcal{Y}^{u}$.

### 3.2 Fine-Tuning Extractor

Figure: Figure 3: Comparison of performance between CE-based features and SC-based representations. We selected three typical generative ZSL methods to evaluate these two fine-tuning losses. We found that CE is better than SC in most cases, except for the ZSL T1 accuracy in AWA2.
Refer to caption: /html/2406.02929/assets/x3.png

Before we start our diffusion-based generation, we want to emphasize the importance of our fine-tuning stage. Previously, many ZSL works, e.g., [han2021contrastive], generally assume that Supervised Contrastive (SC) learning can produce more discriminative representations than those extracted using Cross-Entropy (CE).
However, other influential works have shown that representations learned from SC are more sensitive to data imbalance [cui2021parametric, jiang2021improving, zhu2022balanced, cui2023generalized]. Fewer training samples lead to poorer SC-trained representations. Thus, because unseen classes have no samples for SC training, we raise the question: Is SC really better than CE for ZSL?

To explore this question, we selected ResNet101 [he2016deep] as the extractor, which is the most widely-used architecture in ZSL. We used SC and CE to pretrain two ResNet101 models on ImageNet, respectively. We define the CE-pretrained model as $F_{ce}$ and the SC-pretrained model as $F_{sc}$. Next, we used CE and SC to fine-tune these models on the ZSL training set $\mathcal{D}_{tr}$, and then froze them as $F^{*}_{ce}$ and $F^{*}_{sc}$. Finally, we used traditional generative ZSL approaches to evaluate the ZSL performance of SC-based representations and CE-based features. We found that SC performance is generally worse significantly compared to that of traditional CE. The results can be found in Fig. [3](#S3.F3).

### 3.3 Diffusion-based Representation Generator

Although directly using SC-based representations for classification is not better than using CE-based features for ZSL, existing work has shown that contrastive losses enjoy the ability to capture potential semantics [islam2021broad, ye2021disentangling, li2024return]. For example, as shown in Fig. [2](#S3.F2) (b), the ‘almond eye’ is a significant semantic feature of the fox class. However, the AWA2 dataset does not contain this key semantic annotation. In fact, relying solely on predefined semantics to cover all true semantics is impracticable.

In this work, we leverage SC-based representations to mine potential semantics and address the incompleteness of predefined semantics(^1^11Previous ZSL work, e.g., [han2021contrastive, chen2021semantics], has also explored the power of potential semantics in ZSL. However, most of them only use potential semantics for classifying test samples. Our designed two-branch generation paradigm helps both the final classification and feature generation.). As shown in Fig. [2](#S3.F2), DFG generates visual features from limited data, while DRG aims to enhance the potential semantics to support the DFG. We train the DRG and freeze it before training the DFG. They have the same training procedure, differing only in their input, output, and corresponding discriminators. Thus, in the following, we only describe the DFG training, as it is more complicated. More details can be found in the appendix.

### 3.4 Diffusion-based Feature Generator

Given an image $x^{s}$ from the training set $\mathcal{D}_{tr}$, we use the fine-tuned $F^{*}_{ce}$ and $F^{*}_{sc}$ to extract its clean visual features $\mathbf{v}_{0}=F^{*}_{ce}(x^{s})$ and clean contrastive representation $\mathbf{r}_{0}=F^{*}_{sc}(x^{s})$. Next, following previous diffusion models [ho2020denoising, xiao2022tackling], we adopt the diffusion noising process to obtain infinite data at multiple noise levels from weak to strong according to the diffusion forward chain in Eq. [1](#S2.E1). We set the maximum diffusion time as $T$ and randomly sample a diffusion time $t\sim U(1,\cdots,T)$. We denote the noised visual features as $\mathbf{v}_{t-1}$ and $\mathbf{v}_{t}$ at the diffusion times $t-1$ and $t$, respectively. Next, to model the denoising process, we concatenate the noised feature $\mathbf{v}_{t}$, diffusion time $t$, predefined class semantics $\mathbf{a}$, latent variable $\mathbf{z}$, and contrastive representation $\mathbf{r}_{0}$ as the input of $G$, that is, $\tilde{\mathbf{v}}_{0}=G(\mathbf{a},\mathbf{r}_{0},t,\mathbf{v}_{t},\mathbf{z})$.
After denoising/generating a clean feature $\tilde{\mathbf{v}}_{0}$, it is evaluated from three aspects: adversarial learning, denoising learning, and representation learning.

Adversarial Learning: First, as shown in Fig. [2](#S3.F2)(c), we follow the standard WGAN [martin2017wasserstein] and WGAN-GP [gulrajani2017improved] and train a discriminator $D_{adv}$ that distinguishes clean real features $\mathbf{v}_{0}$ from fake features $\tilde{\mathbf{v}}_{0}$ by predicting the Wasserstein distance $W_{adv}$ between them according to predefined semantics $\mathbf{a}$. Specifically, the discriminator $D_{adv}$ maximizes the following loss $\mathcal{L}_{adv}$:

$$ $\displaystyle\mathcal{L}_{adv}=W_{adv}-\lambda_{gpadv}\mathcal{L}_{gpadv},$ (3) $$

$$ $\displaystyle W_{adv}$ $\displaystyle=\mathbb{E}[D_{adv}(\mathbf{v}_{0},\mathbf{a})]-\mathbb{E}[D_{adv}(\tilde{\mathbf{v}}_{0},\mathbf{a})],$ (4) $$

$$ $\displaystyle\mathcal{L}_{gpadv}=\mathbb{E}[(\|\nabla_{\hat{\mathbf{v}}_{0}}D_{adv}(\hat{\mathbf{v}}_{0},\mathbf{a})\|_{2}-1)^{2}],$ (5) $$

where $\hat{\mathbf{v}}_{0}=\alpha\mathbf{v}_{0}+(1-\alpha)\tilde{\mathbf{v}}_{0}$ with $\alpha\sim U(0,1)$, and $\lambda_{gpadv}$ is a coefficient of the gradient penalty term $\mathcal{L}_{gpadv}$ [gulrajani2017improved] that makes GANs training more stable.

Denoising Learning: Second, as mentioned, due to the limited number of training examples, $D_{adv}$ can easily overfit to the limited training features. To address this, we design the diffusion discriminator $D_{diff}$ to alleviate such instance overfitting, as shown in Fig. [2](#S3.F2)(d). $D_{diff}$ needs to approximate the true denoising distribution $q(\mathbf{v}_{t-1}|\mathbf{v}_{t})$. To this end, we posterior-sample the real noised visual feature $\mathbf{v}_{t-1}$ as well as the fake noised visual feature $\tilde{\mathbf{v}}_{t-1}$:

$$ $\tilde{\mathbf{v}}_{t-1}\sim q(\tilde{\mathbf{v}}_{t-1}|\tilde{\mathbf{v}}_{0},\mathbf{v}_{t})=\mathcal{N}(\tilde{\mathbf{v}}_{t-1};\tilde{\mu}_{t}(\mathbf{v}_{t},\tilde{\mathbf{v}}_{0}),\tilde{\beta}_{t}\bm{I}),$ (6) $$

$$ $\displaystyle\tilde{\mu}_{t}(\mathbf{v}_{t},\mathbf{v}_{0})=\frac{\sqrt{\bar{\alpha}_{t-1}}\beta_{t}}{1-\bar{\alpha}_{t}}\mathbf{v}_{0}+\frac{\sqrt{\alpha_{t}}(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_{t}}\mathbf{v}_{t},$ (7) $$

$$ $\displaystyle\tilde{\beta}_{t}=\frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_{t}}\beta_{t},\quad\bar{\alpha}_{t}=\prod^{t}_{j=1}(1-\beta_{j}).$ (8) $$

Then, we train $D_{diff}$ to learn the Wasserstein distance between them:

$$ $\displaystyle\mathcal{L}_{diff}=W_{diff}-\lambda_{gpdiff}\mathcal{L}_{gpdiff},$ (9) $$

$$ $\displaystyle W_{diff}$ $\displaystyle=\mathbb{E}[D_{diff}(\mathbf{v}_{t-1},\mathbf{v}_{t},\mathbf{r}_{0},\mathbf{a},t)]$ $\displaystyle-\mathbb{E}[D_{diff}(\tilde{\mathbf{v}}_{t-1},\mathbf{v}_{t},\mathbf{r}_{0},\mathbf{a},t)],$ (10) $$

$$ $\mathcal{L}_{gpdiff}=\mathbb{E}[(\|\nabla_{\hat{\mathbf{v}}_{t-1}}D_{diff}(\hat{\mathbf{v}}_{t-1},\mathbf{v}_{t},\mathbf{r}_{0},\mathbf{a},t)\|_{2}-1)^{2}],$ (11) $$

where $\hat{\mathbf{v}}_{t-1}=\alpha\mathbf{v}_{t-1}+(1-\alpha)\tilde{\mathbf{v}}_{t-1}$ with $\alpha\sim U(0,1)$. Our $G$ minimizes $\mathcal{L}_{gpdiff}$, which equates to minimizing the learned divergence per denoising step:

$$ $\displaystyle\sum_{t\geq 1}\mathbb{E}[D_{diff}(q(\mathbf{v}_{t-1}|\mathbf{v}_{t}))\|p_{G}(q(\tilde{\mathbf{v}}_{t-1}|\mathbf{v}_{t}))].$ (12) $$

Representation Learning: Third, the representation discriminator $D_{rep}$ is responsible for distinguishing features via contrastive representation view. It operates in a similar manner to $D_{adv}$, but with the predefined semantics $\mathbf{a}$ replaced by the contrastive representation, i.e.,

$$ $\displaystyle\mathcal{L}_{rep}=W_{rep}-\lambda_{gprep}\mathcal{L}_{gprep},$ (13) $$

$$ $\displaystyle W_{rep}=\mathbb{E}_{q(\mathbf{v}_{0})}[D_{rep}(\mathbf{v}_{0},\mathbf{r}_{0})]-\mathbb{E}_{p_{G}(\tilde{\mathbf{v}}_{0})}[D_{rep}(\tilde{\mathbf{v}}_{0},\mathbf{r}_{0})],$ (14) $$

$$ $\displaystyle\mathcal{L}_{gprep}=\mathbb{E}_{\begin{subarray}{c}q(\mathbf{v}_{0}),\\ p_{G}(\tilde{\mathbf{v}}_{0})\end{subarray}}[(\|\nabla_{\hat{\mathbf{v}}_{0}}D_{rep}(\hat{\mathbf{v}}_{0},\mathbf{r}_{0})\|_{2}-1)^{2}].$ (15) $$

Mutual Learning: Since our three discriminators evaluate features in different ways, they have different criteria for judging them.
If we can enable them to learn mutually, we can obtain stronger discriminators, resulting in better guidance for the generator. For example, the objectives of $D_{adv}$ and $D_{diff}$ are two very similar but distinct tasks: one separates clean features and the other separates noised features. Clearly, separating noised features is a harder task because with more diffusion steps, less information remains. Thus, if we can distill the knowledge from $D_{adv}$ to $D_{diff}$, we can enhance the separation ability on noised features and use the stronger $D_{diff}$ to improve denoising. In contrast, distilling the knowledge from $D_{diff}$ to $D_{adv}$ could prevent $D_{adv}$ from memorizing training samples.

To unify their critiques about the same visual features, we propose our Wasserstein-distance-based knowledge distillation loss:

$$ $\displaystyle\mathcal{L}_{mu}$ $\displaystyle=\kappa_{t}^{\gamma}*(\|W_{diff}-W_{adv}\|_{1}+\|W_{diff}-W_{rep}\|_{1})$ $\displaystyle+\|W_{adv}-W_{rep}\|_{1},$ (16) $$

where $\kappa_{t}$ is the Noise-to-Data (N2D) ratio, represented as $1-\sqrt{\prod^{t}_{j=1}(1-\beta_{j})}$, and $\gamma$ is a smoothing factor.
As $t$ increases, $\kappa_{t}$ also increases, making it harder to distinguish between fake and real noised features, and $W_{diff}$ provides less guidance for $W_{rep}$ and $W_{adv}$. Therefore, we introduce a smoothing factor $\gamma\geq 0$ to control the strength of discriminator alignment.

### 3.5 Optimization

The ZeroDiff model alternately trains $G$ and three discriminators $D_{adv}$, $D_{diff}$, and $D_{rep}$ to optimize the following objective function:

$$ $\min_{G}\max_{\begin{subarray}{c}D_{adv},D_{diff},D_{rep}\end{subarray}}(\mathcal{L}_{adv}+\mathcal{L}_{diff}+\mathcal{L}_{rep}-\lambda_{mu}\mathcal{L}_{mu}),$ (17) $$

where $\lambda_{mu}$ is the hyper-parameter related to the mutual learning.
After completing the training, we freeze them and denote them as $R^{*}$ and $G^{*}$.

### 3.6 ZSL Inference

Finally, we use the ZeroDiff model to train the final ZSL classifier, i.e., for ZSL inference. Since multimodal feature spaces show stronger discriminative power [narayan2020latent, wang2023bi, ye2021disentangling], we use frozen $F^{*}_{ce}$ and $F^{*}_{sc}$ to project the test samples into the visual feature space $\mathcal{V}^{u}$ and representation space $\mathcal{C}^{u}$. Then, we use the frozen $R^{*}$ to sample contrastive representations and feed them into the frozen $G^{*}$ to synthesize visual features $\tilde{\mathbf{x}}^{u}_{0}$ of unseen classes (generating $N_{syn}$ samples for each unseen class). Next, we train the final classifier $F_{zsl}$, i.e. $\mathcal{X}^{u}\rightarrow\mathcal{V}^{u}\rightarrow\mathcal{Y}^{u}$.

## 4 Experiments

### 4.1 Dataset Details

We conduct our experiments on three popular ZSL benchmarks: AWA2 [xian2018zero], CUB [welinder2010caltech] and SUN [patterson2012sun].
AWA2 has 65 animal classes with 85-D attributes and includes and 37322 images.
CUB is a bird dataset containing 11788 images of 200 bird species.
SUN have 14340 images of 645 seen and 72 unseen classes of scenes.
We follow the commonly used setting [xian2018zero] to split seen and unseen classes. We report the average per-class Top-1 accuracy of unseen classes for ZSL. For GZSL, we evaluate the Top-1 accuracy on seen classes ($S$) and unseen classes ($U$), and report their harmonic mean $H=\frac{2\times S\times U}{S+U}$.

Implementation Details.
For visual features, we extract 2048-D features for all datasets using ResNet-101 [he2016deep] pre-trained with cross-entropy on ImageNet-1K [deng2009imagenet]. For contrastive representations, we adopt ResNet-101 pre-trained with PaCo [cui2021parametric] on ImageNet-1K. For semantic labels, we use attribute vectors for AWA2 and 1024-D attributes extracted from textual descriptions [reed2016learning] for CUB and SUN. We use the Adam to optimize all networks with an initial learning rate of 0.0005. For all datasets, $\lambda_{gpadv}$, $\lambda_{gpdiff}$, and $\lambda_{gprep}$ are fixed at 10, and the number of diffusion steps $T$ is set to 4, following DDGAN [xiao2022tackling]. To compute $\beta_{t}$ in Eq. [1](#S2.E1), we use the discretization of the continuous-time extension, known as the Variance Preserving (VP) SDE [song2020score].
All experiments conducted in Quadro RTX 8000. The random seed are fixed to 9182, 3483 and 4115 for AWA2, CUB and SUN, respectively.

**Table 1: Comparisons with the state-of-the-art on standard ZSL. We represent the top-1 accuracy (%) of unseen classes. The best and second-best results are marked in Red and Blue, respectively. $\dagger$ denoted the results using our fine-tune features.**
| Method | Reference | AWA2 | CUB | SUN |
| --- | --- | --- | --- | --- |
| APN [xu2020attribute] | NeurIPS20 | 68.4 | 72.0 | 61.6 |
| I2DFormer [naeem2022i2dformer] | NeurIPS22 | 76.4 | 45.4 | - |
| MSDN [chen2022msdn] | CVPR22 | 70.1 | 76.1 | 65.8 |
| TransZero++ [chen2022transzero++] | TPAMI22 | 72.6 | 78.3 | 67.6 |
| ReZSL [ye2023rebalanced] | TIP23 | 70.9 | 80.9 | 63.0 |
| IAB [jiang2024estimation] | IJCV24 | 74.5 | 78.0 | 65.7 |
| fCLSWGAN [xian2018feature]${}^{\dagger}$ | CVPR18 | 75.9 | 84.5 | 75.5 |
| f-VAEGAN [xian2019fvaegan] ${}^{\dagger}$ | CVPR19 | 75.8 | 85.1 | 75.4 |
| TFVAEGAN [narayan2020latent]${}^{\dagger}$ | ECCV20 | 72.4 | 85.8 | 74.1 |
| SDVAE [chen2021semantics]${}^{\dagger}$ | ICCV21 | 69.3 | 85.1 | 77.0 |
| CEGAN [han2021contrastive]${}^{\dagger}$ | CVPR21 | 72.8 | 84.6 | 74.1 |
| APN+TFVAEGAN [xu2022attribute] | IJCV22 | 73.5 | 74.7 | 66.3 |
| DFCAFlow [su2023dual]${}^{\dagger}$ | TCSVT23 | 74.4 | 83.9 | 77.2 |
| CMC-GAN [yang2023semantics] | IJCV23 | - | 61.4 | 63.7 |
| ZeroDiff${}^{\dagger}$ | Ours | 86.4 | 87.5 | 77.3 |

**Table 2: Comparisons with the state-of-the-art on standard GZSL. $U$, $S$, and $H$ represent the top-1 accuracy (%) of unseen classes, seen classes, and their harmonic mean, respectively. The best and second-best results are marked in Red and Blue, respectively. $\dagger$ denoted the results using our fine-tune features.**
| Method | Reference | AWA2 | CUB | SUN |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| U | S | H | U | S | H | U | S | H |  |  |
| APN | NeurIPS20 | 57.1 | 72.4 | 63.9 | 65.3 | 69.3 | 67.2 | 41.9 | 34.0 | 37.6 |
| I2DFormer | NeurIPS22 | 66.8 | 76.8 | 71.5 | 35.3 | 57.6 | 43.8 | - | - | - |
| MSDN | CVPR22 | 62.0 | 74.5 | 67.7 | 68.7 | 67.5 | 68.1 | 52.2 | 34.2 | 41.3 |
| TransZero++ | TPAMI22 | 64.6 | 82.7 | 72.5 | 67.5 | 73.6 | 70.4 | 48.6 | 37.8 | 42.5 |
| PSVMA | CVPR23 | 73.6 | 77.3 | 75.4 | 70.1 | 77.8 | 73.8 | 61.7 | 45.3 | 52.3 |
| ReZSL | TIP23 | 63.8 | 85.6 | 73.1 | 72.8 | 74.8 | 73.8 | 47.4 | 34.8 | 40.1 |
| IAB | IJCV24 | 70.0 | 83.8 | 76.3 | 70.1 | 78.5 | 74.0 | 46.9 | 42.7 | 44.7 |
| fCLSWGAN${}^{\dagger}$ | CVPR18 | 65.1 | 68.9 | 66.9 | 76.4 | 83.3 | 79.7 | 63.8 | 55.7 | 59.5 |
| f-VAEGAN${}^{\dagger}$ | CVPR19 | 67.3 | 65.6 | 66.4 | 77.4 | 83.5 | 80.3 | 63.6 | 54.1 | 58.4 |
| TFVAEGAN${}^{\dagger}$ | ECCV20 | 54.2 | 89.6 | 67.5 | 79.0 | 83.3 | 81.1 | 51.8 | 53.8 | 52.8 |
| SDVAE${}^{\dagger}$ | ICCV21 | 57.0 | 72.3 | 63.8 | 81.6 | 74.2 | 77.7 | 62.3 | 56.9 | 59.5 |
| CEGAN${}^{\dagger}$ | CVPR21 | 57.1 | 89.0 | 69.6 | 78.9 | 80.9 | 79.9 | 58.1 | 57.4 | 57.8 |
| APN+TFVAEGAN | IJCV22 | 60.9 | 79.1 | 68.8 | 65.6 | 76.3 | 70.6 | 52.6 | 37.3 | 43.7 |
| DFCAFlow${}^{\dagger}$ | TCSVT23 | 67.6 | 81.0 | 73.7 | 77.3 | 82.9 | 80.0 | 63.0 | 59.6 | 61.2 |
| CMC-GAN | IJCV23 | - | - | - | 52.6 | 65.1 | 58.2 | 48.2 | 40.8 | 44.2 |
| ZeroDiff${}^{\dagger}$ | Ours | 74.7 | 85.0 | 79.5 | 80.0 | 83.2 | 81.6 | 63.0 | 56.9 | 59.8 |

### 4.2 Comparison with State-of-the-art

We select representative generative ZSL methods across different generative model types. These methods include:
1. GAN-based methods: f-CLSWGAN [xian2018feature], CEGAN [han2021contrastive] and CMC-GAN [yang2023semantics]
2. VAE-based methods: SDVAE [chen2021semantics]
3. VAEGAN-based methods: f-VAEGAN [xian2019fvaegan], TFVAEGAN [narayan2020latent], APN+TFVAEGAN [xu2022attribute]
4. Flow-based method: DFCAFlow [su2023dual].
For fair comparisons, we reproduce some methods whose code is available with our fine-tuned features and the recommended hyperparameters (e.g., learning rate, number of synthesized unseen features, batch size) from their respective papers.
Additionally, since generative approaches are trained with fine-tuned features, we can also directly compare them to embedding-based SOTA approaches: APN [xu2020attribute], TransZero++ [chen2022transzero], I2DFormer [naeem2022i2dformer], MSDN [chen2022msdn], PSVMA [liu2023progressive], ReZSL [ye2023rebalanced] and IAB [jiang2024estimation].

The ZSL results are in Table [1](#S4.T1) and GZSL results are in Table [2](#S4.T2).
It can be observed from Table [1](#S4.T1) that, in general, the proposed ZeroDiff outperforms the SOTAs, particularly in AWA2.
For ZSL, the new SOTA performance that we have achieved is 86.4% (AWA2), 87.5% (CUB), and 77.3% (SUN), surpassing the second best methods by 10.5% (fCLSWGAN in AWA2), 1.7% (TFVAEGAN in CUB) and 0.1% (DFCAFlow in SUN).
For GZSL, our ZeroDiff still is excellently good at AWA2 and CUB, our $H$ performances are 79.5% and 81.6% while the second bests are only 73.7% and 81.1%.

### 4.3 Ablation Study

**Table 3: Ablation study of our ZeroDiff. In ZSL, $T1$ represents the top-1 accuracy (%) of unseen classes. In GZSL, $U$, $S$, and $H$ represent the top-1 accuracy (%) of unseen classes, seen classes, and their harmonic mean, respectively. $\checkmark$ and $\times$ denote yes and no. $\Circle$ denotes using corresponding versions for DRG.**
| Component | AWA2 | CUB | SUN |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $G$ | $R$ | $D_{adv}$ | $D_{diff}$ | $D_{rep}$ | $\mathcal{L}_{mu}$ | T1 | H | T1 | H | T1 | H |
| $\checkmark$ | $\times$ | $\checkmark$ | $\times$ | $\times$ | $\times$ | 79.9 | 67.0 | 83.7 | 78.5 | 74.7 | 57.8 |
| $\checkmark$ | $\times$ | $\checkmark$ | $\checkmark$ | $\times$ | $\times$ | 82.3 | 72.9 | 84.1 | 79.5 | 75.1 | 58.7 |
| $\times$ | $\checkmark$ | $\Circle$ | $\Circle$ | $\times$ | $\times$ | 83.8 | 67.1 | 78.3 | 66.8 | 71.6 | 47.6 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\times$ | 84.6 | 77.5 | 84.8 | 79.3 | 76.8 | 59.2 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 86.4 | 79.5 | 87.5 | 80.3 | 77.3 | 59.8 |

Figure: Figure 4: Hyper-parameter Sensitivity. (a) The effect of the number of synthetic samples $N_{syn}$. (b) The effect of $\gamma_{mu}$ that controls the smoothing of N2D reweight $\kappa^{\gamma_{mu}}_{t}$ (Eq.[3.4](#S3.Ex2)).
Refer to caption: /html/2406.02929/assets/x4.png

#### 4.3.1 Component Effectiveness

We examine the effect of the proposed components: $G$, $R$, $D_{adv}$, $D_{diff}$, $D_{rep}$, and $\mathcal{L}_{mu}$. The results are reported in Table [3](#S4.T3). We observe that $D_{diff}$ and $\mathcal{L}_{mu}$ consistently improve performance across the three datasets. Another observation is that, in many cases, $G$ benefits from the inclusion of $R$, although $R$ does not perform better than $G$ alone.
It also is a evidence for our motivation that SC-based representations could capture potential features to support features generation.

#### 4.3.2 Hyper-parameter Sensitivity

We also provide the hyper-parameter sensitivity analysis for the number of generated samples per unseen class $N_{syn}$ and the factor $\gamma_{mu}$ that controls the smoothing of the noise-to-data ratio (Eq. [3.4](#S3.Ex2)).
Results are shown in
Fig. [4](#S4.F4).
From Fig. [4](#S4.F4) (a), we can see that a small $N_{syn}$ might undermines GZSL performance, but with $N_{syn}$ increasing, the GZSL performance tends to stabilization.
From Fig. [4](#S4.F4) (b), we can find our method perform robust for the change of $\gamma_{mu}$, and with N2D reweight is consistently better than the counterpart that without N2D reweight.

### 4.4 Limited Training Data

**Table 4: Comparison on limited training data. We evaluate generation methods with 30% and 10% training samples. $T1$ represents the top-1 accuracy (%) of unseen classes in ZSL. In GZSL, $U$, $S$, and $H$ represent the top-1 accuracy (%) of unseen classes, seen classes, and their harmonic mean, respectively. The best and second-best results are marked in Red and Blue, respectively.**
| Method | AWA2 | CUB | SUN |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 30% | 10% | 30% | 10% | 30% | 10% |  |  |  |  |  |  |  |
| T1 | H | T1 | H | T1 | H | T1 | H | T1 | H | T1 | H |  |
| f-VAEGAN [xian2019fvaegan] | 76.6 | 55.0 | 65.1 | 40.0 | 82.1 | 75.3 | 74.7 | 66.4 | 73.4 | 51.5 | 66.9 | 29.3 |
| SDVAE [chen2021semantics] | 70.9 | 66.6 | 62.4 | 45.8 | 82.7 | 77.2 | 84.3 | 77.5 | 75.8 | 52.6 | 70.0 | 34.4 |
| CEGAN [han2021contrastive] | 72.2 | 70.4 | 69.0 | 66.3 | 83.6 | 77.6 | 81.3 | 74.8 | 65.9 | 45.6 | - | - |
| DFCAFlow [su2023dual] | 74.5 | 72.6 | 77.9 | 70.7 | 82.7 | 77.2 | 80.3 | 74.1 | 74.2 | 55.8 | 70.2 | 31.3 |
| ZeroDiff (Ours) | 84.9 | 80.2 | 83.3 | 77.0 | 85.5 | 78.7 | 82.9 | 76.1 | 75.4 | 51.3 | 68.1 | 33.3 |

To investigate the performance of generative ZSL approaches under limited training data, we randomly keep training samples of each seen class by different ratios. As the remaining ratio decreases, ZSL becomes more challenging since overfitting to the training samples becomes much easier. For fairness, we decrease the number of synthesized unseen features proportionally to avoid class imbalance in GZSL. The full set of hyperparameters can be found in the appendix. The results are reported in Table [4](#S4.T4).

The following observations can be made: First, GAN-based approaches, f-VAEGAN and CEGAN, deteriorate severely when only limited training data is available. This indicates that GANs are fragile under limited data conditions. Second, the semantic rectification methods, SDVAE and DFCAFlow, maintain relatively good accuracy. This suggests that semantic rectification is also an effective way to avoid overfitting with limited training data. Third, with the introduction of diffusion models, our ZeroDiff achieves the best performance in many cases. This demonstrates the effectiveness of our method in improving data efficiency.

Figure: Figure 5: Qualitative evaluation with t-SNE visualization on AWA2. We randomly selected 100 generated samples per class and all real samples for 10 unseen classes. We use different colors to denote classes, and use $\circ$ and $*$ to denote the real and synthesized sample features, respectively (Best Viewed in Color).
Refer to caption: /html/2406.02929/assets/x5.png

We also provide a qualitative comparison between the baseline f-VAEGAN and our ZeroDiff using t-SNE visualization of the real and synthesized sample features in Fig. [5](#S4.F5). The visualization shows that as the number of training samples decreases, f-VAEGAN gradually fails to generate unseen classes, while our ZeroDiff maintains a highly robust ability to generate them.

### 4.5 DDGAN v.s. ZeroDiff

We also compare ZeroDiff to DDGAN, a similar work combining GAN and diffusion models.
Since both methods are based on diffusion, we can control the number of denoising/generating steps $t^{te}$ during testing.
The results are reported in Fig. [6](#S4.F6).

We find that DDGAN requires multiple steps to achieve good generation/denoising performance, while our ZeroDiff performs consistently well across different denoising iterations.
Thus, we can denoise/generate fake data by only one-step at inference stage and do not reduce generation performance.

Figure: Figure 6: Comparing ZeroDiff to DDGAN.
Refer to caption: /html/2406.02929/assets/x6.png

### 4.6 Mutual Learning Effectiveness

To further verify the effect of our $\mathcal{L}_{mu}$, we designed an additional experiment to show the change in critic score. Generally, a higher critic score means that discriminators consider the input data more realistic, and vice versa. First, as shown in Fig. [7](#S4.F7) (a), we display the critic score difference of the discriminator $D_{adv}$ between clean real training features $\mathbf{v}^{tr}_{0}$ and test features $\mathbf{v}^{te}_{0}$ of seen classes under the GZSL setting, i.e.,

$$ $\displaystyle\Delta_{adv}(\mathbf{v}^{tr}_{0},\mathbf{v}^{te}_{0})$ $\displaystyle=D_{adv}(\mathbf{v}^{tr}_{0},\mathbf{a}^{tr})$ $\displaystyle-D_{adv}(\mathbf{v}^{te}_{0},\mathbf{a}^{te}).$ (18) $$

A larger $\Delta_{adv}$ indicates that $D_{adv}$ considers real test samples more likely to be fake, indicating memorization of limited real training samples. With $\mathcal{L}_{mu}$, $\Delta_{adv}$ decreases significantly compared to the counterpart without $\mathcal{L}_{mu}$ in CUB and SUN, suggesting that knowledge from $D_{diff}$ helps $D_{adv}$ reduce overfitting in the training set.
Second, conversely, the knowledge from $D_{adv}$ can also benefit $D_{diff}$. As shown in Fig. [7](#S4.F7) (b), we display the critic score difference of $D_{diff}$ between noised real training features $\mathbf{v}^{tr}_{t}$ and noised fake training features $\tilde{\mathbf{v}}^{tr}_{t}$, i.e.,

$$ $\displaystyle\Delta_{diff}(\mathbf{v}^{tr}_{t},\tilde{\mathbf{v}}^{tr}_{t})$ $\displaystyle=D_{diff}(\mathbf{v}^{tr}_{t},\mathbf{v}^{tr}_{t+1},\mathbf{r}^{tr}_{0},\mathbf{a}^{tr},t)$ $\displaystyle-D_{diff}(\tilde{\mathbf{v}}^{tr}_{t},\mathbf{v}^{tr}_{t+1},\mathbf{r}^{tr}_{0},\mathbf{a}^{tr},t).$ (19) $$

After adding $\mathcal{L}_{mu}$, $\Delta_{diff}$ increases significantly. This indicates that the distinguishing ability of $D_{diff}$ is enhanced by the knowledge from $D_{adv}$.

Figure: Figure 7: The effect of $\mathcal{L}_{mu}$ to $D_{adv}$ and $D_{diff}$. (a) indicates that our $\mathcal{L}_{mu}$ mitigates the overfitting of $D_{adv}$ in the training set. (b) shows that the distinguishing ability of $D_{diff}$ is enhanced by our $\mathcal{L}_{mu}$ across different datasets.
Refer to caption: /html/2406.02929/assets/x7.png

## 5 Conclusion

In this study, we explore the Zero-Shot Learning (ZSL) problem in scenarios with limited seen data and propose a data-efficient ZSL method based on a diffusion generative model. We enhance the training samples using a diffusion forward chain and adopt contrastive representation to support feature generation, significantly improving zero-shot capacity with both sufficient and limited training samples. Additionally, our proposed mutual learning loss, based on Wasserstein distance, allows different discriminators to learn from each other. Experiments on three popular datasets demonstrate the data efficiency of our method, showcasing its robustness in cases with both ample and limited training data.

Acknowledgements
This work was partially supported by Research Development Fund with No. RDF-22-01-020 and “Qing Lan Project" in Jiangsu universities, National Natural Science Foundation of China under No. U1804159, and Jiangsu Science and Technology Programme under No. BE2020006-4.

## Declarations

- •
Funding: Research Development Fund with
No. RDF-22-01-020. National Natural Science Foundation of China (No. U1804159). Jiangsu Science and Technology Programme (No. BE2020006-4).
- •
Conflict of interest: The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data Availability Statement. The AWA2, CUB and SUN datasets are publicly available at [http://cvml.ist.ac.at/AwA2/](http://cvml.ist.ac.at/AwA2/), [https://www.vision.caltech.edu/datasets/cub_200_2011/](https://www.vision.caltech.edu/datasets/cub_200_2011/) and [https://cs.brown.edu/~gmpatter/sunattributes.html](https://cs.brown.edu/~gmpatter/sunattributes.html), respectively.
The models and source data generated during and analyzed during the current study are available from the corresponding author upon reasonable request.

## Appendix A Algorithms

We include here pseudo-code for training algorithms (Alg.[1](#alg1) and Alg.[2](#alg2)) and testing algorithm (Alg. [3](#alg3)).
We also discuss used losses not covered before here:

Fine-tuning stage:

For the feature extractor $F_{ce}$:

$$ $\mathcal{L}_{CE}=-\sum^{\|\mathcal{Y}\|}_{i=1}\mathbf{y}_{i}\log(\mathbf{\widehat{y}_{i}}).$ (20) $$

For the representation extractor $F_{sc}$:

$$ $\mathcal{L}_{SC}=-\log\frac{\exp(\mathbf{h}^{\top}\mathbf{h}^{+}/\tau)}{\exp(\mathbf{h}^{\top}\mathbf{h}^{+}/\tau)+\sum_{K}^{k=1}\exp(\mathbf{h}^{\top}\mathbf{h}^{-}_{k}/\tau)},$ (21) $$

where $\mathbf{h}^{+}$, $\mathbf{h}^{-}$, $K$ and $\tau>0$ are the positive example (have the same class label with $\mathbf{h}$), negative example (have a different class label to $\mathbf{h}$), the number of $\mathbf{h}^{-}$ in the batch, a temperature parameter, respectively.

DRG training:

$$ $\displaystyle\mathcal{L}^{\prime}_{adv}=W^{\prime}_{adv}-\lambda^{\prime}_{gpadv}\mathcal{L}^{\prime}_{gpadv},$ (22) $$

$$ $\displaystyle W^{\prime}_{adv}$ $\displaystyle=\mathbb{E}[D^{\prime}_{adv}(\mathbf{r}_{0},\mathbf{a})]-\mathbb{E}[D^{\prime}_{adv}(\tilde{\mathbf{r}}_{0},\mathbf{a})],$ (23) $$

$$ $\displaystyle\mathcal{L}^{\prime}_{gpadv}=\mathbb{E}[(\|\nabla_{\hat{\mathbf{r}}_{0}}D^{\prime}_{adv}(\hat{\mathbf{r}}_{0},\mathbf{a})\|_{2}-1)^{2}],$ (24) $$

$$ $\displaystyle\mathcal{L}^{\prime}_{diff}=W^{\prime}_{diff}-\lambda^{\prime}_{gpdiff}\mathcal{L}^{\prime}_{gpdiff},$ (25) $$

$$ $\displaystyle W^{\prime}_{diff}$ $\displaystyle=\mathbb{E}[D^{\prime}_{diff}(\mathbf{r}_{t-1},\mathbf{r}_{t},\mathbf{a},t)]$ $\displaystyle-\mathbb{E}[D^{\prime}_{diff}(\tilde{\mathbf{r}}_{t-1},\mathbf{r}_{t},\mathbf{a},t)],$ (26) $$

$$ $\mathcal{L}^{\prime}_{gpdiff}=\mathbb{E}[(\|\nabla_{\hat{\mathbf{r}}_{t-1}}D_{diff}(\hat{\mathbf{r}}_{t-1},\mathbf{r}_{t},\mathbf{a},t)\|_{2}-1)^{2}],$ (27) $$

Figure: Algorithm 1 Fine-Tuning of ZeroDiff

Figure: Algorithm 2 Generator Training of ZeroDiff

Figure: Algorithm 3 ZSL Testing of ZeroDiff

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: (1)	More experiments in the ablation study should be conducted to clearly show the contribution of the additional contrastive feature. Specifically, it would be better to see the results of G+R+D_{adv}, G+D_{adv}+D_{diff}+L_{mu}, and etc..

(2)	In recent years, GPT-series models exhibit amazing performances on zero-shot learning tasks, sometimes even surpasses the models specifically designed for ZSL tasks. It would be interesting to discuss the pros and cons of specifically designed ZSL models compared to such large multi-modal models (LMMs). For example, what are the advantages of specifically designed ZSL models? Are they still necessary in the existence of such large multi-modal models? If so, in which scenarios could the ZSL models perform better than the LMMs? Is it possible to combine both ZSL models and LMMs to achieve even better performances?

It would be sufficient to compare the proposed method with LMMs on a small portion of test samples (e.g. 100 images per dataset), and the results and discussions could be included in a separate new section. I believe these experiments and discussions would surely make the paper more insightful.
soundness: 4
strengths: (1)	The proposed method is reasonable and technically solid. To be specific, although addressing the limited data issue by generating more data is quite straightforward, the details of the structure are still novel and effective. For example, the mutual learning mechanism of the discriminators and the incorporation of the diffusion module are well designed.
(2)	The proposed method is effective. As shown in the experiments, the proposed method can achieve better performances than existing methods in most general zero-shot learning situations, and the proposed method consistently surpasses existing generative ZSL methods in data-limited situations. Such results validate the effectiveness of the proposed method.
(3)	The authors provided some visualization results of the learned features, which could provide us with some insights into the possible future direction for further improving ZSL methods.
summary: This paper proposes to exploit diffusion mechanism to enhance the generative models in zero-shot learning. Specifically, existing generative zero-shot learning methods heavily relies on a large quantity of seen class samples, and the performance of such methods degrades with insufficient samples. To address this problem, the proposed method augments the training set by generating more seen-class and unseen-class samples with diffusion mechanism. With more available samples (either real or generated), the proposed method surpasses existing zero-shot learning methods in both general and sample-limited situations.

Besides, the proposed mutual learning mechanism helps learning better discriminators in the generative process, which in turn improves the generators and classifiers, also contributing to the better performance of the proposed framework.
weaknesses: (1)	The main manuscript is quite confusing without the appendix. For example, how to finetune the feature extractors, and the details of training and testing are not clearly presented in the main manuscript, making it somehow hard to understand the proposed method when reading the main paper. It would be better if the authors could briefly explain and highlight such key details in the main paper as well as explain them in detail in the appendix.
(2)	It would be better to conduct more experiments in the ablation study part (Table 3). Since the proposed method adopts two kinds of visual features instead of one kind of feature as in most existing ZSL methods, it would be necessary to show how the additional feature contribute to the final performance. That is to say, it would be better to show the performances of G+R+D_{adv}, G+D_{adv}+D_{diff}+L_{mu}, etc..

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: - How to ensure stable training of such a complex pipeline?
- What are the motivations of the design choices of the key modules?
- How to effectively evaluate the design choices of each module?
soundness: 3
strengths: - The analysis of the performance degradation of ZSL due to a spurious visual-semantic correlation learned from a limited number of seen samples is inspiring.
- The proposed diffusion augmentation and dynamic semantics methods are interesting.
summary: This paper proposes ZeroDiff, a generative framework for zero-shot learning (ZSL). It includes three key components: 1) diffusion augmentation, which transforms limited data into an expanded set of noised data to mitigate generative model overfitting; 2) supervised-contrastive-based representation, which dynamically characterizes semantics of individual samples; and 3) multiple feature discriminators, which evaluate generated features from various perspectives.
weaknesses: - Identify and highlight the 1-2 most critical components that provide the key insights. The proposed pipeline is quite complex. It might be hard to tune the whole model. Moreover, it is also hard to know what is the key insight of the proposed method, since there are too many components.
- Provide more motivations and justifications on the design choices of the key components. For example, 1) a clear explanation of the complementary benefits of using both CE and SC loss-based features should be provided. It claims that the SC loss-based representation contains more instance-level semantics. Then why both CE loss-based features and SC loss-based features are used jointly, rather than simply use SC loss-based representation? 2) The theoretical reasoning behind how the denoising discriminator alleviates instance overfitting is required. 3) A detailed justification for each element in the concatenated input to the DFG, explaining how each contributes to the model's performance. In other word, what is the motivation of taking the concatenation of the semantic label a, latent variable z, diffusion time t, noised feature vt, and SC-based representation r0 as condition of DFG? It would be better to give more insightful and in-depth analysis to these design choices, rather than simply verifying by experiments.
- More detailed ablation study would be beneficial to show the effectiveness of the proposed method. Since the proposed method contains many components, the ablation study may not sufficient. For example, though table 3 gives the ablation study of each component, it is still unclear the effectiveness of the detailed design choice within each component, such as the condition of DFG module.
- Provide a more detailed caption for Fig. 4 that explains the key differences between (a), (b), and (c). It is hard to see the key difference among Fig. 4 (a), (b), and (c). It would be better to give more explanations.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 2
questions: Please see the weakness above.
soundness: 3
strengths: 1. The paper proposes a novel diffusion-based generative method for ZSL.
2. The experiments are comprehensive.
summary: The paper proposes a novel diffusion-based generative framework ZeroDiff for ZSL. Starting with a very motivating indicator, the paper claims that current generative methods learn spurious visual-semantic relationships when training data is insufficient, resulting in the failure of generated features. The comparison results well validate the effectiveness of the proposed method.
weaknesses: 1. The paper claims that generative-based methods learn spurious visual-semantic relationships when training data is insufficient. Is the conclusion applicable to non-generative ZSL methods?
2. As shown in Fig.2, \delta_{adv} increases over the three datasets as the training progresses. However, the paper aims to learn a model with low \delta_{adv} values. Does that mean we are getting a worse model as the training continues? It isn't very clear. The authors need to clarify the relationship between increasing \delta_{adv} values and model performance, and explain how this relates to the goal of learning substantial visual-semantic correlations. It would be helpful to explore what an ideal \delta_{adv} curve should look like.
3. In L262, the authors fine-tune the backbone model by the SC loss. However, the baseline methods do not fine-tune the feature extractor. Therefore, the comparison seems to be unfair. The authors should either apply the same fine-tuning to the baseline methods, or provide results for their method without fine-tuning, to ensure a fair comparison. Additionally, it would be helpful to discuss the impact of this fine-tuning on the overall performance.
4. The whole training process is divided into three stages, which makes it very complicated and hard to read. In addition, the author uses too much information as input for G in L301. Is this reasonable and necessary? The authors need to discuss potential ways to simplify the approach or explain why each component is necessary for the method's performance. It would be helpful to conduct an ablation study that shows the impact of each input on the final performance.

Meta-review: # Summary and Recommendation for Acceptance

---
## **Strengths**
1. **Novel Contributions**:
   - Proposes **ZeroDiff**, a generative framework addressing spurious visual-semantic correlations in zero-shot learning (ZSL). Key innovations include:
     - **Diffusion augmentation**: Expands limited datasets to mitigate overfitting.
     - **Supervised-contrastive representations**: Captures dynamic sample-level semantics.
     - **Mutual-learned discriminators**: Evaluates features from multiple perspectives, enhancing generation and classification performance.
   - Empirical demonstration of robust performance with only 10% of training data.

2. **Experimental Rigor**:
   - Extensive evaluations on popular benchmarks (AWA2, CUB, and SUN) showing superior performance in general and data-limited scenarios.
   - Comprehensive ablation studies validate the effectiveness of individual components.

3. **Theoretical Insights**:
   - Identifies and addresses the degradation in generative ZSL methods caused by spurious correlations.
   - Provides sound theoretical motivations for the proposed design.

4. **Practical Relevance**:
   - Framework is particularly suited for resource-constrained settings due to its data efficiency.

5. **Community Contribution**:
   - Commitment to open-sourcing the code for reproducibility and further exploration by the community.

---

## **Weaknesses**
1. **Complexity**:
   - The model is intricate, with multiple components that complicate interpretation, training, and tuning, potentially deterring adoption.

2. **Presentation Issues**:
   - Initial manuscript was unclear, requiring heavy reliance on the appendix for understanding.
   - Figures (e.g., Fig. 4) lacked sufficient captions to highlight key distinctions.

3. **Incomplete Baseline Comparisons**:
   - Concerns about fairness in baseline comparisons due to differences in fine-tuning methods.

4. **Overlapping Features**:
   - Lack of initial clarity on the complementary benefits of combining SC-based and CE-based features.

---

## **Authors' Mitigation**
1. **Complexity and Presentation**:
   - Revised the manuscript with a clearer, step-by-step explanation of methodology and improved figures.
   - Provided detailed justifications for design choices, particularly the combination of SC and CE features.

2. **Baseline Comparisons**:
   - Clarified the use of fine-tuned features across methods and conducted additional ablation studies to ensure fairness.

3. **Feature Redundancy**:
   - Explained the complementary roles of SC and CE features:
     - **SC** captures instance-level semantics.
     - **CE** provides compact and effective classification performance.
   - Empirical evidence supporting this claim was presented in supplementary analyses.

4. **Evaluation of Design Choices**:
   - Expanded ablation studies to assess the contribution of individual components and inputs (e.g., diffusion time).

---

## **Remaining Weaknesses**
1. **Complexity**:
   - Despite revisions, the framework remains complex and challenging to train, which may limit its accessibility to practitioners.

2. **Baseline Coverage**:
   - While outperforming generative ZSL models, further comparisons to large multimodal models (e.g., CLIP) were not fully addressed, leaving opportunities for future work.

---

## **Justification for Acceptance**
This paper effectively addresses a significant limitation in ZSL by mitigating spurious visual-semantic correlations in limited data scenarios. Its novel contributions, including diffusion augmentation and supervised-contrastive representations, advance the state of the art. The rigorous empirical validation and comprehensive rebuttals to reviewers' concerns further bolster its credibility.

While challenges related to complexity and broader comparisons persist, these are outweighed by the substantial theoretical and practical contributions. Given its strong performance and potential impact on ZSL research, I recommend acceptance.

Decision: Accept (Poster)