Meta-Learning via Classifier(-free) Guidance
Authors: Elvis Nava
Seijin Kobayashi
Yifei Yin
Robert K. Katzschmann
Benjamin F Grewe
Venue: Submitted to ICLR 2023 (2023)
Keywords: deep leaning
meta learning
hypernetworks
generative models
classifier guidance
contrastive learning
clip
classifier-free guidance
latent diffusion
diffusion models

Abstract: State-of-the-art meta-learning techniques do not optimize for zero-shot adaptation to unseen tasks, a setting in which humans excel. On the contrary, meta-learning algorithms learn hyperparameters and weight initializations that explicitly optimize for few-shot learning performance. In this work, we take inspiration from recent advances in generative modeling and language-conditioned image synthesis to propose meta-learning techniques that use natural language guidance to achieve higher zero-shot performance compared to the state-of-the-art. We do so by recasting the meta-learning problem as a multi-modal generative modeling problem: given a task, we consider its adapted neural network weights and its natural language description as equivalent multi-modal task representations. We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second "guidance" model that, given a natural language task description, traverses the hypernetwork latent space to find high-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: "HyperCLIP"-based classifier guidance and a conditional Hypernetwork Latent Diffusion Model ("HyperLDM"), which we show to benefit from the classifier-free guidance technique common in image generation. Finally, we demonstrate that our approaches outperform existing meta-learning methods with zero-shot learning experiments on our Meta-VQA dataset, which we specifically constructed to reflect the multi-modal meta-learning setting.

Under review as a conference paper at ICLR 2023
META- VIA CLASSIFIER(FFREE) GUIDANCE
Anonymous authors Paper under double-bling review
ABSTRACT
State-of-the-art meta-learning techniques do not optimize for zero-shot adaptation unseen tasks, a setting in which humans excel. On the contrary, meta-learning algorithms learn hyperparameters and weight initializations that explicitly optimize for few-shot learning performance. In this work, we take inspiration from recent advances in generative modeling and langunge-conditioned image synthesis to pro- pose meta-learning techniques that use natural language guidance to achieve higher zero-shot performance compared the state-of-the-an We do by recasting the meta-learning problem as multi- modal generative modeling problem: given a task, consider its adapted neural network weights and its natural language description as equivalent multi-modal task representations. We first train an unconditional generative hypernetwork model to produce neural network weights; then train second "guidance" model that, given natural language task description, traverses the hypernetwork latent space find nigh-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: SiypperLLI"--basee classifier guidance and a conditional Hypernetwork Latent Diffusion Model ("HyperL.DM"") which we show to benefit from the classifier-free guidance technique common in image generation. Finally, we demonstrate that our approaches outperform existing meta-learning methods with zero-shot learning experiments on our Meta-VOR VQA dataset, which we specifically constructed to reflect the multi-modal meta-learning setting.
INTRODUCTION
-tate-of-the-art machine learning algorithms often lack the ability to quickly generalize in sample efficient manner to new unseen tasks. In contrast, humans show remarkable capabilities in leveraging previous knowledge for learning new task from just a few examples. Often, not even single example is needed, as all relevant task information can be conveyed the form of natural language instructions. Indeed, humans can solve novel tasks when prompted from variety of different interaction modalities such as visual task observations or natural language prompts. In this work we aim to improve forward generalization of deep networks by developing new models that are capable of a similar multi-modal task inference. In particular, we present two different approaches that utilize text-based task descriptors for rapid zero-shot adaptation new tasks.
The development of deep learning models that quickly adapt and generalize to unseen tasks is the focus of the field of meta-learning, A recent trend stemming from the success of Model-Agnostic Meta-Learning (Finn et al. 2017 MAML) describes meta-learning as a bi-level optimization problem: an outer loop meta-model trained with the goal of improving the few-shot performance of a base model when fine-tuned on a variety of related tasks. MAML was specifically introduced as a gradient- based method to find an unconditional network initialization with good few-shot performance over an entire task universe. Recent progress in large scale deep learning is however challenging this explicit meta-learning framework. Large models trained on huge, rich, and diverse data sets have been shown to possess surprisingly good few-shot learning capabilities (Brown al., 2020). In particular, large scale pre-training and fine-tuning often outperforms explicit meta- meta-learning procedures (Mandi et al.. 2022).
These developments prompted us to develop alternative methods for explicit meta-learning which natively benefit from rich and multi-modal data. Inspired by recent advances in conditional image generation (Ramesh et al. 20022 Rombach et al. (222)., we reframe meta-learning as a modal generative modeling problem such that, given task, its adapted neural network weights and its

---

Under review as a conference paper at ICLR 2023
Figure 1: Schematic of the three main components of our proposed meta- learning approach. A. An unconditional variational autoencoder (VAE) models the latent space adapted network weights W. Its generator hypernetwork h (high- lighted in gray) can be re-used the conditional setting with guidance techniques. B. Our HyperCLIP encoder CLIP is contrastively trained to map work weights W to the space of CLIP embeddings eş. Then, given a new task with descriptor fg, we can use CLIP guid- ance to find a VAE latent vector 2; with embedding e; (H) that has a high cosine
A
x
x
Hypereet
Task
f
W
f
VAE
y
B
x
HyperCLiP Guidance
Task
Cosine
Task
h
W
W
CLIP
CLIP
Desc.
2,
H4
Vj
C
X
similarity to given task embedding e(T), C. Alternatively, our Hypernetwork La- tent Diffusion Model (HyperL.DM) learns, conditional on the task embedding e,, to iteratively denoise a VAE latent vector 0 over T iterations.
Task
Task
CLIP
Desc.
LDM
W
f
V
HyperLDM
1-1
natural language description are considered equivalent multi- molall task representations. What show is that popular techniques for the image domain, such as CLIP-based guidance (Gal et al. 2021 Patashnik al.) 2021). denoising diffusion models (Ho et al./ 2020), and classifier free guidance (Dhariwal & Nichol 2021 Ho & Salimans) 2021 Nichol et al. 2022) can be repurposed for the
meta-learning setting to generate adapted neural network weights instead of images.
Specifically, we approach the generation of neural network weights in two separate phases. In the unconditional pre-training phase, we train generative hypernetwork (Ha al., (016) to map from its latent space to the weight space of base model (Figure A). In the guidance phase, we learn language- conuuag-conditioned models that can be used to traverse the hypernetwork latent space and find zero-shot adapted weights with high performance our task (Figure B and [C).
1.1
OUR CONTRIBUTIONS
We summarise our contributions as follows:
We recast meta-learning as multi-modal generative modeling problem using hypereeworkss providing a theoretical framework for the application of recent research on image synthesis to the field of meta- meta-learning.
We propose HyperCLIP, a contrastive learning method equivalent to Contrastive Language- Image Pre-training (CLIP) (Radford al., 2021), producing CLIP embeddings of fine-tuned neural network weights. We can then use HyperCLIP to: 1) perform Clid-ggud in the latent space of hypernetwork model (Figure B); B)) and 2) perform task inference on fine-tuned models without given task descriptions.
We propose Hypernetwork Latent Diffusion Models (HyperLDM) as an alternative to HyperCLIIP guidance for cask-conditional network generation within the latent space of hypernetwork model (Figure TC). We show how combining this approach with classifier-free guidance (Ho & Salimans, 2021 improves the performance of generated networks.
We demonstrate the soundness our methods on Meta- RQA, our modification of the Pisual-Question-ANe
VQA v2.0 dataset (Goyal et al. 2017) that interprets questions as task descriptions. We show how our guidance methods outperform traditional multi-task and meta-learning techniques for zero-shot learning on this dataset.
2

---

Under review as a conference paper at ICLR 2023
2 META-LEARNING WITH MULTI-MODAL TASK EMBEDDINGS
The setting we investigate is similar to the classic meta-lcarning framework, where we operate within distribution of tasks T; p(T), each associated with loss functiin LT. Using set of training tasks drawn from this distribution, our goal is to train a model such that generally performs well on new unseen tasks drawn from p(T).
2.1 MODEL-AGNOSTIE
We present here a slightly altered formulation of MAMI. (Finn et al. 2017) introduced in (Zintgrat al. 2019), whereby the parameters of the model g are partitioned into parts: context parameters é that are adapted on individual tasks, and shared parameters 0 that are meta-trained and shared across tasks. MAML and its variants focus on the few-shot setting, which aims to learn an initialization for these parameters such that the model g 8 Φ) generalizes well on new tasks after fine-tuming on few data points from that task. To train such model, the data from each task T; split during training into support set DJ and query set DJ The MAML objective aims to optimize the validation score evaluated on the query set when fine-tuning & on the support set, e.g, consider the following optimization problem:
Σ Ln (0(z, (D,,,,)
min ET 0,6
D
(E.)))
where Ay; is some differentiable algorithm, typically implementing a variant few-step gradient descent on the loss computed on the support set, e.g., in the case of one-step gradient descent:
(D, O = " DM
1
Σ
VaLT (00o((000,00) 0)))
(2)
(x' (x',V))D;
with some learning rate The objective from Eq. i itself solved with gradient descent, by iteratively optimizing the parameters in the inner loop the support of sampled task, and updating 0 and the initialization of á with their gradient with respect to the entire inner loop training process, averaged over batches of tasks. Note that the original formulation of MAML considers 0
2.2 NATURAL LANGUAGE TASK EMBEDDINGS
In this work, we assume have access to an additional high-level context embedding e; for each task Tü In practice, such embeddings can come from natural language description f, of the task, which can be encoded into task embeddings using pre-trained language models. While MAML is powerful algorithm for finding universal initializations, the model naturally requires some data at test time to adapt its unconditioned network parameters. In contrast, by incorporating task embeddings, we aim to improve performance on a new task in the no-data (zero-shot), or very low data regime.
simple way to incorporate task embeddings into our model during training is by augmenting the input of the network, concatenating such input with the task embedding during the forward pass. Instead, we consider the use of hypernetworks (Ha et al. 2016), a network that generates the weights of another network given conditioning input. Concretely, given unconditioned neural network parametrized by a weight vector W, we reparametrize the model by introducing a hypernetwork The hypemeework h is parametrized by 6, which generates task-conditioned weight, h(e, 6) = W, The model is then defined as [(, h(e,,
We thus rewrite the MAML. objective with respect the hypernetwork weight 0 as
min TT 0
Σ CT. (AT, (n.))
(3)
D
when (D, 8) e,, recover the classic multi-task objective of hypernetoor optimizing for zero-shot performance. When A7, is instead the gradient descent algorithm on e, the objective aligns with the few-shot performance of h when adapting the embedding initialized at et.
3

---

Under review as a conference paper at ICLR 2023
3 HYPERNETWORKS AS GENERATIVE MODELS
rich literature exists on hypernetworks interpreted as generative models of base network weights (see Section 7). Our work builds upon this interpretation adapt multi-modal generative modeling techniques to the meta-learning domain.
In generative modeling, we aim learn the distribution p(x) over a high dimensional data domain X, such as images, given samples from the distribution. Since classic maximum-likelihodd learning is insufficient to do so, we must resort to techniques such as variational inference, adversarial learning. or diffusion models. It seems apparent that, in meta-learning, the modeling of distribution of diverse high- nigh-dimensional base network weights W may be considered in analogous terms. the Bayesian setting, this distribution is made explicit as we seek to model the posterior p(W|D) [D given data D, but the framework is still useful even when no explicit posterior distribution is assumed, as it the case for deep ensembles. In the present work, we indeed avoid explicit Bayesian inference: given training tasks, we consider their respective fine-tuned network weights W; as training samples for generative model of network weights.
The fundamental building block of our unconditional generative model is the hypernetwork (22,0 0) = W that we can train in two ways: 1) We define Hypernetwork VAE (HVAE) in Figure A which, given samples of fine-tuned base network weights W,, learns a low-dimensional normally distributed latent representation z. The encoder (W,w) = (u2, Σx) with parameters w maps base network weights to means and variances used to sample a latent vector 2, while the decoder (or generator) a classic hypernetwork h(2, 0) = W which reconstructs the network weights from the latent vector. 2) Using MAML, we learn both an embedding and hyperettwrk weights 9 such that, when fine-tuning only the embedding 2 on each task Tú we obtain high tegh-performing base networks with weights W, h=z;; 0). Concretely, we optimize 0 and the initialization of 2 following the objective in Eq. where takes the role of the task ssecifce parameter Φ
4 HYPERCLIP: TRAINING A CLIP ENCODER FOR THE "META-LEARNING MODALITY"
To define our new meta-learning approach, we first borrow from the field of multi-modal contrastive learning. More specifically, we build on top of Contrastive Language-Iman Pre-training (CLIP) (Radford et al. 2021). popular method for joint learning of language and image embeddings with applications to zero-shot and few-shot classification.
In the original CLIP formulation, separate text and image encoders are trained such that, given bi-modal sample (x,, t2) of an image and its corresponding language caption, their representations (CLIP((2)) (1) and (t) =e c,²") are aligned across modalities. Specifically, the formu- lation maximizes the cosine similarity e (UT (T) /llee /
Illle !Ile for pair-wise matches (i = j) and minimizes the cosine similarity for non-matches (i + j). Beyond the original language-image setting, the CLIP approach can be easily adapted to include additional modalities, aligning the representation of more than two encoders at a time. Existing works such as AudioCLIP (Guzhov et al., (022) demonstrate the possibility of training an encoder for additional modality such as audio on the side of the pre-trained frozen CLIP language-iman image encoders.
4.1 CONTRASTIVE LEARNING ON NEURAL NETWORK WEIGHTS
In our work, we consider multi-modal representations of meta-learning tasks Tú, which may be presented in the form of language as task descriptions 4, but potentially also in the form of images, videos, and audio. We fine-tune a base machine learning model f(r, W.) = y for task Tr and consider the base model part of an alternative meta-learning modality for task T. Fine-tuned networks from the meta-learning modality can then be paired in contrastive learning with the other multi-modal deseriptions of T:. We can thus define new HyperCLIP encoder as a "reverse hypernetwork" (H)
CLIP (W)) = e; taking fine-tuned neural network weights W, as input, and outputting a CLIP (H) optimized for high cosine similarity with the CLIP embedding for the textual (and image, video, etc.) description of the task. We point to Figure for an illustration of the approach, and to Algorithm|| for the detailed training procedure.
embedding e
4

---

Under review as a conference paper at ICLR 2023
B
Who is wearing glasses?
How is the weather?
CLIP
Who is wearing glasses?
CLIPT
T,.T,
TN
Where is the child sitting?
T
T
H
C
T, T m TN
CLIP
H
max
CLIP
H
max.
w
Figure 2: Our HyperCLIP encoder CLIP# is contrastively trained to map neural network weights W to the latent space of a pre-trained language encoder CLIPT, which we use to embed the natural language questions associated to the tasks (see A). To perform task inference given an already fine- tuned network, we encode all candidate task questions using the language CLIP encoder (see B), then encode the fine-tuned network weights with HyperCLIP (see C), and finally infer the correct task with a softmax operation over cosine similarities between HyperCLIP and language CLIP embeddings.
Algorithm HyperCLIP Training
sample a batch of tasks Tim N...,N with loss functions LT. training data Diain and text 4 define two N-sized arrays of d-dimensional embeddings T R. xxd and E Rxxx for i 1, .....; N do
Th = CL)Pr(t)) / [[CLIP((t)]
Fine-tune W; with objective: minw Eee (T
H[i = CL (W,) [[CLIP
end for
loss = (Ceross-entropy (T + Laross-entropy (HT /
Update weights of CLIP (r)) using Vloss
4.2 CLASSIFIER- META-LEARNING
On their own, CLIP encoders are most useful for zero-shot or few-shot multi-modal classification as opposed to data generation. Recent popular image synthesis techniques, however, use CLIP encoders or other classifiers to guide generation from pre-trained unconditional generative models. Classifier guidance CLIP guidance (Gal et al. 2021 Patashnik et al.) 2021) gradients with respect to classifier or CLIP encoder to traverse a generative model's 's latent space.
In this work, introduce HyperCLIP guidance, the first algorithm for classifier guidance in the meta-learning setting (Figure B)). Given a previously unseen validation task T7 and an uncon- ditional generative hypernetwork model h(2, 0) W.
use trained HyperCLII encoder CLIP H (W)- (!)) to guide the exploration of the hypernetork's latent space and find set of base weights W; with high zero-shot performance for T. Specifically, as long as we are given a starting hypernetwork latent vector z0 and textual description I, of the task, descent over the guidance loss
update 20 with gradient
CLIP# (h(2, 0)) CLIPT(t))
Lguidance (z) |CIIP (M(2, 0)) [|[LIIP++ + A|[= =01,
and then run the optimized latent vectors 9; through the generative hypernetwork to find adapted zero-shot base network weights h(2, 0) = W; that perform well for the task.
HYPERLDM: TASK CONDDTIONNI DIFFUSION OF HYPERNETWORK LATENTS
Using classifier guidance to explore the latent space traditional generative model with direct CLIP gradients powerful and elegant approach. However, rapid innovation in the image synthesis
5

---

Under review as a conference paper at ICLR 2023
community recently led to this method being largely overcome in favor of applying classifier guidance and classifier-fres guidance during the sampling process of a Diffusion Model (Dhariwal & Nichol 2021 & Salimans 2021; Kim et al.) 2022 Crowson 2022] Nichol al., [022] Rombach et al., 2022). To paint more complete picture, we also investigate this setting in the meta-learning domain, alternative the previously introduced HyperCIIIP guidance.
5.1 (LATENT) DIFFUSION MODELS
Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al. 2015 Ho et al. 2020, DDPM) are
powerful class of generative models designed learn a data distribution p(æ). They do so by learning the inverse of a forward diffuusion process in which samples x of our data distribution are slowly corrupted with additive Gaussian noise over T' steps with a variance schedule BI..... Br, resulting in the Markov Chain
T
3xx = - BB)
q(2 xx)) II [Ig(a|/-1))
(5)
property of such a process is that we can directly sample each intermediate step from x. as ((0,I), I). a, = 11 - 3r and a; = II'' at. Then, to learn the reverse process py (~ |z²), we parametrize the dimeste--deeendett noise function e (2", t) with neural network and learn it by optimizing simplified version of the variational lower bound on p(x)
x' = Va + (vI - 0e)e given e ~ NN
LD (v)) = Eg.e~ V.,..)(0.1),. [[]c (x t)]
Sampling from the reverse process can then be done with
Be VI Of so(x",t)) + ore.
Vac
with { ~ (0, I) and o, chosen between Bg and 3./VI ag Sampling from the learned diffusion
model can be seen analogue to Langevin Dynamics, connection explicitly made in works
exploring the diffusion technique from the "score matching" perspective (Song & Ermon 2019 Song
al. 2020
In our meta-lcarning setting, we aim to train a diffusion model which generates adapted zero-shot base network weights W, that perform well for task T. Thus, our diffusion model has to be conditional on task embedding e;. Moreover, in order to speed up training and leverage our previously trained generative hypernetwork h(z, 2)), we define the diffusion process on latent vectors instead of doing so in weight space, emulating the Latent Diffusion technique from Rombach et al.) (2022).
Satisfying the above requirements, we propose Hypernetwork Latent Diffusson Models (HyperLDM), which learn to sample from the conditional distribution of fine-tuned latent vectors p(2 (e:) given language CLIP embedding corresponding to the task. The HyperLDM neural network models the noise function (2*,, f, e;), and is learned by optimizing the reweighted variational lower bound, which in this setting is
LLDM (b)) = (h.), (W.). (xl,,,
5.2 CLASSIFIER-FREB GUIDANCE FOR META-LEARNING
The classifier guidance technique presented in Section 4.2 can be also adopted together with diffusion models. Even in the case of conditional diffusion models, the gradient of auxiliary classifier (or CLIP encoder) can be added during sampling to induce an effect similar to GAN truncation (Brock ct al. 2018), producing samples that are less diverse but of higher quality. Here, we showcase how the same technique can be adopted when generating adapted base network weights conditioned on task representations, tuning the temperature of sampling to reduce variance and increase base network performance.
The classifier-fres guidance technique (Ho & Salimans 2021 Nichol et al. 2022 allows to leverage a conditional diffusion model to perform the same tempered sampling as above, without the auxiliary classifier. To do so, train the conditional network (00 (,e,) to also model the

---

Under review as a conference paper at ICLR 2023
unconditional case 62(2', t). One way of doing this is with conditioning dropout, simply dropping the conditional imput e, for a certain percentage of training samples, inputting zeros instead. We can then sample at each diffusion iteration with
(x tt e,) (1- cg (2), (x,
For y 0, this recovers the unconditional diffusion model, while for 1 it recovers the standard conk-conditional model. For y > 1, we instead obtain the classifier-ff guidance effect, which we show results in the sampling of latent vectors &; corresponding to higher-performing task-conditional network weights (2,,, +)) W.. We point to more in-depth discussion on classifier-free guidance and its connection to score matching in Appendix A.
EXPERIMENTAL, SETUP AND RESULTS
In this section, we demonstrate the soundness of our two approaches with zero-shot image classifica- tion experiments against a series of traditional meta-learning baseline techniques. Throughout our experiments, we fix the choice of base network model to a CLIP-Adapter model (see Appendix (A.2). only varying the meta-learning techniques employed to obtain adapted base model weights.
6.1 THE META- VQA DATASET
To evaluate the performance of our methods, we utilize a dataset that reflects the setting of meta- learning with modal task descriptors. Existing meta-learning benchmarks such Minilmagenet (Ravi & Larochelle 2016) or CIFAR-FS (Bertinetto et al. 2018) are unsuitable, as they are built for the traditional few-shot learning setting, in which the task 7, is not associated with task descriptors but is meant to be inferred through exposure to the support set D; We thus introduce our own Meta-VQA -ete--uueetttn-eeeeeeee The dataset is composed of training and test tasks 7; each associated with a natural language question t; and mini image classification dataset (x)) vt) E D,. We refer to Appendix A.3 for a more in-tepth discussion.
dataset, modification of the VQA v2.0 dataset (Goyal et al. 2017) for
Task Question (t): Where is the child sitting? Task Data (D))
y
X
Y'2
He sitting on the refrigerator
He is sitting in mom's arms
Figure 3: Example classification task from VQA, adapted from VQA v2 (Goyal et al. 2017). 7).. A single question 4; is associated to multiple image-answer tuples (z) 23).
6.2 ZERO-SHIE SHOT TASK ADAPTATION WITH CLASSIFIER GUIDANCE
In [ we show how our methods compare to a series of baselines when tested on the Meta-VQA dataset the zero-shot setting. For each training task T, the algorithms are given access to the full image/answer support and query sets D;, DJ, together with the question (task descriptor) t;. At test time, in the shoo setting, only the task descriptors t, for each test task T, are given, and the algorithms are tasked with predicting the correct labels of images in the query set DJ. To simulate setting in which we possess larger "unconditional" pre-training dataset, we also train our model while only keeping a fraction of task descriptors from the Meta-V VQA dataset: tasks without language descriptors can still be used to learn the unconditional HNET/HVAE model.
Classic zero-shot CLIP provides provide a 44. 44.99% floor for performance on Meta-VQA model (CLIP-Adapter) consists of adapter layers over its output. We also can derive a -6%024% performance ceiling for zero-shot performance by considering the few-shot setting, in which models have also access to a data support set D for every test task (see Appendix it is apparent that zero-shot techniques cannot surpass this ceiling while keeping the choice of base model fixed.
our base
7

---

Under review as a conference paper at ICLR 2023
Table 1: Zero-Shot learning accuracy averaged Meta-VOL VQA test tasks, with % of available
question descriptors. Results should be interpreted as relative to a performance ceiling of 660.24 obtainable with few-shot learning on our fixed choice base model (see Appendix A.6 ours)
Method
Zero-shot (50% Q.)
Zero-Shot (100% Q.)
Base CLIP
44.99
Uncond. Multitask
53.75 (+: 0.36)
Uncond, MNet-MAML
53.04 (#: 0.69)
Uncond. MNNM-FOMAML
53.04 (#: 0.42)
Uncond. HNet-MAML
53.37 (#: 0.29)
Cond. Multitask
51.68 (#: 0.33)
54.12 (#: 0.80)
Cond. HNet-MAML
51.54 (d: 0.63)
53.02 (#: 0.20)
Cond. Multitask FiLM
51.60 (#: 0.56)
53.84 (# 0.61)
HNet HyperCLIP Guidance
53.51 (+: 0.22)
53.98 (+ 0.54)
HVAE + HyperCIIIP Guidance
53.82 (+: 0.07)
53,91 (+ 0.08)
HNet HyperLDM y
53.66 (#: 0.25)
54.06 (# 0.21)
HNet HyperLDM 7 = 1.5
54.08 (#: 0.11)
54.30 (# 0.27)
HVAE + HyperLDM y
54.72 (#: 0.23)
55.03 (= 0.32)
HVAE + HyperLDM ? = 1.5
54.84 (#: 0.24)
55.10 (= 0.08)
a) Performance of classifier-fre free guidance
b) Performance on partial conditional data
51.6
53
Set Accurncy
Query Set Accuracy
64.4
53
64.2
52
64.0
51
03.8
Cund. Multiakk
..... HVAE + Hyperl.DM
33.6
Uncund Multiaak HVAF Hypert
IIAE + IlyperLDM 7 1.5
49
Unoord. Multitsk
33.4
0.0
0.5
1.0
10
Classifier-free guidance strength y
% of available task descriptors
Figure 4: a) Performance of HyperLDM over different classifier-free guidance parameters 7. For 0 we sample from unconditional latent diffusion model. For = we sample with classic conditioning. For y > 1, we are in the classifier-fre free guidance regime. b) Performance HyperLDM against baselines in the setting where only fraction of natural language task labels given.
We then benchmark several unconditional and conditional methods, with only conditional methods having access language task descriptors. We apply MAML and its first-order variant FOMAML (Nichol et al. 2018) directly to the base network (MNet-MAML. MNee-FOMAML), and to both an unconditional hypernetwork (Uncond. HNLL-MAML, as in Section 3 and a conditional one (Cond. MAML) We also benchmark against standard multitask learning (Uncond. Multitask, Cond. Multitask), and conditional multitask learning with the classic FiLM layer (Perez al. 2017) (Cond. Multitask Fi. It is apparent that the multitask approach, at least in this setting, leads to better zero-shot models than MAMI,, which instead optimizes for few-shot performance. We refer to Appendix A.2 and A.4 for more details on each model.
We then test HyperCLIP Guidance and HyperLDM when trained on top of either a hypernetwork or VAE generator (see Appendix A.2 and A.5 3) for more detail). HyperCLIP Guidance allows for faster sampling than HyperL.M but is generally less performant, still, performs on par with or slighly improves upon all other zero-shot baselines except for Cond. Multitask. The best performing model for the zero-shot setting HVAE + HyperLDM, and specifically for classifier free guidance with 7 1.5. As illustrated in Figure 4 a, to further show the effectiveness of the classifier
8

---

Under review as a conference paper at ICLR 2023
guidance technique, we sweep over several candidate 7 parameters to find that the optimum occurs for y > 1. As shown in Figure{]j when training our model while only keeping 50% or 10% of task descriptors, traditional Cond. Multitask learning is heavily impacted, while HyperLDM is almost not affected due to its two-phased training regime based on an uncondtional VAF. The gap between the multitask baseline and our HyperI.DM technique is particularly striking in this setting.
RELATED WORK
By introducing multiplicative interactions within neural networks (Jayakumar et al. 2019), hypernetworks (Ha et al. (016) have been shown to allow the modeling of diverse target network weights in, e-g., continual learning, even in the compressive regime (von Oswald et al.) 2021a 2020) without loss of performance. For a given supervised problem, hypernetworks have been used to model the complex Bayesian posterior of the weights conjunction with variational inference (Hening et al. 2018 Krueger ct al. 2018). Similar approaches have been used for continual learning (Henning et al. 2021 Another vein of work consists using hypernetworks distill ensembles of
Hypernetworks
diverse networks (Wang et al.] 2018 Ratzlaff & Fuxin 2020
von Oswald et al. 2021a).
Meta learning In the context of meta-learning, hypernetworks have been successfully used in combination with popular gradient-based meta-learning methods (Finn et al. 2017: Zintgraf et al.) 2019; ZZaa et al., 22000; Flennerhag et al.) 2020). More generally, related works have shown the usefulness of learning low dimensional manifold in which to perform task-specific gradient-based adaptation at meta test time (Rusu et al. 2018] von Oswald et al.) 2021b; Lee & Choi (2018), instead of directly adapting in weight space. Recent works bypasses the formal bi-level formulation of meta-learning by, e.g., using transformers to directly map the few-shot examples to the weights of the target network (Zhmoginov et al. 2022).
Generative Modeling and Classifier guidance A plethora techniques have been proposed for the generation of samples from high-dimensional domains such as images, such as Generative Adversarial Networks (Goodfellow et al., 2014, Brock et al. 2018 GANs) and Variational Autoen- coders (Kingma& & Welling [014] VAEs). Denoising Diffusion Probabilistic Models (Sohl-Dickstein jet al., 2015; Ho et al.] 2020 DDPM) overcome common issues in generative modeling using a simple likelihood- based reconstruction loss for iterative denoising, and have been shown achieve
state-of-the-art results in high resolution image generation (Dhariwal & Nichol 2021 Rombach
al. 2022). Several techniques have been proposed for effective conditional sampling in generative
and diffusion models, such as classifier/LLIP guidance (Dhariwal & Nichol 2021 Gal et al. 2021
Patashnik et 2021) and classifier-fre free guidance (Ho & Salimans 2021 Crowson [0022] Nichol et 2002)). Diffusion models with classifier-fre free guidance have also been successfully applied in
non-visual domains, such as audio generation (Kim et al. 2022) and robotic planning (Janner et al.
2022).
Zero-shot learning There exists a large literature on zero-shot learning. including both established benchmarks and well known methods (Han et al.) [202]] [Su et al. 2022] Gupta et al. [02I). While these zero-shot learning works consider the zero-shot performance on unseen class labels within single classification task, our setting considers that of the zero-shot performance where test tasks themselves are unseen, thus raising the zero shot problem to the task-level.
CONCLUSION
In this work we introduced framework that re-interprets meta-learning as multi-modal generative modeling problem. Our HyperCLIP guidance and HyperLDM methods leverage this insight to gener- ate task-adapted neural network weights in zero-shot manner given natural language instructions, and constitute the first application of the CLIP guidance and classifier-free guidance techniques from image generation the meta- meta-learnin domain. Our experiments show that methods successfully make use external task descriptors to produce high- perf-rrffooaccee adapted networks in the zero-shot setting.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our contribution, despite showcasing experimental results, remains within the domain of general methods and thus does not face many immediate ethics concerns common in applied machine learning. Regardless, potential fairness and ethics issues should still be proactively investigated. A potential issue is that some of the methods described in the present work rely on pre-trained large models such as CLIP encoders, which may inadvertently encode harmful stereotypes and biases, manner which may not be immediately apparent to researchers and users making use of our downstream method.
REPRODUCIBILITY STATEMENT
We point to the supplementary material file attached to the submission for the complete codebase and Meta- VQA dataset tools, which can be used to reproduce our results. This code will be released online more polished form after the requirement for anonymity is dropped. The code repository contains README. md file with instructions for installing the necessary software environment. For a complete discussion of the hyperparameters used during our experimental runs, we refer to Appendix AA.5] As for the Meta-VQA dataset, must first download the original VQA v2 dataset (https:: http://viiualee www.ualla....o org/ down oad. html) and place it the data /VQA/ folder, and then run:
seriptes/peeccmppueee..gggeeetts py
sctipss/precmmptteeeeee..eettsse scriptss/peeccmpputt text features. py
to re-generate the pre-computed CLIP embeddings.
REFERENCES
Luca Bertinetto, Joao E. Henriques, Philip Torr, and Andrea Vedaldi. Mcta-leaming with diffcrentiable closed-form solvers. September 2018. URL http:://opeerreveee..etttt
HyxnZh0ct7
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity
Natural Image Synthesis. October 2018. URL URU/nttsss//opeeeezzee net forum? ?
B dredduuneemallutt www.cceeeeeeeeeeeeeeallt........
utm medi
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language Models Few-Shot Learners. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- neurips ec cccpappee/222//aaa.. html
ciates, Inc., 2020. URL 24l.ffaaaaa222222--aaaaart-aaea0 1457c0d cb4 1bt.r//4446611111186a111111111
Bbf
Katherine Crowson. v-diffusion, May 2022. URL https:://ttttaiaae www.tttwww..oo///wwssst..o
v- c.lffusion-pytore. original-date. 2021-12-16T11::44333
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthe- sis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780 8780-8794, Curran Associates, Inc., 2021. URL httpss://papptt html
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-Agnostic Meta-Leaming for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference Machine Learn- ing. pp. 1126-1135. PMLR, July 2017. URL finn 7a h ISSN: 2640-3498.
mlr. mttp
10

---

Under review as a conference paper at ICLR 2023
Sebastian Flennerhag, Andrei A. RUsu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell, Meam-ngg with Warped Gradient Descent, February 2020. URL http:ttt:aaaat
org/abs aa////bb/9999 1909. 00025 arXiv:1909.0000 [cs, stat].
Rinon Gal, Or Patashník, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. Technical Report arXiv:2108.00946, arXiv, December 2021.
00946 arXiv:2108.00946
[cs] type: article.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and
Yu Qiao. CLIP-Adaples Better Vision- Lansuun- Models with Feature Adapters, 1rXiv:2110.045:
[cs], October 2021. URL/attp://rxxi//
04544
arXiv: 2110.04544.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of
the ACM, (3(11):1391144 2014. ISSN 0001-0782. doi: 10.1145/342262.. URL https
7/doi .
10 0ttpg////111111412225551
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. pp.
thecvf. com/ wwwotentttvvtpeoooe 2017/ htm1/Goya Ma whw_thh___hhheeee_eeeeeeeeee CVPPP 201 papeppppppp___ html.
6904-6913, 2017. httptttts://ppeaaeaaaaa
Akshita Gupta, Sanath Narayan, Salman Khan, Fahad Shahbaz Khan, Ling Shao, and Joost van de Weijer. Generative Multi-Label Zero-Shot Learning, January 2021. URL http:ttt////.v..oo/
abs/2101. 11606
arXiv:2101. 1166 [es] version: 2.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending Clip to Image,
Text and Audio. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980, May 2022. doi: 10.1090CCASPP499222000944444 ISSN: 2379-190X.
David Ha, Andrew Dai, and Quoe V. Le. HypenNetworks. Technical Report arXiv: wwwp:.6669.911100 100. arXiv, December 2016.
09106) arXiv: 1609009106
[cs] type: article.
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive Embedding for Generalized Zero-Shot Learning. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2371 2371-2381, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-50921 doi:
URL https::pps.....m...tt. //te:///eeeeeeeeeeee//e/ee eeee. org/document/
9578587/
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Leven Performance on ImageNet Classification, February 2015. URL http://////axi
org/abs/1502. 01852 arXiv:1502.01852 [cs].
Christian Henning, Johannes von Oswald, Joao Sacramento, Simone Carlo Surace, Jean-Pascal Pfister, and Benjamin Grewe. Approximating the Predictive Distribution via Adversarially- Trained Hypernetworks. In Bayesian Deep Learning Workshop, NeurIPS 2018, Montréal, Canada, December 2018. Yarin. doi: 10.5167/uzh- 168578. URL 2tt//2118pppers// pdf
Christian Henning, Maria R. Cervera, Francesco D' Angelo, Johannes Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, João Sacramento, and Benjamin F. Grewe. Posterior Meta- Replay for Continual Learning, 1:X.v:2103.0113: [cs], June 2021. URL htttp:///axiv...////////// arXiv: 2103.01133.
abs/ 2103. 011133
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. November 2021. URL
hagp://weae.abbbruuuuuuaaaaaaag. campaign-
The cht htw.shauuuummmoocehhhaaaa source=hs email mww. hsenc=
b2ANgtz-Bxln811vazze
FoR8xSKJCoNN3kaue

---

Under review as a conference paper at ICLR 2023
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Ad- vances in Neural Information Processing Systems, volume 33, pp. 6840 0840-6851. Curran Asso- ciates, Inc., 2020. URL https: https::/ppoceeeainnneeeooooo//ee www.riininnnnnnnnnneo.eommm....e CC/ htcppaeer/220/haah.
html
Jie Hu, Li Shen, and Gang Sun. Squuezz--mddEExiitttio Networks. pp. 7132-7141, 2018. httpa//teee/geaaccc..oooooo..a/a httpa/accnnooooooooooooeee////// Sguiaeze-anddPxciitttioonnneeee http://ccc.ttttttoeeeetteeeerraa [rg.epaccccaaaaacaaarro.oororrrr 1we00__ppper.... .html
cvpr 2018/html/Hs
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kemel: Convergence and
Generalization in Neural Networks, February 2020.
07572 1EXv::880.007722 [cs, math, stat].
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with Diffusion for Flexible Behavior Synthesis. Technical Report arXiv:2205.0990 arXiv, May 2022. URL harg/abb. abs /bbs/2255 09991 arXiv:2205.000 109999295009991
http://arxi: org/oagggo.
[es] type: article.
Siddhant M. Jayakumar, Wojeiech Czarnecki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative Interactions and Where to Find Them. September 2019. URL httpttss::////tttttttt https://ppeneeviww.ntt/tt EyInK6VEDH
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-TTS: Diffusion Model for Text- to-Speech via Classifier Guidance. In Proceedings of the 39th International Conference
Machine Learning, pp. 11119-11133. PMLR, June 2022. URL lnttps:
mlr. wtt.ppres/vv1///// wttps/ssvvv22kkmmmmmma http//.m2ddd..m html
ISSN: 2640-3498.
Diederik P. Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization, January 2017.
URL http://arxiv. org abs/ 6980 arXiv:1412.6988 [cs].
Diederik Kingma and Max Welling Auto-Encoding Variational Bayes. arXiv: 1312.6114 [cs, stat], May 2014. URL
11///3111111 6114
arXiv: 1312.6114.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian Hypernetworks, April 2018. URL
04759 arXiv:1710.04759 [cs, stat].
Yoonho Lee and Seungjin Choi. Gradient-Based Meta-Leaming with Learned Layerwise Metric and Subspace, June 2018. URL stat].
05558 arXiv:1801.05555 [cs,
Zhao Mandi, Pieter Abbeel, and Stephen James. On the Effectiveness of Fine-tuning Versus Meta- reinforcement Learning, Technical Report arXiv:2206.03277 033271, arXiv, June 2022. URL http! //arxiv.org//aeeeev. /206. 03271 2XXi::2206.03271 [cs] type: article.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms, October 2018. htl.httpp/aaxxv..oom
02999 arXiv:1803.02999 [cs].
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models. Technical Report 1rXiv:2112.1077 arviv, March 2022. URL arXiv:21 12.11441
http:://axxit orgg abs
/ /2122 10741
[es] type: article.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Style- pp. 2085 2094, 2021. URL
CLIP: Text-Driven Manipulation of StyleGAN Imagery.
2021 2a1l1_pper.hit html
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FILM: Visual Reasoning with a General Conditioning Layer, December 2017. URL htttp:/arxxvv.org abs/11099 07871 arXiv:1709.000 [es, stat].
12

---

Under review as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.00 000000 [cs], February 2021. URL BRLhttp::/ /ttp//rxiiiii wttpaaiww...gaaa/a/a/o//aaa..aaa 23. 00020 arXiv: 2103.00020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text- Conditional Image Generation with CLIP Latents. Technical Report arXiv:2204.06122 arXiv, April 2022. URL http:://axxvvv.o///..oa. article.
06125 arXiv:2204.06125 [es] type:
Neale Ratzlaff and Li Fuxin. HyperGAN: A Generative Model for Diverse, Performant Neural 1058, arXiv, July 2020. URL abs/1901. 11058 1rXiv:1901.11058 [es, stat] type: article.
Networks. Technical Report arXiv: 1901,
Sachin Ravi and Hugo Larochelle. Optimization as a Model for Few-Shot Learning. October 2016. oow.ooormmmmmoooooooooo
URL
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis With Latent Diffussion Models. pp. 10684-10695, 2022. URL htttps://opeaacae thw.cccessssstcceeeceeoooeeeeeee thecvf com/ cont http://onn/////////////////// http:///2///2////////// /Rombaac... Becch-Resolutonn Image Synthesis Sates_tthhhhh__hhhh Latent Diffusion Models CVPR pa1t22_apprhhtmmm
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.). Medical Image Computing and Computer-Assisted Intervention MICCAI 2015, Lecture Notes in Computer Science, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-9-319-24574-4 doi: 10. 0111///788--119-55---4- 28.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. September 2018. URL
net htte:/oouuu22cccBBggLAAAAA/////
page
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamies. In Proceedings of the 32nd Interna- tional Conference on Machine Learning, pp. 2256-2265. PMLR, June 2015. URL httpss //proceedings.nut mlr wwwpse/v31111 /sol1/ 2ttp://sooodiictteiiii15- wwwikksssinn...mm ntml
ISSN: 1938-7228.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Dis- tribution. In Advances in Neural Information Processing Systems, volume 32, Curran Asso- ciates, Inc., 2019. URL 2rtolefff.ooccaaaaaaaacaaaod.a.
www.ddddiiinnnnnenne.peee..... wwt..eiis.....ppe......a. wttpcpppp///111133 11200999haa 9/hash/
Abstract html
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Er- mon, and Ben Poole. Score-Based Generative Modeling through Stochastic Differen-
tial Equations. September 2020. URL bttpss://ppezeevee net
PxTIGI 2RRFS& campaaaaaaaaaaaaaaaaaa wtt.a//aiii.anteeeeeeeeeeeeeeeen wtteeaannsttssssteeeeeee
Revue&20news letter)
Hongzu Su, Jingjing Li. Zhi Chen, Lei Zhu, and Ke Lu. Distinguishing Unseen from Seen for Generalized Zero-shot Learning. In 2022 JEEE/VVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 7875-7884, New Orleans, LA, USA, June 2022, IEEE ISBN 978-1- URL htttthtttss:seeeppeeee
66546-946-3 doi: 10. 10.1109/C9P5226888888 202.22228888220777777
httg/docuent/// 98791497
Johannes von Oswald, Christian Henning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations (ICLR 2020), May 2020. doi: 10.48550/arXiv. (tt./a/Xiiv.100.0666660 1906.0069.. URL http:://aaaae orq /abb/////66 [cs, stat].
00695
arXiv: 1906.0000.000 006900006555
13

---

Under review as a conference paper at ICLR 2023
Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans, Christian Henning, Benjamin F. Grewe, and João Sacramento. Neural networks with late phase weights. arXiv:2007. 12927 [cs, stat], April 202 URL http:: org. absa 12927
arXiv: 2007.12927.
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where learm: Gradient sparsity in meta and continual learning. arXiv:21100 S.kiv:2110.114400: [cs]. October 2021b. URL http:://arxs org/ abs 2110
14402 arXiv: 2110.14402.
Kuan-Chieh Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Ad- versarial Distillation of Bayesian Neural Network Posteriors. In Proceedings of the 35th
International Conference Machine Learning, pp. 5190-5199. PMLR. July 2018. URL ISSN: 2640-3498.
wwwpeeddingsmm..... wwt.mmll..ppr..... wwwphang..............m bacress/v0//e 11000/wang888 html
Dominic Zhao, Seijin Kobayashi, João Sacramento, and Johannes von Oswald. Meta-Learning via Hypernetworks. In 4th Workshop on Meta-Learning at NeurIPS 2020 (MetaLearn 2020), Virtual
Conference, December 2020. IEEE. doi: 10.5167/uzh-2002986 URL https:/ /tttps://ww.. zora
uzh. httpcch////eerrt/2223555555
Andrey Zhmoginov, Mark Sandler, and Max Vladymyrov. HyperTransformer: Model Generation for Supervised and Semi-Supervisee Few-Shot Learning. July 2022. URL htt.::tpp///ii....tt/t org
abs/2201. 04182
arXiv:2201.044 20002200..41822 [cs].
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast Context Adaptation via Meta-Learning June 2019. URL /ttp://axvvvvv//aa//ea//e..eaaav /ttp//aaw...rgaa////aaaaa
03642 arXiv:1810.0348 [cs, stat].
A APPENDIX
A.1
CLASSIFIER-FREB GUIDANCE
We hereby provide rationale for the use of classifier guidance and classifier-free guidance during diffusion model sampling. As per the "score matching" interpretation of diffusion models, we assume that our trained noise network approximates the score function of the true conditional latent distribution p(zlex) as (z" t, e4) x -o, V se log (e.). For classifier guidance, we can perturb our diffusion sampling by adding the gradient of the log likelihood of CLIP encoder (e (ze) to the diffusion score follows
(p''', = (=((" (,),))))))) nor 21 logl)(ee)=") 2. [logp(='[ss) + log P (es/2
(10)
We can rewrite this as classifier guidance on the unconditional score V. log (o*) with
o, V. [logp(>') [ogp((e]2=) with ? 1+1
(11)
= using Bayes' rule, as logp(2 (e))) log logp(e;(2*) + +og - log pl (e:), and thus V log p(2))e)) 2. logp(e; + V., logp(2") p(2²) 2²).
For classifier-free guidance, we aim to perform the above sampling without access to a classifier, as long we possess a conditional diffusion model (2 ,t e,) that doubles as an unconditional model (2 f, 0), as illustrated in Section 5.2
Using Bayes' rule again, we can see that a.. logp(e,(2*) V logp(="(ee) V logp(2"). we substitute this into Eq. 11 we obtain
o. V.V. [logp(>') + (logr(=' lez) - logy(='))]
(12)
o. 2. st [(1 7) p(=))()))))))))))))))))))) 1og
(13)
which can be implemented with conditional network
(p'=', - c ( (,e)).
(2
(14)
14

---

Under review as a conference paper at ICLR 2023
A.2 NETWORK ARCHITECTURES
Base Network (f) Our choice for a base model is CLIP-Adapter (Gao et al. 2021). which consists of frozen CLIP image encoder with added learned fully- tully-connected layers refining the output embedding. Specifically, we use the ViT-L/14@ 336px CLIP encoder type with embedding size of 768. The advantages of this model choice lie in its combination of high base performance (due to pre-trained knowledge contained in the CLIP component) and relatively small parameter count, enabling agile medium-small scale experiments. This base ClIP-Adapter network purely works as base model and is not to be confused with HyperCLIP, which is employed at the meta-level. In Section 6.2, when benchmarking the base model alone in the zero-shot setting, we drop the Adapter and use pre-trained zero-shot CLIP (Radford et al. 2021
Hypernetwork (h) For the hypernetworks used in our baseline as well as as the generative model, we use MLP with one hidden layer of 256 units, which are followed by rectified linear activation. For the unconditioned hypernetwrk,, the embedding to the hypernetwork is a vector of dimension 64, while for the conditioned counterpart, the task embedding is used. In order to ensure that the generated weights are properly normalized at initialization, we use the Kaiming initialization (He) al. 2015) for the hypemettwrk weights, initialize the embedding as sample from multivariate
standard gaussian distribution (for unconditioned models), and use the NTK parametrization (Jacot
al. 2020 for the target network.
Variational Autoencoder For the variational autoencoder used as our unconditioned generative model, we use an MLP of hidden layers of size 512 and 256, each followed by the rectified linear non-linearity. We chose 32 as the latent code dimension. We use the same architecture for the decoder, except for the dimensionality of the hidden layers being swapped. We use the Kaiming initialization (He et al. 2015) to initialize the weight of both the encoder and decoder.
HyperCLIP We parametrize our HyperCLIP model as fully-connected MLP with a single hidden layer of dimension 256, taking as input the flattened weight of the corresponding CLIP encoding. We chose the tangent hyperbolic function as the activation function in the hidden layer.
base network and outputting the
HyperLDM While the original LDM makes use of time- conditional UNet (Ronneberger et al.. 2015) to parametrize the noise network, we are unfortunately unable to make use of spatial information and convolutions due to the non-spatial nature of our latent space. We parametrize our HyperI.DM as fully-connected network with residual connections and squeeze-and-cxcitatae layers (Hu et al. 2018). The time index t is embedded into a vector with a 150-dimensional sinusoidal positional embedding, and is concatenated together with the task-conditional embedding e; at the input layer and at intermediate activations. Hidden layer dimensions are 8192, 16384, 8192.
A.3 THE META-VQA DATASET
The original VQA problem is about choosing a suitable natural language answer as when prompted with both a natural language question and an image 4t- Our observation is that the VQA problem can then easily be reformulated as a meta-learning image classification problem with natural language task descriptions: given question-imat iewiiiiaaaaaaaaeee unique questions qi (which will serve as task descriptor fz),, each of which can then be associated with supervised image classification tuples (T}, at) D; To make sure the designed tasks are meaningful, we filter out question-anss -anseer pairs with questions in choosing form, e.g., "A or B?" or yes/no answers. From the remaining questions we keep the ones which appear at least 20 times throughout the dataset, such that each task contains enough samples. In the end our Meta- VQA dataset is composed of 1234 unique tasks (questions), split into 870 training tasks and 373 test tasks, for a total of 104112 12 image-anse answer pairs. There are on average 9.13 answer choices per question/task. The average size of the support set is 57.85 examples, while the average size of the query set is 25.9 examples.
triples (4k. L2, ax) E D, we can group the data by
15

---

Under review as a conference paper at ICLR 2023
A.4 BASELINE METHODS
We detail an overview of the baseline methods we benchmark in table 2 together with algorithm tables detailing each baseline method.
Training: The number of epochs each model is trained on, the learning rate 1r of the optimization, well the learning rate and number of steps of the adaptation algorithm used for each method can be found in 3 For all methods using an adaptation AT. the dataset from the task is randomly split into support set and query set during training, every time task is sampled. The support set is then used to perform the adaptation (see Section Se.tion[.II]. while the query set is used to compute the loss on which the meta-parann pera-parseeters are updated. When no adaptation used, all the data is used for this update. Unconditional methods do not have access to the task embedding e,, while conditioned methods do. When the percentage of available task descriptor reduced, conditioned methods are trained only on the tasks which descriptor is available.
Evaluation: Evaluation is performed on a fixed query set on the predefined query set of the held- out test tasks of the Meta- VQA dataset. Zero-shot performance is evaluated before applying the adaptation procedure AT,- For the few shot performance, all adaptation is performed on the support set of the test tasks. For MAMI. baselines,
keep the same adaptaiton-time learning rate during training, while we always adapt for 50 steps. For each multitask baselines, we use the same adaptation scheme (steps, learning rate, adapting parameters) as their MAML counterpart.
Algorithm 2 Unconditional Multitask Training
Define the base network f with parameters W. for epoch 1...., .N do Sample training batch of image-answes pairs (I', yk) from mix of random training tasks T,
Update W with gradient descent computed with respect the classification loss the sampled batch.
end for
Algorithm 3 Unconditional MNet-MAML Training
Define the base network ff with parameters W.
for meta-epoch = 1,, ........ .N do
Sample a training task T, and data D;. Randomly split D; into support set D and query set D
Run inner-loop adaptation A7. using the support set D;, fine-tuning W into task-adapted
W: = AT, (W).
Use MAML gradient update to adapt W given the inner-loop adaptation.
end for
Algorithm 4 Unconditional HNet-MAML Training
Define the base network f with parameters W.
Define a hypernetwork h with meta-parameters 0, mapping a latent vector 20 base network weights W.
for meta-epoch = I,..., .N do
Sample training task T. and data D.. Randomly split D, into support set D; and query set D.-
Run inner-loop adaptation Ar, using the support set D;. fine-tuning d0 into task-adapted
2, Ar, (z").
Use MAML. gradient update to adapt 20 and 0 given the inner-loop adaptation. end for
16

---

Under review as a conference paper at ICLR 2023
Algorithm 5 Conditional Multitask Training
Define the base network with parameters W. Define hypernetwork h with meta- perammerssssers 0 mapping the clip embedding of the language task descriptor base network weights W,.
for epoch = ......N .N do Sample a training batch of task clip embedding, image and answer triples (ER, Thi ys) from mix of random training tasks Tí
Update 9 with gradient descent computed with respect to the classification loss over the sampled batch. end for
Algorithm 6 Conditional HNet-MAML Training
Define the base network f with parameters W.
Define hypernetorkk h with meta-parameters 0, mapping the clip embedding e; of the language task descriptor to base network weights W..
for meta-epoch- :,..... . do
Sample a training task Tr data D; and the clip embedding e; of the task descriptor. Randomly split D; into support set Dj and query D Run inner-loop adaptation Ay, using the support set D;, fine-tuning e; into task-adapted
é, AT, (ez).
Use MAMI gradient update to adapt 0 given the inner-loop adaptation.
end for
Table Overview of the different methods trained on MetaVQA. The parameters are optimized via the task loss evaluated on the output of the function, averaged over minibatches of tasks. The adaptation A7, implements few step gradient descent algorithm applied on the argument parameter, w.r.t the task loss evaluated on the support set.
Method
Function
Parameters
Unconditional Multitask
f(. W) Al, (Wo)) ((=))
W
Unconditional MNet (FO)MAML
W
Unconditional Hypernetwork MAML
0,
Conditional Multitask
fl, Q)) (C.)(A
0
Conditional Hypernetorrk MAML
0
Table 3: Hyperparameters used for the baseline methods. All methods are trained with the Adam (Kingma & Ba] 2017) optimizer, with meta-batch size of 32 tasks. We use gradient norm clipping for all optimization, with the maximum norm set to 10. Note that when the adaptation algorithm has range of possible steps, the number of step is sampled uniformly from the range for every adaptation.
Method
epochs
1r
A-1r
A-steps
Unconditional MNet Multitask
300
0.0001
Unconditional MNet (FO)MAMI.
500
0.00003
0.01
0-10
Unconditional Hypernetwork MAMI.
100
0.00003
0.1
0-10
Conditional Multitask
60
0.0001
Conditional Hypernetwork MAMI.
200
0.00001
0.1
0-10
17

---

Under review as a conference paper at ICLR 2023
A.5
GUIDANCE MODELS
A.5.1 GENERATIVE HYPERNETWORK
To enable our guidance methods, we need to first train a generative hypernetorkk h as in Section 3 either in the form of an Unconditional Hypernetwork, of a Hypernetwork VAE:
For HNet HyperCLIP guidance and HNet HyperLDM, we meta-learnt an uncondi- tioned hypernetwork with the exact same hyperparameters as the baseline Uncond. HNet- MAML, and used it as the generative hypernetwork.
For HVAE HyperCLIP guidance and HVAE + HyperLDM, we trained an unconditioned VAE on samples of fine tuned network weights W; using the architecture specified in [.2.2 In order to be able to quickly sample new adapted weights, and to reduce the complexity of the manifold from which such weights are sampled, we use adaptations from our unconditional MAML baselines W. Specifically, Variant (Algorithm involves adaptations from Uncond. HNet-MAML, using 50-step adaptation AT, with learning rate 0.1, on support set stochastically sampled for every adaptation phase. Variant (Algorithm S involves adaptations over the base network (initialized from a learned Uncond. MNet-MAN initial- ization), using 50-step adaptation AT, with learning rate 0.01, on support stochastically sampled for every adaptation phase. We trained the VAE on 2000 epochs where each epoch is a single pass through all the tasks, with the Adam (Kingma Ba (017) optimizer and 0.0001 learning rate and batch size 32. We used gradient norm clipping independently for both the encoder and decoder, with the maximum norm capped at 1000. For the final results presented in our main results table (Table II. we use Variant 2.
Algorithm HVAE Training, Variant
Define the base network f with parameters W.
Define an encoder parameters 0.
((Www) with parameters ws and hypernetwork decoder W 1(2, 0) with
Obtain a previously learned HNet Ti(=,, 0) according to Uncond. HNet-MAMI, (Algorithm) 6). for epoch = ,,...,N N do
Create empty batch B = O.. for b= 1.... M do
Sample training task T; and data D:.
Randomly split D; into support set D and query set D.
Run inner-loop adaptation Ar, using the support set DJ, fine-tuning 20 from the HNet into adskedd 5; (z").
Given fine-tuned 2,, produce task-adapted weights W; = h(2,,) 8). Add the fine-tuned weights to the batch: B B (W.).
end for
Train the HVAE encoder and decoder using the VAR loss reconstruct the weight batch B. end for
18

---

Under review as a conference paper at ICLR 2023
Algorithm 8 HVAE Training, Variant
Define the base network with parameters W.
Define an encoder 2 = w) with parameters w and hypernetwork decoder W = h(2, with parameters 0.
Obtain a previously learned base network initialization W0 according to Uncond. MNet-MAML (Algorithm
6
for epoch
N do Create an empty batch = ( for b = .,..., M do
Sample training task T; and data D;- Randomly split D; into support D and query set D.
Run inner-loop adaptation Ar, using the support set D; fine-tuning W; = AT, (WO). Add the fine-tuned weights the batch: B (W:).
end for
Train the HVAE encoder and decoder using the VAE loss to reconstruct the weight batch B.
end for
A.5.2 HYPERCLIP
Training In order to train the HyperCLIP model, we need samples of fine tuned network weights W,. Similarly to HVAE, used adaptations from Uncond. HNet-MAML, using 50-step adaptation Ar, with learning rate 0.1, on a support set stochastically sampled at every adaptation phase, as this would allow We trained our HyperCLIP model for 600 epochs, with the Adam (Kingma & Ba 2017) optimizer, 0.0003 learning rate and batch size 64, for all our experiments.
the same HyperCLIP model for doing guidance both HNet and HVAE.
Guidance We use 10 steps guidance with À 0.01 and learning rate 0.1, for both when performed on HNet and HVAE.
Evaluation Evaluation is performed on a fixed query set on the predefined query set of the held- out test tasks of the Meta-VQA dataset. Zero-shot performance is evaluated on the output of the generative hypernetwork after applying latent space guidance. For the few-shot performance, all adaptation is performed on the support set of the test tasks, the latent space initialized at the output of the guidance procedure. Similarly to our baselines, we use 50-steps gradient descent adaptation with learning rate 0.1.
Algorithm 9 HNet HyperCLIIP Training
Learn unconditional hypernetwork li(20,0 with the Uncond. HNet-MAML, procedure from Algorithm 6
Learn HyperCLII network CLIP=(W) using the HyperCI.IP training procedure from Algorithm
For sampling fine-tuned W,, fine-tune the base-network on training tasks.
Algorithm 10 HVAE HyperCLIP Training
Learn unconditional hypernetwork h(z, 0), as the decoder of a HVAE (Algorithm g
Learn HyperCLIP network CL using the HyperCLIP training procedure from Algorithm For sampling fine-tuned W,, fine-tune the nese-network on training tasks.
19

---

Under review as a conference paper at ICLR 2023
Algorithm 11 HyperCLIP Guidance (Inference time)
Define a learned unconditional hypernetwork h:2, 0). as either a HNet h(2²,0) (Algorithm 6) or the decoder of HVAE (Algorithm 8).
Define a learned HyperCLIP network CW).
Define unseen task T; with natural language task descriptor f..
Randomly sample 2 (O, I) if using the decoder of HVAE, or set = 20 where 2n is the meta learned embedding initialization Optimize with gradient descent over Lguidaace (Eq. 4 obtaining guided z;-
the Hnet.
Obtain guided base weights W; h(2,, 8).
Use adapted base network f with weights W; to classify examples from the unseen task Tr
A.5.3 HYPERLDM
Training Similarly to HyperCLIP, to train HyperLDM we need samples of fine tuned network adaptations from Uncond. HNet-MAML, using 50-step adaptation
weights W,, for which Ar. with learning rate 0.1, on a support set stochastically sampled at every adaptation phase. We parametrize the diffusion process with a linear noise schedule, starting 0.0001 and ending 0.06, and 350 diffusion timesteps. We train the HyperL.DM for 1000 epochs with the Adam optimizer, 0.00025 learning rate and 128 epochs, for all our experiments.
Evaluation Evaluation is performed as for HyperCLIP guidance, except for the fact that adaptation performed natively through sampling from the learned reversed diffusion process, with parameters derived from the chosen & schedule. The guidance parameter >> 0 can be tuned during inference to accentuate the effect of classifier-free guidance.
Algorithm 12 HNet + HyperLDM Training
Learn unconditional hypernetwork h(=9,0) with the Uncond. HÑet-MAML procedure from Algorithm
Learn the HyperL.DM network (,(2', (0(2,, es) using the HyperL.DM training procedure, optimizing
reconstruction of =! with loss from Eq. 8 For sampling fine-tuned 2;, fine-tune the base-network on training tasks, then encode the weights using the HNet.
Algorithm 13 HVAE + HyperL.D Training
Learn unconditional hypernetwork h(2,0) (), the decoder of a HVAE (Algorithm g)
Learn the HyperL.DM network (2', t.. ez) using the HyperL.DM training procedure, optimizing reconstruction of =! with loss from Eq, For sampling fine-tuned 2,, fine-tune the base-network training tasks, then encode the weights using the HVAE.
Algorithm 14 HyperLDM Inference
Define learned unconditional hypernetwork h[2, 0), as either a HNet h(20,0) (Algorithm 6 or the decoder of HVAE (Algorithm(8)
Define learned HyperLDM network eØ (=" t,)).
Define an unseen task T. with natural language task descriptor 4, with clip embedding e4. Randomly sample AO0 I).
Iteratively modify with diffusion sampling using the learned ep network. obtaining guided 2,.
Obtain guided base weights W; h(22, 0)
Use adapted base network ff with weights W; to classify examples from the unseen task Tz.
A.6 FEW-SHOT LEARNING
For completeness, we include in Table results for few-shot learning on the test split of Meta- VQA. Our technique, unlike classic MAML., does not optimize specifically for the few-shot learning
20

---

Under review as a conference paper at ICLR 2023
setting. Instead, the few-shot learning results meant to contextualize performance gains in the zero-shot setting: zero-shot performance gains should be interpreted as relative to the few-shot performance ceiling of 60.24, the maximum attained with our fixed choice of base model.
Table 4: Few-Shot learning accuracy averaged over Meta- VQA test tasks. (* ours)
Method
Few-Shot
Base CLIP-Adapter
54.93 (# 0.11)
Uncond. Multitask
55.53 (#: 0,40)
Uncond. MNet-MAML
60.24 (#: 0.84)
Uncond. MNet-FOMAML
60.03 (# 0.48)
Uncond. HNet-MAML
58.70 (# 0.10)
Cond. Multitask
59.46 (#: 0.31)
Cond. HNet-MAML
59.48 (#: 0.03)
HNet HyperCLIP Guidance
58.82 (# 0.27)
HVAE + HyperCLIP Guidance
58.75 (#: 0.29)
HNet HyperLDM 7
58.70 (#: 0.11)
HNet + HyperLDM 7 1.5
58.60 (#: 0.09)
HVAE HyperLDM 1=
58.97 (# 0.09)
HVAE + HyperLDM 7= 58.89 (# 0.07)
21


Reviews:
[{"review_id": "xQPoPlpZ4kp", "reviewer": "Reviewer_TMAm", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for adapting a base model (in their case, a VQA model) given a task descriptor. The authors propose to perform such adaptation via a hypernetwork layer, and learn a latent representation through which one can control the modulation of the base model by training a VAE on said layer’s weights. This manipulation is achieved by either 1) using a task descriptor learned through joint embedding of task description-parameter pairs for classifier-like guidance in the hypernetwork latent space or 2), fitting a generative model on task descriptor-parameter pairs and using classifier-free guidance on the hypernetwork latent. They demonstrate that these techniques allow them to condition the base model on in-distribution test task descriptions so as to achieve a multi-task learning effect. While the methods considered are somewhat novel and interesting, their merit is not clearly demonstrated experimentally, as it does not clearly outperform the straight-forward yet strong baseline of conditional multi-task learning. Moreover, the setting considered in experiments may not exactly require meta-learning, since the task inference problem is not present at test-time (task description is given), so that the problem reduces to conventional multi-task learning.\n", "summary_of_the_review": "While the methods proposed are fairly original, such approaches contribute considerable added complexity without demonstrating clear advantages to the straight-forward baseline of conditional multi-task learning. Perhaps more importantly, a key issue of clarity is the positioning of the work as pertaining to meta-learning -- which involves both task inference and subsequent adaptation -- since the task inference problem is not present in their train and test settings. ", "strength_and_weaknesses": "Strengths:\n\nThe authors propose a novel approach for modulating a base model in a task specific manner by guiding the prediction of dynamic weights with a learned representation of the task description.\n\nThe authors propose a novel approach for how said task representations can be learned by contrastive learning for joint embedding of task description and dynamic weights.\n\nThe technical background relevant to the method -- hyper-networks, generative hyper-networks, guidance with a contrastive classifier, as well as classifier-free guidance -- is clearly written.\n\nWeaknesses:\n\nThe experimental setting used to validate the effectiveness of the approach does not necessarily require meta-learning (task inference followed by adaptation), since the model is conditioned on an explicitly provided task description. The work thus seems incorrectly positioned, insofar as it relates more to multi-task learning rather than meta-learning. This results in a lack of clarity, which could be resolved by positioning the approach as a solution for multi-task learning.\n\nThe proposed method is not shown to outperform the straight-forward yet strong baseline of conditional multi-task learning, despite being significantly more complex. More experiments are needed that clearly demonstrate the advantage of introducing such complexity.\n\nPopular relevant approaches such as FiLM (Perez et al, 2017)  and AdaIN (Huang et al, 2017) are not discussed, but would serve as strong baselines for base model modulation with a conditioning signal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hfzCVoAsRO", "reviewer": "Reviewer_HRh8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- This paper proposes generative models that predicts (or generates) high-performing task-specific weights conditioned on text description of the task.\n- Specifically, three \"Hyper\"-components are proposed: \n1) Hyper-\"decoder\": a decoder of VAE trained with (unconditional) auto-encoding of weight vector,\n2) Hyper-\"CLIP\": task and text encoders trained to maximize cosine similarity of positive pair,\n3) (optional) Hyper-\"LDM\": diffusion model for the latent space of 1) Hyper-\"decoder\".\n- After training the above three components, we can sample task-specific weights through:\n $t \\xrightarrow{\\text{HyperCLIP}} e \\xrightarrow{\\text{HyperLDM}} z \\xrightarrow{\\text{HyperDecoder}} w$.\n- The experimental results show that the proposed method performs better than the relevant baselines.\n\n\n", "summary_of_the_review": "This paper proposes multi-modal generative models learn to generate ``neural network weights'' conditioned on text description on new tasks, and provides a kind benchmark to this research direction. Some of details can be improved.", "strength_and_weaknesses": "Strength:\n1. This multi-modal generative modelling of \"neural network weight\" is novel .\n2. The task of Meta-VQA is firstly proposed in this paper, and it is very interesting for me.\n3. This paper clearly benchmark the given task including convincing baselines and the proposed method, which largely contributes to (multi-modal) meta-learning community.\n\nWeakness:\n1. The proposed method consists of three components, and their usage is still confusing for me (they are used in two ways: w or w/o Hyper-LDM). What I understand is: a) $z_0$ is firstly sampled, e.g. $z_0\\sim N(0,I)$, b) $z_0\\rightarrow z_t$: the code is iteratively modified through either gradient descent on equation 4 (w/o Hyper-LDM) or LDM, and c) Hyper-Decoder decodes $z_t$ to obtain the weight. If this is not wrong, I would like to recommend either algorithm tables or figure explaining how they are used in the meta-test (inference time).\n2. I can't find any information about how exactly the base model perform each task. Does the base model find a word from vocabulary by measuring the cosine distance? Or is it a classifier? Then, how the generated weights are used for the tasks? These questions can not be resolved from this paper. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "JqzFjlVdGz", "reviewer": "Reviewer_mvke", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents two new approaches to zero-shot learning via building generative models of classifier weights. Contrastive learning is used to build a CLIP-style model which matches task description with network weights. In one approach, this CLIP-style model is used at test time to maximize match between provided task description and the network weight. Here, the search space for network weights is constrained to the latent space of the generative model. \nIn the other approach, a DM is trained on the latent space of the generative model, conditioned on the CLIP embedding of the task. Classifier-free guidance is used at sampling time to improve quality. A new dataset Meta-VQA is also proposed for evaluating zero-shot classification methods.", "summary_of_the_review": "Given the above concerns, I think this work is not suitable for publication in its current state, and hence I suggest rejection.", "strength_and_weaknesses": "While the use of hyper-networks in meta learning is not new, this particular choice of architecture and loss (VAE+ELBO) in this paper has not been investigated before. The proposed methods for zero-shot learning, in particular the LDM based variant, is novel to my best knowledge. \n\nI have some concerns regarding the design of the method and the experiments, listed below:\n\n1. The VAE has the critical role of compressing and linearizing network weights into an euclidean-like latent space. Have you investigated the choice of architecture for both the encoder and decoder beyond shallow MLPs? Often we see adaptive-normalization layers being more appropriate than feedforward MLPs for outputting to non-euclidean vector spaces, have you experimented with any such models?\n\n2. There exists a large existing literature on zero-shot learning, including both established benchmarks (e.g. OpenImages, Cub200-2011, Animals with Attributes AWA, imagenet 21k etc.) and well known methods (e.g. https://arxiv.org/abs/2101.11606v2, https://openaccess.thecvf.com/content/CVPR2022/html/Su_Distinguishing_Unseen_From_Seen_for_Generalized_Zero-Shot_Learning_CVPR_2022_paper.html, https://openaccess.thecvf.com/content/CVPR2022/html/Zhu_Dual_Cross-Attention_Learning_for_Fine-Grained_Visual_Categorization_and_Object_Re-Identification_CVPR_2022_paper.html, https://openaccess.thecvf.com/content/CVPR2021/html/Han_Contrastive_Embedding_for_Generalized_Zero-Shot_Learning_CVPR_2021_paper.html). Have you 1. experimented with running either of your proposed methods on standard ZSL benchmarks? And 2. Have you tried running any previous works (besides CLIP) on your proposed dataset? The related works section also do not cite any prior works in zero-shot learning.\n\n3. Very few details are provided regarding the definition of the baselines that you compare to. For example, how is MAML used without any test-time adaptation? How do you solve the class assignment problem in that case? What is the “standard multitask learning method” referring to? How are the multitask methods adapted for the few-shot setting? \n\n4. Characteristics of the proposed meta-VQA datasets are missing: how many choices are there on average for each question? How do you split the training and test tasks (e.g. did you check if semantically identical questions exist between training and testing)? How big is the support set in the few-shot setting?\n\n5. It is concerning that the performance of the proposed methods do not significantly out-perform baselines on the only zero-shot dataset evaluated, and is even out-performed on the few-shot task. I would suspect that if baselines designed for ZSL are included, the comparisons would appear even worse. Can you provide any hypothesis as to under what circumstances will the proposed method exhibit quantifiably better performance than existing methods?\n\n6. Diffusion models, particularly when imbued with sufficient capacity, are known to overfit rapidly to small datasets, where latents are all mapped to known training examples. Have you validated the generative capabilities of your diffusion model? Is it really working as intended?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0xCLpxYrG0y", "reviewer": "Reviewer_5YoJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a hypernetwork that generates the adapted weights for each task and runs the experiment on a adapted dataset called “meta VQA” for testing zero-shot and few-shot performance. The results suggest that the added benefit of the proposed module is tiny.", "summary_of_the_review": "In summary, the experimental results show that it is probably unnecessary to introduce so many model complications. The newly introduced dataset is a highlight, but the paper could have been evaluated on existing few-shot / zero-shot datasets too. Therefore, I think it is definitely below the bar of acceptance.\n\n---\nI acknowledge that I have read the rebuttal. I thank the authors for providing additional results and raise my score from 1 to 3. Overall, the paper lacks a great amount of clarity in terms of model design and dataset evaluation, and the model design seems intimidating while the added benefits are small.", "strength_and_weaknesses": "Strengths:\n- The authors made a huge effort designing many model components and have spent time writing a paper that describes their idea.\n- The authors also contributed a new evaluation dataset based on existing data.\n\nWeaknesses:\n- Complexity vs. significance. Despite a plethora of new components proposed, e.g. VAE, HyperClip, Guidance, Hyper latent diffusion etc, the method does better than baselines only by 0.5% in zero-shot meta-VQA, which is surprisingly marginal. I suggest that the authors find a minimal set of modifications with the maximal difference of results.\n- Although it is good to see a more realistic dataset on VQA proposed, the paper could also be evaluated on traditional zero-shot and few-shot tasks.\n- The paper lacks a thorough description of how the meta-VQA dataset is created. And also there is no visualization of the examples, nor is there qualitative analyses. I also checked Appendix A.3 and it is still very unclear. A few more examples with model predicted outputs/groundtruth will be very helpful to understand the setup.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject