Meta-Learning via Classifier(-free) Diffusion Guidance
Authors: Elvis Nava
Seijin Kobayashi
Yifei Yin
Robert K. Katzschmann
Benjamin F Grewe
Venue: Accepted by TMLR ()

Abstract: We introduce meta-learning algorithms that perform zero-shot weight-space adaptation of neural network models to unseen tasks. Our methods repurpose the popular generative image synthesis techniques of natural language guidance and diffusion models to generate neural network weights adapted for tasks. We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second "guidance" model that, given a natural language task description, traverses the hypernetwork latent space to find high-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: "HyperCLIP"-based classifier guidance and a conditional Hypernetwork Latent Diffusion Model ("HyperLDM"), which we show to benefit from the classifier-free guidance technique common in image generation. Finally, we demonstrate that our approaches outperform existing multi-task and meta-learning methods in a series of zero-shot learning experiments on our Meta-VQA dataset.

Published in Transactions on Machine Learning Research (08/2023)
Meta-Learning via (lassifier(-free) Diffusion Guidance
Elvis Nava"
enavaGelha.ch ETH A1 Center & INI & Soft Robolics Lab, ETH Zurich
Seijin Kobayashi"
seijink@etha.ch Dept. of Computer Seience, ETH Zurich
Yifei Yin
vftyin Dept. of Computer Science, ETH Zurich
Robert K. Katzschmann
rkköethz.ch Soft Robolics Lab, D-MAVT, ETH Zurich
Benjamin F. Grewe
bgrewe@ktha.cc Institute of Neuroinformatics, University of Zurich & ETH Zurich
Reviewed on OpenReview: https: https: //https: // openreview. net/ pet/forum? id= 16rV)57A3w
Abstract
We introduce meta-learning algorithms that perform zero-shot weight-space adaptation of neural network models to unseen tasks. Our methods repurpose the popular generalive image synthesis techniques of natural language guidance and diffusion models to generate neural network weights adapted for tasks. We first train an unconditional generative hypernetwork model to produce neural network weights; then we train a second "guidance" model that, given a natural language lask description, traverses the hypernetwork latent space find high-performance task-adapted weights in a zero-shot manner. We explore two alternative approaches for latent space guidance: "HyperCLIP"" classifier guidance and conditional Hypernetwork Latent Diffusion Model ("HyperLDM"", which show to benefit from the classificr-free guidance technique common in image generation. Finally, we demonstrate that our approaches outperform existing multi-task and meta-learning methods in a series of zero-shot learning experiments on our Meta- VQA dataset.
1 Introduction
State-of-thee-rt machine learning algorithms often lack the ability generalize in a sample efficient manner to unscen tasks. In contrast, humans show remarkable capabilitics in loveraging previous knowledge for learning new task from just few examples. Often, not even single example is needed, all relevant task information can be conveyed in the form of natural language instructions. Indeed, humans can solve novel tasks when prompted by a variety of different interaction modalities such as visual task observations or natural language prompts. In this work, we present new meta-learning techniques that allow models to perform a similar kind of multi-modal task inference and adaptation in the weight space of neural network models. In particular, we present two different approaches (HyperCLIP guidance and HyperLDM) that utilize natural language task descriptors for zero-shot task adaptation.
The development of deep learning models that "Learn to learn" is the focus of the field of meta- learning., Meta-learning can be defined as bi-level optimization problem, trend stemming from the success of Model-Agnode Mela-Learning (Finn et al., 2017, MAML): an outer loop meta-model is trained with the goal of improving the performance of a base model when fine-tuned on a variety of related tasks. MAML was specifically introduced as a gradient-basod method to find a network initialization with high few-shot performance over an entire set of tasks. Recent progress in large-scale transformer networks is however challenging this explicit meta-learning framework grounded in optimization models trained on huge, rich, and diverse data sets have been shown to possess surprisingly good few-shot
model weights. Large
"Equal Contribution

---

Published in Transactions on Machine Learning Research (08/2023)
D
x
Y,
Figure Given a task T, and a network initialization W with few-shot task data D; one can use traditional gradient descent to perform task adaptation and obtain fine-tuned weights W This adaptation requires that at every step t the gradi- ent Lw,(PP) is computed. Our methods (green) instcad do not require few-shot data Di, but use natural language descrip- tors to generate a surrogate adaptation towards W in zero-shot manner. In summarize this adaptation step as <WW, 4),, which depends only on 4 and not on Di, which may be unavailable.
Gradient Descent
He sitting on the refrigerator
x
X
f
V. ((D)
w
W'
y
y
f
e(W,t)
f
w'
W
the Figure,
y
Guided
t: Where is the child sitting?
Diffusion
learning capabilities through in-context learning (Brown al., 2020). Moreover, large-scale pre-training and fine-tuning often outperforms explicit meta-learning procedures (Mandi et al., 2022). Brown et al. (2020) dispense with the bi-level optimization formulation and use the word "Meta-Learning" to generally describe structure, and use the words "vero-shot", "one-shot",
problem settings with an "few-shot" depending on how many demonstrations are provided in-context at inference time.
These developments in transformer networks prompted us to develop alternative methods for meta-learning in weight-space which natively benefit from rich and multi-modal data like in-context learning. Inspired by recent advances in conditional image generation (Ramesh al., 2022; Rombach al., 2022), we rocast meta-learning as multi-modal generative modeling problem such that, given lask, few-shot data and considered equivalent conditioning modalities for adaptation (Figure 1). What we show is that popular techniques for the image domain, such as CLiP-basod guidance (Gal et al., 2021; Patashnik et al., 2021). denoising diffusion models (Ho et al., 2020), and classifier-fre free guidance (Dhariwal & Nichol, 2021; Ho & Salimans, 2021; Nichol et al., 2022) can be repurposed for the meta-learning setting to generate adapted neural network weights. Using multi-step adaptation instead of traditionally conditioning the model on the natural language task information allows our models achieve higher performance on each task by breaking down computations into multiple steps.
natural language descriptions
We approach the generation of neural network weights in two separate phases. In the unconditional pre-training phase, we train generative hypernetwork (Ha et al., 2016; Schürholt et al., 2022) to map from its latent space to the weight space of a base model (Figure 2.A). In the guidance phase, we learn models that can be used to traverse the hypernetwork latent space and find zero-shot adapted weights with high performance on our task (Figure 2.B and 2.C). Our methods can thus benefit from large scale data through the pre-training phase, even when natural language descriptions are not available for all tasks.
We summarise contributions as follows:
1) We roduce HyperCLIP, contrastive learning method equivalent to Contrastive Language- Pre-training (CLIP) (Radford et al., 2021), producing CLIP embeddings of fine-tuned neural network weights. Using HyperCLIP as a guidance model then allows us to find task-adapted networks in the latent space of a hypernetwork model (Figure 2.B).
2) We introduce Hypernetwork Latent Diffusion Models (HyperLDM) as a costlier but more powerful alter- native to pure HyperCLIP guidance to find task-adapted networks within the latent space of hypernetwork model (Figure2.C) We show how combining this approach with classifier-free free guidance (Ho & Salimans, 2021) improves the performance of generated base networks.
1 See the footnate page of Brown et (2020).
2

---

Published in Transactions on Machine Learning Research (08/2023)
Figure 2: Schematic of the three main components of our proposed meta-learning approach. A. A task- unconditional variational autoencoder (VAE) mod- cls the latent space of adapted weights W for the network f trained data (z), Its generator hy- pernetwork h producing the weights (highlighted in gray) can be re-used in the task-conditional setting with our guidance techniques. B. Our HyperCLIP encoder CLIP# is cont rastively trained map net- work weights W the space of CLIP embeddings e,- Then, given new task with descriptor 4, can CLIP guidance to find VAE latent vector 2, with embedding e!²T) that has high cosine similarity a given task embedding e!?) C. Alternatively, our Hypernetwork Latent Diffusion Model (HyperLDM) learns, conditional on the task embedding e, to iter- atively denoise a VAE latent vector 2/ ,..,, 2 over T itcrations.
A
x
Hypernet
f
W
d
h
W
VAE
y,
B
HyperCLIP Guidance
Cosine sim.
W
W
CLIP
CLIP
Task
W
H
Desc.
C
Task
CLIP
Desc.
LDM
h
W
HyperLDM
Y
3) We demonstrate the usefulness of our methods on Meta- VQA, our modification of the VQA v2.0 dataset (Goyal et al., 2017) built to reflect the multi-task setting with natural language task descriptors. We show how our guidance methods outperform traditional multi-task and meta-learning techniques for zero-shol learning on this dataset.
2 Meta-Learning with Multi-Modal Task Embeddings
The setting we investigate is similar to the classic leaaanln framework, where we operate within a distribution of tasks T ~ p(7), each associated with loss function Cy, Using a set of training tasks drawn from this distribution, our goal to train model such that it generally performs well on new unseen lasks drawn from p(T).
2.1 Background Model-Agnostic Meta-Learning
In Zintgraf et al. (2019)'s version of MAMT., model g composed of context parameters e that are adapted to individual tasks, and shared parameters 0 that meta-trained and shared across tasks. MAML and its variants focus on the few-shot setting, which aims to learn an initialization for these parameters such that the model g( 0, Φ) generalizes well on new tasks after fine-tuning á on a few data points from that task. To train such a model, we sample training data D; from each task T, and split it into a support set Dj and a query set D.. The MAML objective aims to optimize the validation score evaluated on the query set when fine-tuning & on the support set:
min E 0.0 T
CT. (T, (L..))DD
Σ
(1)
D
where Ar, is some differentiable algorithm, typically implementing a variant of few-step gradient descent on the loss computed on the support set, e.g., in the case of one-step gradient descent:
(T, 0) = " "
Σ
VoLr,o(z',0,00 0.0).
(2)
with some learning rate 77. The objective from Eq. is itself solved with gradient descent. This is done by iteratively optimizing the parameters @ the inner loop on the support set of sampled task, and updating and the initialization of with their gradient with respect to the training process of the entire inner loop, averaged over batches of tasks.
3

---

Published in Transactions on Machine Learning Research (08/2023)
2.2 Natural Language Task Embeddings
At lest time, MAML-based mela-learning requires few-shot data Dj from lest lasks to adapt its unconditioned network parameters through gradient descent. In contrast, in this work, to perform zero-shot task adaptation, we utilize an additional high-level context embedding e, for cach task Ti. In practice, such embeddings can come from a natural language description t, of the task, which can be encoded into a task embedding using pre-trained language models.
A simple baseline for incorporating task embeddings into model during training is by augmenting the input of the network, concatenating such imput with the task embedding during the forward pass, using custom conditioning layers such as FiLM (Perez et al., 2017). We instead consider the use of hypernetworks (Ha et al., 2016; Zhao et al., 2020), a network that generates the weights of another network given a conditioning input.
Hypernetworks introduce multiplicative interactions between neural network model weights, similar to how the attention mechanism in transformer models allows for the efficient mixing and propagation of information across self-attention layers. In fact, transformers can be viewed as composition of small hypernet works. The key difference between adopting transformers for in-context learning and our approach is that we deliberately fix the architecture of our small base model f, and use hypernetwork to sample such models. Effectively, we decouple the multiplicative hypernetwork mechanism from the downstream network specialized for a lask.
2.3 Meta-Learning and Hypernetworks
Given neural network f(., W) parametrized by a weight vector W, we reparametrize the model by introducing a hypernetwork h. The hypernetwork h is parametrized by 0, and generates weights h(z, 0) = W from an embedding z, The overall model then defined as ((,,0))))) 0)). In the multi-task setting, one can use a coali-oondiiooned" hypernetwork, in which the input embedding z directly depends on the task T (e.g. 2 = =;). In this work, we will also consider "unconditional" hypernetworks, trained as generative models (see Section 3), with input embeddings z that don't depend on the task, but may for example be normally distributed.
Before introducing our new zero-shot techniques, construct hypernetwork- baseline by rewriling the MAML objective (Eq. 1) with respect to the hypernetwork weight 0 as
min E T,
1 D
Σ
(3)
(aN))D
Forcing (D) es, we recover the simple multi-task objective of a task-conditioned hypernetwork optimizing for zero-shot performance, laking e, directly as input. When AT, instead the gradient descent algorithm on 2, the objective corresponds to variant of MAML, optimizing the few-shot performance of h when only adapting the embedding in the inner loop, initialized at 2. For more details related to the basclines, see Appendix A.6.
3 Hypernetworks as Generative Models
A rich literature exists on hypernetworks interpreted as generative models of base network weights (see Section 7). Our work builds upon this interpretation to adapt multi-modal generative modeling techniques to the meta-learning domain.
In generative modeling, we aim to learn the distribution p(z) over high dimensional data domain X,, such images, given samples from the distribution. To do so, resort to techniques such variational inference, adversarial training, or diffusion models. Our meta-learning sctting can analogously be framed as modeling a distribution ion of diverse high-dinensional base net work weights W. In the Bayesian setting, this distribution is made explicit as we seek to model the posterior p(W)D) D) given data D. However, the framework is still useful even when no explicit posterior distribution is assumed, as it is the case for deep ensembles. In the present work, we indeed avoid explicit Bayesian inference: for each training task T, we fine-tune the base model (x. W) y on it, and use the resulting W; as sample to train generative model of network weights.
4

---

Published in Transactions on Machine Learning Research (08/2023)
The fundamental building block of our unconditional generative model is the hypernetwork h(2, W that we can train in two ways: 1. HVAE: We define a Hypernetwork VAE (iguue2.A), which, given samples of fine-tuned base network weights W,, learns a low-dimensional normally distributed latent representation 2; The encoder d(W,,,) () = (Mz,, ) with parameters w maps base network weights to means and variances used to sample a latent. vector z,, while the decoder (or generator) is a classic hypernetwork h(z,,), = W; which reconstructs the network weights from the latent vector (See Appendix A.7.1). This VAE setup is analogous to that proposed in recent work on hepper-represeenattions (Schürholt et al., 2022). 2. HNet: Using MAML, we learn both an initialization embedding z and hypernetwork weights 9 such that, when fine-tuning only the embedding on each task Ti, we obtain high-performing base networks with weights W; = h(z;, 0). Concretely, we optimize 0 and the initialization of = following the objective in Eq. 3 (see Section 2.3).
Up to this point, we trained an unconditional hypernetwork generative model of neural network weights, comprising the unconditional pre-training phase of our meta-learning approach. This gives us a powerful generator h(2,6) = W, which maps from its latent space to the weight space of our base network. In the next Section, we investigate how to then perform lask-conditional guidance within this latent space, finding adapted latent embeddings 2 for our test tasks in zero-shot manner.
4 HyperCLIP: Training a CLIP Encoder for the "Maaee-Parameters Modality"
The first of our two meta-learning guidance techniques, HyperCLIP Guidance, consists of these steps:
1. We train an unconditional generative hypernetwork h(2, 0) = W over family of training tasks (as in Section
2. We train a HyperCLIP encoder, mapping fine-tuned neural net weights W, for tasks To to multi-modal CLIP embeddings e,
(H)
3. Given new task Tú we use HyperCLIP guidance to guide the exploration of the hypernetwork latent space, in order to find base model weights W; with high zero-shot performance for the task.
To define HyperCLIP guidance, we borrow from the field of multi-modal contrastive learning. More spocifically, we build on top of Contrastive Language-Image Pre-training (CLIP) (Radford et al., 2021), a popular method for joint learning of language and image embeddings with applications to zero-shot and few-shot classification.
In the original CLIP formulation, separate text and image encoders are trained such that, given bi-modal sample (ar,, 44) of an image and its corresponding language caption, their representations (CI.IP, = o/ and CLIP (4))44)))) = (P") are aligned across modalities. Specifically, the formulation maximizes the cosine similarity (T) (T) /lle, (2) (Ile, for pair-wise matches (i () and minimizes the cosine similarity for non-matches (). Beyond the original language-image setting, the CLIP approach can be easily adapted to include additional modalities, aligning the representation of more than two encoders a time. Existing works such AudioCLIP (Guzhov et al., 2022) demonstrate the possibility of training encoder for additional modality such as audio on the side of the pre-trained frozen CLIP language-image encoders.
4.1 Contrastive Learning on Neural Network Weights
In our work, we consider multi-modal representations of meta-learning tasks Tr A descriptor of task may come from the language modality (4), but potentially also from image, video, audio modalities. When we fine-tune a base machine lcarning model f(x, W;) y for task T; we then also consider the fine-tuned base model weights W; as being part of an alternative model--aaramelers modalily that describes task T.. Fine-tuned network weights from the other multi-modal descriptions of T. We thus define our new HyperCLIP encoder CLIPH(W.) = c (H) taking finc-tuned neural network weights W, as input, and outputting a CLIP embodding (!))) optimized for high cosine similarity with the CLIP embedding for the textual (or image, audio, etc.) descriptor of the task. 3 and Algorithm 1 illustrate the approach.
modality can then be paired in contrastive learning with the
5

---

Published in Transactions on Machine Learning Research (08/2023)
B
Who is wearing glasses?
How the weather?
CLIP
Who is wearing glasses?
CLIP
T ,T,
TN
Where is the child sitting?
T
T
x
H
C
x
T,
2 " T
f
CLIP
max.
f
CLIP
H,
max.
W
H
H
Figure A. Our HyperCLIP encoder CLIP h is contrastively trained to map neural network weights W the latent space of a pre-trained language encoder CLIPr, which we use to embed the natural language questions associated to the tasks. B. To perform task inference given an already fine-tuned network one can encode all candidate task questions using the language CLIP encoder. C. We encode the fine-tuned network weights with HyperCLIP, and finally infer the correct task with a softmax operation over cosine similarities between HyperCLII and language CLIP embeddings.
Algorithm HyperCLIP Training
sample batch of lasks Ti=i....N with loss functions LT,, training data Dirain and text define two N-sized arrays of d-dimensional embeddings T E RNxd and H e Nxx for i 1, .....N do
T CLIP+(() [[CLIP((4]|
Fine-tune W; minw Egr D(r' L.. (T((
Hi = (W,) |[CIPP (W;)l
end for loss (feress-eutropy' (TH + ecross-entropy (HTT)) / Update weights of CLIP (()) using Vloss
4.2 Guide Meta-Learning
On their own, CLIP encoders are not capable data generation. Recent popular image synthesis techniques, however, use CL.P encoders or other classifiers to guide generation from pre-trained unconditional generative models. Classifier guidance or CLIP guidance (Gal et al., 2021; Patashník et al., 2021) use gradients with respect to classifier or CLIP encoder traverse a generative model's latent space.
In this work, we introduce HyperCLIP guidance, the first algorithm for classifier guidance in the meta- learning setting (Figure 2.B). Given previously unscen validation task T; and an unconditional generative hypernetwork work model h(2,0) 0) = W, we use a trained HyperCLIP encoder CLIP= = e(H) to guide the exploration of the hypernetwork's latent space and find a set of base weights W, with high zero-shot performance for T: Spccifically, as long as we are given starting hypernetwork latent vector 20 and a textual description 4 of the (ask, we can update 20 with gradient descent over the guidance loss
(x(2) CLIP (h(2, ()) CLIPT(t))
|[CLP, (h(z, 8)) |||[LIPr (t+)l Allz -2
(4)
and then run the optimized latent vectors & through the generative hypernetwork find adapted zero-shot base network weights h(,, 0) W. that perform well for the task.

---

Published in Transactions on Machine Learning Research (08/2023)
5 HyperLDM: Taskiconditional Diffusion of Hypernetwork Latents
Due to rapid innovation in the image synthesis community, simple CLIP guidance has been largely overcome in favor of applying classifier guidance and classifier- free guidance during the sampling process of a Diffusion Model (Dhariwal & Nichol, 2021; Ho Salimans, 2021; Kim et al., 2022; Crowson, 2022; Nichol al., 2022; Rombach al., 2022). To investigate whether these advances also apply our meta-learning setting, we introduce HyperLDM, diffusion-based technique alternative to the previously introduced HyperCLIP guidance.
In summary, our HyperLDM technique for network parameter generation consists of the following steps:
We train an unconditional generative hypernetwork h(z, 0) W over family of training tasks (as in Section
2. We train a conditional HyperLDM model, able to sample latent vectors & for high-performing base model neural networks, conditioned on the task embedding e, for a task Tí.
3. To further improve the generative quality of our HyperLDM model, we use Classificr-free Guidance during conditional sampling.
5.1 (Latent) Diffusion Models
Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho et al., 2020, DDPM) are powerful class of generative models designed to learn a data distribution p(z). They do so by learning the inverse of a forward diffusion process in which samples dº of the data distribution are slowly corrupted with additive Gaussian noise over T steps with a variance schodule BI,..., Br, resulting in the Markov Chain
q(x"|x=' (a=(x==1== = (N"";; v1 B,x=11 3x11)
(5)
T
g(x I: 220)
II g( ((a)(==11)
(6)
A property of such a process is that we can directly sample each intermediate step from x² as x² Vajz" + - given e ~ IN,,I), a = 1 3r and ã; = II'm ay. Then, to learn the reverse process Po (x")) we parametrize the tinestep-dependent noise function (x () with a neural network and learn it by optimizing a simplified version of the variational lower bound on p(x:)
(DM( Eg,...(.. [lle fle--en [|le-ep(x' 10)1)
Sampling from the reverse process then be done with
Be VI- -
(n 1) + o.S,
x
(8)
Vai
with & ~~ (0, I) and chosen between Br and B./VI- 11---.. ã. Sampling from the learned diffusion model can be seen as an analog to Langevin Dynamics, a connection explicitly made in works exploring the diffusion technique from the "score matching" perspective (Song & Ermon, 2019; Song et al., 2020).
In our meta-learning selting, we aim to train a diffusion model which generates adapted zero-shot base network weights W; that perform well for task T. Thus, our diffusion model has to be conditional on a task embedding ex. Moreover, in order to speed up training and leverage our previously trained generative hypernetwork f), we define the diffusion process latent vectors instead of doing so in weight space, emulating the Latent Diffusion technique from Rombach et al. (2022).
We then propose a Hypernetwork Latent Diffusson Model (HyperLDM) which learns to sample from the conditional distribution of fine-tuned latent vectors pe="le,) given a language CLIP embedding corresponding to the task. The HyperL.DM neural network fits the noise function cs ez), and learned by optimizing the reweighted variational lower bound, which in this setting
LLDM (v²) a(W.) [[le-ep(e 1,)))
(9)
7

---

Published in Transactions on Machine Learning Research (08/2023)
5.2 Classifier- Guidance for Meta-Learning
To improve the quality of sampled networks, the classifier guidance technique presented in Section 4.2 can be also combined together with diffusion models. The gradient of an auxiliary classifier (or CLIP encoder) can be added during sampling to induce effect similar to GAN truncation (Brock et al., 2018), producing samples that are less diverse but of higher quality.
The classifier-free guidance technique (Ho & Salimans, 2021; Nichol et al., 2022) allows us to leverage a conaitional diffusion model to obtain the same effect as above, without the auxiliary classifier. To do so, we train the conditional network cx'2", e4) also model the unconditional case eg (2', One way of doing this is with conditioning dropout, simply dropping the conditional input 6, for certain percentage of training samples, and inputting zeros instead. We can then sample at each diffusion iteration with
= 11-1100 (w
+ rew(z',t,
(10)
For 7 = 0, this recovers the unconditional diffusion model, while for = il recovers the standard task- conditional model. For 7 > 1, we instead obtain the classifier-free free guidance effect, which we show results in the sampling of latent vectors & corrosponding to higher-performing task-conditional network weights h(a;, v) W. We point to more in-depth discussion classifier-free guidance and its connection to score matching in Appendix A.
6 Experimental Setup and Results
In this section, we demonstrate the competiliveness of our two approaches in wero-shot image classification experiments against a series of traditional meta-learning techniques. Throughout our experiments, we fix the choice of base network model f to CLIP-Adapter model (see Appendix A.2), only varying the meta-learuing techniques employed to obtain adapted base model weights. The ELIptAdapter base model makes use of pre-trained CLIP encoders obtain high base performance on image classification with textual labels while maintaining relatively small trainable parameter count. It should not be confused with the usage of CLIP encoders to produce lask embeddings, or to train HyperCLIP, all of which happens at the mela-level. Our base model f effectively performs classification in the same way that CLIP (Radford et al,, 2021) does, by using natural language labels as opposed to onc-hot vector for classes, then encoding both image and answers with CLIP encoders, and scoring each answer using a dot product between its embedding and the image embedding.
6.1 The Meta- VQA Dataset
To evaluate the performance of our methods, we utilize a dataset that reflects the setting of meta-learning with multi-modal task descriptors. Existing meta-learning benchmarks such as Minilmagenet (Ravi & Larochelle, 2016) or CIFAR-FS (Bertinetto et al., 2018) are unsuitable, as they are built for the traditional few-shot learning sctting, in which the task T; is not associated with task descriptors but is meant to be inferred through exposure to the support set Dj- We thus introduce our own Meta-VQA dataset, modification The dataset is composed of
of the VQA v2.0 dataset (Goyal et al., 2017) for Sesuul-QuetioonAnssweee training and test tasks Tr, cach associated with a natural language question f; and mini image classification datasel wz)),;) E D,. We refer to Appendix A.5 for a more in-depth discussion, and to Figure 4 for an illustrative example of a task from the dataset.
6.2 Zero-Shot Task Adaptation Experiments
In Table we show how our methods compare to a series of baselines when tested on the Meta-VQA dataset in the zero-shot setting. For each training task T., the algorithms are given access to the full support and query sets D,, DJ, together with the question (task descriptor) t,, At test time, in the zero-shot setting, only the task descriptors t, are given, and the algorithms are tasked with prodicting the correct labels of images in the query set D.-
8

---

Published in Transactions on Machine Learning Research (08/2023)
Task Question (t)): Where is the child sitting? Task Data (D,):
Figure 4: Example classifica- lion task from Meta- VQA, adapted from VQA v2 (Goyal et al., 2017). A task Ti is sociated to single question and multiple image-answer tuples (z),
X
y',
x
y'2
He is sitting on the refrigerator
He is sitting in mom's arss
Table 1: Zero-shot accuracy (mean + s.d.) averaged over Meta-VQA test tasks. Results should be interpreted relative to performance ceiling of 60.24% obtainable when task data is available (few-shot learning), and with our fixed choice of base model (see Appendix A.8). The columns separate the setting in which only half of task cerrripoorsqquettinin are given (50% Q.), and that in which all the task descriptors are given (100% Q-). (* ours)
Method
Zero-shot (50% Q.)
Zero-Shot (100% Q-)
CLIP as Base Model
44.99
Uncond. Multitask
53.75 (#: 0.36)
Uncond. MNet-Mi MAML
53.04 (# 0.69)
Uncond. MNct-FOMAMM
53.04 (#: 0.42)
Uncond. HNel-MAML
53.37
(d: 0.29)
Cond. Multitask
51.68 (+ 0.33)
54.12 (+ 0.80)
Cond. Multitask FILM
51.60 (+ 0.56)
53.84 (#: 0.61)
Cond. HNet-MAML
51.54 (+: 0.63)
53.02 (#: 0.20)
HNet + HyрerCLIP guidance
53.51 (+ 0.22)
53.98 (+ 0.54)
HVAE HyperCLIP guidance
53.82 (#: 0.07)
53.91 (#: 0.08)
HNet HyperLDM y =
53.66 (# 0.25)
54.06 (# 0.21)
HNet HyperLDM 7 = 1.5
54.08 (+: 0.11)
54.30 (#: 0.27)
HVAE HyperLDMM =1
54.72 (+: 0.23)
55.03 (+ 0.32)
* HVAE -
HyperLDM =15
54.84 (#: 0.24)
55.10 (#: 0.08)
In addition, we also simulate a setting in which we possess larger "unconditional" pre-training dataset. Our two-phased approach, which separates generative model pre-training and guidance, benefits from unconditional data: tasks without language descriptors can still be used learn the unconditional HNet/HVAI model. To test this, we conduct additional runs in which we train our model while only keeping fraction of task descriptors from the Meta-VQA dataset.
Using the original CLIP for zero-shot image classification (CLIP as Base Model) provides 44.99% floor for performance on Meta-VQA All other techniques will use CLIP-Adapter as the base model, as previously mentioned. We also obtained an approximate 60. 24%24% performance ceiling from the best method in the few-shot sctting, which models have also access a data support set D for every test task (scc Appendix A.8). Our zero-shot techniques cannot surpass this ceiling while keeping the choice of base model fixed. The zero-shot scores should then be judged within range between 44.99% and 60.24% accuracy.
We then benchmark several unconditional and conditional methods, with only conditional methods having access to language lask descriptors. We apply MAML and its first-order variant FOMAML (Nichol et al., 2018) directly to the base network (MNet-MAML, hypernetwork (Uncond. HNet-MAML, as in Section 2.3) and conditional one (Cond. HNet-MAML) We also benchmark against standard multitask learning (Uncond. Multitask, Cond. Multitask) on the base model without hypernetworks, and conditional multitask learning with the classic FiLM layer (Perez
L.Ne-FFOMALL) and to both an unconditional
9

---

Published in Transactions on Machine Learning Research (08/2023)
A
55
Query Set Accuracy
54.4
Query Set Accuracy
544
53
54.2
52
54.0
51
53.8
Uncond. Multiask
50
Cond. Multitask
53.6
HVAE + HyperLDM y
Uncond. Multitask
49-
HyperLDM y=
HVAE HyperLDM
53.4
0.0
0.5
1.0
1.5
2.0
2.5
48
10
100
Classifier-free guidance strength
% of available task descriptors
Figure 5: A. Zero-shot performance of HyperLDM (mean + s.d.) over different classifier-fre guidance parameters y. For y 0 sample from unconditional latent diffusion model. For y 1 sample with classic conditioning. For > 1, are in the classifier-free guidance regime. B. Zero-shot performance of HyperI.DM (mean + s.d.) against baselines in the setting where only a fraction of natural language task labels are given.
et al., 2017) (Cond. Multitask FiLM). Note that the multitask approach, at least this setting, leads better zero-shot models than MAML, which instead optimizes for few-shot performance, We refer Appendix A.2 and A.6 for more details on each algorithm.
We then test out two approaches, HyperCLIP guidance and HyperLDM, when trained on top of either a hypernetwork or a VAE generator (Section see also Appendix A.2 and A.7 for more detail). HyperCLIP guidance allows for faster sampling than HyperLDM but is generally less performant. Still, HyperCLIP guidance performs on par with or slighly improves upon all other zero-shot baselines except for Cond. Multitask. The best-performing model for the zero-shot setting HVAE HyperLDM, and specifically for classifier-free guidance with 7 = 1.5. As illustrated in Figure5.A, further show the effectiveness of the classifier-free guidance technique, candidate parameters to find that the optimum occurs for 7 > 1. As shown in Vigure5.13 when training our model while only keeping 50% or 10% of task descriptors, traditional Cond. Multitask learning heavily overfits, while HyperLDM is almost not alfected due to its two-phased training regime based on an unconditional VAE. The gap between the multitask baseline and our HyperLDM technique is particularly striking in this setting.
run different experiment sweeping over several
7 Related Work
Hypernetworks By introiigg multiplicative interactions within neural networks (Jayakumar al., 2019), hypernetworks (Ha al., 2016) have been shown to allow the modeling of diverse target network weights in, e.g,, continual learning, even in the compressive regime (von Oswald et al., 2021a; 2020) without loss of performance. For a given supervised problem, hypernetworks have been used to model the complex Bayesian posterior of the weights in conjunction with variational inference (Henning et al., 2018; Krueger et al., 2018). Similar approaches have been used for continual learning (Henning et al., 2021). Another vein of work consists using hypernetworks to distill ensembles of diverse networks (Wang et al., 2018; Ratzlaff & Fuxin, 2020; von Oswald et al., 2021a). Recent work also explored the properties of hypernetworks autocncoder generative models of network weights (Schürholt et al., 2022).
Meta Learning In the context of meta-learning, hypernetworks have been successfully used in combination with popular gradient-base meta-learning methods (Finn al., 2017: Zintgraf et al., 2019; Zhao al., 2020; Flennerhag et al., 2020). More generally, related works have shown the usefulness of learning low
10

---

Published in Transactions on Machine Learning Research (08/2023)
dimensional manifold in which to perform task-specific gradient-based adaptation at meta test-time (Rusu et al., 2018; von Oswald et al., 2021b; Lee & Choi, 2018), instead of directly adapting in weight space. Recent works bypass the formal bi-level formulation of meta-learning (Brown et al., 2020) by, e.g,, using transformers to directly map the few-shot examples to the weights of the target network (Zhmoginov et al., 2022).
Generative Modeling and Classifier(-free) Guidance A plethora of techniques have been proposed for the generation of samples from high-dimensional domains such as images, such as Generative Adversarial Networks (Goodfellow et al., 2014; Brock et al., 2018, GANs) and Variational Autoencoders (Kingma Welling, 2014, VAEs). Denoising Diffusion Probabilistic Models (Sohl-Dickstein et al., 2015; Ho al., 2020, DDPM) overcome common issues in generative modcling using simple likelihood-based reconstruction loss for iterative denoising and have been shown to achieve seate-of-thheart results in high-resolution image generation (Dharíwal & Nichol, 2021; Rombach et al., 2022). Several techniques have been proposed for effective conditional sampling in generative and diffusion models, such as classifier/CLii guidance (Dhariwal & Nichol, 2021; Gal et al., 2021; Patashnik et al., 2021) and classifier-free guidance (Ho & Salimans, 2021; Crowson, 2022; Nichol et al., 2022). Diffusion models with classifier-free guidance have also been successfully applied in non-visual domains, such as audio generation (Kim ct al., 2022) and robotic planning (Janner al., 2022).
Zero-shot Learning There exists large literature on zero-shot learning, including both established benchmarks and well-known methods (Han et al., 2021; Su et al., 2022; Gupta et al,, 2021). While these zero-shot learning works consider the zero-shot performance on unseen class labels within single classification task, our setting considers that of the zero-shot performance where test tasks themselves are unseen, thus raising the zero-shot problem to the task level.
8 Conclusion
In this work, introduced framework that re-interprets meta-learning multi-modal generative modeling problem. Our HyperCLIP guidance and HyperLDM methods leverage this insight to generate task-adapted neural network weights in zero-shot manner given natural language instructions and constitute the first application of the CLIP guidance and classifier-free guidance techniques from image generation to the meta-learning domain. Our experiments show that our methods successfully make use of external task descriptors to produce high-performance adapted networks in the zero-shot setting.
Acknowledgments
We are grateful for funding received by the ETTH AI Center, Swiss National Science Foundation (B.F.G. CRSH15- 173721 and 315230 189251), ETH project funding (B.F.G. ETH-20 19-01), the Human Frontiors Science Program (RGY0072/2019), and Credit Suisse Asset Management.
Code Release
Our code is available at http:htwww..../..aaaaaaaaa/eaaaa
Author Contributions
Elvis Nava * Original idea, implementation, and experiments (guidance techniques and diffusion), paper writing (overall manuscript structure, generative modeling and guidance sections, experimental results section).
Seijin Kobayashi Implementation and experiments (MAML and hypernetworks, HyperCLIP guidance), paper writing (meta-learning background and related work).
Yifei Yin Creation of the Meta-VQA dataset, initial experiments on the CI.IP-dappte base model.
11

---

Published in Transactions on Machine Learning Research (08/2023)
Robert K. Katzschmann Conceptualising of the project idea with focus down-stream learning applications in robotics, project co-lead. Participating in project meetings, giving feedback and conceptually guiding the approaches, Revising the manuscript and figures.
Benjamin F. Grewe Conceptualisation of the project with a focus on using language task descriptors, senior project lead, participating in project mectings, providing conceptual input for developing the two approaches, giving feedback on the manuscripl, developing inluitive schematics for Figures and 3.
References
Luca Bertinetto, Joao F. Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. September 2018. URL httpss//pppeeerrssss.////rrtroop
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large Scale GAN Training for High Fidelity Natural Image
Synthesis. October 2018. URL http:///ww.ppoooooo......o//c//a utm_ http://www.p......oooo///oo//o.a
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Win- ter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sulskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neurol Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL sttps://proceedings. wwtp.ppps.ccppppp/////////////// / M
Abstraattttttttt. .ttml.
Katherine Crowson. v-diffusion, May 2022. URL https: http://tww...ooocccoooooo//aaaaa
original-date: 2021-12-1671::44:33...
Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthosis. In Advances in Neural Information Processing Systems, volume 34, pp. 8780-8794. Curran Associates, Inc., 2021. URL Abstraat..ttteee.......eee html..
https: //papers//pppr...../
Chelseaa Finn, Picter Abbeel, and Sergey Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1126 1135. PMLR, July 2017. URL tttt::://proceedeie pttpeeaccciggglllppppee///////ta
mlr wwt.:ess/////fimmlll..l...
ISSN: 2640-3498.
Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-Learning with Warped Gradient Descent, February 2020. URL arXiv:1909.00023 [cs, stat).
Rinon Gal, Or Patashník, Haggai Maron, Gal Chechik, and Daniel Cohen-Or. StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generators. Technical Report arXiv:2108.009: 000466, arXiv, December 2021. URL 00946. arXiv:2108.009. [cs] type: article.
http://arxiv org/
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongleng Zhang, Hongsheng Li, and Yu Qiao. CLIP-Adapter Better Vision- Models with Feature Adapters. arXio:2110.045.1 fes), October 2021. URL http://arxivv httpaaawwwvv..rgaa///aaaa 04544. arXiv: 2110.04544.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11): 149-144, 2014. ISSN 0001-0782, doi: 10.1145/3422622. URL https://doi.orgg: 10. 2rt...00/000.11//3222222222 1144//422266..
Yash Goyal, Tojas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. Making the v in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. pp. 6904-6913, 2017. URL https:
/ //opemaccess.. thecvf www.//////ttct///eeooooeo _ 20172000712.. 2017/htmm//oyall Maatmm//Gooallaaaingg...... Beckinggghe____ cche_v_CVPP 2017 1aperppperr_... html.
12

---

Published in Transactions on Machine Learning Research (08/2023)
Akshita Gupta, Sanath Narayan, Salman Khan, Fahad Shahbaz Khan, Ling Shao, and Joost van de Weijer. Generative Multi-Tabel Zero-Shot. Learning, January 2021. URL http://aaaii.oorra////eea 11606. arXiv:2101.11606 [cs] version: 2.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: Extending Clip to Image, Text and Audio. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980, May 2022. doi: 10.
ISSN: 2379-190X.
David Ha, Andrew Dai, and Quoe V. Le. HyperNetworks. Technical Report arXiv:1609.099 00900,999,,, arXiv, December 2016. URL http://arxiv orggeiiiir.....ooooooooooo hts//bb//16000 09106. 3XXiv:669.00106 [cs] type: article.
Zongyan Han, Zhenyong Fu, Shuo Chen, and Jian Yang. Contrastive Embedding for Generalized Zero-Shol Learning. In 2021 IEEE/CVE Conference Computer Vision and Pottern Recognition (CVPR), pp. 2371- 2381, Nashville, TN, USA, June 2021. IEEE. ISBN 978-1-66544-509-2 doi: 10.1000/CVPR4463332220000 URL https://ieeexploor
ieee. org/ http///mmmmtt997599999
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving Deep into Rectifiers: Surpassing Human- Level Performance on ImageNet Classification, February 2015, URL http:://arxdv.oorrabb/////e 01852.
arXiv:1502.01$8 01222 [cs].
Christian Henning, Johannes von Oswald, Joao Sacramento, Simone Carlo Surace, Jcan-Pascal Pfister, and Benjamin F. Grewe. Approximating the Predictive Distribution via Adversarially-TTaiine Trainedd Hypernetworks. In Bayesion Deep Learning Workshop, NewrIPS 2018, Montréal, Canada, December 2018. Yarin. doi: 10.5167 115167/uzh-165777 /t////hhhh6677
URL http: httt::htbbbyysiaaaaeee http::/bbaessaneeeeaaamnggg otg///10/ http1///ppers//21111 pdf..
Christian Henning, Maria R. Cervera, Francesco D' D'Angolo, Johannes von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, João Sacramento, and Benjamin F. Grewe. Posterior Meta- Reta-Replay for Contimual Learning. arXiv:2103.011 033 [es], June 2021. URL http:://rx/tt wwwaariiv..ooovaaaee harg/abb/200 waa.////...//./../
arXiv: 2103.01133.
Yusuke Hirota, Yuta Nakashima, and Noa Garcia. Gender and Racial Bias in Visual Question Answering Datasets. In 2022 ACM Conference on Fairness, Accountability, and Transporency, pp. 1280 1292, June 2022. doi: 10. 11555/5331166.53330
URL http:://trii http://ww..rggb///////////// 08148. arXiv:2205.08145 [cs].
Jonathan Ho and Tim Salimans. Classifier-Free Diffusion Guidance. November 2021. URL https: //openreview.c wwwt/eeeeweiddeeeeeeee httpr//ewwgggaaaaaoroo.a.o.aaaar
wwwrcesssueceuceeeeeeeee email&
utm utm_mediee zpRGx3KjCoNR3kavee.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. In Advances in Newral
Information Processing Systems, volume 33, pp. 6840-6851. Curran Associates, Inc., 2020. URL https:
/ //proceedings.nuure iaciiingsnneeeeen http://tw..aaaae///.aa////a//aaa
Jie Hu, Li Shen, and Gang Sun. Squueze-and-Fxciitttion
Networks. pp. 1132-7141, 2018. Squeeze Exciitiitatinn
URL https://openaccess. thecvf.
Networks_ Netxorks_CVPR_ 2018_2018_________0101111111 1aai11_ppee____ . html.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural Tangent Kernel: Convergence and Generalization in Neural Networks, February 2020. URL
wttp://wwww//bb///....oooo 07572. 180.077772 [cs,
math, stat].
Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with Diffusson for Flexible Behavior Synthesis. Technical Report #xXiv22205.0999 arXiv, May 2022. URL http://arxiv..r wwwpii...r///bs/////e/ /bs// 2205. 09991. arXiv:2205.0999
[cs] type: article.
Siddhant M. Jayakumar, Wojeiech M. Czarnocki, Jacob Menick, Jonathan Schwarz, Jack Rae, Simon Osindero, Yee Whye Teh, Tim Harley, and Razvan Pascanu. Multiplicative Interactions and Where to Find Them. September 2019. URL https: http:p///ppppeeeeee..eee///errrs wwt.f//oormm//drr..n.......
13

---

Published in Transactions on Machine Learning Research (08/2023)
Heeseung Kim, Sungwon Kim, and Sungroh Yoon. Guided-TTS: A Diffusion Model for Text-to-Speedt via Classifier Guidance. In Proceedings of the 39th International Conference on Machine Learning, pp- 11119 11133. PMLR, June 2022. URL https: https://pproceedngg..ll.. http:/ddiii........pre////////// pwwpss/v62//kim2222.. html. ISSN: 2640-3498.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization, January 2017. URL http///ai....ooaa////a//aaa org/abs/1412/ 6980. arXiv:1412.6980 [es].
Diederik P. Kingma and Max Welling. Eatoo-nnoding Variational Bayes. arXiv:1312.6114 fes, stat], May /b111/6..11111111....111 arXiv: 1312.6114.
2014. URL
David Krueger, Chin- Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian Hypernetworks, April 2018. URL http://arxia httpp////rrii.orrraa///11000 04759. arXiv:1710.04759 [cs, stat].
Yoonho Lee and Seungjin Choi. Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace, June 2018. URL http://arxiv. wwtp://////a////////// 1801. 05558. arXiv:1801.0555 [cs, stat].
Zhao Mandi, Picter Abbeel, and Stephen James. On the Effectiveness of Fino-tuning Versus Mota-reinforcement Learning. Technical Report arXiv:2206.0321 032711 arXiv, June 2022. URL arXiv:2206.0327 [cs] type: article.
http://orr/abbb//////a 03271.
Alex Nichol, Joshua Achiam, and John Schulman. On First-Order Meta-Learning Algorithms, October 2018.
URL http:://axttii httt:://ardd..rr/aab///BB/ orgg/bs///00/.../ 02999. arXiv:1803.02990 [es].
Alex Nichol, Prafulla. Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Πlya Sutskever, and Mark Chen. GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffussonn Models. Technical Report arXiv:2112.17711, arXiv, March 2022. URL hattp://aree www.aawwwwww.aaaaaeeeeeeeaoeeeee wttp/o//////....o..///b. 10741.
arXiv:2112.107A1 [cs] type: article.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. CLIP: Text-Driven Manipulation StyleGAN Imagery. pp. 2085 2094, 2021.
Style- URL
StttaahhiiikSSyyeIIr https://openaccess.. thecvf. Bttpeey__cco_222__pppeppee html.
StyleCLIP BeepleeLIP_Text-DeP Text-Driven,
Manipulation_of. _t__StyleeAA__marrr__atto Iaaaaa_Immmrrr_cccaaeaoo
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer, December 2017. URL http://arxiv, http:://aaxdv.orr/aa////0000 wttp:/a////99..07777777777 arXiv:1709.07871 [cs, stat).
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv:2103.0003 [cs], February 2021. URL http:://axixi /ttp:v//axvv...vm http:://aarvv..o//aab/////s/ /203.. 00020. arXiv: 2103.00020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Text-Conditional Image Generation with CLIP Latents. Technical Report arXiv:2204.06112,, arXiv, April 2022. URL 2204. 06125. arXiv:2204.06125 [cs] type: article.
http:///rxxe wwwaaiii.raaaaaaaaaaaaaaaaaaaaae
Ncale Ratzlaff and Li Fuxin. HyperGAN: A Generative Model for Diverse, Performant Neural Net- works. Technical Report arXiv:1901. 11058, arXiv, July 2020. URL arXiv:1901. [cs, stat] type: article.
11180./../1111118....... 11058.
Sachin Ravi and Hugo Larochelle. Optimization as Model for Few-Shot Learning. October 2016. URL https://openreviie http:///pppaaoeeeeeeeeeee/ndddee
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis With Latent Diffusion Models. pp. 10684-10695, 2022. URL https://openaccess. thecvf. watevvf.co//cee htt///coent//vPPO000/ httt///maaaaaaaatttn HigheHeeoogtteeeeeeeeeeeeeeeettt Resolution Image_ Synthesis With Latent_Diie Diffusion Models CVPR, 2022 _02e_ppper... html.
14

---

Published in Transactions on Machine Learning Research (08/2023)
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi (eds.), Medical Image Computing and Computer- Assisted Intervention MICCAI 2015, Lecture Notes in Computer Science, pp. 234-241, Cham, 2015. Springer International Publishing. ISBN 978-9-319-245744. doi: 10.1007/978---319-24455
28.
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-Learning with Latent Embedding Optimization. September 2018. URL https:
Konstantin Schürholt, Boris Knyazev, Xavier Giró-i Nieto, and Damian Borth. Hypee-Repressnlations Generative Models: Sampling Unseen Neural Network Weights, September 2022. URL http://arxiv. org/abs/2200 14733. arXiv:2209.1444 14733 [es].
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsupervised Learn- ing using Nonequilibriun Thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, pp. 2256 2265. PMLR, June 2015. URL https://prooeeedinas mlr wtw.:/ngsmm......ms./eeeeeeeee press/v37/ sobl-dickksteinl5.trt html. ISSN: 1938-7228.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. In
A Advances in Neural Informalion Processing Systems, volume 32. Curran Associales, Inc., 2019. URL https:
/
Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and
Ben Poole. Score-Based Generative Modeling through Stochastic Differential Equations. Septem- ber 2020. URL attps://ppeearrriie..eee// wtt.:///eooooo////11111111//a/a ottp://ccin.nattaaaaaaaaaaaaaaaa campaiigu-ALLPP//0ooosssssuu htt.://aaaaaaatttoooooooeoeeeeea
Hongzu Su, Jingjing Li, Zhi Chen, Lei Zhu, and Ke Lu. Distinguishing Unscen from Scen for Generalized Zero- shot Learning. In 2022 IEEE/CVP Conference Compuler Vision and Pallern Recognition (CVPR), pp. 7875-7884, New Orleans, LA, USA, June 2022. IEEE. ISBN 978-1-66546-946-3 doi: D0.1199/CVPR52688. 2022.00773. URL https://1eeexplore ieee /tg///ocmmmntt/8994499///
Johannes von Oswald, Christian Hemning, Benjamin F. Grewe, and João Sacramento. Continual learning with hypernetworks. In International Conference on Learning Representations (ICLR 2020). May 2020.
doi: 119550/rrXiv.1000 00695. URL http:// wwwaaaaivvooor... www./org//////aeeeeeoeeee aa//a/bb///0...////// 00695. arXiv:1906.006%: 196999 [cs, stat]..
Johannes von Oswald, Seijin Kobayashi, Alexander Meulemans, Christian Henning, Benjamin F. Grewe, and João Sacramento. Neural networks with late-phase weights. arXiv:2007. 12927 [cs, stat), April 2021a. URL
http://arxiv.orgg hts//bbs/2000./ 12927.
arXiv: 2007.12927.
Johannes von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. ar arXiv:2110.14400 [es], October 2021b. URL http://arxiv. http://rrxiv..og/aa///1//..... ab11110../0101111////11 14402. arXiv: 2110.14402
Kuan-Chich Wang, Paul Vicol, James Lucas, Li Gu, Roger Grosse, and Richard Zemel. Adversarial Distillation of Bayesian Neural Network Posteriors. In Proceedings of the 35th Internalional Conference on Machine Learning, pp. 5190-5199. PMLR, July 2018. URL https://proceedings mlr. Sassss/v80vaanglen html. ISSN: 2640-3498.
Dominic Zhao, Seijin Kobayashi, João Sacramento, and Johannes von Oswald. Meta-Learning via Hyper- networks. In 4th Workshop on Mela-Learning al NeurIPS 2020 (MelaTearn 2020), Virtual Conference, December 2020, IEEE, doi: (05167/uzh--0 200298. URL https: http:://wwwwzz/ zora. uzh ch/id//prrnt//200955
Andrey Zhmoginov, Mark Sandler, and Max Vladymyrov. HyperTransformer: Model Generation for Supervised and Supi-iipee Few-Shot Learning, July 2022. URL http:://axixi http://aaxxi..oo/////220000 04182. arXiv:2201.041 [cs).
15

---

Published in Transactions on Machine Learning Research (08/2023)
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast Context Adaptation via Meta-Learning June 2019. URL org [cs, stat].
httpg/bb//811....000 18810.0662, arXiv:1810.03642
A Appendix
A.1
Classifier- Free Guidance
We hereby provide a rationale for the use of classifier guidance and classifie--free guidance during diffusion model sampling. As per the "score matching" interpretation of diffusion models, we assume that our trained noise network approximates the score function of the true conditional latent distribution p(zle: ew(=', & -o, 2 px='((e))) For classifier guidance, the gradient of the log- likelihood of our CLIP encoder pa (eslzs) to the diffusion score as follows
perturb diffusion sampling by adding
Ep(z', = e ,4,44) nor a (og V.. [logp(="|e]] + mlog
(11)
We can rewrite this classifier guidance the unconditional score a logp(z") with
Vo [logp(>") logp(es(z") with y 7=1+1
(12)
using Bayes' rule, as log
(e)p()))) (ex) logp(e,1>*) + logp(>') - logp(e;), and thus V Voclog((="(e) Vee logp(e,(z")+ + ye log)(=²).
For classifier-free guidance, we aim to perform the above sampling without access to classifier, as long we possess conditional diffusion model (2 ,t,e; that doubles as an unconditional model 6, (2" (,,",t,, 0), as illustrated Section 5.2.
Using Bayes' rule again, we can see that a logp(e;(<*) v logp(="|e;) V. logp(='). If we substitute this into Eq. 12 we oblain
oe v [log((>') + (logp(>")ez) - logp(>'))]
(13)
oe at [[1 - 7) log)(=") + llggp(z"(e.))
(14)
which can be implemented with our conditional network as
tu(zt,t,e,) = (1 (1-7) (1-7))c(() 0, + re
(15)
A.2 Network architectures
Base Network (f) Our choice for a base model is a CLIP-Adapter (Gao et al., 2021), which consists of a frozen CLIP image encoder (Radford et al., 2021) with added learned fully-connected layers refining the output embedding. Specifically, we use the FVT-L/14336pp CLIP encoder type with an embedding size of 768, and for the adapter linear activation, and new linear output layer of size 768. The advantages of this model choice lie in its combination of high base performance (due to pre-trained knowledge contained in the CLIP component) and relatively small parameter count, enabling agile medium-small scale experiments. This base CLIP-Adapter network purely works as a base model and not to be confused with HyperCLIP, which is employed at the meta-level. In Section 6.2, when benchmarking the base model alone in the zero-shot setting (CLIP as Base Model), we drop the Adapter and use pre-trained zero-shot CLIP.
an MLP with one hidden layer of 256 units, which are followed by rectified
Hypernetwork (h) For the hypernetworks used in our baseline well as the generative model, use an MLP with one hidden layer of 256 units, which are followed by a rectified lincar activation. For the unconditioned hypernetwork, the embedding to the hypernetwork is vector of dimension 64, while for the conditioned counterpart, the task embedding is used. In order to ensure that the generated weights are properly normalized initialization, we use the Kaiming initialization (He et al., 2015) for the hypernetwork weights, initialize the embedding as sample from multivariate standard Gaussian distribution (for unconditioned models), and the NTK parametrization (Jacot et al., 2020) for the target network.
16

---

Published in Transactions on Machine Learning Research (08/2023)
Variational Autoencoder The variational autoencoder in our unconditioned generative model uses decoder an MLP of 2 hidden layers of size 512 and 256, each followed by the rectified linear non-linearity. We chose 32 as the latent code dimension. We use the same architecture for the decoder, except for the dimensionality of the hidden layers being swapped. We use the Kaiming initialization (He et al., 2015) to initialize the weight of both the encoder and decoder.
HyperCLIn We parametrize our HyperCLIP model as a fully-connected MLP with a single hidden layer of dimension 256, taking as input the flattened weight of the base network and outputting the corresponding CLIP encoding. We chose the tangent hyperbolic function as the activation function in the hidden layer.
HyperLDM While the original LDM makes use of a time-conditional UNet (Ronneberger et al., 2015) to parametrize the noise network, we are unfortunately unable to make use of spatial information and convolutions due to the non-spatial nature of our latent space, We parametrize our HyperLDM as a fully- connected network with residual connections and squoeze-and-ecciatton layers (Hu et al., 2018). The time index t is embedded into vector with 150-dimensional sinusoidal positional embedding and is concatenated together with the lask-conditional embedding at the input layer and at intermediate activations. Hidden layer dimensions are 8192, 16384, and 8192.
A.3 Notes and Limitations
While optimal parameter counts for the task-condiitonal techniques vary between our techniques, as well between our techniques and the baselines, all of the investigated approaches ultimately produce adapted weights for the same base network f, with the same architecture. The average performance of this base network with this fixed architecture when adapted and deployed on each individual test task is what allows us to fairly compare all meta-learning and multi-task algorithms. We acknowledge that, as limitation of our work, the comparisons hold up when comparing relatively small fixed base nerworks f, and our approach might not be scalable to compete with massively pre-trained large-scale multi-task models. In any case, we believe that the weight space generation of compact models can be useful in a variety of contexts, such as when the adapted base model needs to be deployed in embedded systems and other domains with limited compute resources. The scaling behavior of our techniques is still an open problem, which can be of interest. for future research.
A.4 Fairness and Bias
While our overall proposed methods belong to the realm of general-purpose techniques, their specific application may inadvertently raise issues related to gender and racial bias. As our MetaVQA dataset is a simple modification of VQA v2 (Goyal et al., 2017), it straightforwardly inherits gender and racial bias problems that have been found to exist within this dataset (Hirota et al., 2022). For example, answer distributions have been found to be different for the same question when the subject is a man woman. Such imbalances will necessarily exactly transfer to our MetaVQA datasct. Morcover, question-answer pairs have been found that express gender stereotypes. In any case, these problems are dataset-specific and can be ameliorated by future work on the source material from which MetaVQA is constructed, or by taking gender bias issues into account when constructing new dataset from scratch.
A.5 The Meta- VQA Dataset
The original VQA problem is about choosing suitable natural language answer yk when prompted with both natural language question t; and an image zg. Our observation is that the VQA problem can then easily be reformulated as a meta-learning image classification problem with natural language task descriptions: given question-image--nssee will serve as task descriptor), each of which can then be associated with supervised image classification tuples v E D, To make sure the designed tasks are meaningful, we filter out question-answer pairs with questions in a choice format, e.g., "A or B?" or "yes/no" answers. From the remaining questions, we keep the ones that appear at least. 20 times throughout the dataset, such that each task contains enough samples. Tn
triples (4,, 5k) E D, we can group the data by unique questions 4; (which
17

---

Published in Transactions on Machine Learning Research (08/2023)
the end, our Meta-VQA dataset is composed of 1234 unique tasks (questions), split into 870 training tasks and 373 test tasks, for total of 104112 12 image-answer pairs. There are average 9.13 answer choices per question/tase ttakk. The average size of the support set is 57.85 examples, while the average size of the query set is 25.9 examples.
10
Quessoon How many train tracks are there? Quesion: How many trains are on track?
Questor: How tracks there?
Question What zebras doing? Quaetion Where the zabras?
Gussitio:: What sport is being played? What playing?
What coat?
jackel?
Figure 6: UMAP projection for CLIP embeddings of Meta VQA questions (using the VIT-L/14 0336pz CLIP encoder). Clusters of similar questions exist in the dataset, similarly to how they exist in VQA v2.
A.6 Baseline methods
We detail overview of the baseline methods benchmark in table 2, together with algorithm tables detailing each baseline method.
Training:
The number of epochs each model is trained on, the learning rate ir of the optimization, as well as the learning rate and number of steps of the adaptation algorithm used for each method can be found in table 3. For all methods using an adaptation Ar, the dataset D, from a sampled task Ti is randomly split into a support set Dj and a query set. DJ during training. The support set is then used to perform the adaptation, while the query set is used to compute the loss on which the petameerssssers are updated (see Section 2.1). For baselincs with inner-loop adaptation Ar. all the data D, from sampled task T; is used in training. Unconditional methods do not have access to the task embedding e,, while conditional methods do. When the percentage of available task descriptors is reduced, conditional basclines are trained only on the tasks for which the descriptors are available, as they require such descriptors during training, unlike our two two-phased techniques.
Evaluation: For cach held-out test task T₁ from the Meta-VoC VQA datasot, we perform a zero-shot model evaluation on the fixed predefined query set D for the task. Zero-shot performance is evaluated before applying any adaptation procedure AT,-
18

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm Unconditional Multitask Training
Define the base network f with parameters W. for epoch = ,....... NN do
Sample training batch of image-answer pairs (2t, va) from mix of random training tasks T. batch. end for
Update W with gradient descent computed with respect to the classification loss over the sampled
Algorithm Unconditional MNet-MAMM Training
Define the base network f with parameters W. for meta-epoch = 1,.... do Sample a training task Tr and data D;. Randomly split D; into support set Dj and query set D
Run inner-loop adaptation AT, using the support set D;, fine-tuning W into task-adapted W; AT,
Use MAML gradient update to adapt W given the inner-loop adaptation.
end for
A Algorithm Unconditional HNet-MAML, Training
Define the base network f with parameters W.
Define a hypernetwork with meta-parameters 8, mapping a latent vector 20 to base network weights W. for meta-epoch = 1,....N N do
Sample a training task T, and data D;. Randomly split D; into support set Dj and query set D.S.
Run inner-loop adaptation AT, using the support set Dř, fine-tuning 20 into task- Ar, (2.).
Use MAML gradient update to adapt 20 and 0 given the inner-loop adaptation.
end for
Algorithm 5 Conditional Multitask Training
Define the base network / with parameters W. Define a hypernetwork h with meta-parameters 0, mapping the clip embedding of the language task descriptor to base network weights W.
for epoch = = 1,..., N do
Sample a training batch of task clip embedding, image and answer triples (es, Tk yk) from a mix of random training tasks T.
Update f with gradient descent computed with respoct the classification loss over the sampled batch. end for
Algorithm Conditional Multitask FiLM Training
Define the base network f with parameters W.
Define a FiLM layer, mapping the clip embedding e; the language task descriptor to modulation signals for the hidden activation layer of f. for meta-epoch = 1, ......,. N do Sample training batch of task clip embedding, image and answer triples (es, Tkk yk) from mix of random training tasks T.
Update 0 with gradient descent computed with respect to the classification loss over the sampled batch. end for
19

---

Published in Transactions on Machine Learning Research (08/2023)
Algorithm 7 Conditional HNe(-MAML Training
Define the base network f with parameters W.
Define a hypernetwork h with meta-parameterss mapping the clip embedding et of the language task descriptor to base network weights W,.
for meta-epoch = ....... N do
Sample a training task Ti, data D; and the clip embedding e, of the task descriptor. Randomly split D, into support set D and query set D
Run inner-loop adaptation AT; using the support set Df, fine-tuning & into task-adapted & AT, Use MAML gradient update adapt 0 given the inner-loop adaptation.
end for
Table 2: Overview of the different methods trained on MetaVQA. The parameters are optimized via the task loss evaluated on the output of the function, averaged over mini-batches of tasks. The adaptation Ar, implements a few-step gradient descent algorithm applied on the argument parameter, w.r.t the task loss evaluated on the support set.
Method
Function
Parameters
Uncond. Multitask
f(,W)
W
Uncond. MNet- (FO)MAL
f(, AT,AA (2)
W
Uncond. HNet-MAML
0,
Cond. Multitask
f(,b(er,))
0
Cond. Multitask FiLM
flye, W) ((,
W
Cond. HNel-MAML
0
Table 3: Hyperparameters used for the baseline methods. All methods are trained with the Adam (Kingma & Ba, 2017) optimizer, with meta- meta-batch size of 32 tasks. We use gradient norm clipping for all optimization, with the maximum norm set to 10. Note that when the adaptation algorithm A has a range possible steps, the number steps is sampled uniformly from the range for every adaptation.
Method
epochs
1r
A-lr
A-steps
Uncond. MNet-Mullitask
300
0.0001
Uncond. MNet- MNNet-FFOMMAM)
500
0.00003
0.01
0-10
Uncond. HNet-MAMI.
100
0.00003
0.1
0-10
Cond. Multitask
0.0001
Cond. Multitask FILM
300
0.0001
Cond. HNet-MAML
200
0.00001
0.1
0-10
A.7
Guidance Models
A.7.1
Generative hypernetwork
To enable our guidance methods, we need to first (rain a generative hypernetwork h as in Section 3, either in the form of an Unconditional Hypernetwork, or of a Hypernetwork VAE;
For HNet + HyperCLIP guidance and HNet + HyperLDM, we mota-learned an unconditional hypernetwork with the exact same hyperparameters the bascline Uncond. HNet-MAML, and used it the generative hypernetwork.
20

---

Published in Transactions on Machine Learning Research (08/2023)
For HVAE + HyperCLIP guidance and HVAE + HyperLDM, trained VAE samples of fine-tuned net work weights W; using the base network architecture specified in Appendix A.2. We detail the procedure in Algorithm 8 and, as training samples W,, we use adaptations over the base network (initialized from a learned Uncond. MNet-MAML initialization), using 50-step adaptation AT; with learning rale 0.01 on randomly split support sels. We trained the VAE for 2000 epochs where each epoch is a single pass through all the tasks, with the Adam (Kingma Ba, 2017) optimizer and 0.0001 learning rate and batch size 32. We used gradient norm clipping independently for both the encoder and decoder, with the maximum norm capped at 1000.
Algorithm НVAE Training
Define the base network f with parameters W. Define an encoder 2 = w(W,w) with parameters and hypernetwork decoder W h(2, () with parameters
0.
Obtain previously learned base nelwork initialization W0 according to Uncond. MNet-MAML (Algorithm 7).
for epoch = 1..... N do Create an emply batch B = 0- for b= M do
Sample a training task T; and data D,.
Randomly split D; into support D and query set D Run inner-loop adaptation AT, using the support set Dú, fine-tuning W; AT, (WO). Add the fine-tuned weights to the batch: B = B [W.).
end for
Train the HVAE encoder and decoder using the VAE loss to reconstruct the weight batch B. end for
A.7.2 HyperCLIP
Hypemet
x
3
HyperCLIP Guidance
Casine sim.
VAE
h
W
W
CLIP
CLIP
Task
Desc.
z
0
y
Y
L.(z)
2
V. (z)
Who is weaning glasses?
CLIP
X
CLIP
H
H
Figure 7: A comprehensive diagram of the steps involved for training and evaluating the HyperCLIP guidance technique. (1) Training an unconditional HVAF over the training tasks. (2) Training the HyperCLIP model according to the contrastive procedure described in Algorithm 1. (3) Exploring the latent space the HVAE with gradient descent over the HyperCLIP guidance loss, as described in Scction 4.2.
Training To train the HyperCLIP model, we need samples of fine-tuned network weights W.. We use adaptations from Uncond. HNet-MAML, using 50-step adaptation Ar, with a learning rate 0.1, on randomly split support sets. We trained our HyperCLIP model for 600 epochs with the Adam (Kingma & Ba, 2017) optimizer, 0.0003 learning rate, and batch size 64 for all our experiments.
21

---

Published in Transactions on Machine Learning Research (08/2023)
Guidance We use 10 steps guidance with À = 0.01 and learning rate 0.1, to perform guidance within either the HNet or HVAE latent spaces.
Evaluation For each held-out test task Ti from the Meta- VQA dataset, we perform a zero-shot model evaluation the fixed predefined query set DJ for the task. Zero-shot performance is evaluated the output of the generative hypernetwork h after applying latent space guidance.
A Algorithm HNet + HyperCLIP Training
Learn unconditional hypernetwork h(=", ,) with the Uncond. HNct-MAML procedure from Algorithm
7.
Learn HyperCI.IP network CL.P (W) using the HyperCI.IP training procedure from Algorithm 1. For sampling fine-tuned Wi, fine-tune the base network on training tasks.
Algorithm 10 HVAE + HyperCLII Training
Learn an unconditional hypernetwork h(z 8), the docoder of a HVAE (Algorithm 8). Learn HyperCLIP network CLIP (W) using the HyperCLIP training procedure from Algorithm For sampling fine-tuned W,, fine-tune the base network on training tasks.
Algorithm HyperCLIP guidance (Inference time)
Define learned unconditional hypernetwork h(z, 8), as either a HNet h(29,0) (Algorithm 7) or the decoder of HVAR (Algorithm 8).
Define a learned HyperCLIP network (W).
Define unseen task To with natural language task descriptor 4.
Randomly sample 2 ~ ~N( I) if using the decoder of a HVAE, or set == == where 20 is the meta-learned embedding initialization the Hnet.
Optimize z with gradient descent over (guidance (Eq. 4), obtaining guided 24.
Obtain guided base weights W; = h(z, 0).
Use adapted base network f with weights W; classify examples from the unseen task Tr.
A.7.3 HMperLDM
Training Similarly HyperCLIP, to train HyperLDM need samples of fine tuned network weights Wa for which we use adaptations from Uncond. HNet-MAML, using 50-step adaptation A7, with learning rate 0.1, on randomly split support sets. We parametrize the diffusion process with a linear noise schedule, / starting at 0.0001 and ending at 0.06, and 350 diffusion timesteps. For all our experiments, we train the HyperLDM for 1000 epochs with the Adam optimizer, 0.00025 learning rate, and 128 epochs.
Evaluation Evaluation is performed as for HyperCLIP guidance, except for the fact that adaptation is performed natively through sampling from the learned reversed diffusion process, with parameters derived from the chosen 3 schedule. The guidance parameter 7 > 0 can be tuned during inference to accentuate the effect of classifier- guidance.
Algorithm 12 HNet + HyperLDM Training
Learn an unconditional hypernetwork h(22, 7.
(a2,,0 0) with the Uncond. HNet-MAML procedure from Algorithm
Learn the HyperI.DM network c,1 f, ex) using the HyperL.M training procedure, optimizing reconstruc- (,,2",,,,, tion of 2 with loss from Eq. 9. For sampling fine-tuned 24, fine-tune the base network on training tasks, then encode the weights using the HNet.
22

---

Published in Transactions on Machine Learning Research (08/2023)
Hypemet
x
3
HyperLDM Sampling
Task Desc.
CLIP,
e
W
f
VAE
2
y
LDM
LDM
2
HyperLDM Training
2
Z
Z
Gaussian noise
f
d
X
Y;
train inverse noise
W
Task
CLIP
Desc.
Decode final weights
LDM
Y
Figure & A comprehensive diagram the steps involved for training and evaluating the HyperLDM model. (1) Training an unconditional HVAE over the training tasks. (2) Training the HyperLDM model as a conditional reversed diffusion process described in Scction 5.1. (3) Sampling from the conditional diffusion model.
Algorithm 13 HVAE HyperLDM Training
Learn an unconditional hypernetwork h(2, 0), as the decoder of a HVAE (Algorithm 8).
Learn the HyperLDM network ew(2", e,) using the HyperLDM training procedure, optimizing reconstruc- tion of 2f with loss from Eq. 9. For sampling fine-tuned 24, fine-tune the base network training tasks, then encode the weights using the HVAE.
Algorithm MyperL.DM Inference
Define learned unconditional hypernetwork 1(zz 0), as either HNet 1(=2,0) (Algorithm 7) the decoder of HVAE (Algorithm 8).
Define a learned HyperLDM network (,()",t,)))
Define an unseen task To with natural language task descriptor 4, with clip embedding ex.
Randomly sample z ~ (O,
Iteratively modify z with diffusion sampling using the learned Ev network, obtaining guided 24.
Obtain guided base weights W, = h(2, 0).
Use adapted base network f with weights W; classify examples from the unseen task Ti-
Table 4: Few-Shot learning accuracy averaged over Meta-VQA test tasks. (* ouurs)
Method
Few-Shot
CLIP Base Model
54.93 (+ 0.11)
Uncond. Multitask
55.53 (+ 0.40)
Uncond. MINet-MAML
60.24 (4: 0.84)
Uncond. FOMAM..
60.03 (+ 0.48)
Uncond. HNet-MAML
58.70 (# 0.10)
Cond. Multitask
59.46 (# 0.31)
Cond. HNet-MAML
59.48 (# 0.03)
23

---

Published in Transactions on Machine Learning Research (08/2023)
A.8 Few-Shot Learning
For completeness, we include in Table 4 the results for few-shot learning on the lest split of Meta-VQA. Our technique, unlike classic MAML, does not optimize specifically for the few-shot learning setting, Instead, the fow-shot learning results are meant to contextualize performance gains in the zero-shot setting: zcro-shot performance gains should be interpreted as relative to the few-shot accuracy ceiling of 60.24%, the maximum attained with our fixed choice of the base model.
For few-shot learning test time, all adaptation is performed the support set of the test tasks. For MAML baselines, keep the same adaptation-tine time learning rate during training, and we always adapt for 50 steps. For each multitask baseline, we use the same adaptation scheme (steps, learning rate, adapting parameters) as their MAML counterpart.
24


Reviews:
[{"review_id": "z6mPxVbTox", "reviewer": "Reviewer_jJek", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The idea of meta-learning in the weight space for multimodal application is very interesting.\n2. The authors explored several variants which provide a solid foundation for future research.\n\n\nCons:\n1. The current empirical evaluation is limited.\n- a. The main concern is that the method is only evaluated on the meta-VQA dataset with a single way of splitting the original VQA dataset into different tasks. The only difference in terms of tasks is the specific question, which may risk the whole evaluation heavily biased to text. Because different tasks could be also determined based on image only or the context from both text and image.\n- b. More rigorous analysis on the relationship among questions should be provided. The actual semantic of the questions should be considered when determining \"different tasks\". For example, two questions with very different sentence structure may essentially ask the same thing. An actionable item could be analyzing the relationship between semantic relationship (e.g. measured by sentence similarity between support set and query set) and accuracy to truly understand the generalization ability of the evaluated models.\n- c. The current evaluation setting might not be practical as the authors only \"keep the ones which appear at least 20 times throughout the dataset\". But in reality, the questions could be very specific to the query image, which may only appear once in the whole dataset. Therefore, it is doubtful whether the current setting really shows the generalization of the evaluated methods. The reviewer doubt if such questions with less support samples are also evaluated, a simple multitask unconditional baseline may perform better according to the trend shown in Figure 5B.\n- d. Broader possibility of evaluation dataset construction should be considered. For example, beyond VQA, there are also datasets about visual question answering in some specific domains, like knowledge-intensive, sports, medical and etc. If these datasets can be merged together to evaluate the proposed method, it will make this work of much more contribution and potential impact.\n\n2. Further visualization on the latent space will be very interesting. This could help us to interpret the task similarity. It could be also more interesting, if such analysis could be compared with plain sentence similarity from CLIP text encoder, this helps us to understand what is learned throughout the process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the meta learning setting is explored in the weight space towards multimodal application. To model the rich task space, the authors proposed to train a generative network that can produce weights. Then a HyperCLIP encoder is proposed to be contrastively trained to map generated weights to the CLIP embeddings of the task descriptions. Then the authors explored using CLIP guidance or a latent diffusion process to obtain latent vectors during task adaptation. The effectiveness of the proposed method is shown to be effective on meta-VQA.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check cons for details.", "broader_impact_concerns": "It could be actually also very interesting if the authors can analyze the model's behavior towards certain sensitive attributes [a].\n\n[a] Gender and Racial Bias in Visual Question Answering Datasets", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PFUbRZBaR9", "reviewer": "Reviewer_cxHR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n**[S1] Interesting new setup holds a lot of potentials:** The paper proposes a pretty exciting potential; Meta-learning by simply aligning the model weights to the shared semantic multimodal space provided by large-scale \"foundation\" models. The technical definitions and the use of CLIP/LDM guidance appear sound to this reviewer. \n\n**[S2] Promising results (on paper):** The details are a bit unclear (See Weaknesses before), but assuming nothing is amiss, the results look very promising on the dataset prepared by the manuscript. To the degree that I understand it, grouping different questions into \"tasks\" and assuming the answering task as a simple classification task on the space of all answers is a valid setup (with caveats) and the proposed method clearly surpasses baseline and dummy attempts. However, I am unclear on details which makes me put a \"conditional\" label on my assessment. I look forward to hearing from author's feedback.\n\nWeaknesses:\n\n**[W1] Unclear evaluation setup:** Despite 10 pages of content and 6 more pages of supplementary, the paper is still unclear on the construction of the MetaVQA dataset. Is the answer space (classification \"classes\") shared between all tasks? If yes, I think the setup makes sense, but there is risk of cross-talk between tasks, and the effects of this should be studied in more detail. E.g., for task 1: How many cats are there? and task N: How many dogs are there?, the task definition and answer space overlap a lot and the answer can simply be to count the number of \"main\" animals in the picture. It would also tremendously help to see examples of this task. \n\nOn the other hand, if the answer space is not fixed. i.e., class 1 does not represent a fixed concept across tasks but rather changes for each task, then I do not understand how this problem is (even hypothetically) solvable. i.e., there is no semantic link between classes and the answer space for zero-shot evaluation (Unlike something like metric learning, which would also encode answer space into the same space).\n\n**[W2] Unclear motivations and over-arching principles for metaVQA**  The core premise (viewed outside of the proposed evaluation) makes sense, i.e., given a natural language task description, it should be possible to guide the network weights to align to the target task by leveraging large foundation models. However, I am unclear on how this translates to the MetaVQA problem (also see W1). Are there any requirements for the support set to cover all possible answer spaces? Can the paper explain in more detail the composition of the support and query sets and how meta-learning can happen in this setup? What information in the task definition can the HCLIP/LDM guidance leverage to map from the given question \"task\" and the set of answers possible that are not represented in the same space but rather as classes (class-ids)? Do you think MetaVQA captures the core claims made in the introduction?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes two meta-learning algorithms (or two facets of one) that leverage the large-scale multimodal backbones (LDMs and CLIP) to pose the meta-learning problem as a \"weight-generation\" problem. In the \"hyperCLIP\" based solution, the paper trains to align the 'HyperCLIP' network, which takes fine-tuned NN weights and maps them to be trained with contrastive learning to align with the embeddings for \"task definition\". In the LDM \"classifier-free\" CLIP guidance which aims to produce NN weights via diffusion as guided by the \"task embedding\". This is a unique take on meta-learning. The evaluation is done with a constructed \"diagnostic\" dataset made by re-arranging VQA 2.0 datasets into several \"tasks\" with each task containing a unique question. Then, for a given task, the problem can simply become an Image classification problem. The results on this dataset are promising (on paper). However, a few details are unclear and I have some concerns about the core principles of how one would apply the above process of guidance (which appear sound by themselves) to this newly constructed dataset.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Requested changes:\n\n[C1] Please expand on the creation of the MetaVQA dataset and how it is posed as a classification problem. See W1 above. This is central to a better understanding of the evaluation of the paper and is crucial. \n\n[C2] Please clearly explain your guiding principles behind this model and experiment design. See W2 above. This is also crucial to fully facilitate the understanding of the work.\n\nThe reason I marked \"No\" on \"Claims And Evidence\" is that things are not clear currently. However, I think this is easily remedied in a revision/rebuttal and I look forward to hearing back.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "qGniJo0M38", "reviewer": "Reviewer_7YF9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The model structures proposed are novel and interesting, and demonstrate a different way to do things rather than the prompt-direct-to-output that standard transformer LLMs demonstrate. \n2. The results are very good, and as such, the proposed model formulations can serve as a powerful ingredient for future work. \n3. The evaluation while not exhaustive, seems enough to demonstrate a point. \n4. Writing is generally good.\n5. The conceptual and engineering novelty of the work is high.\n\nWeaknesses:\n\n1. The written paper seems to miss the fact that existing transformer models can be viewed as a set of small hypernetworks. Specifically, the key, query and value networks essentially receive inputs, and generate weights that are then dot producted and further regularized with self-attention circuitry. It would be worth acknowledging this and talking about how this work is different. \n2. There seems to be an emphasis on the few-shot and zero-shot subfield of meta-learning, and as a result statements like \"The development of deep learning models that perform quick adaptation to unseen tasks is the focus of the\nfield of meta-learning.\" feel imprecise. Quick adaptation is a type of task that meta-learning can be used to solve. To say that the entirety of meta-learning is aiming for that is imprecise in my own opinion. \n3. The description of the methods involved is hard to parse and digest. I recommend improving the figures a bit, showing the specific optimizations done more explicitly, and breaking down the modules further when described to make it easier to digest. Perhaps starting with a list of ingredient modules that are needed so one can see the whole picture, and then zooming into each one. This is very interesting work, and deserves more detail I believe. \n4. Other than the fact that the models introduce try to do things in a novel way that works, it's not clear where this sort of model could outperform existing standard contrastively trained transformers. This is something you should make clear to the reader.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper appears to be doing a number of things:\n\n1. Demonstrates how one can embed the weights of a neural network in a way compatible with existing contrastive learning. The idea of passing weights through another network is not new, but the application of it in a contrastive setup with good results is novel and very valuable. \n2. Introduces two model formulations, HyperCLIP and HyperLDM, which demonstrate how one can first align contrastively a text embedding with a weight embedding, and then guide simple optimization process to find appropriate model weights given a task-related prompt. \n3. Adapts VQA V2 in a way that allows evaluation of such prompt-based models.\n\nI find the work done to be very interesting from a conceptual and engineering standpoint. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Try to ensure that general statements about meta-learning do not equate it with quick adaptation tasks, but rather the task of learning to learn such that a specific goal is met (whether that is speed, generalization, robustness etc).\n2. Break down the model into key ingredients, list them, and then explain those thoroughly but compactly, so that the reader can appreciate the depth of the work. I have previously played with few-shot hypernets, and I can intuit many of the components you work with, but many others don't have that experience. \n3. Try to make your figures show more explicitly the optimization processes used. You can use something like draw.io to generate consistent but smooth shapes, and colours that are colourblind friendly. Colours can help deliver messages more readily.\n4. Try to discuss how transformers (that may be viewed as a chain of small hypernetworks) differ from your work. \n5. Find arguments or applications where your model could outperform, or be preferable to train/bet on. This should help deliver part of the message you are trying to deliver. ", "broader_impact_concerns": "There are no concerns that surpass the typical level of risk associated with less risky endeavors in the field of Machine Learning.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission looks into leveraging large-scale multimodal backbones for the purpose of meta-learning by framing the problem as a weight generation problem. The core idea consists of i) training an unconditional hypernetwork on meta-training tasks; ii) repurposing CLIP's language backbone by learning some kind of alignment mechanism between the hypernetwork and the CLIP embeddings of the tasks' natural language descriptions; and iii) using the learned alignment mechanism to produce weights for new tasks given their natural language description.

The authors introduce two approaches named HyperCLIP and HyperLDM for which the alignment mechanism in ii) is trained contrastively (HyperCLIP) and generatively (HyperLDM). The two approaches are applied to zero-shot learning on a proposed modification of the VQA v2.0 dataset called Meta-VQA.

All reviewers agree that the submission is of great interest to the TMLR community. While concerns were raised over the scope of the evaluation, the empirical results are sufficient to back the claims made in the paper.
Decision: Accept (poster)