OCD: Learning to Overfit with Conditional Diffusion Models
Authors: Shahar Lutati
Lior Wolf
Venue: Submitted to ICLR 2023 (2023)
Keywords: Local learning
hypernetworks
diffusion processes

Abstract: We present a dynamic model in which the weights are conditioned on an input sample $x$ and are learned to match those that would be obtained by finetuning a base model on $x$ and its label $y$. This mapping between an input sample and network weights is shown to be approximated by a linear transformation of the sample distribution, which suggests that a denoising diffusion model can be suitable for this task. The diffusion model we therefore employ focuses on modifying a single layer of the base model and is conditioned on the input, activations, and output of this layer. Our experiments demonstrate the wide applicability of the method for image classification, 3D reconstruction, tabular data, and speech separation. Our code is attached as supplementary.

Under review as a conference paper at ICLR 2023
OCD: LEARNING TO OVERFIT WITH CONDITIONAL DIFFUSION MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
We present dynamic model in which the weights are conditioned on an input sample x and are learned to match those that would be obtained by finetuning a base model on x and its label y. This mapping between an input sample and network weights is shown to be approximated by a linear transformation of the sample distribution, which suggests that denoising diffusion model can be suit- able for this task. The diffusion model we therefore employ focuses on modifying a single layer of the base model and is conditioned on the input, activations, and output of this layer. Our experiments demonstrate the wide applicability of the method for image classification, 3D reconstruction, tabular data, speech separa- tion, and natural language processing. Our code is attached supplementary.
INTRODUCTION
Here is simple local algorithm: For each testing pattern, (1) select the few training examples located in the vicinity the testing pattern, (2) train a neural network with only these few examples, and (3) apply the resulting network to the testing pattern.
Bottou & Vapnik (1992)
Thirty years after the local learning method in the epigraph was introduced, it can be modernized in few ways. First, instead of training a neural network from scratch handful of samples, the method can finetune, with the same samples, a base model that is pretrained on the entire training set. The empirical success of transfer learning methods (Han et al. (222)) suggests that this would lead to an improvement.
Second, instead of retraining neural network each time, we can learn predict the weights of the locally-trained neural network for each input sample. This idea utilizes a dynamic, input-dependent architecture, also known as a hypernetwork (Ha et al. 2016).
Third, we can take the approach to an extreme and consider local regions that contain a single sample. During training, we finetune the base model for each training sample separately. In this process, which we call "overfitting". we train on each specific sample = (I, y) from the training set, starting with the weights of the base model and obtaining a model fe.. We then learn model that maps between æ (without the label) and the shift in the weights of fo, from those of the base model. Given a test sample x, we apply the learned mapping 9 to it, obtain model weights, and apply the resulting model to x.
The overfitted models are expected to be similar to the base model, since the samples we overfit are part of the training set of the base model. We provide theoretical arguments that support that the mapping from the x part of to fo, can be approximated by a locally convex transformation. As result, it is likely that a diffusion process that is able to generate samples in the domain of would also work for generating the weights of the fine-tuned networks. Recently, diffusion models, such DDPM (Ho et al. 2020) and DDIM (Song al. 2020) were shown to be highly successful
in generating perceptual samples (Dhariwal & Nichol 2021b Kong et al. 2021). We, therefore,
employ conditional diffusion model to model g.
In order to make the diffusion models suitable for predicting network weights, we make three ad- justments. First, automatically select specific layer of the neural model and modify only this

---

Under review as a conference paper at ICLR 2023
layer. This considerably reduces the size of the generated data and, in our experience, is sufficient for supporting the overfitting effect. Second, we condition the diffusion process on the input of the selected layer, its activations, and its output. Third, since the diffusion process assumes unit variance scale (Ho et al. 2020), we separately learn the scale of the weight modification.
Our method is widely applicable, and we evaluate it across four very different domains: image classification, image synthesis, regression in tabular data, and speech separation. In all cases, the results obtained by our method improve upon the non-local use of the same underlying architecture.
2 RELATED WORK
Local learning approaches perform inference with models that are focused on training samples in the vicinity of each test sample. This way, the predictions are based on what is believed to be the most relevant data points. K-nearest neighbors, for example, is a local learning method. [Bottou & Vapnik] (1992) have presented simple algorithm for adjusting the capacity of the learned model locally. and discuss the advantages of such models for learning with uneven data distributions. [AL- paydin & Jordan (196) combine multiple local perceptrons in either a cooperative or a discrimina- tive manner, and Zhang et al. (2006) combine multiple local support vector machines. These and other similar contributions rely on local neighborhoods containing multiple samples. The one-shot similarity kernel of Wolf et al. (2009) contrasts a single test sample with many training samples. We are unaware of any previous contribution that finetunes a model based on a single sample or any local learning approach that involves hypernetworks.
Hypernetworks (Ha et al. 2016) are neural models that generate the weights of a second primary network, which performs the actual prediction task. Since the inferred weights are multiplied by the activations of the primary network, hypernetworks are a form of multiplicative interactions (Jayaku- mar al. 2020), and extend layer-specific dynamic networks, which have been used to adapt neural models to the properties of the input sample (Klein et al. 2015 Riegler et al. 2015).
Hypernetoorks benefit from the knowledge-sharing ability of the weight-generating network and are therefore suited for meta-lcarning tasks, including few-shot learning (Bertinetto et al.] 2016). con- tinual learning (von Oswald al.) 2020p, and model personalization [hamssian et al. (2021) When there is a need to repeatedly train similar networks, predicting the weights can be more efficient than backpropagation. Hypernetworks have, therefore, been used for neural architecture search (Brock et al. 2018 Zhang et al. 2019), and hyperparameter selection (Lorraine & Duvenaud, 2018).
MEND by Mitchell et al. (22T) explores the problem of model editing for large language models, in which the model's parameters are updated after training to incorporate new data. In our work, the goal is to predict the label the new sample and not to update the model. Unlike MEND, our method does not employ the label of the new sample.
Many of the recent generative models for images (Ho et al. 2022 Chen et al.) 2020 Dhariwal & Nichol 2021a) and speech (Kong al.] |021]] Chen al. 2020 are based on de- generate form of the Pocker-Planncl equation. SSoh--Dicsstein et al. (2015) showed that complicated distributions could be learnt using simple diffusion process. The Denoising Diffusion Probablistic Models (DDPM) of Ho et al. (2020) extend the framework and present high quality image synthesis. Song et al. (2020) sped up the inference time by an order of magnitude using implicit sampling with their DDIM method. Wattsnn et al. (2021) propose dynamic programming algorithm to find an efficient denoising schedule Lnd San-Roman al. (202l) apply learned scaling adjustments the noise scheduling. Luhman & Luhman (2021) combined knowledge distillation with DDPMs
Diffusion models
The iterative nature of the denoising generation scheme creates an opportunity to steer the process, by considering the gradients of additional loss terms. The Itcrative Latent Variable Refinement (ILVR) method [Choi et al.) (202T) does so for images by directing the generated image toward low-resolution template. A similar technique was subsequently employed for voice modifica- tion Levkovitch et al.) (2022). Direct conditioning is also possible: [Saharia et al. (2022) generate photo-realistic text-to-image scenes by conditioning a diffusion model text embedding; Amit voice gen- eration, the mel-spectrogram can be used as additional input to the denoising network Chen et al. (2020); Kong et al. (202T); Liu et al. (2021a). as can the input text for a text-to-speech diffusion model Popoy al. (2021).
et al. (202T) repeatedly condition on the input image to obtain image segmentation.
2

---

Under review as a conference paper at ICLR 2023
3 PROBLEM SETTING AND ANALYSIS
We are given a dataset S of samples 2; E X. for i- = I..n, and the associated labels yi ) sampled ii.d from some distribution Pxxy over the composite domain X' V. We consider prediction models f X' x θ ) that are parameterized by weight vectors in the domain θ. Specifically, we first learn a base model [a(x) = [(x, 0). which trained over entire training set S.
PxxY: we further consider finetuned version of So that is optimized to overfit on the sample = (xr, y) i.e., we minimize the loss of the single sample 8, initializing the optimization process with 0. We denote the obtained parameters as 0, E O and the obtained network as fa.. Naturally, the prediction error of fe, on the sample s is expected to be small, improving upon that fo for that specific sample.
For every sample
The meta-learning problem we consider is the one of learning a model that maps r (the input domain of sample and potentially multiple latent representations of in the context of fe, collec- tively denoted as (xx), to a vector of weight differences, such that the following loss is minimized
(1) where MSE is the mean squared error. If generalizes well to unseen samples s² (t",)"), we would expect f(" 0 + g( ((x=""(("))) [(=")) be better prediction of y" than f(x", 0).
[(s) = MSE + (x))),
We rewrite the mapping function gaz, [(x) as a unit-norm component gu gu(2z, [(x)), and a scale factor p(a, ixx)).
g(x, p(z, T(x)) In( [(x)), where (s,, (xx))112= =1
(2)
(3)
Denote the mapping between a sample (z,y) and 0_-0 as H X x y + O. Let HI TT.-00
X' x y O the linear approximation of the operator H, i.e., H(s + 6s) = H(s) +
Hs(s) ôs (dss @ ds), where @ denotes the tensor product along the sample dimensions.
Next, we provide two theoretical arguments without concrete quantization. First, following Klein- berg et al. (2018), if 0 is near a local minima or an inflection point of the finetuning loss, then 0, would converge to this point and (s (s+ + 5s) is expected to be convex.
Second, since 0 is obtained via an SGD optimization process over sample from the distribution Pxxy, and since s ~ Pxxy, it is near either local minima or an inflection point for the training loss of finetuning with sample See, for example, Kleinberg al. (2018).
Combining the two claims, we can expect to be locally convex.
Lemma 1. Since is locally convex, it follows that the distribution PH, (e) in the domain O ob- tained when applying Hy to samples ~ Pxxx takes the following form:
PH P (s) s))det(HH
where H (X x ) ①) (X Y) is the pseudoinverse of the tensor H, (first we compute H1 at given and then compute the tensor pseudoinverse).
See appendix for the proof.
From the lemma, the distribution of H1 is linear transformation of the distribution of s. Ho ct al. (2020) showed that a diffusion process is suitable for estimating samples from complicated sample distributions. Assume that there exists diffusion process, with weights y, over the sample distribution, such that the variational bound is maximized
max Feg (log(p (xo:r)) - log(g((ner(zo)))
(5)
where P., is the model distribution, g(x)) is the data distribution, and Forr forms a Markov chain. From Lemma. [ have that the image of the distribution of H₃ is contained in the image of g, with the same input set. Thus, if a diffusion model has the capacity to capture will also capture H,, which is subset of g.
3

---

Under review as a conference paper at ICLR 2023
4 METHOD
Our method is based on a modified diffusion process. Recall that we denote the training dataset as S {(z,, 3t)]f= and that for every training sample S we run fine-tuning based on that single sample to obtain the overfitted parameters (function) as 92 (fe.). ). In our method, 8, f+ g z. [(r),, where 0 are the base model's parameters which are trained over S, and g(æ, [x)) is mapping function that maps the input, i.e., the part of 8, and multiple latent representations of it, (zx), to the desired shift in the model parameters.
Layer selection Current deep neural networks can have millions or even billions of parameters. Thus, learning to modify all network parameters can be a prohibitive task. Therefore, we opt to modify, via function g. on single layer of fa.
To select this layer, we follow Lutati Wolf (2021) and choose the layer that presents the maximal entropy of the loss, when fixing the samples and perturbing the layer's parameters. Denote the perturbed weights, in which only layer L is perturbed, as 0 The score used for selection
Σ (E.))S
Entropyo, (C(f(x,0)).))
(6)
where L is the loss objective which the function of fa is trained on, and the entropy is computed over multiple draws of 8², The entropy is computed by fitting a Gaussian Kernel Density Estimation (GKDE) (Silverman) 1986) to the obtained empirical distribution of the loss function. Since sam- pling does not involve ve a backpropagation computation, the process is not costly, so 10, 000 samples used.
The conditioning signal The latent representations, (x), has three components. Given a selected layer, L, we denote the input this layer (when passing sample x f(x, 0)), as (x) and the activation of this layer as (x). We also use the output of the base function fo(r). [xx) is, therefore, the tuple
T(x) = (z
4.1 DIFFUSION PROCESS
The diffusion goal is to reconstruct the mapping function (x [(x).. The process iteratively starts random Q7, iterates with 2, where decreasing and is the diffusion step. and returns Sog
The diffuusin error estimation network, is a function of the current estimation, Ωr, the latent representation tuple, I(x), and the diffusion timestep, t. The latter is encoded through a positional encoding network (Vaswani et al. (017)), PE. All inputs, except for Ω4 are combined into one vector: = PE(()= E((i)) + (ar))) + E. (Jxx)), where E,, Ea, E. are the encodings of the layer input, layer activations and network output. Note that most of the components of e do not change during the diffusion process, and can be computed only once. This way, the conditioning overhead is reduced to minimum. The conditional diffusion process is depicted in Fig
The complete training procedure of En is depicted in Alg. I The first phase is overfitting, using simple gradient decent over a single input-output pair, see line The overfitting phase is not demanding, since the backpropagation conducted only over the selected layer and a single sample.
Training Phase
As stated in Sec. while regular diffusion assumes that the input has unit variance, when estimat- ing network weights, scaling has crucial impact. This normalization ensures that the diffusion is trained over unit-variance input. We denote by gnorm the normalized difference between 0g and the parameters 0 of the base model (1.
Following Song et al. 2020), linear scheduling is used for the diffusion process, and Br, Or Cr. B are set in line A training example is then sampled:
2 morrm
2e,
where e NV 1) is normal noise. Since our goal is to recover the noiseless Onomm, the objective is
||e e (I(x), 0)]
4

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Training Algorithm.
((x)
E;
Input: S training set, base network parameters, L the loss of the primary task, T diffusion steps Output: en diffusion network (incl. E, E.).
(z)
E
PE
E,
1: repeat
2: sample (2, S and set 0₂
3: repeat
EN
4: Grad step V fo, (1:) update 0g
5: until L(y, Ja. (x)) converges
2T
Gg
6: (uorm 10,-00
7: to Uniform (1... c= N
Figure The diffusion process. x is the in- put of the base network, (z).. [xx) is tuple of latent representations of æ. F, Fa,, and E. the input, activation, and output encoders, respectively, of the selected layer that is being modified. is the diffusion step, and 2 the current diffusion estimation.
3. = 10- (1-1) aa = 1 Br,
8:
a; k
9:
ai Grad step on Vile En (1(x), t))l up-
10:
dating En. PE and the components of
until ||c en (P,. (I))),t)) converges
gradient step is taken in order to optimize this objective, see line.
Inference Phase Given an unseen input x, [(x) is computed using the base network f(r, 0) and used for all calls the diffusion network eg. The diffusion steps are listed in appendix B
4.2 SCALE ESTIMATION
The Evidence Lower Bound (ELBO) used in Ho et al. (2020) assumes that the generated data has unit variance. In case, in which the generated data reflects difference in the layer's weights, the scale of the data presents considerable variation. Naturally, shifting the weights of network by some vector d or by some scale times can create significant difference.
We, therefore, as indicated in Eq. 2 use an MLP network p(x, (xx) to estimate the appropriate scale factor, based on the same conditioning signal that is used for the network en that implements Iu as diffusion process.
When learning network p. the following objective function is used
pp(z, [(x)) pal?
Cscale
Σ
10 10- logio
(10)
s= (aS
where Ps 110, O|l
4.3 ARCHITECTURE
The network is U-Net (Ronneberger et al., 2015), following Ho et al. (2020). Each resolution level has residual blocks, and an attention layer. The bottleneck contains two attention layers.
The positional encoder is composed of stacked sine and cosine encodings, following Vaswani et al. 2017). The encoders of it, az are both single fully-connected layers, with dimensions to match the positional embedding. The encoder of the base network's 's output fol (x) depends on the output type. In the case of a classification network, where the output is a vector in C where classes, the encoder Eo is single fully-connected layer. In the case of image generation, the output image is first encoded using a VGG11 encoder (Simonyan & Zisserman, 2014). and then the latent representation is passed through single fully-com colnectedd layer, again matching the dimension of the positional encoder. For speech separation, the estimated speech is first transformed to a spectogram with 1024 bins of FFT, then encoded using the same VGG11.
the number of
5

---

Under review as a conference paper at ICLR 2023
5 EXPERIMENTS
In all experiments the UNet éq has 128 channels and five downsampling layers. The Adam optimizer (Kingma & Ba 2014). with a learning rate of 10-4, is used. A linear noise schedule is used based on Song et al. (2020). and the number of diffusion steps is 10. All experiments are repeated three times report the standard deviation (SD) of the success metrics. The inference overhead due to U-Net forward pass is 27[ms] on a low-end Nvidia RTX2060 GPU (there are ten such passes, one per diffusion step).
In addition to the full method, we also show results for the network that overfits on the test data, which serves as an upper bound that cannot be applied without violating the train/test protocol. On some datasets we check to what extent selecting single layer limits our results, by performing the overfitting process on all of the model weights. On all datasets, we ablate the scale component of our "Overfit with Conditional Diffusion models" (OCD) method, by estimating fixed global scale factor p = sps)(ps) as the mean value of the scale factor As over the train-set. An additional ablation selects the model fe, of the training sample with the closest to the test sample. This "nearest neighbor" ablation can be seen as the simplest way to implement the concept of OCD. Finally, we present an ablation that selects the layer with the second highest layer selection score, to evaluate the significance of the selection criterion.
Image Classification Results for the MNIST dataset (LeCun & Cortes 2010) are obtained with the LeNet5 architecture (Lecun et al. 1998). The selected layer is the one next to the last fully connected layer, which, as can be seen in appendix C has the maximal entropy among LeNet5's layers. CIFAR1O images (Krizhevsky et al., 2009) are classified using GoogleNet (Szegedy et al.) 2015). The selected layer was the last fully-connected layer, see appendix For both architectures, the three encoders Er,, Et.,, Eo are simple fully-connected layers, with dimensions to match the number of channels in the UNet (128).
For classification experiments we measure both the cross entropy (evaluted on the test set) and the test accuracy. As can seen in Tab. !I our method reduces the CE loss by a factor of in comparison to the base network and there is a improvement of 0.5% in accuracy. Ablating the scale prediction, the results considerably degrade in comparison to the full method. The Nearest-Neighhor ablation yields slightly better results than the base network. The ablation that selects an alternative layer results in performance that similar or slightly better than the base network. This is congruent with the small difference between fitting the selected layer and fitting all layers, which may suggest that much of the benefit of overfitting occurs in the selected layer.
On CIFARIO, our method improves classification accuracy from 92.9% to 93.7%. As in MNIST, much of the improvement is lost when running the three ablations. In both MNIST and CIFAR, when using the ground truth to overfit a specific example, the accuracy becomes, as expected, 100%. Considering the CE loss, overfitting the entire model instead of the selected layers yields only mild improvement (for MNIST below the standard deviation). This indicates that the added improvement gained by applying our method to all layers (and not just to the selected one) may not justify the additional resources required.
Experiments were also conducted on the TinyImageNet dataset Le & Yang 2015). The bascline network used is the (distilled) Data efficient transformer (DeiT-B/I6-D) of Touvron al. 2002] The selected layer is the weight of the head module. Our method is able to improve the baseline network, achieving 90.8% accuracy on test-set, falling short of the current state-of-the-ane which has double the parameters count, within 0.5%.
The weights in the OCD hypemetworks are obtained by a diffusion process, and multiple networks can be drawn for each sample x. When employing an ensemble of five classifiers using different initialization of the diffusion process noise, the results surpass the current state of the art, yielding 92.00% (4.7% better than the bascline model, and 0. 0.65% better than the current state of the art).
Image Synthesis We further tested our method on the image generation task of novel view syn- thesis, using a NeRF architecture (Mildenhall et al.) 202I) and the Wyynthetic-Blendee dataset. The TiRy-NeRF architecture (Murhyy 2020) employs an MLP network consisting of three fully- connected layers. The input is 3D ray 5D coordinate (spatial location and viewing direction). The output is the corresponding emitted radiance. For each view, a batch of 4096 rays is computed, from which the interpolated image is synthesized.

---

Under review as a conference paper at ICLR 2023
Table 1: Performance classification tasks. CE=Cross Entropy
MNIST (LeNet5)
CIFARIO (GoogleNet)
Method
Test-CE (4)
Accuracy %(()
Test-CE (4)
Accuracy %(1)
Base network
0.080 + 0.009
99.2 + 0.1
0.085 + 0.01
92.85 + 0.40
Overfitting on test
0.002 0.0001
100
0.075 + 0.005
100
Overfitting test (All Layers)
0.002 + 0.0001
100
0.073 + 0.003
100
OCD nearest neighbor ablation
0.073 = 0.010
19.
0.082 + 0.02
93.03 + 0.40
OCD scaling ablation
0.069 = 0.010
9.1
0.084 0.02
93.01 0.33
OCD alternative layer ablation
0.078 + 0.010
0.1
0.084 : 0.01
92.96 + 0.27
OCD (ours)
0.010 = 0.006
09.7=0.1
0.080 + 0.01
93.68 = 0.38
Table 2: Classification accuracy for TinyImageNet dataset.
Method
Test-CE (4)
Accuracy (())
Swin L/4 Liu et al.
2021b)
NA
91.35
DeiT-B/16-D Touvron et al.
2022
0.71
87.29
DeiT-B/16-D + OCD
0.65
90.80
DeiT-B/16-D + OCD, ensemble of five
0.65
92.00
Table 3: Performance (MSE+SD, lower is better) for the TinyNeRF network.
Method
Lego
Hotdog
Drums
Base model
0.076 0.004
0.063 + 0.007
0.068 + 0.006
Overfitting on test
0.043
0.005
0.032 + 0.005
0.049 0.003
OCD no scaling ablation
0.070
0.008
0.060 + 0.005
0.064 + 0.008
OCD (ours)
0.052 0.006
0.047 0.004
0.057 + 0.006
We experimented with three objects from the dataset: Lego, Hotdog, and Drums. For each object different TinyNeRF base model is trained over the corresponding training set. A single overfitting example is produced by considering batch of 4096 rays from the same viewpoint.
Based on the data in appendix C the first layer is selected. We, therefore, changed the layer-input encoder. E, such that the input image is first encoded by the VGG-11 encoder of [Simonyan & Zisserman (2014) (pretrained over MekggNect-lkk, followed by a fully-connected layer, to match the dimensions of UNet channels. The encoders Eus E, simple fully-connected layers, with dimensions to match the number of channels in the UNet (128).
As seen in Tab. 3 our method improves the MSE by 31% the Lego model, by 25% for Hotdog, and 16% for Drums. Without input-dependent scaling, the performance is much closer to the base network than that of our complete method. Sample test views are shown in Fig. Fig-2 and in Appendix D Evidently, our method improves both image sharpness and color palette, bringing the synthesized image closer the one obtained by overfitting the test image.
Tabular Data Gorishniy et al. 22021) have extensively benchmarked various architectures and base network Layers). We were
tabular datasets. We use their simple MLP architecture unable to reproduce the reported transformer, since the hyperparameters are not provided, and our resources did not allow us to run neural architecture search, as Gorishniy et (202T) did. We run on two of the benchmarks listed: California Housing Kelley Pace & Barry ( (1997) (CA), which is the first listed and has the least number of samples, and Microsoft LETOR4.0(M)) (Qin Liu 2013).
which is the last listed and has the largest number of samples.
Fig. 3(d) presents the layer selection criterion, with the first layer chosen for both datasets. As can seen in Tab. 4 for CA the base MLP model falls behind ResNet. Applying method, the simple architecture achieves better results. For MI when applying our method, the simple baseline achieves record MSE of 0.743, surpassing the current best record this dataset, which is 0.745 (Popov) ett al.] 2020). The ablation that removes input- dependet scaling degrades the performance of the base network, emphasizing the importance of accurate scaling per sample.
7

---

Under review as a conference paper at ICLR 2023
Table 4: Tabular benchmarks by Gorishniy et al. (202I). Table 5: Performance of Gated-LSSTM
MSE + SD, lower is better.
with the Hungarian loss on Libri5Mix.
Method
CA
MI
Method
SI-SDRi[dB] (t)
MLP
0.4990 + 0.0030
0.7470 + .0004
Dovrat et al. (2021)
12.7 0.1
ResNet
0.4860 = 0.0030
0.7480 = .0003
Lutati al. (2022)
13.2 + 0.2
Overfit MLP on test
0.4750 = .0020
0.7410 + .0003
Overht on test
13.5 + 0.1
OCD + MLP no scale
0.5000 = .0030
0.7490 + .0006
OCD no scale
12.8 - 0.3
OCD + MLP (ours)
0.4800 + .0020 0.7430 + .0004
OCD (ours)
13.4 0.1
Table Classification accuracy for the NLP task of few-shot classification with 8 samples per class.
Method
SST-5
AmazonCF
CR
Emotion
EnronSpam
Average
T-few 3B LLu et al. 2022
55.0
19.0
92.1
57.4
93.1
63.3
SetFit (Tunstall et al. 2022
43.6
40.3
88.5
48.8
90.1
62.2
SetFit FOCD
47.8
41.0
90.5
50.2
92.2
64.3
Ensemble of SetFit + OCD
48.6
41.2
91.2
50.5
92.7
64.8
Table 7: The Cross-Entropy loss for the NLP task of few-shot classification with 8 samples per class using SetFIT + OCD, comparing one instance the mean network obtained with 5 instances.
Method
SST-5
AmazonCF
CR
Emotion
EnronSpam
One instance
0.45 + 0.19
0.65 + 0.30
0.41 + 0.11
0.63 = 0.31
0.39 + 0.20
Mean network of 5 instances
0.42 + 0.20
0.61 + 0.25
0.39 + 0.11
0.59 = 0.32
0.38 + 0.19
Speech Separation To tackle the task of single microphone speech separation of multiple speak- ers. Nachmani et al. (2020) introduce the Gated-LSTM architecture with MulCat block and Dovrat et al. (2021) introduced a permutation permutation-invariint loss based on the Hungarian matching algorithm, using the same architecture. Lutati et al. (202)) further improved results for this architecture, by employing an iterative method based on a theoretical upper bound, achieving state-of-the-ee art results.
The same backbone and Hungarian meetarian-meehod method loss are used in our experiments, which run on the Libri5Mix dataset without augmentations, measuring the SI-SDRi score. The selected layer was the
the projection layer of the last MulCat block (appendix C). The output of the Gated-L.STM separated sounds, and encode it, we apply the audio encoding described in Sec. 4.3 to each output channel separately and concatenate before applying the linear projection to 128
As can seen in Tab. 5 applying our diffusion model over the Gated-LSTM model, we achieve 13.4dB, surpassing current of-te-of-the-ar results and approaching the results obtained by overfit- ting on the test data. The ablation that removes input-dependens scaling is much closer in perfor- mance to the base network than to our complete method.
Prompt-free Few-Shot NLP Classification Recent few-shot methods have achieved impressive results in label-scarce settings. However, they are difficult to employ since they require language models with billions of parameters and hand-crafted prompts. Very recently, Tunstall al. (0022) introduced a lean architecture, named SetFit, for finetuning a pretrained sentence transformer over small dataset (8 examples per class), and then employing logistic regression over the corresponding embedding. In our experiments, we apply OCD to the last linear layer of the SetFIT sentence transformer. The U-Net has 64 channels, with 5 downsample layers. The size of the last linear layer is 768+768. (xx) E R68821 is the embedding of the Sentence Transformer.
As can be seen Tab. Jhb.Gussh [dssigg OCD and this lean architecture (with 110M parameters) outperforms, on average, the state-of-the-art model of Liu et al.) (2022). which has 3B parameters. The mean performance across the datasets is improved by 1.0% over the state-of-the-art and by 2.1% over the SetFIT model we build upon. Recall that since the weights in the OCD hypereeworks are obtained by a diffusion process, one can draw multiple networks for each sample x. The variability that arises from the random initialization of the diffusion process allows us use an ensemble. As can be in Tab. [ when applying an ensemble of five inferences, the results of OCD further improve by 0.4% (on average) gap of 1.5% over the tftte-of-thearrt.
8

---

Under review as a conference paper at ICLR 2023
(b)
Figure 2:
Sample TinyNeRF results (Lego model). More results can be found in Appendix D (a)
Base model on a test view. (b) Same test view, overfitted using the ground truth (c) OCD (ours).
Training(hrs]
Inference [ms]
Table 8: Runtime on low-end Nvidia RTX2060GPU. Training includes the base Architecture Base Overfit Diffusion Base
OCD
model training, the collection of data for LeNet5
0.5
1.0
6.0
15
288
OCD by overfitting each sample in the train- ing set, and the training of the diffusion model. The main overhead between the base model and the OCD one during inference is a result of running the UNet the diffusion process for ten iterations.
GoogleNet
2.5
1.0
5.0
20
298
TinyNeRF
1.1
0.2
5.2
230
510
MLL
1.2
0.1
6.2
75
355
Gated-LSTM
30
2.5
6.8
450
730
SetFit
0.2
0.1
5.1
125
401
Interestingly, when testing the average network weights of the 5 networks, the accuracy is virtually identical the one obtained with a single OCD network (omitted from the table). However, there is some reduction in Cross-Entropy for the mean network (Tab. 7
Runtime Tab. lists the measured runtime with and without OCD for both training and inference.
Since overfitting each sample includes only 3 gradient descent steps over a single layer, most of the data collection results are shorter than the training time of the base model, with the exception of LeNet5 training that used a batch size of 16 versus batch size of I in the overfitting stage. Training the diffusion model is slower in most experiments than training the base model, except for the speech model, in which training the base model is lengthy. The inference overhead is almost constant across the six models since it consists of running the UNet ég ten times. In absolute terms, the overhead incurred by OCD, even very modest GPU, is only quarter of second, while improving the results substantially. We note that the same ég is used for all experiments, padding the actual weight matrix, which induces an unnecessary computational cost the case of small networks.
CONCLUSIONS
We present what is, as far we can ascertain, the first diffusion-baased hypernetwork and show that learning the scale independently is required for best performance. The hypernetwork is studied in the specific context local learning, in which a dynamic model is conditioned on single input sample. This, too, seems to be a novel contribution. Using the diffusion architecture for the local learning task is justified by a first-order analysis of the change network's weights when performing limited finetuning.
The training samples for the hypernetwork are collected by finetuning a model on each specific sample from the training set used by the base model. By construction, this is only slightly more demanding than fitting the base model on the training set. More limiting is the size of the output of the hypernetwork, which for modern networks can be larger than the output dimensions of other diffusion processes. We, therefore, focus on a single layer, which selected as the one that is most affected by weight perturbations. We extensively tested our method, tackling very diverse set of tasks, using the same set of hyperparameters. We are yet to find a single dataset or architecture on which OCD method does not significantly improve the results of the baseline architecture.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ethem Alpaydin and Michael Jordan. Local linear perceptrons for classification. IEEE Transac- tions on Neural Networks, 7(3):78--77 794, 1996.
Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior Wolf. Segdiff:: Image segmentation with diffusion probabilistic models. arXiv preprint arXiv:2112.00390, 2021.
Luca Bertinetto, João F Henriques, Jack Valmadre, Philip Torr, and Andrea Vedaldi. Learning feed- forward one-shot learners. In Advances in Neural Information Processing Systems, pp. 523-531, 2016.
Léon Bottou and Vladimir Vapnik. Local learning algorithms. Neural Computation, 4(6):888-900, 1992. doi: 10.1162/neco. 1992.4.6.888.
Andrew Brock, Theo Lim, J.M. Ritchie, and Nick Weston. SMASH: One-shot model architecture search through hypernetorrk.. In International Conference on Learning Representations, 2018. URL htttps:://pppeeeeeeeeeeeeeeeeeee
net forum? id=rrddeChrsssss
Nanxin Chen, Yu Zhang, Heiga Zen, Ron Weiss, Mohammad Norouzi, and William Chan. Wave- grad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009. 00713, 2020.
Jooyoung Choi, Sungwon Kim, Yonghyun Jeong, Youngjune Gwon, and Sungroh Yoon. Ilvr: Con- ditioning method for denoising diffusion probabilistic models. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14347-14356. IEEE, 2021.
Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. arXiv preprint arXiv:2/05.05233, 2021a
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021b.
Shaked Dovrat, Eliya Nachmani, and Lior Wolf. Many-speakers single channel speech separation with optimal permutation training. In INTERSPEECH, 2021.
Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932-18843 2021.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 00000, 2016.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Yuan Yao, Ao Zhang, Liang Zhang, al. Pre-trained models: Past, present and future. A/ Open, 2:225-250, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840 68511 2020.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47):1-33, 2022.
Siddhant M. Jayakumar al. Multiplicative interactions and where to find them. In International Conference on Learning Representations, 2020.
R. Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics Probability Let- ters, 33(3):291-29) 1997. ISSN 0167-7152. doi: https//do..ogglll.00066661777773
00140-X. URL https: http:///wwwwwe.. www.:/sccencee.ee www.eecwweeeeeccoooooeeeoc..eeee httpa//ceeeee.etccococcecceeaaaa
S01 001677152960011
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.14/2.680 2014.
10

---

Under review as a conference paper at ICLR 2023
Benjamin Klein, Lior Wolf, and Yehuda Afek. A dynamic convolutional layer for short range weather prediction. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 4840-4848, 2015.
Bobby Kleinberg, Yuanzhi Li, and Yang Yuan. An alternative view: When does sgd escape local minima? In International Conference on Machine Learning, pp. 2698-2707. PMLR, 2018.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient- learning applied document recog- nition. Proceedings of the IEEE, 86(11):2278-2) 1998. doi: 10. 109/5.76679.
Yann
LeCun and Corinna Cortes. 2010.
MNIST handwritten digit database.
URL http://yann. lecun. com/exdb/
mnist/
Alon Levkovitch, Eliya Nachmani, and Lior Wolf. Zero-shot voice conditioning for denoising dif- fusion its models. In INTERSPEECH, 2022.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205.08 05638, 2022.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. arXiv preprint arX5v:2105.02446, 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021b.
Jonathan Lorraine and David Duvenaud. Stochastic hyperparameter optimization through hypemett works. arXiv preprint arXiv: /802.094/9 2018.
Eric Luhman and Troy Luhman. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv:2101.02388, 2021.
Shahar Lutati and Lior Wolf. Hyperhypernetwoork for the design of antenna arrays. In International Conference on Machine Learning, pp. 7214-7223. PMLR, 2021.
Shahar Lutati, Eliya Nachmani, and Lior Wolf. Sepit: Approaching a single channel speech separa- tion bound. In INTERSPEECH, 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications the ACM, 65(1)999-106, 2021.
Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher Manning, Fast model editing at scale. arXiv preprint arXiv:2110.11309, 2021.
Krishna Murthy, nerf-pytorch.
https://iitubb.com/kkiiiii blob/master/tiny_nerr py 2020.
pytorchh/
Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple
speakers. In International Conference on Machine Learning, pp. 7164-7175. PMLR, 2020.
Sergei Popov, Stanislav Morozov, and Artem Babenko. Neural oblivious decision ensembles for
deep learning on tabular data. In International Conference on Learning Representations, 2020. URL htll.httts:://peneee. htt.://eweeeeeeeeeooooooooooeo/e

---

Under review as a conference paper at ICLR 2023
Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad- tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning. pp. 8599 8608. PMLR, 2021.
Tao Qin and Tie- Yan Liu. Introducing letor 4.0 datasets. arXiv preprint
2013.
Gernot Riegler, Samuel Schulter, Matthias Ruther, and Horst Bischof. Conditioned regression mod- els for non-blind single image super-resolution. Proceedings of the IEEE International Con- ference on Computer Vision, pp. 522-530, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp.
Springer. 2015.
M. Rosenblatt. Random Processes. Springer New York, 1974. doi: 10.1077/978-1-6612-9882-- 4612 9852-6
URL http:httts//dooorggg///////
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al. Photorealistic exx-to-image diffusion models with deep language understanding. arXiv preprint 1185 2022.
Robin San-Roman, Eliya Nachmani, and Lior Wolf. Noise estimation for generative diffusion mod- els. arXiv preprini arXiv: 201..1442222000 2021.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypemetworks.. In International Conference on Machine Learning, pp. 9489-99999 990229959..01 PMLR, 2021.
B. W. Silverman. Density Estimation for Statistics and Data Analysis. Chapman & Hall, London, 1986.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1eeXiv::409.14 2014.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing. pp.
PMLR, 2015.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit ili: Revenge of the vit. arXiv preprint arXiv:2204.0711 2022.
Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. Efficient few-shot learning without prompts. arXiv preprint arX9v: I055, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe Continual learning with hypernetworks. In International Conference on Learning Representations, 2020.
Daniel Watson, Jonathan Ho, Mohammad Norouzi, and William Chan. Learning to efficiently sam- ple from diffusion probabilistic models. arXiv preprint urXiv:2106.03802 2021.
Lior Wolf, Tal Hassner, and Yaniv Taigman. The one-shot similarity kernel. In 2009 IEEE 12th International Conference Computer Vision, pp. 897-902. IEFE, 2009.
12

---

Under review as a conference paper at ICLR 2023
C. Zhang, M. Ren, and R. Urtasun. Graph hypemeeworrss for neural architecture search. In Int. Conf. on Learning Representations. 2019. URL hattoss//penneeiiiww..tettttooo EkgWDoA9FX
Hao Zhang, Alexander C Berg, Michael Maire, and Jitendra Malik. Svm-knn: Discriminative near- est neighbor classification for visual category recognition. In CVPR, volume 2, pp. 2126-2136. IEEE, 2006.
13

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Inference Algorithm.
Input: & input sample, the parameters of the base network, 622 diffusion network, T diffusion steps.
Output: 9u ((x) estimated normalized (0, 8) for associates with r.
1: 1-T T
2: N 1)
3: while 2 do
4: B. 10 (r- 10 (t-1) (t-1) at Prs ar = Br I-ai- Be
I
I-
Qr VI-ar (()).)
os
5: 2-11
+ 1 1 / 3.
Vor
6: f=t-1
7: end while
8: return So
A PROOF OF LEMMA
Lemma 1. Since is locally convex, follows that the distribution PH,(e) in the domain @ ob- tained when applying H1 to samples ~ Pxxy takes the following form:
PH((s) (H, (s) & s)
(11)
where (s) (X Q) (X x )) is the pseudoinverse of the tensor H1 Hi(s) (first we compute H1 at given 8, and then compute the tensor pseudoinverse).
Proof. For any random variable r, and mapping function M that is monotonic and with countable number of zeros, the following holds (Rosenblati (1974))
PM(()) = (P (r)
(12)
dr
Where M-1 is the pseudoinverse of M. From the local convexity of H, we have that its' Hessian nonnegative.
H is locally convex - v2 6s) 2 0, võs + 0 Recall that the Hessian of H the gradient of H1, thus its gradient non-negative, leading to monotonicity.
(13)
Hi(s) = VH(s) -
(14)
(s) 2
(15)
In addition, since H1 is a linear operator with finite size, it must have finite dimension of null space.
Ker(H,) < dim(Hz),
(16)
where Ker is the tensor null space, dim is the tensor dimensions. This fulfills the second condition of the mapping function having countable number of zeros.
B THE INFERENCE TIME ALGORITHM
The steps of the inference algorithm are listed om Alg.
C LAYER SELECTION PLOTS
Fig. 3 depicts the layer selection criterion for various experiments.
D TINYNERF SAMPLE RESULTS
Fig. 4 presents sample results for the reconstruction of the Lego Model. Fig. 5 presents results for the Hotdog model. Fig. presents sample test views of the Drums model.
14

---

Under review as a conference paper at ICLR 2023
Larked' Selection Enttertoe
GoogleNet Sclection Crisnion
3.4
Mw Mn
M w w
Lage
(a)
(b)
MubCCe Foe an in Seleccer Crisoren
440
140
(d)
Figure 3: Layer Selection Criterion for different experiments. (a) For LeNet5 on MNIST, the next to last Fully-Connected layer is selected since it has the maximal entropy. (b) For GoogleNet on CIFARIO, the last Fully-Connected layer selected. (c) For TinyNeRF (three datasets), the first Fully-Connected layer is selected. (d) For Tabular MLP the first layer is selected. (c) For MulCat the last projection layer selected.
15

---

Under review as a conference paper at ICLR 2023
a
(b)
Figure 4: Sample TinyNeRF results on the Lego model. (a) Base model on a test view. (b) Same
test view, overfitted using the ground truth (c) OCD (ours).
16

---

Under review as a conference paper at ICLR 2023
(
(b)
Figure 5: Sample TinyNeRF results on the Hotdog model. (a) Base model on a test view. (b) Same test view, overfitted using the ground truth (c) OCD (ours).
17

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
Figure 6: Sample TinyNeRF results on the Drums model. (a) Base model on view. (b) Same
test view, overfitted using the ground truth (c) OCD (ours).
18


Reviews:
[{"review_id": "w6wMc5-wOJ5", "reviewer": "Reviewer_Q7pB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a diffusion model which, given a neural network input x, adapts the neural network weights to specialise the network to this single input. Doing so before evaluating on each test example (x, y) leads to better predictions of y. The diffusion model is trained to imitate an SGD procedure which overfits a pretrained model's weights to a single data point.", "summary_of_the_review": "The method is novel and clearly-presented. The results are good, but their significance is a little unclear to me because of the lack of presentation and discussion of computational cost. I will consider raising my score if this is improved.\n\n~~EDIT: Lowering my score; see \"Additional related work\" comment below.~~", "strength_and_weaknesses": "Strengths:\n - Strong experimental results on a wide variety of problems and neural architectures (image classification, NeRF, tabular data, speech separation). State-of-the-art on some of these.\n - Well-motivated and experimentally-justified innovations include predicting the scale of the weight change separately, and changing weights for only a single network layer.\n - Apart from some minor points listed below, the writing is clear and a pleasure to read.\n\nWeaknesses:\n- There is a lack of discussion of computational cost. Relative to the baselines they consider (mostly just the same networks as for OCD, but without weight adaptation), OCD is presumably considerably more costly at both training time and test time. Since at training time, it requires both performing many \"finetuning\" SGD steps, and training the diffusion model. And the test-time weight adaptations are probably expensive relative to the cost of a single forward pass. Ideally there would be a comparison where the baselines were given larger architectures, and trained for longer, etc., to match OCD for computational cost. I appreciate that this paper can still be a valuable contribution without \"winning\" in such a comparison but this would provide very helpful context. Failing this, the differences in training time and test time should at least be made explicit and discussed. I am willing to raise my score if you address this point.\n\nMinor:\n- Section 3, \"$s$ [...] is near either a local minima or an inflection point for the training loss of finetuning with sample $s$\". I don't think this is necessarily true. E.g. in image classification the loss would be minimised when the \"correct\" output logit is driven to infinity and so it seems that the minimum may be at infinity (or more precisely, there may not be any minimum) when training on a single data point.\n- Equation 10. It is not clear what $x$ refers to here, or what the \"data distribution\" $q(x_0)$  refers to. The variable $x$ previously referred to inputs to the base model (i.e. in Eq. 1) but I think this might now be referring to a variable in the space of the network weights.\n- Section 3, \"without concrete quantization\" - probably not important, but I cannot tell what is meant by this phrase", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qKLtYbxe0M", "reviewer": "Reviewer_Nu1A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use diffusion model to adapt selected layers of a trained model conditioned on an input sample. Authors conducted experiments onclassification, 3D construction, tablular data and speech separation. ", "summary_of_the_review": "This paper proposes to use diffusion model to generate/adapt certain weights of a trained model given a single input. The motivation and the training/inference pipeline could be better structured and explained. ", "strength_and_weaknesses": "Stength\nUsing diffusion model for conditional parameter generation is novel.\n\nWeakness\n1. I find the paper a bit difficult to follow, it could use some rewriting/clearer plots to emphasize the motivation and approach.\n2. I would like to see more details on the experiments: what's the cost of training and what's the overhead for inference. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "mo_SYHH-nm", "reviewer": "Reviewer_A1Ss", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a hypernetwork model where the weights are conditioned on input sample $x$ and are trained to match the model after finetuning. The hypernetwork generator is a conditional diffusion model, conditioned on $x$ (and it's features), and operates in the space of network weights $\\theta$.", "summary_of_the_review": "I think it is an interesting paper, although the presentation is not clear enough. I am willing to amend my scores if my concerns are addressed.", "strength_and_weaknesses": "## strengths\n- The paper presents an interesting idea of using conditional diffusion model as a hypernetwork generator.\n- The experiments covers a wide range of applications including image classification, 3-D reconstruction, tabular data and speech separation.\n## weakness and questions\n- The paper is well-written for the most part but some parts can be improved to make the manuscript clearer. For example:\n\t- In Eq. 1, $\\theta$ is not formally defined before introduction. Is $\\theta$ the \"base network parameters\"? Does this mean the initialized weights or the pretrained weights?\n\t- What are $\\Omega$'s in Sec 4.1? Are they of the same dimensionality as $\\theta$?\n\t- The presentation of figures can be improved. For example, Figure 2 is a little hard to read, the lines are very thin and captions/ticks/legends are very tiny.\n- UNet: What's the dimensionality of $\\Omega$? If we choose a conv layer then should we use a 4-D UNet? In the case shown in Figure 1, is it true that the weight matrix is treated/reshaped as a 2-D \"image\" and we operate a 2-D UNet on it? If so, any explanation for why the inductive bias introduced by UNet is useful here?\n- The experiments on image classification are a little bit limited. It would be better to test the algorithm on larger networks and larger datasets e.g. TinyImageNet.\n- At the inference phase, do we get a single point estimate of $\\theta$? Is it possible to employ an ensemble here? With the proposed OCD formulation, it might be interesting to integrate/compare the Bayesian ensemble here.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VHK0hUzaxe", "reviewer": "Reviewer_wwSB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an approach to use diffusion models as a manner to parameterize a meta-learning algorithm, where the network perbutation prediction by the diffusion process adapts a base model to a separate task.", "summary_of_the_review": "I think the paper is a bit preliminary -- a lot of work needs to go into the writing to clarify the paper and results could be significantly improved.", "strength_and_weaknesses": "# Strengths\n\n**Novelty.** The problem studied by the paper is interesting and is new to my knowledge\n\n# Weaknesses\n\n**Clarity.** The paper is somewhat poorly written -- the introduction does not motivate the problem to be studied, related works are incorrectly referenced and the method section is confusing and does not adequately describe what is happening in the approach. What is the train data used for training the method? How is it obtained? How is the diffusion process used to obtain the data necessary to overfit data?\n\n**Results.** The results provided the paper are somewhat toy -- the NeRF look very poorly fit for example and the paper does not appear to be empirically rigorous. The paper does not explain why the approach only works on a single layer in the network. Also I could not completely understand the different baselines presented in each table. Why is each baseline natural?\n\n**Significance.** The results provided in the paper are not significant in its current state. The proposed approach seems to substantially underperform existing baselines -- the paper could be improved substantially by illustrating an concrete application of the approach on a real world setting that outperforms existing baselines.\n\n==============\nPost Rebuttal Update\n\nI have decided to downgrade my score (see the discussion below for some issues raised). The results for instance on TinyNeRF seem incorrect (even with 32 ray samples the results are still much sharper than those shown in the paper) and generally seem toy. This problem setup seems very similar to existing meta-learning techniques, for instance a meta-learned loss https://arxiv.org/abs/1906.05374 can be used to adapt the model (where we can not input a label y into the learned loss function). Experiments are run only with very limited data on very small architectures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject