Learning to Predict Parameter for Unseen Data
Authors: Shiye Wang
Kaituo Feng
Changsheng Li
Ye Yuan
Guoren Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Parameter Prediction
Training paradigm

Abstract: Typical deep learning models depend heavily on large amounts of training data and resort to an iterative optimization algorithm (e.g., SGD or Adam) for learning network parameters, which makes the training process very time- and resource-intensive. In this paper, we propose a new training paradigm and formulate network parameter training into a prediction task:  given a network architecture, we observe there exists correlations between datasets and their corresponding optimal network parameters, and explore if we can learn a hyper-mapping between them to capture the relations, such that we can directly predict the parameters of the network for a new dataset never seen during the training phase. To do this, we put forward a new hypernetwork with the purpose of building a mapping between datasets and their corresponding network parameters, and then predict parameters for unseen data with only a single forward propagation of the hypernetwork. At its heart, our model benefits from a series of GRU sharing weights to capture the dependencies of parameters among different layers in the network. Extensive experimental studies are performed and experimental results validate our proposed method achieves surprisingly good efficacy. For instance, it takes 119 GPU seconds to train ResNet-18 using Adam from scratch and obtain a top-1 accuracy of 74.56%, while our method costs only 0.5 GPU seconds to predict the network parameters of ResNet-18 achieving comparable performance (73.33%), more than 200 times faster than the traditional training paradigm.

Under review as a conference paper at ICLR 2023
LEARNING TO PREDICT PARAMETER FOR UNSEEN DATA
Anonymous authors Paper under double-blind review
ABSTRACT
Typical deep learning models depend heavily on large amounts of training data and resort to an iterative optimization algorithm (e-g., SGD or Adam) for learning network parameters, which makes the training process very time- and resource- intensive. In this paper, we propose a new training paradigm and formulate net- work parameter training into a prediction task: given a network architecture, we observe there exists correlations between datasets and their corresponding optimal network parameters, and explore if we can learn myper-mapping between them to capture the relations, such that we can directly predict the parameters of the network for put forward a new hypernetwork with the purpose of building a mapping between datasets and their corresponding network parameters, and then predict parameters for unseen data with only a single forward propagation of the hypernetwork. At its heart, our model benefits from a series of GRU sharing weights capture the dependencies of parameters among different layers in the network. Extensive ex- perimental studies are performed and experimental results validate our proposed method achieves surprisingly good efficacy. For instance, it takes 119 GPU sec- onds to train ResNet- 18 using Adam from scratch and the network obtains top-1 1 accuracy of 74.56%, while our method costs only 0.5 GPU seconds to predict the network parameters of ResNet-18 18 achieving comparable performance (73.33%), more than 200 times faster than the traditional training paradigm.
dataset never during the training phase. To do this, we
INTRODUCTION
Deep learning has yielded superior performance in a variety of fields in the past decade, such as computer vision (Keddall & Gal 2017). natural language processing (DBL). reinforcement learning (Zheng et al. 2018 8018:1 Fujimoto et al. 2018), etc. One of the keys to success for deep learning stems from huge amounts of training data used to learn a deep network. In order to optimize the network, the traditional training paradigm takes advantage of an iterative optimization algorithm (e.g.. SGD) to train the model a mini-batch manner, leading to huge time and resource consumption. For example, when training RestNet- 101 (He et al. 2016) the ImageNet (Deng et al.) 2009) dataset, often takes several days or weeks for the model to be well optimized with GPU involved. Thus, how to accelerate the training process of the network is emergent topic in deep learning.
Nowadays, many methods for accelerating training deep neural networks have been proposed (Kingma & Ba 2015 Joffe & Szegedy 2015: Chen al. (018). The representative works include optimization based techniques by improving the stochastic gradient descent (Kingma Ba 2015
Yong et al.] 22020] Anil et al. (220), normalization based techniques (loffe & SSzeedyy [015 Sali-
mans & Kingma 2016: Ba et al. 2016), parallel training techniques (Chen et al. 2018 Kim et al.,
2019), et. Although these methods have showed promising potential to speed up the training of the network, they still follow the traditional iterative-based training paradigm.
In this paper, we investigate new training paradigm for deep neural networks. In contrast to previous works accelerating the training of the network, we formulate the parameter training prob- lem into a prediction task: given a network architecture, we attempt learn a hyper-mapping be- tween datasets and their corresponding optimal network parameters, and then leverage the hyper- mapping to directly predict the network parameters for new dataset unseen during training. A basic

---

Under review as a conference paper at ICLR 2023
assumption behind the above prediction task is that there exists correlations between datasets and their corresponding parameters of given network. In order to demonstrate the rational- ity of this assumption, ing experiment: for a dataset, we first randomly sample 3000 images to train 3-layer convolu- tional neural network until convergence. Then we conduct the average pooling operation the original inputs as vector representation of the training data. We repeat the above experiment 1000 times, and thus obtain 1000 groups of rep- resentations and the corresponding network pa- rameters. Finally, relation Analysis (CCA) (Weenink, 2003) to evaluate the correlations between training data and the network parameters by the above 1000 groups of data. Figure [ shows the results, which illustrates there are indeed correlations between training datasets and their network pa- rameters for given network architecture.
Correlation Coefficient
0.8-
random
learned
0.7-
perform the follow-
0.6-
0.5
0.4
0.3 Fashion-MNIST
CIFAR-100 Mini-ImageNet
Datasets
Figure 1: Correlation coefficients between training datasets and the network parameters on the Pashion-MNIST (Xiao al. 2017). CIFAR-100 (Krizhevsky et al.) (2009). Mini-
utilize Canonical Cor-
ImageNet (Vinyals et al. (2016) datasets, respec- tively. 'learned' depicts correlations between train- ing datasets and the corresponding optimal net- work parameters. 'random' denotes correlations between training datasets and the network param- eters selected randomly from 1000 groups.
In light of this, we propose new hypernet- work, called PudNet, to learn hyper-mapping between datasets and network parameters. Specifi- cally, PudNet first summarizes the characters datasets by compressing them into different vectors as their sketch. Then, PudNet extends the traditional hypernetwork (Ha et al., 2017) to predict net- work parameters of different layers based on these vectors. Considering that parameters among different layers should be dependent, into PudNet to capture the relations among them, so as to improve the performance of PudNet. Fi- nally, it is worth noting that for training PudNet, it is infeasible if prepare thousands of datasets and train networks on these datasets to obtain the corresponding optimal parameters respectively. Instead, we adopt meta-learning based approach (Finn et al., 2017) train the hypernetwork.
incorporate Gate Recurrent Unit (GRU) (Cho et al. 2014)
Extensive experiments demonstrate the surprising effectiveness of our PudNet. For example, takes around 54, 119, 140 GPU seconds train ResNet-18 using Adam from scratch and ob- tain top-1 accuracies of 99.91 % 74.56%. 71.84% on the Fashion-MNIST (Xiao et al.] (017), CIFAR-100 (Krizhevsky et al., 2009 Mini-ImageNet (Vinyals et al. 2016), respectively. While our method costs only around 0.5 GPU seconds to predict the parameters of ResNet- 18 and still achieves 96.24%, 73.33%, 71.57% top-1 accuracies on the three datasets respectively, least 100 times faster than the traditional training paradigm.
Our contributions are summarized as follows: 1) We find there are correlations between datasets and their corresponding parameters of a given network, and propose a general training paradigm for deep networks by formulating network training into a parameter prediction task. 2) We extend hypernet- work to learn the correlations between datasets and their corresponding network parameters, such that we can directly generate parameters for arbitrary unseen data with only a single forward prop- agation. 3) Our method achieves surprisingly good performance for unseen data, which is expected to motivate more researchers to explore along with this research direction.
RELATED WORK
2.1 HYPERNETWORKS
The original goal of hypernetwork proposed in (Ha et al., 2017) is to decrease the number of training parameters, by training a hypernetwork with a smaller size to generate the parameters another network with a larger size a fixed dataset. Because of its promising performance, hypernet-
work has been gradually applied to various tasks (Krueger et al. 2017 Zhang et al.) 2019 von
Oswald al. 2020 Li et al. 2020 Shamsian et al. 2021). von Oswald al. (2020) proposes task conditioned hypernetwork to overcome catastrophic forgetting in continual learning. It learss an embedding for each task and utilizes the task embedding to generate corresponding parameters
2

---

Under review as a conference paper at ICLR 2023
for each task. Bayesian hypernetwork (Krueger et al.) 2017) is proposed to approximate Bayesian inference in neural networks. GHN-2 proposed in Knyazev et al. (202)) attempts to build a map- ping between the network architectures and network parameters, where the dataset is always fixed. GHN-2 leverages graph neural networks to model the information of the network architectures for learning the mapping. Our work is orthogonal to GHN-2, since aim to build mapping between the datasets and the network parameters, given a network architecture. Moreover, we extend the tra- ditional hypernetwork by incorporating GRU to capture the relations among parameters of differnt layers and develop a meta-learning based manner to optimize the hypernetwork.
2.2 ACCELERATION OF NETWORK TRAINING
Many works have been proposed to speed up the training process of deep neural networks in the past decade, including optimization based methods (Kingma & Ba 2015; Yong al. 2020 Anil et al.) 2020), normalization based methods (loffe & Szegedy 221g Ba et al. 2016), paralle lel training meth- ods (Chen et al. [2018 Kim et al. 2019), etc. Optimization based methods mainly aim improve the stochastic gradient descent. For instance, Yong et al. (2020) proposes a gradient centralization method that centralizes gradient vectors improve the Lipschitzness of the loss function. Nor- malization based methods intend to propose good normalization methods to speed up the training process. The representative work is the batch normalization (Loffe & Szegedy (2015) that can make the optimization landscape smooth and lead to fast convergence (Santurkar et al. 2018). Parallel training methods usually stack multiple hardwares to conduct parallel training, which can reduce training time by dispersing calculation amounts to distributed devices. However, these methods still follow the traditional iterative based training paradigm. Different from them, we attempt to explore new training paradigm, and transform the network training problem of into prediction task.
3 PROPOSED METHOD
In this section, we will introduce our PudNet in detail. For better illustration, we first give prelimi- naries and our problem formulation, and then elaborate the details of our method.
3.1 PRELIMINARIES
Notation We denote He our hypernetwork parameterized by 8. Let Diraan {D, be the set of training datasets, where D; is the 8th dataset and N is the number of training datasets. Each sample 2j D; has label yj E C,r, where C is the class set of D,- We use C U C, to denote the whole label set of training datasets. Similarly, we define Dieat as the set of unseen datasets used for testing and Cte as the set containing all labels in Diest
Problem Formulation In contrast to traditional iterative- based training paradigm, we attempt to explore a new training paradigm, and formulate the network training into parameter prediction task. To this end, we propose the following objective function:
arg min B
H,),
(1)
where F(D,, Ho) denotes a forward propagation of our hypernetwork Hg- The input of the forward propagation is the dataset D, and its output is the predicted parameters of network Ω by Hg. Note that the architecture of Ω is always fixed during training and testing, e.g., ResNet-18. This makes sense because we often apply a representative deep model to data of different domains. Thus, it is obviously meaningful if we can predict the network parameters for unseen data using an identical network architecture. S {M²)) denotes the ground -oound-truth parameter set of network Ω corresponding datasets Derain where M³ is the ground-truth parameters for the dataset D;- L is loss function, measuring the difference between the ground-truth parameters M and the predicted parameters.
The core idea in I is to learn hyper-mapping Ha between datasets Dirain and the network param- eter set M on the basis of our finding that there are correlations between datasets and the network parameters, shown in Figure [ However, it is prohibitive if preparing thousands of datasets D,
3

---

Under review as a conference paper at ICLR 2023
Dataset Compression
Coatext-Aware Parameter Prediction
Datasel
GRL
GRU
GRU
Weigh Grnazato
Woyli Genazioo
Weigh Ganeaaoe
connection
weight application
Arrain
panmeters
Dataset
support
Target Network
Class controld
Figure 2: Overview of our proposed PudNet. PudNet first compresses each dataset into sketch with a fixed size, and then utilizes the hypernetwork to generate parameters of a target network based on the sketch. Finally, PudNet is optimized based support set in a meta-learning based manner.
and training network S on D; to obtain the corresponding ground-trut -ruth parameters M respec- tively. To alleviate this problem, we adopt meta-lcarming based (Vinyals et al. (016) approach to train the hypernetwork Ho, and propose another objective function as:
arg min
Σ
C(a,, FD; Ho),, EEEE
Instead of optimizing Ho by directly matching the predicted parameters F(D;, Ωi Ho) with the ground-truth parameters M. we can adopt typical loss, c.g., cross-entropy, to optimize He. where each dataset D; can be regarded as task in meta-learinn (Vinyals al. 2016). By learning on multiple tasks, the parameter predictor Hg is gradually able to learn to predict perfórmant parameters for training datasets Draain During testing. we can utilize F(D, H₂) to directly predict the parameters for testing dataset D never seen in Dtroin with only a single forward propagation.
3.2 OVERVIEW OF OUR FRAMEWORK
Our goal is to learn a hypernetwork Ha, so as to directly predict the network parameters for an unseen dataset by Hgg However, there remains two issues that not solved: First, the sizes of different D; may be different and the dataset sizes may be large, which makes Hg hard to be trained; Second, there may be correlations among parameters of different layers in network. However, how to capture such context relations among parameters has not been fully explored so far.
To this end, propose novel framework, PudNet, shown in Figuire 2 PudNet first introduces a dataset compression module to compress each dataset D; into a small size sketch 8; E R to summarize the major characteristics of D,, where and m are the size and dimension of the sketch, respectively. Then, our context-aware parameter prediction module takes the sketch 8; as input, and outputs the predicted parameters of the target network, e.g., ResNet-18. At its heart, multiple GRUs sharing weights are constructed to capture the dependencies of parameters among different layers in the network. Finally, PudNet optimized based on a support set veta-learning based manner.
3.3 DATASET COMPRESSION
To solve the issue of different sizes of training datasets, we first compress each dataset into sketch with a fixed size. In recent years, many data compression methods have been proposed, such as matrix sketching (Liberty) 2013 Qian al. 2015), random projection (Sarlos 2006 Liberty al. 2007), etc. In principle, these methods can be applied to our data compression module. For simpli- fication, leverage deep neural network to extract feature vector the representation of each
4

---

Under review as a conference paper at ICLR 2023
sample, and then conduct the average pooling operation to generate final vector the sketch of the dataset. The sketch 8; for the dataset D; can be calculated as:
8;
Σ EyED,
To(ay),
(3)
Di
where (-).) denotes a feature extractor parameterized by o. and the structure of the feature extractor used in the experiments can be found in Appendix A.I [Di] is the size of the dataset D.. The parameter @ is jointly trained with PudNet in an end-to-end fashion. In future work, more efforts could be made to explore more effective solutions summarize the information of dataset, e.g.
using statistic network (Edwards Storkey 2017 to compress datasets.
3.4 CONTEXT-AWALE PARAMETER PREDICTION
After obtaining the sketches for all training datasets, we will feed them into the context-aware pa- rameter prediction module, i.e., our hypernetwork, as shown in F the followings, we will introduce this module in detail.
Capturing Context Relations via GRU Since an input of neural network would sequentially pass forward the layers of this network, the parameters of different layers should be not independent. If ignore the context relations among parameters of different layers, the solution may be sub- optimal. Thus, we utilize GRU (Cho et al. 2014) to capture the context-aware parameter relations as shown in 3 Note that we have two changes compared to conventional GRU: 1) conventional GRU utilizes the randomly initialized hidden state hg. Different from this, we set the dataset sketch embedding as the initial hidden state: hg 8,,
provide dataset information for predicting corresponding structure parameters; 2) In each recurrent step, conventional GRU usually takes the next word as input in natural language process field (Cho al. 2014). Instead of that, we exploit the predicted structure parameters of previous layer as imput, enabling the information of shallower layer parameters memorized in GRU to help the prediction of parameters in the deeper layer. By this way, the dependency relations among parameters of different layers can be well captured.
The following comes formal description:
r: = o W, (t--1,a-1l])
hg 5
ht
GRU
GRU
GRU
o (W, [h--1, a=-1)),
h tanh Wh (re h1-1, a1-1)),
000000
hr (1 =1,)= h:-1 + ze hus
4t-1
a, oW. hc),
Figure 3: An illustration of capturing context re- lations via GRU.
where h, could transfer dataset related infor- mation. is a latent vector encoding the context information. W.. W., W., W, are the learnable parameters. As in GRU, the reset gate F. decides how much information in the hidden state h needs to be reset. h2 is a new memory, which absorbs the information of h--1 and a--1- 2, is an update gate, which regulates how much information in h, to update and how much information in h forget. The context-aware output a, is taken as input to the weight generator predict parameters of the 1-th layer of the target network Ω.
Initial Residual Connection To ensure that the final context-aware output contains least a frac- tion of the initial dataset information, we additionally implement an initial residual connection be- tween dataset sketch embedding s; and a; as:
à = a. x (1-1)
17)
(4)
where 1) is the hyperparameter. After obtaining à,, we put à, into weight generator used to generate the parameters of 52.
Weight Generator Since the target network Ω usually has different sizes and dimensions in dif- ferent layers, we construct the weight generator Iv. for each layer t to transform à, of a fixed dimension to network parameter tensor w. with variable dimensions. Here Do, denotes the weight
5

---

Under review as a conference paper at ICLR 2023
generator of the t-th layer, is the learnable parameters of 9e wy is the predicted parameter of t-th layer in f2. We can derive the predicted parameter of the f-th layer as:
Wt Pv. (â,)
(5)
where 9t, consists one linear layer and two x convolutional layers. More details of the weight generator can be found in Appendix A.I When the parameters w, of each layer are predicted, we
can use these parameters as the final parameters of ?
3.5 OPTIMIZATION OF OUR FRAMEWORK
In this section, we introduce how to optimize our PudNet. In contrast to traditional classification tasks where training data and testing data have the identical label space. In our task, the label spaces between training and testing can be different, even not overlapped. Thus, training a classification head on the training data can not be used to predict labels of testing data. Motivated by several metric learning methods (Chen et al.) 2021 Oreshkin al. 2018), we introduce parameter-free classification method to solve the above issue.
Similar to Chen et al.
2021 we obtain a metric-based category prediction on class ct as: exp(T < F(D,, Q; Ho)), us >)
pvy c br Q, D;; Ho)
(6)
Ee exp(T- < F(D,; S2; H,)), ue >)
where uy is the centroid of class which is the average output of the predicted network Q over samples belonging to class C the support set, as Snell et (017)) <, denotes the cosine similarity of two vector, and 7 is a learnable temperature parameter. [(z;; Ho)) is the output of the target neural network Q based on the input zz--
Then, the parameter-fiee free classification loss can be defined as:
N
L1 41= Σ (, H.)
where Vy is the true label of zj, C is the cross-entropy loss.
To further improve the performance of the model, we introduce an auxiliary task for training our hypemnetorr, by adding full classification head 2,p parameterized by s. The classification head aims to map the output of the target network Z to probabilities of the whole classes C+r from Derain The full classification loss is defined as:
NV
C2
Σ Σ ((D, Ho))) myED;
Our parameter prediction task co-trained with a full classification head is related to curriculum learning (Oreshkin et al. 2018). Since learning varying label space is more challenging than learning on a static one, the full classification problem that maps features to a static label set could be regraded obtain the basic level knowledge before handling harder parameter prediction task. Moreover, to make parameter-free based prediction and full classification based prediction consistent, which is motivated by (Chen et al. 2022] Wu et al. (019). we introduce Kullback-Leibler Divergence loss encourage their predicted probabilities be matched:
simpler curriculum. This easier 'prerequisite' could help the hypemnetwrr to
L3
Σ
Σ
i=1 DDED
where KL is the Kullback-Lebller Divergence. p④y (z,)) and g(v (z)) are the predicted probabilities of z, of parameter-free free based and full classification based methods, respectively. The probabilities of the corresponding classes in p(ylæy) ) are padding with zero to match the dimension of (g)lr,)
Finally, we give the overall multi-task loss as:
Listat Li + L2 L3-
(10)
where D; can be regarded as a task similar to that in meta-learning. By minimizing IO).. our hypernetwork be well trained. For unseen data in testing, utilize hypernetwork to directly predict its network parameters, and use the parameter-free based method for classification. The training procedure of our PudNet is in Appendix A.4

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENT
4.1 DATASET CONSTRUCTION
In the experiment, we construct numerous datasets for evaluating our method based on four datasets:
Fashion-MNIST (Xiao al. 2017), CIFAR-100 (Krizhevsky et al. 2009), Mini-ImageNNe (Vinyals)
et al. 2016), Animals- (uupta & Brown 2022). The constructed datasets are summarized as:
Fashion-set: We randomly select 6 classes from Fashion-MNNIST to construct training datasets and the remaining 4 classes for constructing testing datasets, We construct 2000 groups of datasets from the 6-category training set to train PudNet. To verify PudNet's ability directly generate param- eters, we construct 500 groups of datastes from the 4-category testing set to generate 500 groups of network parameters. For each group of network parameters, we also construct another dataset having identical labels but not overlapped samples with the dataset used for generating parameters, order to test the performance of the predicted network parameters. Each dataset contains 600 randomly sampled images with 2 randomly sampled classes.
CIFAR100-set: We randomly choose 80 classes from CIFAR- 100 for constructing training datasets and 20 classes not overlapped with the above 80 classes construct testing datasets. We sample 100000 groups of datasets for training PudNet. Similar to Fashion-set, we construct 500 groups of datastes to directly generate their network parameters by PudNet, and create another 500 groups of datasets for testing the performance of the predicted parameters. Each datasets consists of 500 images with 5 classes randomly selected.
Similar to CIFAR 100-set, the mini- mini-Imagennt dataset is randomly split into 80 classes for creating training datasets and 20 classes for creating testing datsets. We sample 50000 groups of datasets for training PudNet. Similar to Fashion-set, we use 500 groups of datastes to generate their network parameters, and construct another 500 groups of datasets for testing. There are 500 images with 5 classes selected randomly in each dataset.
ImageNet-set:
CIFAR100 Animals10: To further verify our PudNet, we construct a cross-domain dataset. We use CIFAR I 100-set to construct training datasets, and Animals- 10 for testing datasets. There are 100000 groups of datasets from CIFAR100-set to train PudNet. We randomly split Animals10 into two not overlapped subsets: one is used to generate parameters, and the other for testing.
4.2 BASELINES
We compare method with traditional iterative based training paradigm including training from scratch and one training acceleration method, GC (Yong et al.) 2020). We also take the pretrained model as baseline. In addition, we also compare with meta-learning methods, including Match- Net (Vinyals et al.] 20166, ProtoNet (Snell et al.] 2017). Meta-Baseline (Chen et al.) 2021) Meta- DeepDBC (Xie al., 2022). and MUSML (Jing al. 2022). We use two kinds of architectures as our target network Ω: 3-layer CNN, ConvNet-3 and Resnet-18. To ensure fair comparison, we use all training datasets for training meta-learning methods and the pretrained model. For all experiments, we use ACC (Top-1 Accuracy) metric to evaluate the classification performance.
4.3 IMPLEMENTATION DETAILS
We perform the experiments using GeForce RTX 3090 Ti GPU. We set the learning rate as 0.001. For the target network 3env we set the hyperparameter y as 0.2,0.1,0.3.0.. for Fashion-set, respectively. For the target network ResNet-
CIFAR 100-set, ImageNet-set, CIFAR 10AAR000-Anmmall 18, we set the hyperparameter as 0.2.0.5,0.5,0.5 for Fashion-set, CIFAR100-sC ImageNet-set, CIFAR100-AAniaals/ respectively. More details could be found in Appendix A. and A.2
4.4 RESULT AND ANALYSIS
General Performance Analysis Table a Table show the general results of our method. We could find that method consistently outperforms the meta-learning methods and the pretrained method. This demonstrates that learning a dyper-mapping between datasets and corresponding net- work parameters is effective. To demonstrate the time consumption our method could save, also

---

Under review as a conference paper at ICLR 2023
Table I: Results of different methods in terms of the target network ConvNet-3 on the Fashion-see sett CIFARI00-set, ImageNet-set datasets.
Method
Fashion-set
CIFAR 100set time (sec.)
ImageNet-sel time (sec.)
ACC(%)
time (sec.)
ACC
ACC
Pretrained
94.12.10.63
58.35110.61
53.28-0.66
MatchNet
89.93=00.65
47.75=t0.73
43.83=0.83
ProtoNet
92.32-+0.37
51.96-+0.557
49.59-0.88
Mcta-Bas Baseline
94.85+00.31
57.69-t0.38
54.97=0.75
Meta-DeepDBC
95.76.10.39
60.52.10.41
55.36.10.73
MUSML
96.05=+0.32
1.21
56.49-0.55
1.22
54.03=00.4
1.22
1 epochs
91.07+1.11
0.87
49.11
0.99
41.53=1.20
2.67
Adam
Scratch
30 epochs
99.97+0.02
25.99
64.54+0.40
28.37
64.33=0.83
77.08
50 epochs
99.96.10.03
43.36
70.68
49.02
6..25
133.48
epochs
92.36
0.88
50.23++1.23
0.99
40.74+1.32
GC
30 epochs
99.98110.01
26.01
66.76:10.54
29.83
65.45=0.89
77.12
50 epochs
99.97=0.02
43.42
71.56+00.3
50.22
69.44=0.77
133.69
PudNet
96.64.10.34
0.03
64.09-10.40
0.03
59.31
0.03
Table 2: Results of different methods in terms of the target network ResNet-18 the Fashion-set, CIFARI00-seL, ImageNee-set datasets.
Method
Fashion-set
CIFAR100-set
ImageNet-set time (sec.)
ACC(%)
time (sec.)
ACC
time (sec.)
ACC
Pretrained
93.76:10.47
64.58:00.59
65.67=0.73
MatchNet
90.16-+0.53
56.23-+0.71
53.17=0.91
ProtoNet
93.64.10.47
60.29-10.59
58.95
Meta-Bas Octa-Baseline
95.35=#0.29
67.51=0.55
67.16=0.70
Meta- MC
94.28-10.31
69.54-1-0.49
68.48=0.60
MUSML
95.87:10.4
2.55
66.47-10.3
2.59
66.03=0.91
2.60
epachs
93.98.11.21
52.82.11.01
3,96
46.43
Adam
Scratch
30 epochs
99.91
54.22
74.56+00.45
118.87
71.84=0.69
140.37
50 epochs
99.87+00111
91.19
79.85+00.47
198.12
75.98
231.63
1 epochs
94. 1.21
1.88
53.21
4.01
1..35
4.82
GC
30 epochs 50 epochs
99.94-+0.05
54.93
75.74+0.59
119.03
72.89=0.73
140.98
99.96:10.03
91.73
79.58-L0.55
199.61
76.73-0.87
232.57
PudNet
96.24-H0.39
0.50
73.33++0.54
0.49
71.57+0.71
0.50
Table Results of different methods on the cross-domain datasets CIFARI00-AAniaa
Method
ConvNet-3
ResNet-18
ACC (%)
time (sec.)
ACC (%)
time (sec.)
Pretrained
16.43.10.73
33.36:L0.75
DBC
31.93110.81
40.50=10.64
MUSMI.
26.77=t0.67
1.71
36.78+-0.59
3.21
epochs
18.33++0.57
21.37
22.09-t0.58
31.29
Adam Scratch
5 epochs
38.71-10.31
103.25
49.12-10.08
156.56
10 epochs
53.62110118
203.64
66.4411.37
311.74
1 epochs
18.94400.4
21.33
23.01+1.02
30.79
GC
5 epochs
39.43-10.44
102.73
49.77-10.54
155.34
Scratch
10 epochs
55.21+0.51
202.67
68.56:10.39
310.29
PudNet
35.15110777
0.03
43.21110.69
0.49
provide the time of training the model from scratch by a widely-used optimizer Adam (Kingma & Ba 2015) and the training acceleration technique, GC. We could find that takes around 55, 119, 140 GPU seconds to train ResNet- 18 using the accelerated method GC and the network obtains top- accuracies of 99.94%, 75. 15.74 72.89% the Fashion-set, CIFAR-set, IeageNet-set respectively. While our method costs only around 0.5 GPU seconds to predict the parameters of ResNet- 18 and still achieves comparable performance (96.24%. 73.33%. 71.57% top-1 accuracies) the three datasets respectively, at least 100 times faster than the accelerated method.
Performance on Cross-domain Datasets We further evaluate our method on the cross-dome doms-n datasets, CIFAR 100 - Table
the results the CIFAR 100 Animals 10 dataset.
8

---

Under review as a conference paper at ICLR 2023
Table 4: Ablation study of our method.
ConvNet-3
ResNet-18
Setting
Fashion-set CIFAR100-set ImageNet-sel|
Fashion-set
CIFAR100-set ImageNet-set
PudNet-w.o-Contexl
03.13:10.36
59.51110.42
54.36=0.67
(93.08.10.44
65.35.10.51
61.42:10.70
PudNet-Ramdom
90..11+0.32
57.26+H0.39
52.88=0.59
189.09--0.400
60.56:10.50
55 55.93--0.70
PudNet-w.o-iniiRRe
95.44::1.355
61.5610.40
57.81=0.66
995.42::0.44
71.0::0.511
68.98-10.3
PudNet-metric
195.311:0.45 60.97
50.15_0.87
194.751:0.44
68.60.10.61
0..2
PudNeetw.o.-kl
195.92=0.34
61.29::041
57.64=0.62
99,.44=0.388
70.27=0.54
67.53=0.73
PudNet
96.6440..4
64.09+0040
59.31 59.4 (96.24-0.39
73.33+-0.54
71.57+0.71
can obtain comparable accuracy when training the model from scratch at around 5 epochs, while our method is at least 300 times faster than the traditional training methods. In addition, we also that generally achieves better performance based Table [ and 2 and with MUSMI. which is proposed very recently. Our model still outperforms them in a large margin. We expect that such result could motivate more researchers explore along this direction.
compare with Meta-DeepDBi
Ablation Study We design a variant of our method to analyse the effect of the context relation information. PudNet-w.o. www.t.t denotes method directly feeds the dataset sketch into the weight generator without using GRU. The results are listed in Table + Our PudNet outperforms PudNett-woo-Conneex in large margin, demonstrating the effectiveness of capturing dependencies among parameters of different layers.
Moreover, we design two variants of our method to further study the contribution of the dataset information. PudNet-Random denotes our method randomly initializes the hidden state of GRU. PudNet-w.o.-iniie denotes our method does not utilize initial residual connection for initial dataset information complementary. As shown in Table 4 the performance of our method decreases when randomly initializing the hidden state of GRU. It indicates that exploiting the initialization of the hidden state to deliver the information of datasets is effective. Moreover, PudNet is better than PudNet-w.o. -inii-.....ii.es.. illustrating the effectiveness of the initial residual connection.
Finally, we design another two variants of our methods to study the impact of the auxiliary task. PudNet-metric means our method only using the parameter-free loss. PudNet-w.o.-ke means out method without using the KI. Divergence. As shown in Table A PudNet-w.o.-K has better per- formance than muetec-metric, demonstrating it is effective for the auxiliary full classification task. PudNet outperforms PudNet-w.o.-kl This illustrates it is effective to encourage the predicted prob- ability distribution parameter-free method and full classification method to be matched.
Fine-tuning Predicted Parameters Since typi- cal strategy for applying a pretrained model to a new dataset is to fine-tune the model. Thus, intend to evaluate the performance of fine-tuning our method and baselines the CIFAR-set dataset. To do this, we first incorporate an additional linear classifica- tion layer our method and baselines, except 'From scratch'. Then, we randomly select 10000 samples from 20 classes for fine-tuning the models, and use the remaining 2000 samples for testing. Table 5) shows the results. Here 'From Scratch' means directly training the target network with random initialized parameters from scratch. Our method achieves the best performance. This indicates the predicted parameters by our PudNet can well serve as a pretrained model.
Table 5: Performance of finetuning all meth- ods using 50 epochs.
Method
ConvNet-3
ResNet-18
From Scratch
44.85=0.23
48.25=:0.30
Pretraincd
49.40=0.17
59.75-+0.24
Meta- DeCa-DeepDBC
50.05=0.21
61.35+00228
PudNet 55.21
65.19.10.22
5 CONCLUSION
In this paper, we found there are correlations among datasets and the corresponding parameters of given network, and explored a new training paradigm for deep neural networks. We proposed new hypernetwork, PudNet, which can directly predict the network parameters for an unseen data with only single forward propagation. Essential to our hypemnework is the construction of series of GRU, to capture the relations among parameters of different layers in a network. Extensive experimental results demonstrated the effectiveness and efficiency of our method.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rohan Anil, Vineet Gupta, Tomer Koren, Kevin Regan, and Yoram Singer. Scalable second order optimization for deep learning. arXiv preprint 2xXi:20022000188. 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. leni:o100065555 2016.
arXiv preprint
Pierre Baldi and Peter J Sadowski. Understanding dropout. Advances in neural information pro- cessing systems, 26, 2013.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. Advances in neural information processing systems, 31, 2018.
Jianguo Chen, Kenli Li, Kashif Bilal, Keqin Li, S Yu Philip, et al. A bi-layered parallel training architecture for large-scale convolutional neural networks. IEEE transactions on parallel and distributed systems, 30(5):965-995)) 9966, 2018.
Yinbo Chen, Zhuang Liu, Huijuan Xu, Trevor Darrell, and Xiaolong Wang. Meta-baseline: Explor- ing simple meta-learning for few-shot learning. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9062-9071, 2021.
Zekai Chen, Xiao Zhang, and Xiuzhen Cheng. Asm2tv: An adaptive semi-supervised multi-task multi-view learning framework for human activity recognition. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 36, pp. 6342-6349, 2022.
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçchree Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pp. 1724-1734, 2014.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. L2 regularization for learning kernels.
arXiv preprint 1205..:200.2666 2653, 2012.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810::8100048800 2018.
Harrison Edwards and Amos J. Storkey. Towards a neural statistician. In Proceedings of the Inter- national Conference on Learning Representations, 2017.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference Machine Learning, pp- 1126-1135, 2017.
Zhenyong Fu, Tao Xiang, Elyor Kodirov, and Shaogang Gong. Zero-shot object recognition by semantic manifold distance. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2635-2644, 2015.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. Proceedings of the International Conference on Machine Learning, pp. 1587- 1596, 2018
Arthur Gretton, Dino Sejdinovic, Heiko Strathmann, Sivaraman Balakrishnan, Massimiliano Pontil, Kenji Fukumizu, and Bharath K Sriperumbudur. Optimal kernel choice for large-scale two-sample tests. Advances neural information processing systems, 25, 2012.
Shesh Narayan Gupta and Nicholas Bear Brown. Adjusting for bias with procedural data. arXiv preprint ar Xrxiv22204.01 108, 2022.
10

---

Under review as a conference paper at ICLR 2023
David Ha, Andrew M. Dai, and Quoe V. Le. Hypemettorks. In Proceedings of the International Conference on Learning Representations, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey, IEEE transactions on pattern analysis and machine intelligence, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the International Conference on Machine Learning, pp. 448-456, 2015.
Weisen Jiang, James Kwok, and Yu Zhang. Subspace learning for effective meta-learning. In International Conference on Machine Learning, pp. 10177-10194, 2022.
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in Neural Information Processing Systems, 30, 2017.
Soojeong Kim, Gyeong-In Yu, Hojin Park, Sungwoo Cho, Eunji Jeong, Hyeonmin Ha, Sanha Lee, Joo Seong Jeong, and Byung-Gon Chun. Parallax: Sparsity-aware data parallel training of deep neural networks. In Proceedings of the Fourleenth EuroSys Conference, pp. 1-15, 2019.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Boris Knyazev, Michal Drozdzal, Graham W Taylor, and Adriana Romero Soriano. Parameter prediction for unseen deep architectures. Advances Neural Information Processing Systems, 34:29433-29448, 2021.
Elyor Kodirov, Tao Xiang, and Shaogang Gong. Semantic autoencoder for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3174-3183, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypermetworks. arXiv preprint arXiv: 2017.
Yawei Li, Shuhang Gu, Kai Zhang, Lue Van Gool, and Radu Timofte. Dhp: Differentiable meta pruning via hypernetworks. In Proceedings of the European Conference on Computer Vision, pp. 2020.
Edo Liberty. Simple and deterministic matrix sketching. In Proceedings of the 19th ACM SIGKDD international conference Knowledge discovery and data mining, pp. 581-588, 2013.
Edo Liberty, Franco Woolfe, Per-Gunnar Martinsson, Vladimir Rokhlin, and Mark Tygert. Ran- domized algorithms for the low-rank approximation of matrices. Proceedings of the National Academy of Sciences, 104(51)220167-2017, 2007.
Kaifeng Lyu. Zhiyuan Li, and Sanjeev Arora. Understanding the generalization benefit of normal- ization layers: Sharpness reduction. arXiv preprint arXiv:2206.0708 2022.
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems, 31, 2018.
Farhad Pourpanah, Moloud Abdar, Yuxuan Luo, Xinlei Zhou, Ran Wang, Chee Peng Lim, Xi- Zhao Wang, and QM Jonathan Wu. A review of generalized zero-shot learning methods. JEEE transactions on pattern analysis and machine intelligence, 2022.

---

Under review as a conference paper at ICLR 2023
Chao Qian, Yang Yu, and Zhi-Hua Zhou. Subset selection by pareto optimization. Advances in neural information processing systems, 28, 2015.
Matthew Richardson and Pedro Domingos. Markov logic networks. Machine learning, 62(1):107- 136, 2006.
Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learn- ing. In International conference on machine learning. pp. 2152-2166 PMLR, 2015.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterization to accelerate training of deep neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry, How does batch normal- ization help optimization? Advances in Neural Information Processing Systems, 31, 2018.
Tamas Sarlos. Improved approximation algorithms for large matrices via random projections. In 47th annual IEEE symposium foundations of computer science (FOCS), pp. 143-152, 2006.
Aviv Shamsian, Aviv Navon, Ethan Fetaya, and Gal Chechik. Personalized federated learning using hypernetworks. In Proceedings of the International Conference on Machine Learning, pp. 9489- 9502, 2021.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning, Ad- vances Neural Information Processing Systems, 30, 2017.
Tatiana Tommasi and Barbara Caputo. The more you know, the less you learn: from knowledge transfer one-shot learning of object categories. In BMVC, number CONF, 2009.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In Proceedings of the International Conference on Learning Rep- resentations, 2020.
Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10 (2):1-37, 2019.
David Weenink. Canonical correlation analysis. In Proceedings of the Institute of Phonetic Sciences of the University of Amsterdam, volume 25, pp. 8 81-99, 2003.
Colin Wei. Jason D Lee, Qiang Liu, and Tengyu Ma. Regularization matters: Generalization and optimization of neural nets vs their induced kernel. Advances in Neural Information Processing Systems, 32, 2019.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Yongqin Xian, Tobias Lorenz, Bernt Schiele, and Zeynep Akata. Feature generating networks for zero-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pp. 5542-5551, 2018.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint 1708.7777, 2017.
Jiangtao Xie, Fei Long, Jiaming Lv, Qilong Wang, and Peihua Li. Joint distribution matters: Deep brownian distance covariance for few-shot classification. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 7972-7981, 2022.
Jingjing Xu, Xu Sun, Zhiyuan Zhang, Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Under review as a conference paper at ICLR 2023
Huaxiu Yao, Xian Wu, Zhiqiang Tao, Yaliang Li, Bolin Ding, Ruirui Li, and Zhenhui Li. Automated relational meta-learning. arXiv preprint arXiv:2001.00745, 2020.
Hongwei Yong, Jianqiang Huang, Xiansheng Hua, and Lei Zhang. Gradient centralization: A new optimization technique for deep neural networks. In Proceedings of the European Conference on Computer Vision, pp. 635-652, 2020.
Chris Zhang, Mengye Ren, and Raquel Urtasun. Graph hypernetworks for neural architecture search. In Proceedings of the International Conference on Learning Representations, 2019.
Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, and Zhenhui Li. Drn: A deep reinforcement learning framework for news recommendation. In Pro- ceedings of the world wide web conference, pp. 167-176, 2018.
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1): 43-76, 2020.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 ARCHITECTURE DETAILS
Dataset Compression Module Details. The feature extractor in our framework contains several basic blocks, where each basic block consists of a 5 x 5 convolutional layer, leakyReLU function and a batch normalization layer. For generating parameters of ConvNet-33, we use one basic block the feature extractor. For ResNet-18 or ResNet-34, extractor. Note that our feature extractor is jointly trained with PudNet in an end-to-end manner.
stack five basic blocks the feature
Structure of the Weight Generator Figure shows the architecture of the weight generator. The weight generator takes input vector with the dimension of d.. and outputs a tensor with the size of dous dsn x f x f as the parameters of the convolutional layer.
Weight Generator
Activation
Ix1 Conv
Activation
on din
fXf
2dh
XXX
dg
Figure 4: Architecture of the Weight Generator.
Structures of the Target Networks. We use ConvNet-3, ResNet- 18, ResNet-34 as the target net- works. The structure ConvNet-3 shown Figure 5 For ResNet- 18 and ResNet-34, we use the same architectures with He et al. (2016).
MexPonZi
5x5 Cone
ButchNorm20
MaxPuel2d
ReLU
Relkil
3
5x5
Figure 5: Structure of the Target Network ConvNet-3.
A.2 ADDITIONAL IMPLEMENTATION DETAILS
For all experiments, use ACC (Top-1 Accuracy) metric evaluate the classification performance. All experiments are optimized by Adam optimizer. We set learning rate to 0.001 and train PudNet until convergence.
metric-based learning process, following Chen et al. (2021). the temperature = 10 in Eq.( (6) is fixed. 10 labeled samples per class are used as support sets to deduce the class centroid. As mentioned in the main paper, we introduce auxiliary tasks assist optimization. We add an full classification linear layer (c.g. 80-way linear head in CIFAR-100 to maintain static class during training. We also introduce consistency loss, while the dimension between logits deduced by metric-based classification re...5-dimensional in CIFAR-100) and the logits produced by full linear head (e.g.8t 30-dimensional CIFAR-100) are not matching. thus we transpose the 5- dimensional logit to 80-dimensional logit, by padding the rest values with zero. We search v from {0, 0.1, 0.2, 0.3, ... 0.9). For target network ConvNet-3, we set = 0.2 for Fashion-set. 7) = 0.1 for CIFAR 100-set, set, 77 0.3 for ImageNet-set, r7 0.1 for Animmllalsl For target network ResNet-18, 18, we set = 0.2 for Fashion-set, 77 = 0.5 for CIFAR 100-set and ImageNet-set,
1 0.5 for CIFAR100 Animall
14

---

Under review as a conference paper at ICLR 2023
Table 6: Results of different methods dataset CIFAR- -CIFAR-IO 10.
Method
ConvNet-3
R8
ACC(%)
(sec.)
(%)
time (sec.)
Pretrained
34.32:10.61
40.93:1.448
Meta-DeepDBB
41.28:10.59
47.15:10.63
MUSML
37.45:10.53
2.37
45.37:10.57
5.34
epochs
19.98.10.63
51.57
21.37:1..71
60.79
Adam
5 epochs
44.67.10.4
253.16
33.76.10.44
310.29
Scratch
10 epochs
48.69+0048
506.63
48.21=00255
621.75
20 epochs
58.34+0.53
1015.26
65.17=0.55
1248.23
1 epochs
20.03-10.83
51.74
21.44.11.02
60.99
GC Scratch
5 epochs epochs
44.89:+0.67
253.56
34.41=0.78
310.33
49.36:10.55
507.12
49.8
622,01
20 epochs 58.74:10.49
1015.87
66.78=0047
1248.98
PudNet
43.76.10.64
0.05
51.05:10.56
0.56
Table 7: Results of different methods on the cross-domain datasets on ResNet- 18.
Method
ImageNet-AAiialls
ImageNet-CIIFARO
ACC (%)
time (scc.)
ACC (%)
time (sec.)
Pretrained Meta-Des CBC
34.79=0.49
34.54:20.63
38.57=0.55
40.93=00.71
MUSMI.
36.89-0.44
3.21
37.12++0.65.
5.34
1 epochs
22.09=0.58
31.29
21.37
60.79
Adam Scratch
epochs
49.12=0.08
156.56
33.76:10.44
310.79
10 epochs
66.44-0.37
311.74
48.21+:0255
621.75
20 epochs
73.47=0.67
623.92
65.17
1248.23
epochs
23.01-1.02
30.79
21.4411.02
60.99
GC Scratch
5 epochs
49.77=0.54
155,34
34.41=0.78
310.33
10 epochs
68.56-0.39
310.29
49.89=0.53
622.01
20 epochs 75.04=0.61
623.33
66.78:10.47
1248.98
PudNet
42.43-0.58
0.48
45.07=20.70
0.57
A.3 ADDITIONAL EXPERIMENTS
Performance on Other Full Dataset Analysis: Here we utilze CIFAR100-set to train the model and evaluate the performance on CIFAR-I0. Note that the classes in CIFAR- are mutually ex- clusive with the classes in CIFAR- CIFAR- (riiZiiyyyykky al. 2009). Since Meta-Deepa Detta-DeeDDBCC achieves better performance among all meta-lcarning methods based on Table I and 2 here we only report the results of Me and pretrained model for clarity. 6 shows the results CIFAR- 100->CIFAR-10 10. We observe that our model outperforms the Pretrained and Meta-DeepDBBC com- petitors. This further verifies the effectiveness of our method. We also provide the time consumption of training the target network from scratch with Adam optimizer on CIFAR-10, It worth noting that takes around 622 GPU seconds train ResNet-18 using Adam from scratch and obtain top-1 curacy of 488.11% % on CIFAR-10, 10. While our method costs only around 0.56 GPU seconds to predict the parameters of ResNet-13 18 and still achieves 51.05% top-1 accuracy, which is 1000 times faster than the training from scratch method.
Performance on Cross-domain Datasets: We construct another two cross-domain datasets. We use ImageNet-set to construct training datasets, and Animals-10 and CIFARIO for testing datasets. There are 50000 groups of datasets from ImageNet-set to train PudNet. We randomly split CIFAR I0 into two nonoverlapping subsets separately: one is used to generate parameters, and the other for testing. The separation process for Animals is analogy. We further evaluate our method on these two cross-domain datasets: ImageNet-AAnine Animalls,, ImageNet-CIIFARIO Table 7 shows the sults. our method still achieves surprisingly good efficiency. For instance, it takes 622.01 GPU seconds to train ResNet- 18 on the InAgeRet--CIFARO dataset using GC from scratch and obtain
15

---

Under review as a conference paper at ICLR 2023
top-1 accuracy of 49.89%, while our method PudNet costs only 0.57 GPU seconds to predict the network parameters of ResNet-18 achieving comparable performance (45.07%), more than 1000 times faster than the traditional training paradigm. In addition, we could find that method also outperforms state-of-the-art meta-learning methods in a large margin. We expect that such a result could motivate more researchers explore along this direction.
We perform another experiment to directly predict pa- rameters of ResNet-34 by PudNet on CIFARI The results listed in Table We observe that our method achieves comparable performance to that of GC at 30 epochs, while our method is more than 250 times faster than GC. This further demonstrates the efficiency of our method.
Performance on Deeper Target Network:
Table 8: Results of different methods in terms of the target network ResNet-34 34 on CIFAR100-see
Method
ACC(%)
time(sec.)
Pretrained
65.03-L0..53
Meta-baseling
67.40:10.69
Meta- BBC
69. 69.64:10.75
MUSML
66.39:10.59
3.11
1 epochs
1..3
5.47
Adam
30 epochs
71.17:10.53
153.87
Scratch
50 epochs
78.72:10.71
263.25
epochs
48.44:1.1.4
5.52
GC Scratch
30 epochs
72.37.L0.75
154.19
50 epochs
79.85110.83
264.03
PudNet
72.87-10.64
0.59
Effect of Different Dataset Embedding: To further study the effect of dataset embedding, de- sign three variants to predict parameters for Conv on CIFAR 10. "Sum" denotes summing up the representations of all samples in dataset as the dataset embedding. "Geometric means" denotes using the geometric mean of sample representations as the dataset embedding. "Mean+Vars denotes concatenating the mean and the variance of sample representations as the dataset embedding. The results are reported in Table We find that these four dataset embedding methods have comparable results. Here we only explore some simple dataset embedding methods. In the future, more compli- cated data compression methods could be explored, such as matrix sketching (Liberty) [003 Qian al., 2015), random projection (Sarlos 2006 Liberty et al. 2007) and statistic
& Storkey 2017 etc.
Table 9: Effect of Different Dataset Embedding
Method
Sum
Geometric mean
Mean+Var
Mean(Ours)
Acc(%)
62.81
64.05
65.22
64.09
Effeet of Different Number of Training Classes with Varying Groups: We analyze the effect of different number of classes on training set. We utilize PudNet to predict parameters for CenvNet-3. The results are shown in Table TO Train-C20 denotes that the training set involving 50000 groups of datasets contains 20 classes in total. Similarly, Train-C80 denotes that the training set involving 50000 groups of datasets has 80 classes. We find that with more classes included training set, the performance of our PudNet is improved as the number of dataset groups increases.
Table 10: Effect of different number of training classes with varying groups on CIFAR -et.set.
Groups
50
100
500
1000
5000
10000
50000
Train-C20
45.77=0.33
49.25=0.31
51.31
51.11
50.93+0.29
51.02+0.22
51.10+1.19
Train-C80
45.37
$1.49
58.9810.44
60.44.1..5
62.42.1.39
62.84.10.41
64.0910.40
16

---

Under review as a conference paper at ICLR 2023
102
65
42
50
180
570
2000
S.6
3900
5100
16.00
150
19:0
880
900
(8) Fashion-set
(b) CIFAR100-sel
(c) ImageNet-set
Figure 6: Effect of Different Groups Datasets for Training to Predict Parameters for ConvNet-3.
102
20
60
50
570
200
200
(a) Fashion-son sct
(b) CIFAR100-Set
ImageNet-set
Figure 7: Effect of Different Groups of Datasets for Training Predict Parameters for ResNet-18.
Effect of Different Number of Stacked GRU:
We exploit stacked GRU to transfer context-aware information of parameters for 18sNet-18. We conduct experiments to investigate the effect dif- ferent number of layers in stacked GRU and report the results in Table 11 We can observe that the performance falls after rising, as the number of layers increases.
Table 11: Effect of different layers of stacked GRU.
Stacked GRU
num_layers22 Target Network lamessy
nam_layers=4
layerse
nam_ray
CIFAR100-set
ResNet-18
68.8910.52
73.33.10.5
71.7810.57
70.15.00.49
70.81
ImageNet-set
ResNet-18
60.81
68. 13.669
71.57:00.1
70.54:0.74
69.26:10.71
Effeet of Different Structure of Feature Extractors: To study the influence of feature extractors with different architectures, add experiment on CIFAR100-set 100-set in terms of ConvNet-3, The Table 12 "2conv", "3conv", "4conv" denote stacking 2, 3, convolution layers the feature extractor respectively. "1linear" denotes adding linear layer after the convolution layer. We observe that our method obtains comparable performance with the feature extractors of different structures. Thus, our method sensitive the structure of feature extractors.
results are listed in Table T2
Table 12: Effect of different structure of feature extractors
Method 2conv
3conv 4conv Iconv- linear 2conv-1linear 3conv-
Acc(%)
63.87
63.49
62.44
64.09
64.01
63.43
We analyze the effect of different groups of datasets for training. The results utilizing PudNet to predict parameters for ConvNet-3 are shown in Figure For the target network ConvNet-3, we construct 2000, 100000, 50000 groups of datasets for training PudNet on Fashion-set, CUFAR100-SES Imagenet-set respectively. Figure rrpport the results of ex- ploiting PudNet to predict parameters for ResNet-18. We construct 2000, 100000, 100000 groups of datasets for training our PudNet on Fashion-set. CIFAR 100-set, Imagenet-see respectively. find that with more datasets for training, PudNet could obtain better performance. This is because
Effect of Different Groups of Datasets:
17

---

Under review as a conference paper at ICLR 2023
100
70
65
65
60
60
55
90
55
50
85
50
45
80
45
40
750
205
40
35
0
(a) Fashion-set with ConvNet-3 (b) CIFARI00-set with ConvNet- (c) ImageNet-set with ConvNet-
3
3
100
75
75
70
70
65
65
60
60
55
55
750
50
0.9
50
0.1
(d) Fashion-set with ResNet-18 CIFARI000sseiii ResNet- (1) ImageNet-set with ResNet-18
Figure 8: Sensitivity Analysis of
r).
with larger groups of datasets to learn the hyper-mapping relation, our PudNet could obtain better generalization ability. However, when the number of group becomes large, the performance increase becomes slow.
Parameter Sensitive Analysis: We analyze the effect of different values of the
Recall that y controls the percent of dataset complementary information in the initial residual connection. Figure X (b)(() show the results in terms of ConvNet-3, and Figure 8 (f) give the results for ResNet-18 18. We observe that our model obtain better performance when >> 0 in general. Additionally, our method is not sensitive to 7) relatively large range.
We discuss the convergence property of the proposed method by plotting the loss curves with increasing iteration. Here we utilize PudNet to predict parameters for ConvNet- 3, based on Fashion-set, CIFARI00 -00-st and ImageNet-set respectively. As shown in Figure 9 the training metrie-based loss and training total loss first decrease rapidly as the number of iterations increases, and then gradually decreases to convergence.
Convergence Analysis:
A.4 ALGORITHM PSEUDO-CODE
We provide the training procedure of our PudNet as listed in Algorithm For each training dataset D; Dtrein we first derive the skecth 8; of dataset D, and set the initial hidden state hg in GRU. Then, we predict the parameters of each layer the target network Ω. Finally, we optimize the the learnable parameters 0, by the the overall multi-task loss Liotel.
A.5 DISCUSSION WITH SOME RELATED TOPICS
The generalization ability of model an important research topic the machine learning community. To generalize well on unseen data, many methods have been proposed. For example, normalization methods such as batch normalization [Bjorck et al. 2018) and layer nor- malization (Xu et al. 2019) could improve the generalization ability (Lyu et al. 2022). Besides, some regularization techniques such as L2 regularization (Cortes et al. 2012) and dropout (Baldi & Sadowskil 2013) could also help the generalization (Wei et al. 2019 What's more, some works resort to unsupervised pretraining on large-scale data to obtain a model with great generalization
Generalization:
ability DDevlin et al. 2018)
18

---

Under review as a conference paper at ICLR 2023
14
206
00
6000
1088
Training
(a) metric- metric-based loss Fashion-sed set
(b) total loss Fashion-set
7.0
g
125
1.00
0.75
4.
2000
100000
metric- based loss on CIFARI00-
(d) total loss CIFARI00-set
Sct
2.29
3.09
10100
(e) metric-hased loss ImageNet-set
(f) total loss ImageNet-set
Figure 9: Training Loss of PudNet.
Algorithm The Training of PudNet
Input: a set of training datasets Diraen [D:) target network architecture Z Initialize the learnable parameter 0 of Hg-
Initialize the learnable parameter yp the auxiliary full classification. while not converged do for E {1, [1,) do
Obtain a skeeth s; of dataset D; via the dataset compression module; Initialize the hidden state ho = S;:
for each layer in f2 do
Predict the parameters wy of the 1-th layer by (5) Fix the predicted parameters wy to Ω;
for each batch in D, do
Compute the parameter-fe free loss C, with batch size of b;
Compute the full classification loss L2 and consistency loss L3 with batch size of b;
Update the learnable parameters by overall multi-task loss Listat (10):
Output: The PudNet Ho.
19

---

Under review as a conference paper at ICLR 2023
Transfer Learning: The key idea of transfer learning is to transfer knowledge from source domains to a different but related target domain to improve the performance of the target learner (Zhuag jet al.] 2020). There are considerable methods on transfer learning, including feature-basee meth- ods (Gretton et al. 2012) parameter- based methods (Tommasi & Caputo] 2009) and relational- based methods (Richardson & Domingos 2006), etc. reature-based approaches usually transform the original sample features in different domains into a common latent feature space. Parameter- based methods usually learn finetune the parameters of the last few layers across different domain datasets. Relational-batt based methods transfer the logical relationship or rules learned in the source domain to the target domain.
Meta-Learning: Meta-Learmin introduces the mechanism of "learning to learn ", which intends to train a model on a variety of learning tasks, such that it can solve new learning tasks using only a small number of training samples (Hospedales et al. 2021). Meta-Learning is usually divided into three categories: optimization-basee based methods, metric-based methods, and model-basee Yao et al. 2020). Optimization-
methods methods usually train the model to be easy to fine-tune by a small number of gradient steps with a small amount of training data (Finn et al. 2017p. Metric- based methods learn compare validation points with training points and predicting the label of matching training points (Hospedales et al. (021)) Note that few-shot learning can be regarded as the applications of metric-based meta-learning Hospedales et al.] 2021). Model-based methods embed the training data into activation state, making predictions for test data based on this state.
Zero-shot Learning: Zero-shot learning aims to learn classifier that could classify never seen
classes during training without knowing any labeled data of novel class (Wang et al.
2019 The
representative zero-shot learning approaches includes: ESZSL (eoses((RRrr--reeeeee & Torr
2015 SAE
(Koirrvv et al. (017)) f-CLLWGAAN (Xian et al. 2018), etc. The core 1dea of zero-shot is to transfer the learned knowledge of seen classes to the classes unseen during training (Pourpanah al. 2022
Since there is no label information for the unseen class, the auxiliary information for each unseen class is necessary to solve zero-shot learning problem (Wang et al., (019). For example, given the auxiliary information for a unseen class zebra: "look like horse, with stripes", the zero-shot learning model could use this semantic information to recognize the zebra class as long as the model known the pattern of "horse" and "stripes" (Fu et al. 2015).
Since our work aims to learn a hyper-mapping between datasets and their corresponding network parameters and directly predict the parameters for unseen dataset based the hyper-mapping, our task is totally different from the above works. Despite this, there are some relations between method and the above works. First, since it is prohibitive prepare thousands of datasets and training networks on them to obtain the corresponding ground-truth parameters, we develop new learning manner, motivated by meta-learning. Besides, because our method can directly predict network parameters for an unseen dataset, it is potential to couple with zero-shot learning, which is worthy be further studied.
20


Reviews:
[{"review_id": "sDb0FRnD8C", "reviewer": "Reviewer_AS8e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To overcome heavy burden of computational cost and time to learn network parameters, this work propose a method that predicts parameters of the network on the given unseen dataset. For this, they allow the new hypernetwork to directly predict the network parameters with forward process, by learning mapping between datasets and their corresponding network parameters. With only 0.5 GPU seconds, this method can predict the network parameters of ResNet-18 which is competitive performance compared with the parameters trained on the dataset from scratch.", "summary_of_the_review": "I think the experimental setting is rather limited to support or validate the proposed method. As I mentioned in the strengths and weaknesses section, I hope the authors address my concern by showing the solid and extensive experimental results on more realistic settings.", "strength_and_weaknesses": "- Strengths\n    - Writing is clear and easy to follow.\n    - Novel concept idea. This work reveals that there is correlation between datasets and the network parameters.\n    - With a single hypernetwork, this work can predict the parameters of multiple datasets. I think this is quite new.\n- Weaknesses\n    - The main concern is that the experiments to validate the proposed method are limited and weak.\n        - While experimental setting is similar with the existing meta-learning methods for the classification task, the most recent or powerful meta-learning methods are not compared.\n        - In addition, I think that such similarity with the existing few-shot classificaiton tasks for the meta-learning are not attractive and not challenging. The most architectures are small (3 CNN layer ~ ResNet-18) and the small number of classes in each task, most unseen tasks are sampled from the in-distribution dataset with the seen tasks. I think this work need to more focus on the cross-domain setting or different architectures.\n        - For the fair comparison, I think Meta-dataset [1] can be used.\n     - There is a related work [2] that a single hypernetwork can predict parameters for unseen architectures. I recommend to add discussion with this related work.\n\n[1] Meta-Dataset: A Dataset of Datasets for Learning to Learn from Few Examples, ICLR 2020\n[2] Parameter Prediction for Unseen Deep Architectures, NeurIPS 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DZmv6m3unH", "reviewer": "Reviewer_y6gY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to predict the parameter of a neural network via a hyper-network.\nIt mainly consists of two components, a compression method to capture the feature of a dataset, and a GRU-based network to generate the per-layer parameter.\nSome empirical results are very interesting and surprising.", "summary_of_the_review": "The paper is novel and interesting to me in general. However, there are some missing empirical evaluations and intuitions, which I would like to hear from the authors in the feedback phases.", "strength_and_weaknesses": "### Strengths\n- I'm not familiar with the area of generating network weight for unseen datasets. But to me, this paper is well-motivated and novel.\n- The organization of this paper is pretty good, first using an experiment to demonstrate the correlation between the dataset and the trained weight, then three main components of the method, it's easy to follow and interesting to read.\n- Empirical evaluation is thorough and have some interesting results. Although the performance still lay far behind a fully-trained network, it can achieve reasonable results.\n\n### Weaknesses\n- The core of the proposed approach is that it can generate parameters for unseen datasets. The paper only shows CIFAR-100 -> Animals-10, I would expect some other evidence. For example, CIFAR-10 -> ImageNet.\n- One interesting I notice in the method is that it trains a weight generator, which allows generating parameters of different sizes. However, I don't see any evidence in the experiment part. For instance, can you use the network to generate a larger ResNet-50 and show the accuracy?\n- The ablation study reveals the effectiveness of different parts. Despite this, it could be better if the authors can provide more insights and intuitions in the method part. Some part is a bit ad-hoc to me. For instance, adding an additional classification head and trying to match the two predictions.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "568lW8yjeJ0", "reviewer": "Reviewer_VS2g", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to predict parameters of the model for unseen datasets given a fixed model. To do so, the paper proposes to leverage hypernetworks to predict model parameters for unseen datasets. The proposed pipeline is validated on three sets of datasets simulated from standard ML datasets (CIFAR-100, Fashion-MNIST, and ImageNet) and three simple CNN architectures (3-layer CNN, ConvNet-3, and a Resnet18). The reported results highlight the benefits of the proposed pipeline. Moreover, the authors stress-test the pipeline on one cross domain setup by training the model on CIFAR100 and testing it om Animals 10.", "summary_of_the_review": "Overall, the paper is interesting, well written and easy to follow. However, prior to recommending the paper for acceptance the reviewer would like to see improved validation and better positioning of the introduced idea/pipeline.", "strength_and_weaknesses": "Strengths:\nThe topic studied in the paper is of interest to the ML community.\nThe paper is well written and easy to follow\n\nWeaknesses (details in next section):\nThe validation of the paper could be strengthened by considering more challenging setups and models.\nPositioning the paper w.r.t to transfer learning, few-shot and zero-shot literature is missing\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject