Meta-ticket: Finding optimal subnetworks for few-shot learning within randomly initialized neural networks
Authors: Daiki Chijiwa
Shin'ya Yamaguchi
Atsutoshi Kumagai
Yasutoshi Ida
Venue: NeurIPS 2022 Accept (2022)
Keywords: Meta-learning
Few-shot learning
Lottery ticket hypothesis

Abstract: Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with a few data. The main challenge is how to avoid overfitting since over-parameterized NNs can easily overfit to such small dataset. Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks. On the other hand, one conventional approach to avoid overfitting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision. However, although such manually-designed sparse structures are sample-efficient for sufficiently large datasets, they are still insufficient for few-shot learning. Then the following questions naturally arise: (1) Can we find sparse structures effective for few-shot learning by meta-learning? (2) What benefits will it bring in terms of meta-generalization? In this work, we propose a novel meta-learning approach, called Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task. Due to this task-wise adaptation ability, Meta-ticket achieves superior meta-generalization compared to MAML-based methods especially with large NNs.

Meta-ticket:
Finding optimal subnetworks for few-shot learning within randomly initialized neural networks
Daiki Chijiwa**
Shin'ya Yamaguchi
Atsutoshi Kumagai"
Yasutoshi Ida²
NTT Computer and Data Science Laboratories, NTT Corporation Kyoto University
Abstract
Few-shot learning for neural networks (NNs) is an important problem that aims to train NNs with few data. The main challenge is how to avoid overfitting since over-parameterized NNs can easily overfit to such small dataset. Previous work (e.g. MAML by Finn et al. 2017) tackles this challenge by meta-learning, which learns how to learn from a few data by using various tasks. On the other hand, conventional approach to avoid overfitting is restricting hypothesis spaces by endowing sparse NN structures like convolution layers in computer vision. sample-efficient
However, although such manually-designe for sufficiently large datasets, they are still insufficient for few-shot learning. Then the following questions naturally arise: (1) Can find sparse structures effective for few-shot learning by meta-learning? (2) What benefits will it bring in terms of mntta-geneaalization?? In this work, propose novel meta-leaming approach, called Meta-ticket to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. We empirically validated that Meta-ticket successfully discover sparse subnetworks that can learn specialized features for each given task. Due this task-wiss adaptation ability, Meta-ticket achieves superior meta- generalization compared to MAML-based methods especially with large NNs.
sparse structures
Introduction
Recent neural networks (NNs) have achieved remarkable performance for solving complex recogni- tion tasks such image recognition and natural language understanding [20, 58]. These successful performance are mainly due to carefully designed NN structures (inductive biases) and large amount of training data. However, it still takes sigmificant cost for collecting such large amount of data, Thus, few-shot learning has been an important problem that aims to enable NNs to learn from few samples or experiences like humans can do. Since NNs have strong memorization capacity [67], the main challenge of few-shot learning is how to avoid overfitting to small number of data.
Meta-learning is of the promising approaches for few-shot learning, which automatically learn how to learn without overfitting, by using a bunch of tasks from a so-called meta-training dataset. Meta-learning consists of two optimizations: adapting to a given task (inner optimization) and improving the inner optimization for each task (metta-ptimizatioon) Gradient-based meta- learning [14] is approach modeling the inner optimization as gradient descent steps, which leads to sample-efficient meta-learning compared to black-box models like recurrent NNs [22]. Model- Agnostic Meta-Learning (MAML. proposed by Finn et [13]) representative method of the gradient-basee based meta- meta-learning. It optimizes initial parameters in a given NN with stochastic gradient descent (SGD) so that it can adapt to a novel task by gradient descent starting from the leaanle initial parameters. However, Raghu et al. [50] pointed out that, in classification tasks with NN,
Corresponding author: daiki. seeriichiiiiieee www.wwiatttttcooooo
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

mo Φ1
VC
Φ
m
Φ
00
00
m
Randomly
Initialized
Find subnetwork (Meta-ticket)
V22
Task Adaptation
Figure 1: For randomly initialized neural network (z), instead of optimizing the initial parameters 6, Meta-ticket finds an optimal subnetwork ( by meta-optimizing binary parameters m so that the subnetwork can successfully adapt tasks with a few samples.
MAML tends to fix its feature extractor and learn only its classification layer (feature reuse) rather than learning both the feature extractor and the classification layer (rapid learning) during inner optimization. In other words, MAML directly encodes information of the meta-training datasel into the meta-learned initial parameters in the feature extractor, which results in suboptimal solutions that cannot sufficiently adapt to tasks in unknown domains [44].
On the other hand, another conventional approach to avoid overfitting is to limit hypothesis spaces for learning. In the case of NNs, this is corresponding to designing domain-specitf NN structures. The structure convolutional neural networks (CNNs) is one of the most successful examples. Neyshabur [42] pointed out that the superior generalization capacity of CNNs is mainly due their sparse structures. However, although such hand-designed sparse NN structures are sample-effiiies with sufficiently large dataset, they are still insufficient for few-shot learning [59].
In this paper, we focus on sparse NN structures for few-shot learning rather than the initial parameters like MAML. Our research questions are: (1) Can we automatically find sparse NN structures that are effective for few-shot learning by meta-learning? (2) What are the strengths and weaknesses of the meta-learned NN structures compared to the meta-learned initial parameters of MAML? To answer these questions, we exploit techniques to identify sparse subnetworks within a randomly initialized NN from the literature of strong lottery ticket hypothesis [51]. Based on them, formulate novel gradient-based meta-learning framework called Meta-ticket in a parallel way to MAML, which finds an optimal subnetwork for few-shot learning within given randomly initialized NN (Figure 1). Specifically, Meta-ticket meta-optimizes the subnetwork structures instead of the initial parameters. Our findings summarized as follows:
We validated that Meta-ticket can discover sparse NN structures achieving comparable few-shot performance to MAML few-shot image classification benchmarks (Section 4), even though Meta-ticket never directly meta-optimize the randomly initialized parameters in NNs. Although Meta-ticket often degrades with small NNs on easy tasks, outperforms MAMI. when using larger NNs on more difficult tasks. This indicates that the meta-learned NN structures enable the scaled-up NNs exploit their increased capacity with less overfitting in few-shot learning.
Moreover, we found that Meta-tickel prefers rapid learning rather than feature reuse in contrast to MAML. This remarkable property may be emerged from the different meta-optimization. suggested by rough theoretical arguments in Section 3.2. As a result, Meta-ticket outperforms MAML--ased methods by a large margin (up to more than 15% accuracy) cross-domain evaluations (Section 4. 4.1.3,
Also we analyzed the effect of sparse NN structures themselves for few-shot learning. To remove the effect of initial parameter values, we consider the setting that parameters are initialized with the same constant. Even in such an extreme setting, Meta-ticker achieves over 70% accuracy while random subnetworks only do nearly 30% (Section 1.2)
Background
2.1 Meta-learning for few-shot learning
Let X be an input space and Y an output space. A distribution T over x Y called a task. In standard learning problem, given task T, consider model fo(z) X + Y parameterized

---

by ò E In and optimize & to reduce the expectation value of a loss function L: Y.Y. X Y + R. For simplicity, we denote the mean value of the loss function over dataset D X Y as L(f:: D) (iwiDDD (z)
In meta-learning, consider distribution PTT) over the set of tasks T. Meta-leaming consists of two optimization stages: adapting to a given task T (inner optimization) and improving the inner optimization for each task mnett-optimization.. To model the inner optimization, we consider meta-model (D) parameterized by meta-parameter 0 E R.N. where 'D is a training data sampled i..dd from the task T. Given lo N, consider the setting of k-shot learning where the training data DD consists of samples, and denote a task with l samples over (X x Y)* by T Then a meta- meta-model for k-shot learning is given in the form of Fg : (X x Y)* + R" with a meta- parameer 0 E I. The aim of meta- meta-learning for k-shot learning is to optimize the meta- eeta-parameter as follows:
min BT~p(T))DD DTi Dralt (for: D 1 where ø' = Fo(D"").
(1)
BERN
The meta-paramees is optimized to minimize the expected validation loss L(Jw; Dval) with d' adapted training data Dtr by the meta-model Fe.
2.2 aodel-agnostic meta-learning (MAML)
MAML [13] is a representative method of gradient- based meta-lcarning [14] which optimizes hyperparameters appearing in gradient descent procedure for k-shot learning. Specifically, MAML learns an initial parameter Φ E R" for a model fz). Here we focus on the case fo (z) is an L-layered neural network parametrized by Ø = (Ø") (=1)_= LL E "a.....-__. where ny is the number of parameters in the [-th layer. The
of MAML (with S gradient steps) for neural
network fa)) is described as follows:
PMAML (D'1)
= ØS-1
=(0b 0 a' (lfe:
(2)
1SISL
where we set 6, s as the collection of parameters i-th step, and of := 0 a'V ot i Dtr) as an i-th gradient step for (-th layer. a' E R2o is called an inner learning rate for 1-th layer, and the gradients V Ve= C(fo; Dtr) in the inner optimization are called task gradients.
2.3 Rapid learning vs. feature reuse
Raghu et al. [50] investigated what makes the MAMI. algorithm effective few-shot classification with neural networks. Let fz) h&z gx( be an L-layered neural network, where we denote an (L lllyeeee neural network go (z) (feature extractor) and a linear classifier haz (outpur layer). Under these settings, they considered two phenomenon: rapid learning and feature reuse. Rapid learning means that the neural network fo(x) adapt to each task with large changes in the overall parameters é. Feature reuse means the feature extractor go (xx) meta-learns extracting reusable features for all tasks and is barely changed during the inner optimization. Thus only the output layer he (æ) adapts to each task in the feature reuse case. The question posed by Raghu et al. [50] is as follows: Which of rapid learning and feature reuse is dominant in meta-models trained with MAML?
They empirically showed that the task gradients of the feature extractor in MAML approximately vanish, and thus the feature reuse is dominant in MAML, on few-shot classification benchmarks. Also they introduced variant of MAML, called ANIL, that lets all parameters of the feature extractor go (x) be frozen in inner optimization (i.e. setting the inner learning rates a' = 0 for 1 < ( < 1). Since ANIL fully satisfies feature reuse by definition, it performs similarly to MAML.
On the other hand, Oh al. [44] investigated the rapid learning aspects of MAML. They proposed another variant of MAML, called BOIL, that let the parameter of the output layer be frozen in inner optimization (i.e. at = 0). In other words, contrary to ANIL, BOIL forces the parameters in the feature extractor to largely change for each task since the parameter of the output layer cannot adapt anymore during inner optimization. Although this formulation seems counterintuitive, they showed that BOIL actually works well. Moreover, BOIL outperforms MAML and ANIL cross-domain
3

---

adaptation benchmarks, i.c. evaluating meta-leamers with datasets that differ from the one used for meta-learning. The result indicates that the feature reuse MAML may be suboptimal from the viewpoint of metta-generalization
3 Methods
In this section, we propose novel gradient- based meta-learing method called Meta-ticket (Algo- rithm 1). The aim of Meta-ticket is to meta-learn an optimal sparse NN structure for k-shot learning, within a randomly initialized NN. The algorithm can formulated in parallel to MAML as we will see in Section 3.1. In contrast to MAMI, which directly encodes information of a meta-training dataset into the NN parameters, Meta-ticket instead meta-learns which connections between neurons are useful for learning new tasks. Due to this difference in meta-optimization, it is shown that Meta-ticket prefers rapid-learning rather than feature reuse by analyzing task gradients in Section 3.2, which leads to better cross-domane adaptation demonstrated in Section 4.
3.1 Algorithm: Meta-ticket
Let fa( be an L-layered neural network parametrized by Φ (0') E BICIZLR R", where nt is the number of parameters in the I-th layer and := Eisist n. We define sparse structure for fx( as a binary mask m (m"))sss E Sıs1Zz 13 {0, 13" whose dimension is same as Ø. The corresponding sparse subnetwork is (2oo(() where @ means an element-wise product in R", The meta-model FM (D'r) for Meta-ticket (with S gradient steps) is described as follows:
FMT (Dtr) = m O Øs-1
S-1
m O (%) a Σ 1=0
(3) where a' E F is an 1-th inner learning rate, of = - oVo [(fmor Drr) is an i-th gradient step, and o; := (d))si The differences in Eq. (3) from Eq. (2) of MAML are the parts involving the mask m. In contrast to MAMI optimizing the initial parameter do. Meta-ticket optimizes only the binary mask m and keeps Φo being a fixed randomly initialized parameter.
V of
C ((mmor:D")
ISISL
The whole algorithm of Meta-ticket is described in Algorithm 1, which involves two functions CalculateMask() and UpdateScore() to deal with the mnraa-aaraeeter m. Since m is a discrete variable, we cannot directly apply the standard NN optimization methods to it. Instead, we optimize a latent continuous parameter s = (s') E R" corresponding to m, called a score parameter, inspired by Ramanujan et al. [51].
More specifically, we introduce hyperparameter Pinit E [0, 1] called an initial sparsity, which determines the ratio of how many components in m become zero at the beginning of the algorithm. Also we randomly initialize the score parameters s. Using the initial sparsity Pinit and the initialized score 8, for each layer 1 E {1.... T.}. we set the layer-wise score threshold [y J-th -th smallest score s," in the score parameters {s, S i S m4 in the 1-th layer, before the
E IR as the
algorithm starts. During the algorithm, using the fixed thresholds, Calcul ateMaak(sss returns the layer-wise masks (m') )ısIs where each component m of m' is given by
[0
(s] S o))
mf
(4)
1
(sf o.). o))
Although m is a non-differentiable function the latent variable s, we can optimize s by straight-through estimator [7] previous literature [51]. The optimization step is given by UpdateScore(s, Lm) == s vm + (momentum), i.e. gradient descent with the gradient for with respect to the meta-objective L. defined by Eq. (1), (3), and an outer learning rate B.
3.2 Analysis on task gradients
The meta-models learned with Meta-ticket could have different properties from the ones with MAMI. since their meta-optimization targets are different. One is discrete variable m (Meta-ticket) and

---

Algorithm Meta-ticket
(The colored codes are unique to this algorithm.)
Require: a, 220, Pinit E [0
Or 8: inner/outer learning rates, Pinit an initial sparsity. Randomly initialized parameters. b. Score thresholds determined by Pinit and the initial score > Initialize the binary mask m using the initial score
Require: @ E R" R
Require: (di)isiSL R
1: m f CalcullateMas((s)
2: while s is not converged do
3: Sample B tasks T1,
TB P(T)
4: while b < 1. do
5:
Sample datasets D Dial ~ T for k-shot learning of the task To
6:
% t FMT (D))
7:
end while
8: Calculate the losa-loss Lm
B Eb ((So (()))
9: UpdateScore(s, Lm)
10: m f CalculateMask(()
11: end while
12: return m
0.000
0.000
0.300
0.000
Hob
0.003
0:006
0.000
0.03
0.208
0.000
0.002
0.03
0:082
0.000
0.000
10000
20000
10000
Figure 2: We plotted the mean absolute values of task gradients for the last layer of the feature extractor in a 5-layered MLP (meta-trained on CIFAR-FS as Section 4.1.3) with varying inner learning rates.
another a continuous variable do (MAML). As discussed in Section 2.3, feature reuse dominant in the task gradients in MAML, and it may be harmful for meta-gentn gentaaeeneaalzztioon capacity. Then a natural question arises: What about the task gradients Meta-ticket? Here we compare the dynamics of task gradients during meta-training between MAML and Meta-ticket, both empirically and theoretically.
First of all, performed experiments meta-training, for different inner learning rate setting (Figure 2). We find that the task gradients in both methods behave similarly in very early phase of meta-training. However, in the rest, the task gradients in Meta-ticket either stop decreasing or even increase, while ones in MAML converge to nearly zero in contrast. These results indicate that Meta-ticket prefers rapid learning rather than feature reuse.
how task gradients of the feature extractor change during
In addition the above empirical observation, further investigate why Meta-ticket prefers rapid learning by roughly theoretical argument. For simplicity, we assume that the inner optimization consists of single gradient step (S = 1). Now consider the situation that MAML for k-shot learning converges, i.e. the meta meta-gradient for Eq. (1) with respect to op in Eq. (2) approximately vanish follows:
DralTT | du -flooolooollssee D:: Dval)|| SE.
(5)
for some > O. By Taylor expansion at a = O, we have V V C(fon; Dval) o(a). Now we assume |a| << 1 and approximate Eq. (5) by
Dval)
=
E
V 4a C(f ) Se.
(6)
By simply replacing the symbol Dval in Eq. (6) by t it follows that
E | do C(for D S.
(7)
The left-hand side of Eq. (7) is nothing but the norm of task gradients MAMI. In other words. when the meta-gradiiens of MAML converges to zero during meta-training, the task gradients too.
5

---

On the other hand, if we consider the situation that Meta-ticket converges, we have || BT
w (anea/mmmmee-aVV(mmmD Din) S..
(8)
By the same argument Eq. (5-7) and by Leibniz rule, we have
ED V D = EDHNTH Øo o V vo C(fo: D'n S.
(9)
The last inequality indicates that, if every absolute values of components in do smaller than (e.g. when using Kaiming initialization), the norm of the task gradients V VoL(Smoo: Dtr) in Meta-ticket can be larger than E. This is contrary the MAML case (Eq. (7)) where the task gradients should be smaller than E. In conclusion, the empirical observation that Meta-ticke ticket prefers rapid-learnin (Figure 2) can be explained by the Taylor expansion analysis under the assumption of the convergence of meta- mneta-gradients.
3.3 Some notes on Meta-ticket
Finally, we note some implementation details of Meta-ticket used in our experiments (Section 4).
Parameters that should not be pruned In typical NN architectures, some kind of parameters should not be pruned by Meta-ticket. One type is non-matrix parameters (e.g. bias terms in linear/convolutional layers, element-wise scaling parameters in Batch/Layer normalization layers and etc). They often unsuitable for being pruned since pruning them only hurts the adaptation capability of the NN. Another type is the parameters of the output layer for classification tasks. Pruning them means that choosing which neurons in the previous layer can contribute to some i-th output of the output layer. However, unlike standard supervised learning setting, i-th output can represent an arbitrary class in meta-learning setting. Therefore, pruning the output layer never reflect useful information for the classification tasks and only hurts adaptation capability of the NN. For the parameter Φi of these types. we initialize and fix the corresponding mask m (i.e. such o, is totally not pruned) throughout the whole Meta-ticket algorithm.
As known in the studies of strong lottery ticket hypothesis, pruning randomly initialized NNs results in less approximation capacity than standard training by directly optimizing parameters. Similarly, when the size of given NN is small, Meta- ticket tends to achieve degraded results as we will see in Section 4. For such cases, however, we can adopt the iterative randomization technique (IteRand [9]]) proposed in the lottery ticket literature. which enables us to prune randomly initialized NNs with the same level of approximation capacity as standard training. We will refer the algorithm combining Meta-ticket with the iterative randomization as Meta-ticket+IteRann in our experiments (Section 4.1.1).
Boosting approximation capacity for small NNs
20-way Accuracies
4 Experiments
92.5
90.0
87.5
In this section, we conduct experiments to evaluate meta- generalization performance of Meta-ticket on multiple bench- marks of few-shot image classification. As baseline methods, we choose MAML [13] and its variants (ANIL [50] and BOIL [44]) since Meta-ticket is formulated in a parallel way to MAML and can be considered as another approach of gradient-basee based meta- learning. The details on every datasets, NN architectures and hyperparameters in our experiments given in Appendix A.
82.5
80.0
MAML
Ticket
77.5
1.0
2.0
4,0
8.0
5-way Accuracies
97
06-
4.1 Experiments with MLPs
35
94
In this subsection, we investigate the properties of Meta-ticke using 5-layered multilayer perceptrons (MLPs) with the ReLU activation function and batch normalization layers [24]. Since MLPs are fully-coneected neural networks (NNs) designed for general tasks, we can see how Meta-ticket performs on NNs involving little inductive biases of task information. In other
03
1.0
2.0
4.0
8.0
Figure 3: 5-shot accuracies on
Omniglot with MLPs of various sizes. The x-axis is width factor.

---

11
20
12
Figure 4: We visualize 28 x 28 binary masks m for the first fully- cully-connected layer in 5-MLP with the same constant parameters. The light (resp. dark) color represents my (resp. my 0).
Table I: Cross-domain evaluation for 5-shot 5-way classification with 5-MLP.
Meta-train
CIFAR-FS
VGG-Flower
Meta-test
CIFAR-FS
VGG-Flower
Aircraft
CIFAR-FS
#Go-Flower
Aireraft
MAML [13]
53.48 0.36%
58.60 + 0.59%
28.95
0.76%
36.67 = 0.86%
59.16 0.09%
28.50 + 0.75%
ANIL [50]
50.37 0.68%
53.32 +: 0.62%
26.86
0.77%
34.94 1.14%
58.87 1.30%
27.07 +: 0.73%
BOIL. [44]
55.47 + 1.13%
62.46 + 0.15%
33.36 + 0.84%
36.49 = 0.98%
58.15 0.86%
28.74 + 0.97%
Meta-ticket
55.51 = 0.47%
62.34 +: 0.17%
31.85
0.31%
40 82 + 0.43%
64.09 + 0.28%
30 60 0.63%
BOII.
56.06 0.57%
0.07%
33.02 0.32%
0.41%
0.20%
0.36%
words, Meta-ticket is expected to learn inductive biases from a given meta-training dataset and encode them as sparse structures in fully-connected layers of the MLPs.
We use the following small-scale datasets for meta-learning with MLPs: Omniglot [29, 59], CIFAR- FS [8]. VGG-Flower [43, 30] and Aircraft [37, 30]. Omniglot is relatively easy dataset of image classification tasks, which consists of monochrome 28 x 28 images of handwritten characters from different alphabets in the world. CIFAR-FS, VGG-Flower and Aircraft are also datasets of classification tasks, with more natural and colorful 32 32 images.
4.1.1 Experiments with various network sizes
As known in the literature [51, 49] of strong lottery tickets, in general, the pruning-only optmiizatinn like Meta-ticket tends suffer from the lacked model capacity especially when using small NNs. To check how Meta-ticket suffers from the lacked capacity, we evaluate Meta-ticket with MLPs of various network widths, comparing MAML as a reference. In Figure 3, we plot 5-shot meta- test accuracies on Omniglot. As we guessed in the above, Meta-ticket suffers from the degraded performance with small network widths. On the other hand, we can see that Meta-ticket even achieves superior accuracies when using larger networks more difficult tasks of 20-way classification. Also, as we noted in Section 3.3, the results with small NNs is largely improved by applying the iterative randomization [9] (Meta-ticket+IteRann from the lottery ticket literature.
4.1.2 The effect of sparse structures themselves
While Meta-ticket never directly meta-optimize the randomly initialized parameters in NNs, can be considered that the parameter selection by Meta-ticket may produce a similar effect to the parameter optimization like MAML. Now the following question arises: Can sparse NN structures solely be effective for few-shot learning without any information on initial parameters? To answer this question, we consider an extremely restricted setting that all parameters initialized with the same constant value, which prevents parameters from carrying any information about tasks. We conducted experiments in this setting on Omniglot 5-shot 5- -aw classification with 3 adaptation steps. The results are as follows: The subnetworks obtained by Meta-ticket achieve 71.59 + 1.51 % in meta-test accuracy while random subnetworks achieve only about 33 %. This shows that Meta-ticket can find subnetworks being effective for few-shot learning even without leveraging any parameter signals. Also we visualize the sparse structures obtained by Meta-ticket in Figure 4. It indicates that the above success is due to finding sparse structures that exploit the locality of the images in Omniglot.
7

---

Table 2: Cross-domain evaluation for 5-shot 5 5-way classification with large CNNs, which are
meta- deta-trained on minilmageNet and meta-tested on minilmageNet, CUB and Cars.
Networks
ResNer-12
WideResNet-259 10
Meta-train
minilmageNet
minilmageNet
Mela-lest
minilmageNet
CUB
Cars
minilmageNet
CUB
Cars
MAML.
67.47 + 1.31%
54.44 + 0.235
43.68 1.44%
62.83 + 2.47%
46.46 3.54%
37.09 + 2.49%
ANIL
66.88 1.59%
53.90 + 1.17%
40.87 3.95%
63.01 0.82%
46.65 0.20%
37.51 4: 1.95%
BOIL
69.67 + 0.66%
58.79 + 148%
47.11 = 1.10%
65.82 = 5.26%
$5.92 0.78%
12.08 + 0.50%
Meta-ticket
71.31 0.29%
57.97 + 0.53%
45.90 + 0.50%
74.63 + 0.22%
63.26 + 0.86%
52.54 0.70%
+ BOIL
74.23 0.30%
64.06 1.05%
55.20 + 0.64%
73 0.17%
62.19 + 0.69%
51 41 0.83%
4.1.3 Evaluation on different domaín datasets
Finally, we evaluate the cross-domain adaptation ability of Meta-ticket with a 5-layered MLP (denoted by 5-MLP). As discussed in Section 2.3 based on the previous works [50, 44], MAML tends to suffer when adapting to tasks in unknown domains due to its feature reuse property. In contrast, since Meta-ticket prefers rapid learning rather than feature reuse (see Section 3.2), it is expected that the subnetworks obtained by Meta-ticket can generalize tasks in unknown domains.
We conduct the cross-domain evaluation on CIFAR-FS, VGG-Flower and Aircraft. CIFAR-FS consists of classification tasks with relatively general classes from CIFAR-100 [28], and VGG-Flower and Aircraft have specialized classes. Table shows the results. Meta-ticket outperforms MAML by a larger margin especially for tasks in unknown domains (i.e. different from the meta- veeta-training dataset), which indicates that Meta-ticket successfully discovered Noomain-generalinn
sparse structures. Also, we compare with ANIL [50] and BOIL [44] as baselines, which variants of MAML with modified inner learning rates (see Section 2.3 for details). In particular, BOIL satisfics the rapid learning property by definition and thus better meta- geta-generallzzttion than MAML and ANIL. In parallel to MAML, we can consider the variant of Meta-ticket with a₁ = 0 (i.e. the same modification as BOIL) which is denoted by Peta-ticckt+BBII. Since Meta-ticket+Bol or better accuracies than BOIL, we find out that the Meta-ticket-bart based method also benefits from the modification and still achieves superior meta-generalizztton than the MAML-based methods.
achieves almost the same
4.2 Experiments with CNNs on mini-Imagent
Meta-ticket also can be applied to convolutional neural networks (CNNs). Although CNNs already have sparse NN structures as convolutional kernels. they can still benefit from the rapid learnable nature of Meta-ticket. To evaluate the practical benefits as well as meta-generalle veta--eneralliztion ability, we conducted cross-domain evaluation on near real-world datasets. We employed minilmagenet dataset [59, 52], which consists of image classification tasks with 84 84 images and general classes, for meta-trainingttestnn and specialized datasets such as classification of birds (CUB dataset [63, 21)) and cars (Cars dataset [27, 57]) for meta-testing. For CNN architectures, we chose near-practin near-practical deep ones: ResNet-12 [20] and WideResNet-28-11 10 [66].
The results are shown in Table Meta-ticket and its BOIL-variant consistently outperform MAML- based methods, by large margin especially when evaluated on specialized domains, probably because Meta-ticket can adjust its feature extractor to recognize fine- ane-grained differences by the rapid learning. Moreover, while MAML struggles to exploit the potential of larger network such as WideRessNet22--U1 10, Meta-ticket can benefit from such scaling up of network size.
5 Related Work
Gradient-based meta-learning Meta-learning [6, 54, 55] has been studied for decades to equip computers with learning capabilities. There are several approaches depending on how the inner optimization is modeled. Black-box meta- meta-learning approaches model the inner optimization by recurrent [22, 3, 52] or feedforward [41, 19] neural networks. Gradient-basee meta-learning [14] models the inner optimization by one or multiple steps of gradient descent, which leads to sample- efficient meta-learning due to the strong inductive bias on the inner optimization. MAMI. [13] is a representative method of the gradient- based approach, which meta-learns initial parameters for the
8

---

inner optimization. There are also extensions of MAML, including meta-learnable learning rates [34] and transformation matrices [47, 15, 5, 60] of the inner optimization. Some works [50, 44] discussed on feature reuse of MAMI., and pointed out that it may hurt the gettaagenealiizttoon capacity of MAMI.. Since our proposed method (Meta-ticket) another gradient-basee meta-learning method formulated in parallel to MAMI,, most of extensions of MAMI. can also be applied to Meta-ticket. Finally, we note on related works involving both meta-learning and pruning. Tian et al. [56] and Gao et al. [18] investigated magnitude-based pruning methods for meta-learning of MAML, while our work focused on meta-learning by pruning only. Alizadeh et al. [2] exploits meta--gradiett to prune randomly initialized neural networks. Although the methodology is very similar to ours, their target is pruning at initialization [33] for standard learning, not few-shot learning.
Learning neural network structures from data Some prior works [17, 42, 68] attempt to auto- matically learn inductive biases on neural network structures conventionally designed by human. Neyshabur [42] pointed out that the superior generalization capacity of CNNs mainly due to sparse structures, which enables to leverage locality of image data. Also they proposed pruning scheme to automatically find such sparse structures, but does not involve meta-learning and not designed for few-shot learning. Another line of research is neural architecture search (NAS [70]). In particular, meta-lcarning beanning-based based NAS [36. 31, 65] can efficiently search neural architectures. Also, NAS for few-shot learning [25, 35, 12, 62, 61] is closely related to work. Since NAS aims to automatically design how to connect pre-defined blocks and not to find sparse structures in each block, it is orthogonal to our approach of finding sparse NN structures.
Lottery ticket hypothesis Lottery ticket hypothesis (LTH) is originally proposed in Franckle and Carbin [16], which claims the existence of sparse subnetworks in randomly initialized NNs such that the former can achieve good accuracy the latter after training. Inspired by the analysis in Zhou et al. [69]. Ramanujan et al. [51] proposed to train NNs by pruning randomly initialized NNs (weight- pruning optimization) without any optimization of the initial parameters (weight optimization). Their hypothesis are formulated as strong LTH: There exist sparse subnetworks in randomly initialized NNs that already achieves good accuracy without training. The follow-up works [38, 45, 49] theoretically proved it under assumption on the network size. From the viewpoint of LTH, Meta-ticket can be considered as a meta-learning method with weight-pruning optimization, and MAML as one with weight optimization. Surprisingly, even though weight-pruning optimization tends to achieve worse accuracies than weight optimization in general, Meta-ticket outperforms MAML by large margin in cross-domain evaluation.
Applications of sparse NN structures to continual learning Continual learning [55, 46] aims to learn new tasks sequentially without forgetting the capacity learned from past tasks. To handle such multiple tasks, some works [40, 39, 23, 64] leverage sparse NN structures in continual learning. In particular, the approach by Wortsman al. [64] shares similar spirit ours in that they leverage subnetworks of a randomly initialized neural network. Unlike our approach, which finds an optimal subnetwork common to multiple tasks, they leverage task-specific subnetworks separate learned results from multiple tasks.
6 Limitations
As suggested from the theory of strong lottery tickets [38, 45, 49], Meta-ticket also tends to degrade with small NNs compared to MAML (see Section 1.). However, the degradation can be reduced largely by iterative randomization technique [9] as explained in Section 3.3. Moreover, Meta-ticket actually outperforms MAML when using larger NNs. We regard Meta-ticket as an alternative baased meta-learning of MAML, and thus one can switch either of them to use depending on the NN scale. Also, since our theoretical analysis on task gradients (Section 3.2) heavily depends on Taylor series approximation, it holds only when the inner learning rates are sufficiently small. For general cases, it might require the arguments involving the theory on gradient- meta-learning with SGD. Hence, proving the feature reuse in MAML still remains an open problem.

---

7 Conclusion
In this work, we investigated sparse NN structures that effective for few-shot learning. We proposed a novel gradient-based meta-learning method called Meta-ticket, which discovers optimal subnetworks for few-shot learning within randomly initialized neural networks. While Meta-ticket formulated in parallel to MAML, we found that Meta-ticket prefers rapid learning rather than feature reuse in contrast MAML. As a result, since the resulting feature extractor obtained by Meta-ticket can recognize specialized features for each task, Mleta-ticket can achieve superior meta generalization especially on unknown domains. Moreover, we found that Meta-ticket can benefit from scaling up of the size of neural networks while MAML struggles to exploit the potential of large NNs. We hope that this work opens up new research direction in gradient-bar based meta-learning, focusing on sparse NN structures rather than initial parameters.
References
[1] toreh.nn.init Torch 1.11.0 documentation. init. html. Accessed: 2022-05-01.
[2] Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost Amersfoort, Sebastian Far- quhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations, 2022.
[3] Marein Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29, 2016.
[4] Sébastien M R Amold, Praateck Mahajan, Debajyoti Datta, Ian Bunner, and Konstantinos Saitas Zarkias. learn2learn: library for Meta-Learning research. August 2020.
[5] Sungyong Baik, Scokil Hong, and Kyoung Mu Lee. Learning to forget for meta-learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2379-2387, 2020.
[6] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning a synaptic learning rule. Univer- sité de Montréal, Département d'informatique et de recherche opérationnelle, 1990.
[7] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1101.388833333 2013.
[8] Luca Bertinetto, Joao Henriques, Philip Torr, and Andrea Vedaldi. Meta-learning with differentiable closed-form solvers. In International Conference on Learning Representations, 2019.
[9] Daiki Chijiwa, Shin ya Yamaguchi, Yasutoshi Ida, Kenji Umakoshi, and Tomohiro Inoue. Pruning randomly initialized neural networks with iterative randomization. Advances Neural Information Processing Systems, 34, 2021.
[10] Tristan Deleu, Tobias Würff, Mandana Samiei, Joseph Paul Cohen, and Yoshua Ben- gio.
Torchmeta: A Meta-Learning library for PyTorch, 2019. Available at:
[11] Guneet Singh Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. In International Conference on Learning Representations,
2020.
[12] Thomas Elsken, Benedikt Staffler, Jan Hendrik Metzen, and Frank Hutter. Meta-learning of neural architectures for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12365-12375 12375, 2020.
[13] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135. PMLR, 06-11 Aug 2017.
[14] Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018
10

---

[15] Sebastian Flemnerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
[16]
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable
neural networks. In International Conference on Learning Representations, 2019.
[17]
Adam Gaier and David Ha. Weight agnostic neural networks. Advances in neural information
processing systems, 32, 2019.
[18] Dawei Gao, Yuexiang Xie, Zimu Zhou, Zhen Wang, Yaliang Li, and Bolin Ding. Finding meta winning ticket to train your maml. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 41 1-420, 2022.
[19] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713, PMLR, 2018.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778. TEEE Computer Society, 2016.
[21] Nathan Hilliard, Lawrence Phillips, Scott Howland, Artäm Yankov, Courtney D Corley, and Nathan O Hodas. Few-shot learning with metric-agnostic conditional embeddings. arXiv preprint arXiv:1802.04376 2018.
[22] Sepp Hochreiter, A Steven Younger, and Peter R Conwell. Learning to learn using gradient descent. In International Conference on Artificial Neural Networks, pages 87-94. Springer, 2001.
[23] Ching-Y Yi Hung, Cheng-Hao Tu, Cheng- En Wu, Chien-Hune Chen, Yi-Ming Chan, and Chu- Song Chen. Compacting, picking and growing for unforgetting continual learning. Advances in Neural Information Processing Systems, 32, 2019.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[25] Jachong Kim, Sangyeul Lee. Sungwan Kim, Moonsu Cha, Jung Kwon Lee, Youngduck Choi, Yongscok Choi, Dong- Dong-Yeon Cho, and Jiwon Kim. Auto-meta: Automated gradient based meta learner search. arXiv preprint arXiv: 800606697, 2018.
[26] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[27] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[28] Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
[29]
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept
learning through probabilistic program induction. Science, 133-226666:333-33366 2015.
[30] Hae Beom Lee, Hayeon Lee, Donghyun Na, Sachoon Kim, Minseop Park, Eunho Yang. and Sung Ju Hwang. Learning to balance: Bayesian meta-learning for imbalanced and out-of- distribution tasks. In International Conference on Learning Representations, 2020.
[31] Hayeon Lee, Eunyoung Hyung, and Sung Ju Hwang. Rapid neural architecture search by learn- ing to generate graphs from datasets. In International Conference on Learning Representations, 2020.
[32] Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10657-10665, 2019.
[33] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr. SNIP: SINGLE-SHOT NETWORK PRUNING BASED ON CONNECTION SENSITIVITY. In International Conference Learning Representations, 2019.
[34] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for sew-shot learning. arXiv preprint 100::0907909333 2017.
11

---

[35] Dongze Lian, Yin Zheng, Yintao Xu, Yanxiong Lu, Leyu Lin, Peilin Zhao, Junzhou Huang. and Shenghua Gao. Towards fast adaptation of neural architectures with meta learning. In International Conference on Learning Representations, 2020.
[36] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search.
In International Conference on Learning Representations, 2019.
[37] Subhransu Maji. Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification of aircraft, 2013.
[38] Eran Malach, Gilad Yehudai, Shai Shalev-Schwartz, and Ohad Shamir. Proving the lottery ticket hypothesis: Pruning is all you need. In International Conference on Machine Learning, pages 6682-6691. PMLR, 2020.
[39] Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision (ECCV), pages 67-82, 2018.
[40] Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEFE conference on Computer Vision and Pattern Recognition, pages 7765-7773, 2018.
[41] Nikhil Mishra, Mostafa Rohaninejad. Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference Learning Representations, 2018.
[42] Behnam Neyshabur. Towards learning convolutions from scratch. Advances in Neural Information Processing Systems, 33:8078-8088, 2020.
[43] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[44] Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. Boil: Towards representation change for few-shot learning. International Conference on Learning Representations, 2021.
[45] Laurent Orseau, Marcus Hutter, and Omar Rivasplata. Logarithmic pruning is all you need. In
H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2925-2934. Curran Associates, Inc., 2020.
[46] German I Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 11::54-71, 2019.
[47] Eunbyung Park and Junier B Oliva. Meta-curvature.
Advances in Neural Information
Processing Systems, 32, 2019.
[48]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Deiito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019.
[49] Ankit Pensia, Shashank Rajput, Alliot Nagle, Harit Vishwakarma, and Dimitris Papailiopoulos. Optimal lottery tickets via subset sum: Logarithmic over- sece-paramtterization is sufficient. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2599-2610. Curran Associates, Inc., 2020.
[50] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference on Learning Representations, 2020.
[51] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What's hidden in randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11893-11902, 2020.
[52] Sachin Ravi and Larochelle. Optimization as a model for few-shot learning. In International Conference Learning Representations, 2017.
12

---

[53] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (ICV), 115(3)2111-522 2015.
[54] Jürgen Schmidhuher. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131-139 1992.
[55] Sebastian Thrun. Lifelong learning algorithms. In Learning to learn, pages 181-209. Springer, 1998.
[56) Hongduan Tian, Bo Liu, Xiao-Tong Yuan, and Qingshan Liu. Meta-learning with network pruning. In European Conference on Computer Vision, pages 675-700. Springer, 2020.
[57] Hung-Yu Tseng, Hsin-Ying Lee, Jia-Bin Huang, and Ming-Hsuan Yang. Cross-domain few-shot classification via learned feature-wise transformation. In International Conference on Learning Representations, 2020.
[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[59] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[60] Johannes Von Oswald, Dominic Zhao, Seijin Kobayashi, Simon Schug, Massimo Caccia, Nicolas Zucchet, and João Sacramento. Learning where to learn: Gradient sparsity in meta and continual learning. Advances in Neural Information Processing Systems, 34, 2021.
[61] Haoxiang Wang. Yite Wang, Ruoyu Sun, and Bo Li. Global convergence of maml and theory- inspired neural architecture search for few-shot learning. arXiv preprint vrXiv:2203.09133 2022.
[62] Jiaxing Wang, Jiaxiang Wu, Haoli Bai, and Jian Cheng. M-nas: Meta neural architecture search. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6186-6193 2020.
[63] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. California Institute of Technology. CNS-TR-2010- 001, 2010.
[64] Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Raste- gari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. Advances in Neural Information Processing Systems, 33:15173-1511 2020.
[65] Haoran You, Baopu Li, Zhanyi Sun, Xu Ouyang, and Yingyan Lin. Supertickets: Drawing task aggkstigc lottery tickets from supernets via jointly architecture searching and parameter pruning. In European Conference on Computer Vision, pages 674-690. Springer, 2022.
[66] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016.
[67] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, 2017.
[68] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterization.
In International Conference on Learning Representations, 2021.
[69] Hattie Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[70] Barret Zoph and Quoe Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix in the supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix in the supplemental material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes)
(b) Did you mention the license of the assets? [Yes] See Appendix in the supplemental material.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The code for reproducing experimental results is included in the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yH5Wqfv2OSa", "reviewer": "Reviewer_xhDJ", "summary": "This works presents a few-shot learning framework named Meta-Ticket based on gradient-based meta learning. The meta learning is applied to select optimal sub-network from a randomly initialised network, so it’s easier to transfer to a novel domain, as similar to MAML. Meta-ticket has shown to achieve comparable, most of the time better performance than MAML, particularly in larger NN setting, and with more difficult datasets. The paper additionally provides interesting discussions on learning behaviour of few-shot adaptation in comparison with MAML, both in empirical and theoretical perspective. ", "questions": "See the weaknesses.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very easy to follow and supported with a clear and interesting motivation. In general, the authors present a very novel way of doing few-shot learning, combing some of different research areas of machine learning of Lottery ticket hypothesis, randomly initialised network and meta learning.  The discussion on rapid learning v.s. feature reuse is one hightlight of this work, to better understand the learning strategies of different few-shot learning methods. The authors also support the observation with theoretical argument.\n\nSome additional comments which may further improve this work:\n\n-- Search the optimal sub-network is a well-studied technique in transfer/multi-task learning strategy. I would suggest adding relevant related works and slightly discuss the difference of the proposed mask searching strategy: [1,2,3].\n\n[1] Mallya et al. Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights, 2018.\n\n[2] Hung et al. Compacting, Picking and Growing for Unforgetting Continual Learning 2019.\n\n[3] Wortsman et al. Supermasks in Superposition 2020.\n\n-- After searching the optimal subnetwork, I assume the rest of the network parameters are remained unused? If so, the baseline methods were trained with a larger parameter size? This actually leads to unfair comparison, as in the few-shot learning literature, it may leads to degraded performance due to overfitting. \n\n-- Is there any hyper-parameter controlling the size of the subnetwork? It would be interesting to see how different size of subnetwork can relate to the final few-shot learning performance.\n\n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ul2idaS2CI-", "reviewer": "Reviewer_jF2G", "summary": "The authors proposed a novel meta-learning approach referred to as Meta-ticket, to find optimal sparse subnetworks for few-shot learning within randomly initialized NNs. \n\nThey empirically validated that Meta-ticket successfully finds subnetworks that can learn specialized features for each given task. \n\nDue to this rapid task-wise adaptation ability, Meta-ticket achieves superior meta generalization compared to MAML-based methods, especially with large NNs. \n", "questions": "The Authors stated that Meta-ticket prefers rapid learning rather than feature reuse. On similar cross-domain meta-tasks, feature reuse might be preferred over rapid learning. \n\nIs there any case that feature reuse is better than rapid learning in the Meta-ticket (adaptations such as specific to general or specific to specific)? A plot like Fig.2 would be helpful for better analysis of other adaptations.\n\nThe effects of the initial sparsity in B.2 (Fig.1 and Table 6) were a good analysis. The layerwise threshold hyperparameter seems to be sensitive to the initialization of weight scores. The authors could prepare the sensitivity analysis on various kinds of initialization methods with their iteration-validation performances.\n\n", "limitations": "The authors showed the limitation of subnetworks; Meta-ticket tends to degrade with small NNs compared to MAML.\n\nTo the issue, this work has found the solution to the degradation caused by the small NNs, providing iterative randomization techniques.\n", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors proposed a new Meta-ticket for few-shot learning, inspired by Lottery Ticket Hypothesis.\n\nThe Meta-ticket finds optimal sparse subnetworks which tend to adapt to each task with a large change in the overall parameters. \n\n(+) The Meta-ticket is technically sound and seems to be supported by several experimental results. \n\n(+) For fair comparisons with others, the authors prepared the cross-domain experimental settings and baselines – MAML, ANIL, and BOIL. \n\n(-) The authors only showed the parts of adaptations such as general to specific and general to general.\n\n(-) There is no ablation study on the layerwise threshold hyperparameter which seems to be sensitive to the initialization of weight scores.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OaaOxQmy6E0", "reviewer": "Reviewer_hLtC", "summary": "This paper introduces a novel gradient-based meta-learning method, Meta-Ticket, as an alternative of MAML. For meta-objective, Meta-Ticket aims to optimize a sparse subnetwork hidden within a randomly initialized neural network (meta-mask, called meta-ticket), while MAML optimizes the initial parameters (meta-initialization). \n\nThe experiments show that, Meta-ticket can achieve superior performance than MAML on cross-domain few-shot settings (over different datasets), although it tends to achieve worse accuracies than MAML in general setting.", "questions": "If ProsPr [1] is applied to a single few-shot task, can it achieve the same results as Meta-ticket (that is trained over many few-shot tasks) on testing few-shot tasks?", "limitations": "See above", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. If focusing on cross-domain few-shot applications, the proposed Meta-Ticket achieves better meta-generalization ability rather than MAMLs.\n2. Sufficient experiments over many datasets (domains) focusing on different cross-domain tasks and different types of NNs.\n\nCons:\n1. Novelty is limited. The proposed Meta-Ticket algorithm is almost the same as ProsPr [1], which also optimizes a subnetwork (learns a mask $\\mathbf{m}$) AND uses meta-gradients to take into account that we are going to train the weights after we apply mask to get subnetwork. \n\nHowever, there are two differences between this paper and [1]:\n    (1) The x <- UpdateScores(s, L) in the paper replaces the \"saliency scores computation\" in [1] with the technique in [37].\n    (2) The training steps after pruning, in this paper, is obtained from the few-shot validation data from multiple tasks. But in [1], the training steps after pruning is standard training from large data of single task. \n\nDespite the two differences, I still think the novelty of the proposed algorithm is not enough.\n\n2. As pointed out in the paper (Section 6 and 4.1.1), the Meta-Ticket may tend to optimize larger network to achieve similar results as MAML, as it tries to find a hidden prune-mask within the fixed random parameters. This could be a limitation (i.e., memory problem, speed issue, etc) if training machines do not have enough resources. \n3. In experiments, missing a lot of SOTA gradient-based meta-learning methods. Especially, there are many task-specific gradient-based methods (e.g., LEO, MMAML) that is good for cross-domain few-shot learning. Maybe the authors should consider including them.\n\n[1] Milad Alizadeh, Shyam A. Tailor, Luisa M Zintgraf, Joost van Amersfoort, Sebastian Far- quhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initialization using meta-gradients. In International Conference on Learning Representations, 2022.\n\n[37] Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, and Mohammad Rastegari. What’s hidden in a randomly weighted neural network? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11893–11902, 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "D_GSZvuUrV", "reviewer": "Reviewer_KHZB", "summary": "The authors propose Meta-ticket as a meta-learning approach that finds optimal sparse subnetworks for few-shot learning within the randomly initialized neural network. The paper can be seen as a MAML modification. ", "questions": "1. How does the method work on the pretrain model?\n2. Is it possible to train the weight of the network simultaneously with mask m?\n3. It is a good way to compare original MAML and Meta-ticket with some architectures. Does it seem that MAML requires fewer parameters to solve a few-shot task than Meta-ticket?", "limitations": "The limitation section is well written.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths \n1. The paper is well written and easy to follow.\n2. The idea looks fresh and interesting.\n3. The analysis of rapid learning and feature reuse is well done.\nWeaknesses\n1. It is unclear why the authors refer only to MAML (and its modification). If the paper introduces a new method for a few shots, the method should be compared with the state-of-the-art model using the gaussian process, transformers, and hyper networks. \n2. On the other hand, the paper can be seen as another MAML-based method analysis. But the contribution in the paper should be changed.\n3. It is not clear why authors use randomly initialized neural networks. How does the method work on the pretrain model?\n4. The fig. 1 should be better described.\n5. Section 3.1 is not clear. Functions CalculateMask() and UpdateScore() are only mentioned and do not described in detail. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept