Localized Meta-Learning: A PAC-Bayes Analysis for Meta-Learning Beyond Global Prior
Authors: Chenghao Liu
Tao Lu
Doyen Sahoo
Yuan Fang
Kun Zhang
Steven Hoi
Venue:  (2021)
Keywords: localized meta-learning
PAC-Bayes
meta-learning

Abstract: Meta-learning methods learn the meta-knowledge among various training tasks and aim to promote the learning of new tasks under the task similarity assumption. Such meta-knowledge is often represented as a fixed distribution; this, however, may be too restrictive to capture various specific task information because the discriminative patterns in the data may change dramatically across tasks. In this work, we aim to equip the meta learner with the ability to model and produce task-specific meta-knowledge and, accordingly, present a localized meta-learning framework based on the PAC-Bayes theory. In particular, we propose a Local Coordinate Coding (LCC) based prior predictor that allows the meta learner to generate local meta-knowledge for specific tasks adaptively. We further develop a practical algorithm with deep neural network based on the bound. Empirical results on real-world datasets demonstrate the efficacy of the proposed method.

Under review as a conference paper at ICLR 2021
LOCALIZED META-LEARNING A PAC-BAYES ANAL- YSIS FOR META- BEYOND GLOBAL PRIOR
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-learning methods learn the meta-knowledge among various training tasks and aim to promote the learning of new tasks under the task similarity assumption. Such neet-knnowledge is often represented as a fixed distribution; this, however, may be too restrictive to capture various specific task information because the discriminative patterns in the data may change dramatically across tasks. In this work, we aim to equip the meta learner with the ability to model and produce task-specific meta knowledge and, accordingly, present a localized meta-learning framework based the PAC-Bayes theory. In particular, Coordinate Coding (LCC) based prior predictor that allows the meta learner to generate local -eotl-knowwegge for specific tasks adaptively. We further develop practical algorithm with deep neural network based on the bound. Empirical results real-world datasets demonstrate the efficacy of the proposed method.
propose Local
11
13
INTRODUCTION
Recent years have seen resurgence of interest in the field of meta-learning, or learning-to-ledt to-rinngt---earn (Thrun & Pratt 2012), especially for empowering deep neu-
kootlkkwwwedgee
Localized Meta-knowledge
15
Task Instances
17
ral networks with the capability of fast adapting to unseen tasks just humans (Finn al. 2017 Ravi
18
Adaptation Task
19
& Larochelle| 2017). More concretely, the neural networks are trained from sequence of datasets, associated with different tasks sampled from meta- distribution (also called task environment (Baxter 2000, Maurer 2005)). The principal aim of meta
20
21
22
23
24
learner is to extract transferable meta-knowledge from observed tasks and facilitate the learning of new tasks sampled from the same meta-distribution. The performance is measured by the generalization ability from a finite set of observed tasks, which evaluated by learning related unseen tasks. For this reason, there has been considerable interest in theoretical bounds on the generalization for meta-learning algorithms (Denevi et al. 2018b a).
Figure 1: Illustration of the localized meta- learning framework. Instead of using global meta-knowledge for all tasks, tailor the meta-knowledge for various specific task.
26
28
30
31
One typical line of work (Pentina Lampert 2014 Amit & Meir 2018 uses PAC-Bayes bound to analyze the generalization behavior of the meta learner and quantify the relation between the expected loss new tasks and the average loss on the observed tasks. In this setup, it formulates meta-learning
32
33
34 as hierarchical Bayes. For each task, the base learner produces posterior based on the associated
35 task data and the prior. Each prior a reference w.r.t. base model that generated by the meta leaner and must be chosen before observing task data. Accordingly, meta-knowledge is formulated as
37 global distribution over all possible priors. Initially, it is called as hyperprior since it is chosen before observing training tasks. To learn versatile meta-knowledde across tasks, the meta learner observes 39 sequence of training tasks and adjusts its hyperprior into a hyperposterior distribution over the set of priors. The prior generated by the hyperposterior is then used solve tasks.
41 Despite of its great success, such global hyperposterior rather generic, typically not well tailored to various specific tasks. In contrast, in many scenarios the related tasks may require task-specific 43 meta- keotu-nnwwledee.. Consequently, traditional meta- neeal-nnoweedee may lead to sub-optimal performance for any individual prediction task. As motivational example, suppose have two different

---

Under review as a conference paper at ICLR 2021
45
tasks: distinguishing motorcycle versus bicycle and distinguishing motorcycle versus car. Intuitively, each task uses distinct discriminative patterns and thus the desired meta- veeta-nnoledge is required to extract these patterns simultancously. It could be a challenging problem to represent it with global hyperposterior since the most significant patterns in the first task could be irrelevant or even detrimental to the second task. Figure schematically illustrates this notion. Therefore, customized meta-knowledge such that the patterns are most discriminative for given task is urgently desired. Can the meta-knowledge be adaptive to tasks? How can one achieve it? Intuitively, we could implement this idea by reformulating the meta-knowledge as a maping function. Leveraging the samples in the target task, the meta model produces tasks specific meta-knowleege.
46
48
50
51
52
53
54
Naturally yet interestingly, one can see quantitatively how customized prior knowledge improves generalization capability, in light of the -Ay-B literature on the data distribution dependent-prion (Catoni 2007 Parrado-Hernan Hernandez et al. 2012, Dziugaite & Roy| 2018). Specifically, PAC-Bayes
55
56
57
bounds control the generalization error of Gibbs Classifiers. They usually depend on a tradeoff between the empirical error of the posterior Q and KL-divergence term K PL)) where / is the prior. Since this KL-divergence term forms part of the generalization bound and is typically large in standard PAC-Bayee approaches (Lever et al. 2013), the choice of posterior is constrained by the need to minimize the KL-divergence between prior P and posterior Q. Thus, choosing an appropriate prior for each task which is close the related posterior could yield improved generalization bounds. This encourages the study of data distribution-deeeedeent priors for the PAC-Bayes analysis and gives rise principled approaches to localized PAC-Bayes analysis. Previous related work are mainly discussed in Appendix A
58
59
61
62
63
64
65
66
Inspired by this, we propose a Localized Meta-Learning (LML) framework by formulating meta- knowledge as a conditional distribution over priors. Given task data distribution, we allow a meta learner to adaptively generate an appropriate prior for new task. The challenges of developing this model are three-fold. First, the task data distribution is not explicitly given, and our only perception for it is via the associated sample set. Second, it should be permutation invariant - the output of model should not change under any permutation of the elements in the sample set. Third, the learned
67
68
69
70
71
72 model could be used for solving unseen tasks. To address these problems, we further develop a prior
73 predictor using Local Coordinate Coding (LCC) Yu et al.) 2009). In particular, if the classifier in
74 each task is specialized to a parametric model, e.g. deep neural network, the proposed LCC- based prior predictor predicts base model parameters using the task sample set. The main contributions include: (1) A localized meta- meta-learning framework which provides a means to tighten the original PAC-Bayes meta-learning bound (Pentina & Lampert 2014 Amit & Meir 2018) by minimizing the task-complexity term by choosing data-dependent prior; (2) An LCC-based prior predictor, an implementation of conditional hyperposterior, which generates local keeelenno for specific task; (3) A practical algorithm for probabilistic deep neural networks by minimizing the bound (though the optimization method can be applied to large family of differentiable models); (4) Experimental results which demonstrate improved performance over meta-learning method in this field.
76
77
78
79
80
81
82
83
2 PRELIMINARIES
84
85
Our prior predictor was implemented by Local Coordinate Coding (LCC). The LML framework was inspired by PAC-Bayes theory for meta learning. In this section we briefly review the related definitions and formulations.
86
87
88
2.1
LOCAL COORDINATE CODING
Definition 1. (Lipschitz, Smoothness Yu et al. 2009).) (.) A functinn f(x) in Rdd is a (a, ().s))-ipschhtz
89
smooth w.r.t. a norm ifl.(()) -((x') a X x'| || and (x(( (1xx))-((( (x)) (x' -x]] S
90
0||x - x'1?2.
91
92
Definition 2. (Coordinate Coding Yu et al. (2009). .) A coordinate coding is a pair (7,, C), where
C is a set of anchor voints(bases)) and Y is a map of x E Rd to [7u(x) E RRC such that Σ. 7x() = 1. It induces the following physical approximation of x in Rdd :x= Eucc Luechu(x)u (u(x),. Definition 3. (Latent Manifold Yu et al. (2009 A subset M C. Rdd is called a smooth manifold with an intrinsic dimension (C) : d.y if there exists constant CM such that given any x E M.
93
2

---

Under review as a conference paper at ICLR 2021
there exists |C|
anchor points u1 (xx)........... ...,, (x)
R so that vxx M:
CI
inf || -x - (x)2 S cmllx' x
PERICI
j=1
where y = [71 Mail" are the local codings w.r.t the anchor points.b 95
Definition 2 and 3 imply that any point in Rd can be expressed as a linear combination of a set of anchor points. Later, we will show that high dimensional nonlinear prior predictor can be approximated by simple linear function w.r.t. the coordinate coding, and the approximation quality ensured by the locality of such coding (each data point can be well approximated by a linear combination of its nearby anchor points).
96
97
98
99
100
101
2.2
PAC-BAYES REGULAR HOITA-LEARNIG
102
In order to present the advances proposed this paper, we recall some definitions in PAC-Bayes theory for single-task learning and meta-learning (Catoni 2007 Baxter 2000 Pentina & Lampert Amit & Mcir 2018). In the context of classification, we assume all tasks share the same input space X. output space V space of classifiers (hypotheses) H. (h : X - ))} and loss function f y x y + [0. 1]. The meta learner observes n tasks the form of sample sets Si...., S,. The
104
105
106
number of samples in task is denoted by m,, Each observed task i consists of a set of i.idd samples S. - (x } m! which is drawn from a data distribution S; ~ D. Following the meta-learning setup (Baxter 2000), we assume that each data distribution D; is generated i.i.d. from the same meta distribution T. Let h(x) be the prediction of x. the goal of each task is to find classifier h that minimizes the expected loss ( y). Since the underlying 'true" data distribution D, is unknown, the base learner receives a finite set of samples S; and produces an "optimal" classifier As(S:) with a learning algorithm A-(-) that will be used to predict the labels of unseen inputs.
108
109
110
111
112
113
114
PAC- ACCBaays theory studies the properties of randomized classifier, called Gibbs classifier. Let Q be posterior distribution over H. To make a prediction, the Gibbs classifier samples a classifier h E H according to Q and then predicts a label with the chosen h. The expected error under data distribution D and empirical error on the sample set S are then given by averaging over distribution Q. namely (Q)))= (x)-)
115
117
1 y) and ér Q) En~q m Σソ= {(h(z,), )) 3/y respectively.
In the context of meta-learning, the goal of the meta learner is to extract meta- wntal-nnoweddge contained in the observed tasks that will be used as prior knowledge for learning new tasks. In each task, the prior knowledge P is in the form of distribution over classifiers H. The base learner produces a posterior Q = A, (S P) over H based on a sample set and a prior P. All tasks are learned through the same learning procedure. The meta learner treats the prior P itself as random variable and assumes the meta- knowledge is in the form of a distribution over all possible priors. Let hyperprior P be an initial distribution over priors, meta learner uses the observed tasks to adjust its original hyperprior P into hyperposterior Q from the learning process. Given this, the quality of the hyperposterior Q is measured by the expected task error of learning new tasks using priors generated from it, which is formulated as:
er( () QlE D. m) Dm (Q- As,S, P)). Accordingly, the empirical counterpart of the above quantity is given by:
( Ep
êqQQ Ao(S,, P)).
119
2.3 PAC-BAYES REGULAR META- BOUND WITH GAUSSIAN RANDOMIZATION
120
Based on the above definitions, Pentina & Lampert 2014) and Amit & Meir (011)) present regular
121
meta-learning PAC- BBC-B generalization bounds w.r.t. hyperposterior Q. Notably, the proof technique in Amit & Meir (2018) allows to incorporate different single task bounds. Consider the benefit of
122
123
Catoni's bound (Catoni 2007) (the minimization problem derived from the bound is a simple linear
combination of empirical risk plus regularizer), here we instantiate regular meta-learning bound with Gaussian randomization based on that. To make fair comparison, we will adopt the same Catoni's bound to analysis the proposed LML framework later. Particularly, the classifier h is parameterized as hw with w E Rdw. The prior and posterior are a distribution over the set of all possible parameters
125
127
w. We choose both the prior P and posterior Q to be spherical Gaussians, i.e. P = N(w". o Ta..)
128
3

---

Under review as a conference paper at ICLR 2021
Task
Task
Task
Task-
Task 1
Task 2
Task n
Future Task-
$ D
$2-D2 D2
ft D
S-Dnew
S D1-D1
S2 Dz=Dz
Sm Di
S- D--Din
Prior
Prior
Learner
Localized Meta
Base Learner
Aglibul dotributon
N (WWR,021a
W
S ( new
Wnew
W W
Figure 2: Comparison between PAC-Bayes regular meta-learning (left) and LML (right) In regular meta-learning, the mean of prior w is sampled from a global hyperposterior distribution Q = N(w o Idw In LML,, w is produced by prior predictor Φ (D
and N(w ou J. The mean w is random variable distributed first according to the hyperprior' P, which we formulate as (O0, o Td..) and later according to hyperposterior Q, which
129
130
we model (Ww Liw When encountering a new task i, we first sample the mean of prior w from the hyperposterior N Vww2.oz Ld.)), and then use it as basis to learn the mean of posterior
131
132
Wi A& P)) as shown in Figure 2 Then, could derive the following PAC-Bayes meta-learning bound.
134
Theorem 1. Consider the regular meta-learning framework, given the hyperprior p NO ( law) Then for any hyperposterior Q. any C1, C2 > 0 and any 8 (0, 1] with probability 2 we have,
cí
cí
(r(@) Scieee())
202nmood +
2cino 1| 12
WP E w 1
202nm,0
1
2m,
d
2
+ log
log cino
(3)
canmiow 1-e-or and d2
2
where e1 and obtain that
I-e--z To get better understanding, we further simplify the notation
di
& 2c:no
dd
er( Sddee(()
1
|E 1 -
20mmm,ow
20mnm
losk-compleaing
+ const(6, n, Mi. ow, 02).
(4)
135 See Appendix D.4 for the proof. Notice that the expected task generalization error is bounded by the 138 empirical multi-task error plus two complexity terms which measures the -ooplllent and the cosk-compl,x,ty, respectively.
137
PAC-BAYES LOCALIZED META-LEARNING
138
3.1 MOTIVATION AND OVERALL FRAMEWORK
140
Our motivation stems from a core challenge PAC-Bayes meta-learning bound in 4 wherein
EEw 2 which measures the closeness between the mean of posterior and the mean of global hyperposterior for each task, is typically vital to the generalization bound. Finding the tightest possible bound generally depends minimizing this
the task-complexity term D -i=1 Zoynm,o
142
143
diCWE?
term It is obvious that the optimal 2 is Σ 2ezam,ng This solution for global hyperposterior is required satisfy the task similarity assumption that the optimal posteriors for each task are close together and lie within small subset of the model space. Under this circumstance, there exists global hyperposterior from which good prior for any individual task reachable. However, if the optimal posteriors for each task are not related even mutually exclusive, i.c., optimal posterior has negative effect on another task, the global hyperposterior may impede the learning of some tasks. Moreover, this complexity term could be inevitably large and incur large generalization error.
145
147
149
151
Note that @ is the mean of hyperposterior Q and this complexity term naturally indicates the divergence between the mean of prior w." sampled from the hyperposterior Q and the mean of posterior w in each task. Therefore,
propose adaptively choose the mean prior w!" according to task i. It obvious that the complexity term vanishes we set w." = w but the prior P; in each task has be chosen independently of the sample set S,. Fortunately, the PAC-Bayes
153
154
4

---

Under review as a conference paper at ICLR 2021
theorem allows us to choose prior upon the data distribution D,. Therefore, we propose prior predictor & : D + w" which receives task data distribution D and outputs the mean of prior w". In this way, the generated priors could focus locally on those regions of model parameters that are of particular interest in solving specific tasks.
157
159
Particularly, the prior predictor is parameterized hy with v E Rdv, We assume to be a random variable distributed first according to the hyperprior P. which we reformulate as (O, oaTa.). and later according to hyperposterior 2, which we reformulate as ((v @ ; ). Given a new task i, we first sample v from hyperposterior ((ve, o la,) and estimate the mean of prior w by leveraging prior predictor w = (D."). Then, the base learner utilizes the sample set S and the prior
160
161
162
165
184
P: = NW(... ow to produce a mean posterior w/ (S; Pt), as shown in Figure right..
165
To make w close to w in each task, what properties are the prior predictor is expected to exhibit? Importantly, is required to (i) uncover the tight relationship between the sample set and model parameters. Intuitively, features and parameters yield similar local and global structures in their respective spaces in the classification problem. Features in the same category tend to be spatially clustered together while maintaining the separation between different classes. Take linear classifiers as an example, let Wa be the parameters w.r.t. category k. the separability between classes is implemented as x Wa. which also explicitly encourages intra-class compactness. A reasonable choice of W is to maximize the inner product distance with the input features in the same category and minimize the distance with the input features of the non- non- con-ellnginng categories. Besides, the prior predictor should be (ii)
166
167
168
169
170
171
172
175
175
since it will be used continuously as new tasks and hence new categories become available. Lastly, it should be (iii) invariant under permutations of its inputs.
176
177
3.2
LCC-BASED PRIOR PREDICTOR
There exists many implementations, such set transformer (Lee et al. 2018). relation network (Rusu) et al. 2019), (ae2vee((Achlle et al. [2019), that satisfy the above conditions. We follow the idea of nearest class mean classifier (Mensink al. 2013), which represents class parameter by averaging
179
181
its feature embeddings. This idea has been explored in transductive few-shot learning problems (Snell
182
et al. 2017| Qiao et al. 2018). [Snell et al. (2017) learn metric space across tasks such that when represented in this embedding, prototype (centroid) of each class can be used for label prediction in the new task. Qiao et al. (2018) directly predict the classifier weights using the activations by exploiting the close relationship between the parameters and the activations in a neural network associated with the same category. In summary, the classification problem of each task is transformed
189
184
185
186
187
as a generic metric learning problem which is shared across tasks. Once this mapping has been learned on observed tasks, due to the structure- prrettrrrrr property, could be easily generalized to new tasks. Formally, consider each task as K-class classification problem, and the parameter of the classifier in task i denoted as w, = [w:[1] [...,,,... w; [k].k..... w; [K]], the prior predictor for class can be defined as:
188
190
...,
[k] th (D
E
1
Σ (x(x))
(5)
SunD mikk xyESS,
where (.).): IRd - Rdw is the feature embedding function, ma is the number of samples belonging to category k. Sia and Dis are the sample set and data distribution for category k in task i. We call this function the expected prior predictor. Since data distribution Dia is considered unknown and our only insight as to Dis is through the sample set Sik. we approximate the expected prior predictor by its empirical counterpart. Note that if the prior predictor is relatively stable to perturbations of the sample set, then the generated prior could still reflect the underlying task data distribution, rather than the data, resulting in generalization bound that still holds perhaps with smaller probability (Dziugaite & Roy 2018). Formally, the empirical prior predictor is defined as:
w/'kk] (Sax) 1
Σ (x,)).
(6)
mik X,ESit
Although we can implement the embedding function (-) with a multilayer perceptron (MLP), both input x E ded and model parameter w E gdd are dineniiimniionnl,, making the empirical prior predictor (-) difficult to learn. Inspired by the local coordinate coding method, if the anchor points are sufficiently localized, the embedding function de (xz) can be approximated by a linear function w.r.t. set of codings, [7u (u(xz))uecc Accordingly, we propose LCC-based prior predictor, which
5

---

Under review as a conference paper at ICLR 2021
is defined as:
wf (Sas))
Σ
Σ
%u(x)) (u),
Makk
X, EStt DEC
192 where @v @v( E Rdw is the embedding of the corresponding anchor point u E C. As such, the parameters of LCC-based prior predictor w.r.t. category A can be represented Vş
194 1 (uz), Ov, (uz),.... dv. (u]ci)] Lemma] illustrates the approximation error between empirical prior predictor and LCC-based prior predictor.
Lemma 1. (Empirical Prior Predictor Approximation) Given the definition of w [k] and w! [K] in Eq. O6) and Eq. 7)) ler (r, C) be an arbitrary coordinate coding on I2 and (.(.) be an (ox, -Liipschht smooth function. We have for all E Rd
l! [b]] w [k]]] I Oa,
(8)
198
where 0 (7, - MILE 2x, ES
[||> -
+1 Euec ||x;- ull?) andx; EuEc Tu (x(x;)u
J.
197
See Appendix D.I for the proof. Lemma] shows that good LCC-based prior predictor should make x close to its physical approximation and should be localized. The complexity of LCC coding scheme depends on the number of anchor points |C). We follow the optimization method in Yu et al.
198
200
2009) to find the coordinate coding (y, C), which is presented in Appendix B
201
3.3
PAC-BAYES LOCALIZED META- LEARNING BOUND WITH GAUSSIAN RANDOMIZATION
In order to derive PAC-Bayes generalization bound for localized meta-learning we first bound the approximation error between expected prior predictor and LCC-based prior predictor.
203
Lemma 2. Given the definition of P and w" in Eq.
5 and 77, let X' be a compact set with radius
R. i.e. x' E X, 1|x - x'1l S
R. For any ő E [0,1) with probability 2 1 - 6, we have
l|w"" w 113
aR (1+ + V log())) O0,e1.
Vmisk
See Appendix D.2 for the proof. Lemma shshows that the approximation error between expected prior predictor and LCCC-based prior predictor depends on (i) the concentration of prior predictor and (ii) the quality of LCC coding scheme. The first term implies the number of samples for each category should be larger for better approximation. This is consistent with the results of estimating
205
208
207
208
the center of mass (Cristianini & Shawe-Taylor 2004). Based on Lemma using the same Catoni's bound. we have the following Payes LML bound.
Theorem 2. Consider the localized meta-learning framework. Given the hyperprior P N (O of (dy) ), then for any hyperposterior Q, any C1,02 02 > 0 and any å E (0,1) with probabil- ity 2 8 we have,
dí 202nm.o3
&
er()) (c)
+
1 + +Σ 2ctno?
IEW - eS)))l
commo
An aR (1 log 5
+ +0a.01,.)) + dw K( 22)
Mis
k
log An
di
2 8'
log
(9)
Conmiow 8
2
where ci T-e-sy and c2 --- and obtain that
To get better understanding, we further simplify the notation
d
diis Conm,
er(2) Sc(cher())
)[[ 91
EE
T. e
2
2cino
taak-complexrity
const(a, 3, R, n, mi, dy, ow, C1 02).
(10)
See appendix D.3 for the proof. Similarly to the regular meta-learning bound in Theorem I the expected task error er( Q) is bounded by the empirical task error ér Q)) plus the cask-complextty and
211
enviromment-complexin terms. The main innovation here is to exploit the potential to choose the 213 mean of prior w' adaptively, based on task data S. Intuitively, the selection of the LCC-based
prior predictor is appropriate, it will narrow the divergence between the mean of prior w.? sampled

---

Under review as a conference paper at ICLR 2021
from the hyperposterior Q and the mean of posterior w. in each task. Therefore, the bound can be tighter than the ones in the regular meta-learning (Pentina & Lampert] 2014: Amit & Meir (0188). Our empirical study in Section 4 will illustrate that the algorithm derived from this bound can reduce task-complexity and thus achieve better performance than the methods derived from regular
216
218
219
meta-learning bounds.
220
When one is choosing the number of anchor points |C], there is a balance between accuracy and simplicity of prior predictor. As we increase |C*, it will essentially increase the expressive power of
221
(-) and reduce the task-complexity term [|Ew ( However, at the same time, it will
222
increase the petiormmmnttcoomllen term || and make the bound loose. If we set |C| to 1, it degenerates to the regular meta-lcarning framework.
224
225
3.4 LOCALIZED META- ALGORITHM
Since the bound in 9 holds uniformly w.r.t Q, the guarantees of Theorem also hold for the resulting learned hyperposterior Q (vvv oe ), so the mean of prior w'" sampled from the learned hyperposterior work well for future tasks. The PAC-Bayes localized meta-learning bound in ( can
be compactly written as プ (Qr; : = As, P)) + on || | + Σ dz
|lE - @ (1.)
my v
where 01, 0/ > 0 are hyperparameters. For task i, the learning algorithm As(-) can be formulated as w = arg min (Q; = (N(?? o )). To make fair comparison and guarantee the benefit of w
the proposed LML is not from using an improved optimization method, we follow the same learning algorithm in (Amit& & Meir 2018). Specifically, we jointly optimize the parameters of LCC-based prior predictor v and the parameters of classifiers in each task WT, W2, .... Wa. which formulated as
n
Σ Ber.(w.)) + on 112 + IE o. i=1
arg min
(11)
V.WI WWW.....WW.... Wi.
i=1
We optimize and via mini mini-batch SGD. The details of the localized meta- meta-learning algorithm is given in Appendix(F [F] The expectation over Gaussian distribution and its gradient can be efficiently estimated by using the e--arameterization trick Kingma & Welling (2014); Rezende al. (201)..
227
228
For example, to sample w from the posterior Q = NNw aw ldw) we first draw & &~ NO, Ta,,)
229
and then apply the deterministic function w + f C or, where @ is an element-wise multiplication.
230
4
EXPERIMENTS
232
Datasets and Setup. We use CIFAR-100 and Caltech-256 in our experiments. CIFAR- Krizhevsky (2099) contains 60,000 images from 100 fine-grained categories and 20 coarse-level categories. As in Zhou et al. (2018), we use 64, 16, and 20 classes for meta-training, meta-validation, and meta-testing, respectively. Caltech- 256 has 30,607 color images from 256 classes Griffin et al. 2007). Similarly, we split the dataset into 150, 56 and 50 classes for meta-training, meta- validatal,,,, and meta-testing- We consider 5-way classification problem. Each task is generated by randomly sampling 5 categories and each category contains 50 samples. The base model uses the convolutional architecture in Finn al. (2017). which consists of 4 convolutional layers, each with 32 filters and a fully-connected layer mapping to the number of classes on top. High dimensional data often lies on some low dimensional manifolds. We utilize an auto-encoder to extract the semantic information of image data and then construct the LCC scheme based on the embeddings. The parameters of prior predictor and base model are random perturbations in the form of Gaussian distribution.
234
236
238
239
240
241
242
243
244 We design two different meta-learning environment settings to validate the efficacy of the proposed method. The first one uses the pre-trained base model as an initialization, which utilizes all the mraiinga classes (64-class classification in CIFAR- 100 case) to train the feature extractor. The second one uses the random initialization. We compare the proposed LML method with ML-PL method Pentina & Lampert 20014), ML-AM method Amit & Meir (018)) and ML-A which is derived from Theorem| [| In all these methods, we use their main theorems about the generalization upper bound to derive the objective of the algorithm. To ensure fair comparison, all approaches adopt the same network architecture and pre-trained feature extractor (more details can be found in
245
246
247
249
251
Appendix F)
252
7

---

Under review as a conference paper at ICLR 2021
CaRtech-256
Accuracy on New Taskg%))
CIFAR-100
Caltech-256
CIFAR-100
LNL
MLAM
UNL
ML-4N
Accura
Accuran
LML
Number Tasks
Number Tasks
Number Tasks
Number Tasks
(a) With pre-trained feature extractor
(b) Without pre-trained feature extractor
Figure 3: Average test accuracy of learning a new task for varied numbers of training tasks ((C) 64).
253
Results. In Figure g we demonstrate the average test error of learning a new task based on the
number of training tasks, together with the standard deviation, in different settings (with without a pre-trained feature extractor).
255
is obvious that the performance continually increases as we increase the number of training tasks for all the methods. This is consistent with the generalization bounds that the complexity term converges to zero if large numbers of tasks are observed. ML-A consistently outperforms ML-PL and ML-AM since the single-task bound used in Theorem [ ML-A)
257
258
converges at the rate of ( while the bounds w.r.t. MI.PII. and ML-AM converge at the rate
259
of O ). This demonstrates the importance of using tight generalization bound. Moreover, our 260
proposed LML significantly outperforms the baselines, which validates the effectiveness of the proposed LCC-based prior predictor. This confirms that LCC-based prior predictor is a more suitable representation for meta-knowledge than the traditional global hyperposterior in ML-A, ML-AM, and ML-PL. Finally, we observe that if the pre-trained feature extractor is provided, all of these methods do better than meta-training with random initialization. This because the pre-trained feature extractor can be regarded as a dependent hyperpior. It is closer to the hyperposteior than the randomly initialized hyperprior. Therefore, it is able reduce the environment complexity term and improves the generalization performance.
262
263
264
265
266
267
Accuracy on New Taak(5))
Callech-256
New Ta
LIFAR-100
Caltech-256
CIFAR-100
82.
20
Number of Bases
Number of Bases
Number of Tasks
Number of Tasks
Figure 4: (a) The impact of the number of anchor points |C| in (C) The divergence value (normalized) between the mean generated prior wF and the mean of learned posterior w
(a)
(b)
268
269
4(b we show the divergence between the mean of generated prior w from meta model and the mean of learned posterior w for LMI and MI.-A. This further validates the effectiveness of the LCC-based prior predictor which could narrow down the divergence term and thus tighten the bound. In Figure 4(a) we vary the number of anchor points CCI in LCC scheme from 4 to 256, the optimal value is around 64 both datasets. This indicates that LML is sensitive to the number of anchor points |C), which further affects the quality of LCC-based prior predictor and the performance
In Figure
270
271
273
275
of LML.
CONCLUSION
276
277
This work contributes a novel localized meta-learning framework from both the theoretical and computational perspectives. In order to tailor meta- meta-knowledde to various individual task, we formulate meta model as mapping function that leverages the samples in target set and produces task specific knell-nnoweedge as prior. Quantitatively, this idea essentially provides a means to theoretically tighten the PAC-Bayes meta-learning generalization bound. We propose LCC-based prior predictor
278
279
280
281
282
to output localized meta-knowledge by using task information and further develop a practical algorithm with deep neural networks by minimizing the generalization bound. An interesting
284
topic for future work would be to explore other principles to construct the prior predictor and apply the localized meta-learning framework to more realistic scenarios where tasks are sampled non-i.i.d.
206
from an environment. Another challenging problem is to extend our techniques to derive localized
287
meta-learning algorithms for regression and reinforcement learning problems.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
289
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-lcarning. In Proceedings of the IEEE International Conference on Computer Vision, pp.
290
2019.
Ron Amit and Ron Meir. Meta-learning by adjusting priors based on extended PAC-Bayes theory. In International Conference on Machine Learning, pp. 205-214, 2018.
293
284
Balcan, Mikhail Khodak, and Ameet Talwalkar. Provable guarantees for gradient-based leta-learning. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Califormia, USA, pp. 424-433, 2019.
295
296
Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research, 12: 149, 2000.
298
289
O Catoni. PAC-Bayesian supervised classification: The thermodynamics of statistical learning. institute of mathematical statisties lecture notes-monograph series 56. IMS, Beachwood, OH. MR2483528, 2007.
300
301
302
Nello Cristianini and John Shawe- Taylor. Kernel methods for pattern analysis, volume 173. Cam- bridge University Press Cambridge, 2004.
303
304
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Incremental learning-to- learn with statistical guarantees. In Proceedings of the Thirty-Fourth Conference Uncertainty in Artificial Intelligence, UAI 2018, Monterey, California, USA, August 6-10, 2018, pp. 457-466, aa.
306
307
308
Giulia Denevi, Carlo Ciliberto, Dimitris Stamos, and Massimiliano Pontil. Learning to learn around a common mean. In Advances in Neural Information Processing Systems, pp. 10169-10179, 2018b.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Leaming-to-leea stochas- tic gradient descent with biased regularization. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 1566-1575, 2019.
311
312
313
314
Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent PAC- PAC-Bayes priors via differential privacy. In Advances Neural Information Processing Systems, pp. 1430-8441, 2018.
315
316
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Vollue 70, pp. 1126-11355 JMLR. org, 2017
318
319
Tomer Galanti, Lior Wolf, and Tamir Hazan. A theoretical framework for deep transfer learning. Information and Inference: Journal of the IMA, 5(2) 159-209, 2016.
320
321
Gregory Griffin, Alex Holub, and Pietro Perona. Caltech- 256 object category dataset. 2007.
322
Benjamin Guedj. A primer on pac-bayesian learning. arXiv preprint arXiv:1901.05353 2019.
323
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
324
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
327
328
329
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Juho Lee, Yoonho Lee, Jungtack Kim, Adam R Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation 2018
332
inmutaaion-invariant invariant neural networks. arXiv
333
preprint

---

Under review as a conference paper at ICLR 2021
Guy Lever, François Laviolette, and John Shawe-Taylor. Tighter PAC-Baye bounds through distribution-dependenn priors. Theoretical Computer Science, 473:4-28, 2013.
335
336
Andreas Maurer. Algorithmic stability and meta-learning. Journal of Machine Learning Research, (Jun))9679994, 2005,
337
338
Thomas Mensink, Jakob Verbeek, Florent Perronnin, and Gabriela Csurka. Distance-based image classification: Generalizing to new classes at near-zero cost. IEEE transactions on pattern analysis and machine intelligence, 35(11):2624 2637, 2013.
339
340
341
Emilio Parrado--ernnnder, Amiran Ambroladze, John Shawe-Taylor, and Shiliang Sun. PAC-Bayes bounds with data dependent priors. Journal of Machine Learning Research, 13(Dec):350733531, 2012.
343
344
Anastasia Pentina and Christoph Lampert. A PAC-Bayesian bound for lifelong learning. In Interna- fional Conference on Machine Learning, pp. 991-9991 2014.
345
346
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan L. Yuille. Few-shot image recognition by predicting pa- rameters from activations. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT. USA, June 18-22, 2018, pp. 7229-7238, 2018
347
348
349
Sachin Ravi and Hugo Larochelle. Optimization as model for few-shot learning. In 5th Interna- lional Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
351
352
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, pp. 1278-1286, 2014.
354
355
356
Omar Rivasplata, Csaba Szepesvari, John S Shawe-Taylor, Emilio Parrado-Hernandez, and Shiliang Sun. PAC-Bayes bounds for stable algorithms with instance- -eeendded priors. In Advances in Neural Information Processing Systems, pp. 9214 9224, 2018.
357
358
359
Andrei A. Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019.
361
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, pp. 4077-4087, 2017.
354
385
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
386
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, pp. 3630-3638, 2016.
367
368
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv:111..07172, 2018.
Xin Wang, Fisher Yu, Ruth Wang, Trevor Darrell, and Joseph E Gonzalez. Tafe-net: Task-aware feature embeddings for low shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1831-1840, 2019.
371
372
373
Kai Yu, Tong Zhang, and Yihong Gong. Nonlinear learning using local coordinate coding. In Advances in neural information processing systems, pp. 2223-2231, 2009.
374
Fengwei Zhou, Bin Wu, and Zhenguo Li. Deep meta-learning: Learning to learn in the concept space. arXiv preprini (00..::800033500 2018
376
377
Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via leamnng. In Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pp. 7693-7702,
379
380
2019.
10

---

Under review as a conference paper at ICLR 2021
Supplementary Materials for Localized Meta- Learning: A PAC-Bayes Analysis for Meta-Learning Beyond Global Prior
381
383
384
This supplementary document contains the discussion of previous work, the technical proofs of theoretical results and details of experiments. It is structured as follows: Appendix A gives a detailed
385
386
discussion of previous work. Appendix B presents the optimization method for LCC. Appendix C
387
presents notations for prior predictor. Appendix D gives the proofs of the main results. Appendix
show the approximation error between LCC-based prior predictor and empirical prior predictor, expected prior predictor, respectively. They are used in the proof of Theorem [ Next, in
D.I and D.2
389
390
Appendix D.3 and [D.4 we show the PAC-Bayes generalization bound of localized meta-learning in Theorem and also provides the PAC-Bayes generalization bound of regular meta-learning in
391
Theorem Details of experiments and more empirical results are presented in Appendix L Finally, we summarize the localized lett-llerning algorithm in Appendix
393
A RELATED WORK
394
395
Mernnna Meta-learning literature commonly considers the empirical task error by directly
optimizing loss of meta learner across tasks in the training data. Recently, this has been successfully
387
applied in a variety of models for fho-shot learning Ravi & Larochelle (2017); (Seell et al. (201 7);
Finn et al. 2017); Vinyals et al. (2016). Although Vuorio et al. (2018); Rusu al. (019); Zintgraf
399
ct al. (2019) Wang et al. (0119) consider task adaptation when using meta-knowledge for specific tasks, all of them are not based on generalization error bounds, which is the in the same spirit as
401
our work. lett-lea in the online setting has regained attention recently Denevi et al. (2018b a Balcan et al.) (2019), J, in which online-to- to-batch conversion results could imply generalization bounds. Galanti et al. (2016) analyzes transfer learning in neural networks with PAC-Bayes tools.
402
403
404
Most related our work are Penntnna & Lampert (2014); Amit & Meir (20T8), which provide
405
PAC- Bayes generalization bound for meta-learning framework. In contrast, neither work provides a principled way derive localized knowledge for specific tasks.
408
407
Localized PAC-Bayes Learning, There has been prosperous line of research for learning priors
408
to improve the PAC-Bayes bounds Catoni (2007); Guedj (2019). Parrado- -errandee et al. (2012) showed that priors can be learned by splitting the available training data into two parts, one for
409
learning the prior, one for learning the posterior. [ever et al. (013) bounded the KL divergence by
410
term independent of data distribution and derived an expression for the overall optimal prior, i.e. the
412
prior distribution resulting in the smallest bound value. Recently, RRivasplata et al. (2018) bounded the KL divergence by investigating the stability of the hypothesis. Dziugaite & Roy (2018) optimized
414 the prior term in a differentially private way. In summary, theses methods construct some quantities that reflect the underlying data distribution, rather than the sample set, and then choose the prior P based on these quantities. These works, however, are only applicable for single-task problem and could not transfer knowledge across tasks in meta-learning setting.
416
417
418 B OPTIMIZATION OF LCC
We minimize the inequality in g to obtain a set of anchor points. As with Yu et al. (2009), we simplify the localization error term by assuming x = x, and then we optimize the following objective function:
arg min
Σ
Σ allx; x 1
Σ
|x,
u|l
s.t.Vx,
[x(())= =
(12)
1= EE;
uec
uEC
419 where Euec 7u (x)u. In practice, we update C and 7 by alternately optimizing a LASSO 420 problem and a least-square regression problem, respectively.
C NOTATIONS
Let (.).) Rd + Rd be the feature embedding function. mik denotes the number of samples belonging to category k. Six and Da are the sample set and data distribution for category A: in task i,
11

---

Under review as a conference paper at ICLR 2021
respectively. Then, the expected prior predictor w.r.t. class & in task i is defined as:
w [k] (D St mil x,ESi
E
Σ ox(x)).
The empirical prior predictor w.r.t. class & in task is defined as: [g] Φ (Six)
Σ (x,).
mak xy
The LCC-based prior predictor w.r.t. class in task i is defined as;
w.' [k] = (Sax)
Σ
Σ
Tu(x))ov (u).
mik xj ESi NEC
422
D
THEORETICAL RESULTS
423
D.
PROOF OF LEMMA T
This lemma bounds the error between the empirical prior predictor wr[kk] and the LCC-based prior predictor w.[[k]
424
425
Lemma Given the definition of w. [k] and w/ [k] in Eq. 6 and Eq. (7: C) be an arbitrary coordinate coding on Rdd and Φ be an (a, 3)-Lisschitz smooth function. We have for all RR.
[w][ wf [Klll S
Σ
[<||x, x,11 + 3 Σ ||& ull²) = On, C)2 (13)
mik x5 E
uEC
where x, EuEo (x(x))u.
Proof. Let xj = EEueo nu(x))u. We have
((tk) (Sck))ll22
1
Σ [|xv(x;)- - Σ (x)()))v(u)ll
mik
XyESu
uEC
1
Σ
(10v(())) (x(22)1122 +
Σ (u(x))( (x,))11
mis
XyESik
uEC
1
Σ
(10v(x))) - (x,)111 Σ 7x(x)) de Σ
Ta(x))u) - Vil((,)) -xy)1l))
mik
E,ES
uEC
uEC
1
Σ
(1)v(x)) (2,))11 + Σ hr((x))l((ev( dv( Σ
(u(x))u))= (u- x2)112)
mik XjESu
uEC
uEC
1
Σ
(allx, - Xy112 3
Σ |x, u1l3 - 0a,
mik
X,ES
uEC
427
In the above derivation, the first inequality holds by the triangle inequality. The second equality
428
holds since LuecTu(x)) 1 for all x,. The last inequality uses the assumption of (a, -iipshii smoothness of (.).). This implies the desired bound.
429
This lemma demonstrates that the quality of LCC approximation is bounded by two terms: the first term
X; - X; |l2 indicates x should be close its physical approximation x, the second term |X ul implies that the coding should be localized. According to the Manifold Coding Theorem in Yu et al. (2009), the data points x lie on a compact smooth manifold M. Then given any >
there exists anchor points C' C M and coding such that
1
Σ
(alxx, (alxx
x + 3
Σ ||, -
< |aCM + (1 + 5v dl
(14)
mik
XJES
DEC
430 shows that the approximation error of local coordinate coding depends on the intrinsic dimension of the manifold instead of the dimension of input.
12

---

Under review as a conference paper at ICLR 2021
D.2 PROOF OF LEMMA
438
In order to proof Lemma we first introduce relevant theorem.
Theorem 3. (Vector-valued extension of McDiarmid's inequality Rivasplata et al. (2018)) Let
X1...., Xm be independent random variables, and f X 1R. be vector-valued mapping function. If. for all i E m.,, and for all Xi....; Xm: X E X, the function f satisfies sup [[ (Xi:i Xi, Xi+1m) - f(xt--1,, X;x Xi+ 1:m)
S Ci
(15)
X, x
Then [XI
[ 1m))l 1 ii= G. For any å E (0, 1) with probability 2 1 we
have
m
m i= c
[ - (m
log
(16)
2
434 The above theorem indicates that bounded differences in norm implics the concentration of f(Xim)) around its mean in norm, i.c., (X [XI is small with high probability.
436 Then, we bound the error between expected prior predictor w!" and the empirical prior predictor w."
Lemma 3. Given the definition of w [k] and : in 5 and 6 let X be compact set with
radius R, i.e, VX, E N, ||x x' x/11 S R. For any E (0, with probability 2 1 ö, we have
|| [k] [K]] S aR (1 + V
1
log (
(17)
/mas
2
Proof. According to the definition of (.) the sample set Sik, have
(-)in in 66. for all points X1, xI.....Xjj1,,K Xj+1, Xma X/ in
sup x
d (x1: x xm+mm
& (x (X1:- X x xj+1:mm
RR
sup (x,) - 2v (xf]l S sup al|x; -x1l x,
(18)
makx,
where It denotes the domain of x, say R = supx ||x|| The first inequality follows from the Lipschitz smoothness condition of o. and the second inequality follows by the definition of domain X. Utilizing Theorem
3 for any d E (0, 1] with probability 2 we have
R (1 1 log 8).
w [k] w [[]]]] | (S
<
(19)
Vmick
437 This implies the bound.
L
438 Lemma 3 shows that the bounded difference of function (.(.)) implies its concentration, which can be further used to bound the differences between empirical prior predictor w. [k] and expected prior predictor w." [k]. Now, we bound the error between expected prior predictor w!" and the 1.CC-based prior predictor w.P.
Lemma Given the definition of w." and w." (5) and 7 let X' be a compact set with radius R, i.., Vx, x' E X, x x'1l S R. For any â E (0, with probability 2 1 ō, we have
K
aR
1 (1 + log( (3)) + 00,
l/wPP
w!l1
Σ
(20)
mmss
A=
13

---

Under review as a conference paper at ICLR 2021
Proof According the definition of wP, w. and w., we have
w
[ [k:] w. [+]]
K
[ (Six] - (Sax) + (S(a)) (S((2))
[I (S:x)] 1
(5 ( -
(Six)) ( (fx(
@. (S(x))
[11[| (S:a)] (Su))l (| (S.) 2|100 (SW))ll (
(21)
Substitute Lemma and Lemmal
iinto the above inequality, we can derive
Mmuk (1 + V V log( 1 ()) 0a.0(100) )
aR
P SI
l|w 111
2
(22)
442
This gives the assertion.
Lemma Sanumaa]showe shows that the approximation eπтоr between expected prior predictor and LCC-based prior predictor depends on the number of samples in each category and the quality of the LCC coding
443
444
445
scheme.
D.3 PROOF OF THEBREM 2
Theorem 3 Let Q be the posterior of base learner Q = -NWWe,oTaa Iar and P be the prior
(S), o Jaw).. The mean of prior produced by the LCC-based prior predictor . (S) in Eq. (7) and its parameter v is sampled from the hyperposterior of meta leamer Q = N(v o la,). Given the hyperprior
.N(0,, ((0,a Ja,), then for any hyperposterior Q, any cı, c2 and any 5 E (0, with probability 2 1 8 we have,
n
&
(r()) (c + C 202nm,0
11
|Ew - कh (S:))1
201n0?
conm, 2
aR
(1 + V
4n logl 8
+ O 00.0(1:C)
+ dw Ki 2x)2
O w
0
Miik
Ow
k=1
4n
cj
2
+
log
log 8'
(23)
c2
8
20, 2c
==1
where o
I-e- and C₂
We can simplify the notation and obtain that
I-c-or
&
c
(r(2) (c(ger(()) +
)| 2
|E
2esumoo?
2cino
Conm,
+ consila, B, R, 8, m.)
(24)
447
Proof Our proof contains two steps. First, we bound the error within observed tasks due to observing limited number of samples. Then we bound the eπror the task environment level due to observing a finite number tasks. Both of the two steps utilize Catoni's 's classical PAC- -Cyaaye bound Catoni (007)) to measure the error. We give here general statement of the Catoni's classical PAC-Bayes bound.
449
451
Theorem 4. (Classical PAC-Bayes bound, general notations) Let X be a sample space and X be some distribution over X, and let F. be a hypotheses space of functions over X. Define a loss function x),,X) : F x X 1), and let XG 2 {X1,.... -xx) be sequence of G independent random
14

---

Under review as a conference paper at ICLR 2021
variables distributed according X. Let т be some prior distribution over FF (which must not depend on the samples X1....,Xo .G).. For any ò E (0, 11, the following bounds holds uniformly for all posterior distribution p over F (even sample dependent),
G
E ((,, + KL(pl(T) + log 1 jup
P X 12
E E. gJ.
c
:VP]
~
XmXf~p
1-
Gxc
6.
(25)
First step We utilize Theorem 4 to bound the generalization error in each of the observed tasks. Let E I.....7 n be the index of task. For task i, we substitute the following definition into the Catoni's PAC-Bayes Bound. Specifically, Xg - (Xij.Wi)) K - my denote the samples and X _ D;
453
454
455
denotes the data distribution. We instantiate the hypotheses with a hierarchical model f _ (v, w). where E Rdd, and w E Rddw are the parameters of meta learner (prior predictor) (-) and base learner h())) respectively. The loss function only considers the base learner, which is defined as of,, X) & ((hw((x), y). The prior over model parameter is represented as - (P,P) & (0, of I.,)).. Jd. a Gaussian distribution (hyperprior of meta learner) centered at O ow
456
459
and Gaussian distribution (prior of base learner) centered at w respectively. We set the posterior to p & (Q.Q)= & (N(v2 d. o [) Gaussian distribution (hyperposterior of
460
meta learner) centered at 2 and Gaussian distribution (posterior of base learner) centered at w According Theorem 4 the generalization bound holds for any posterior distribution including the one generated in our localized meta-learning framework. Specifically, we first sample v from hyperposterior N 2vv oo and estimate w by leveraging expected prior predictor w = (D). The base learner algorithm As(S, P) utilizes the sample set S and the prior P = NNW"",'"""" o Then we sample base learner parameter from posterior N(w Tá. and compute the incurred loss (() y). On the whole, meta- ... Sn and adjusts its hyperprior o la,) into hyperposterior Q = Am ASI,, (SI,..., Sn, P) N,ve,o La,)..
462
463
464
465
486
to produce a posterior Q- As, P) N (
467
468
learning algorithm A,, (S1........ (S,,..,, Sn, P) observes a series of tasks SI....,
469
P N(v
The KI. divergence term between prior x and posterior p is computed as follows:
E log p()) tmp
N(v2, of W(W oatTa) log NN0,0 W(wP, oz
(I(p)T)=
E v-Nivv (oo2,,, N(v2.00
E
(ff)
N(w?, o IyI
E v=
E log o(0,o vnN(vR aoVva, WWW (g
E
log N(wP o law)
1|v 20
/ - -
(26)
v>V Mo 2
In our localized meta-lcarning framework, in order to make K [L() small, the center of prior distribution w" is generated by the expected prior predictor w" P 2 (D). However, the data distribution D is considered unknown and our only insight as to Dae is through the sample set Su. In this work, we approximate the expected prior predictor (D), with the LCC-based prior predictor wf = (S). Denote the term
F
w - w''l' by B [| w/'l2
i-Nive,ogTa,
20
v
for convenience, we have
1
EE 1 w 11
E v
w
+ w - w 11
20
w
w
EE
w/112+ + || w/11 +2( P [( wP)]
2o
<R
wP112 + || w1 2//w@ w||l| wFll |1] [|| 9
v 20
w
2 Q
(.) 12 o
Bll w
(27)
15

---

Under review as a conference paper at ICLR 2021
Since w.- (S.) @ (S,1 ................................ (S (S.kk)),, we have
K
Bllw? - (S.)]1
Σ
E
8 [k]
(Sx))l
K
BBl
(Qvo(Su)) + ((
[1000 (
(S
K
Eww [k]] vee (Sit)l +
101 o2)
dv
Amm
= EE T (5:) Ko
(28)
where (Sx) denotes the variance @ (Sax I|-
The last equality uses the fact that dy = 2 for any 8' E (0, 1] with probability 2 - 8' we have
[Cldw. Combining Lemma
1 E
w/'l
2
2
EE (S:))l + dw
K( d
Fx
aR
(1 +
1 log( {)) + 00 C7)
ow
Ow
Mmisk
2
k=1
(29)
Then, according to Theorem
4 > 0 we obtain that for any
P P.
E
E
(lh (x),
D.v
ob Law
E vnNivR oa WwWW Jaw)
(Xw((x))3))
e C2
Mi j=l
1
2
8
2
E v~ vNN(yR,0laa, 20
w w/ log
(1 - e-2)) -
m,
20
2
(30)
for all observed tasks = ,,...,,.. n. Define 8' and combine inequality 29
& 2
we obtain
P S
mh
E Dx a w~W/W ,0 Taw)
E
E ((ww(x)) y)
mi
C2
E vvN((v
e(h (x,):,83)))
e e-
C2
ms j=1
www.Ww9.ar law)
+
1
1
Ew
e
@Se(1111 + +log
2
+dwK( (r)
(1
e
20
o2
8,
Fw
K
aR
(1+
+
1
log(
2 )) +
O 02 c1)
ve 2 8,
(31)
2 kml
m
8,
Using the notations in Section
3 the above bound can be simplified
TP
D
er(Ab(S;) P.))
(v-Nive
P (D). 2L
C2
E R=N(" (D)
(2lm)) (rs,ssS P.)
e 12 N
1
1
Ew 2 2 (S:)11 + +log 2
+
112
+
dw K(
)m;
2002
Fw
K
aR
2) + O (, ve }
2
(1
log J,
8z.
(32)
mmak
Second step Next we bound the error due to observing a limited number of tasks from the environment. We reuse Theorem with the following substitutions. The samples are (D,, mi, St), = 1...., n,
16

---

Under review as a conference paper at ICLR 2021
where (D,, m²) are sampled from the same meta distribution and S; D The hyposthe- sis is parameterized (D) with meta learner parameter v. The loss function is g(f, X) - [hww x), where w As Pt). Let स - W000,olaa be the prior
E
E
DWWpD law) meta learner parameter, the following holds for any do 0,
E
E
E
(x)((x),))
i SS
of WwW o0 law (e.e)
Σ 1- - e e
C1
E Vi o WwwWWwWWWWW.....
Tew (z. {()m(x),
i=
vo
log
2 21-
(33)
(1 - e
2
60
Using the term in Section
3 the above bound can be simplified
P P(D""
er(2)
< C1 e -
(An P.))
v.Ni(v 1o w O.)),, (all)))
| + log 1
21 60:
(34)
-- )n
20
50
Finally, by employing the union bound, we could bound the probability of the intersection of the events in 32 and (34 For any 8 >> O, set do and 6; for . n, have
P PD )Tr,
er(2)
C122
< (1 -
E N ,a (Ab(sS; P.))
(1 [1 - )2)
i=1
C
1
1
11
1
Ew Tve + log
4n
c
E
n1 i=l
---e- e1-e-
20
8
K
aR
log( (m )} + O0.0(1.C))
2
- dw 0x)2
(1 +
w
k=1
miik
2
ow
1| + log v@ 2 21-6.
(35)
(1 (1-6-ci )
20?
We can further simplify the notation and obtain that
P )ur,S j er(2) S (
n 011
C
Ew
(
20znmoo?
2crno?
conm,o
+consl(ar, 3, R, n, m.), 4Q
21-
(36)
where &f = T-e-o and c/ - This completes the proof.
472
D.4 PROOF OF THEOREM 1
Theorem 2 Let Q be the posterior of base learner Q NNw (Wwa.02Iaw) and P be the prior N(wP o I The mean of prior is sampled from the hyperposterior of meta learner Q N(we o Id Given the hyperprior P NO oz Ja then for any hyperposterior Q, any
17

---

Under review as a conference paper at ICLR 2021
02 > 0 and any 8 E (0. 1] with probability 2 1 -1-6 8 we have,
GP
Cj
GC
er(@) Sc(Zer( 2 +
+
)/
11/
? w
w
202mm100
201 no
2020mm:0
1=1
w
462
1
2n
ci
2 8
log
8
log
(37)
Conm;
2
2
cino
where cj
and C2 I-e
Proof Instead of generating the mean of prior with prior predictor, the vanilla meta-learning frame- work directly produces the mean of prior w by sampling from hyperposterior Q = N NVwwe dy Idw) ). Then the base learner algorithm As P) utilizes the sample set S and the prior P = N NN(w", I to produce a posterior Q = As, P) = N N (dw) Similarly with the two-steps proof in Theorem
2
we first get intra-task bound by using Theorem 4
For any o, 0, we have
P D
E (x (x.))>D.w P WNWWWWWNNWW... ol Www saw)
E
(lw(()),))
m
Ellw (xi): Us)
1 e
C2
mi j=l
w
WoPNNWWW,o oz Now Ww~N o WwWWW Tow)
1
1
1
+
w
21
E w. NN w 2 law
|
w! 1 log 8,
vQ
2 1 - 8,
(1 1=)3)=
m,
20
20
(38)
The term
E
|W Q
can be simplified as w/ll
wC~N(WW aa Iin!
2
E
1! w/
wf" wP'>N(w ( .2 Tew
20
E w/ W
2( - Q 112
WT Tw + 1| V. [Illl
20
wP
E ? wP
1112 + oN
(39)
20
where V V.lllw(l) denotes the variance of w/ ||.
Then we get an inter-task bound. For any 8n >
w.
have
P P(D")))r,,S,-DD" 11
E (D. P NNW a law Jw-N (wD a Jaw (ar.0)- Dr
E
E
E
E E(hw (x),)x)))
1
<
E WnnN(w oz Low www .z law (a.D))D
EE
E
El1w (x),
1 c
w
1=1
1
+ (1-6-c1 )n
w @1 + log
v2 2 1 80.
(40)
20
50
For any 8 > 0, set 6g
and 6, for i 1 .,..., Using the union bound, we finally get
P P(D.'
er())
C22
(0,,,))))., 1211aw (AA,
E
(1e E 02)
+ 1- - l
Σ
1
1| +
1
- WP
E
1 +
w
1 + log
2n
(1 - c 02) ---))- mm
20
20
2
8
i=l
1
| 12 +
log
.4
2 1 - 6.
(41)
- 11 )n
20
18

---

Under review as a conference paper at ICLR 2021
Similarly, further simplify the notation and obtain that
P P(D tt
er er(2) S (c2er(2)
212
)| Q 1112 n
c/
+
E
w
w012
20ynmoo
20 20
20g
o
w
+const(6, n. m4),
21-6,
(42)
where ej 1---- and C₂ --e 12
This completes the proof.
475 E DETAILS OF EXPERIMENTS
476
While the theorems consider bounded-loss we use an unbounded loss in our experiments, we can have theoretical guarantees variation of the loss which is clipped to [0; 1J. Besides, in practice the loss function almost always smaller than one.
478
479
E.
DATA PREPARATION
480
We used the 5-way 50-shot classification setups, where each task instance involves classifying images from 5 different categories sampled randomly from one of the meta-sets. We did not employ any data augmentation or feature averaging during meta-training, or any other data apart from the corresponding training and validation meta-sets.
482
483
494
E.2
NETWORK ARCHITECHTURE
485
Auto-Encoder for LCC For CIFAR100, the encoder is 7 layers with 66-3266-64-1288
486
channels. Each convolutional layer is followed by a LeakyReLU activation and a batch normalization layer. The stt, 3rd and 5th layer have stride and kernel size (3, 3). The 2nd, 4th and 6th layer have stride 2 and kernel size (4, 4). The 7th layer has stride 1 and kernel size (4, 4). The decoder is the same as encoder except that the layers are in reverse order. The input is resized to 32 x 32. For Caltech-256. the encoder is 5 layers with 32-64-128 10l-118-225225 channels. Each convolutional layer is followed by a LeakyReLU activation and a batch normalization layer. The first 4 layers have stride and kernel size (4, 4). The last layer has stride and kernel size (6, 6). The decoder is the same as encoder except that the layers are in reverse order. The input is resized to 96 96.
487
489
491
493
Base Model The network architecture used for the classification task is small CNN with 4 con- volutional layers, each with 32 filters, and a linear output layer, similar to Finn et al. (2017). Each convolutional layer is followed by Batch Normalization layer, Leaky ReLUU layer, and
485
497
pooling layer. For CIFARI00, the input is resized to 32 x 32. For Caltech-256, the input is resized to 96.
498
96
499
E.3
OPTIMIZATION
500
Auto-Encoder for LCC As optimizer we used AdamKingma Ba (2015) with B1 0.9 and 32 = 0.999. The initial learning rate is 1 x 10-4, The number of epochs is 100. The batch size is
502
512.
LCC Training We alternatively train the coefficients and bases of LCC with Adam with 3 = 0.9 and 3g 0.999. In specifics, for both datasets, we alternatively update the coefficients for 60 times and then update the bases for 60 times. The number of training epochs is 3.The number of bases is 64. The batch size is 256.
504
506
Pre- Traininnn of Feature Extractor We 64-way classification in CIFAR-100 and 150-way classification in Caltech-256 to pre-train the feature embedding only on the meta-tran veta-training dataset. For both CIFAR 100 and Caltech-256, L2 regularization term of 5e was used. We used the Adam optimizer. The initial learning rate is 10 B, is 0.9 and 33 is 0.999. The number of epochs is
508
510
50. The batch size is 512.
19

---

Under review as a conference paper at ICLR 2021
Meta-Training We use the cross-entropy loss in & Meir (0118). Although this is inconsistent with the bounded loss setting in our theoretical framework, we can still have a guarantee on a variation of the loss which is clipped to [0, 1]. In practice, the loss is almost always smaller than one. For CIFAR100 and Caltech-256, the number of epochs of meta-training phase is 12; the number of epochs of meta-festing phase is 40. The batch size is 32 for both datasets. As optimizer we used Adam with 31 = 0.9 and B2 = 0.999. In the setting with pre-trained base model, the learning rate is 1 x 10-S for convolutional layers and 5 10 for the linear output layer. the setting without a pre-trained base model, the learning rate is x 10 3 for convolutional layers and 5 x 10 for the linear output layer. The confidence parameter is chosen to be 8 0.1. The variance hyper-pane ppper-parameters for prior predictor and base model are dw
513
515
517
518
519
520
0.01. The hyperparameters on, in LML and ML-A are
522
set to 0.01.
523
E.4
MORE EXPERIMENTAL RESULTS
We also compare with two typical meta-learning few-shot learning methods: MAML (Finn et al. 2017) and MatchingNet (Vinyals et al. (0166). Both two methods use the Adam optimizer with initial 0.0001. In the meta-training phase, we randomly split the samples of each class into support set (5 samples)
525
526
learning
527
and query set (45 samples). The number of epochs is 100. For MAML, the
528
learning rate of inner update is 0.01.
Caltech-256
Accuracy on New Task(%))
CIFAR-100
Caltech-256
Accuracy on New Task(%))
CIFAR-100
80
accuracy on New
2
70
-8-
AM
MAME
AM
HANL
Accuracy
Mi
AM
t :
HOrE
Number Tasks
5
Number Tasks
Tasks
(a) With pre-traines feature extractor
(b) Without pre-trained feature extractor
Figure 5: Average test accuracy of learning a new task for varied numbers of training tasks ((C) 64).
529
In Figure 5
we demonstrate the average test error of learning new task based on the number of training tasks, together with the standard deviation, in different settings (with or without a pre- trained feature extractor). We can find that all PAC- BaC-e baselines outperform MAML and MatchingNet.
530
531
532
Note that MAML and MatchingNet adopt the episodic training paradigm to solve the few-shot learning problem. The meta-training process requires millions of tasks and each task contains limited samples, which is not the case in our experiments. Scarce tasks in meta-training leads to severely meta-overfitting In our method, the learned prior serves both as an initialization of base model and 535
533
regularizer which restricts the solution space in soft manner while allowing variation based on specific task data. It yields model with smaller error than its unbiased counterpart when applied to a similar task.
537
F
PSEUDO CODE
539
20

---

Under review as a conference paper at ICLR 2021
Algorithm Localized Meta-Learnin (LML) algorithm
Input: Data sets of observed tasks: SJ.... Sp.
Output: Learned prior predictor parameterized by v. Initialize Rdd, and W; E R&w for i 1. 11..,,
Construct LCC scheme (7, C) from the whole training data by optimizing Eq. 12 while not converged do for each task é E .n)...n) do
Sample random mini-batch from the data S Sy.. Approximate (vr;(w;) using S.
end for Compute the objective in 11 i.e. J Σ Eer;(w:) 01|lv + E.= m,
|Ew?
eve 2.
Evaluate the gradient of J w.r.t. (v, Wi...., w using backpropagation. Take an optimization step.
end while
21


Reviews:
[{"review_id": "bPzJ3N9H6Yu", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents an algorithm for offline meta-learning, where tasks are drawn from a distribution and presented to a learner sequentially, the objective being to use accumulated knowledge in order to facilitate the learning of new tasks. The algorithm is motivated from the PAC-Bayes theory of generalization, extended in recent years to the meta-learning setup, and provides a new localized approach where the prior used for a novel task is allowed to depend on the data from the new task in addition to all previous tasks. They make use of a previously proposed local learning method (LCC) that leads to extra flexibility and to a tightening of the meta-learning bounds, and provide an algorithm based on these bounds from which a learning algorithm is derived based on minimization of the upper bound. Empirical results are provided demonstrating the efficacy of the method and comparing to other recent approaches.  \n\n\nStrong points The problems of meta-learning is of significant current interest and importance, and the theoretical formulation within the widely used and proven PAC-Bayes framework is well founded. Moreover, the authors go beyond current bounds by introducing a localized approach to data-dependent prior learning, deriving an algorithm directly from theoretical bounds, and demonstrating its utility.  \n\nWeak points The paper is difficult to follow making it hard to assess its true novelty. For example, the notion of 'localized learning' is key to the work, but is never defined clearly. Given its important it warrants a clear definition. For instance, in the caption of image 1 they state ''Instead of using global\nmeta-knowledge for all tasks, we tailor the meta-knowledge for various specific task.\" This is not very helpful. If I understand correctly, the main issue is using the LCC approach for determining a data-dependent prior for the new task. Moreover, given the most work on PAC-Bayes theory specifically requires that the prior be data-independent, and that task-dependent priors have only very recently been introduced and analyzed (e.g., Rivasplata et al., PAC-Bayes analysis beyond the usual bounds, NeurIPS 2020 ), this is a major issue that requires clear elaboration.\n\nSpecific issues\n\nThe definition of coordinate coding and the latent manifold require some further elaboration, perhaps through an example. How do local anchor points relate to the standard notion of a basis? \n\nThe authors assume that both the prior and posterior (lines 128-129) are spherical Gaussians with the same variance, i.e., they do not allow the variance to adapt through learning. Isn't this a significant limitation? The same holds for the distribution of v in line 160. Some discussion of this is called for. \n\nThe authors present a bounds based on Catoni's work claiming, correctly, that it allows for faster convergence at a rate of 1/m rather than 1/\\sqrt{m}. However, they refrain from noting that the empirical error is multiplied by a constant that is larger than 1, as opposed to previous bounds. While this may not be an issue for deep networks that have very low empirical error, this should be clearly stated. \n\nThe form of the task-complexity term in eq. (3) and (9) requires further explanation and elaboration given its pivotal role in the work.\n\nIn Lemma 1 and Theorem 2, it is not clear whether you used the  Latent Manifold assumption (Def 3) , and where does it come into play in the bound and in the analysis?\n\nThe optimization of LCC is not explained sufficiently.  Since it is the main part of the innovation, it should be explained more than is done in Appendix B.  Eq. 12 uses L2 distance on the data points, how effective this is in real data sets?\n\nGiven the multiple definitions required for the algorithm, I would urge the authors to include all necessary definitions and equations in Appendix F that contains the pseudo-code. This will greatly facilitate understanding the exact form of the algorithm. \n\nMinor points\nLine 49 - missing Fig number, Definition 3 - change order to  d_M:=|C|, Line 95 - delete b, Line 96 - any point in M (not in R^d). \n\nConcerns \nThe overall idea of the paper is interesting, and, according to my understanding novel and promising. However, I cannot be certain that I have fully understood the paper, given the concerns raised above about lack of clarity. I believe that a clear re-writing of the paper, highlighting the novel contributions of the work and putting it in context (e.g., re new results by Rivasplata mentioned above) is called for. I believe that such a rewriting, assuming the correctness of the proofs in the appendix that I have not checked, will greatly enhance the paper. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QWBV_TOaXXC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- Summary and Contributions\n    - In this paper, the authors proposed a localized version of PAC-Bayes bound for meta-learning. This paper is a direct extension of (Amit and Meir 2018) by customizing the prior of each task with Local Coordinate Coding. \n\t\t\n- Strengths\n    - The main text is easy to follow, with major theorems and intuitions well illustrated. \n    - The idea of localized prior for specific tasks is supported and inspired from the PAC-Bayes theory.\n\t\t\n- Weaknesses and questions:\n   - The theoretical novelty of this paper is limited - it is a combination of the theoretical analyses in (Amit and Meir 2018) and those in the work of LCC. \n   - Why do the hyperprior in Line 130 and the prior sampled from the hyper-posterior in Line 128 share the same variance? It does not make any sense. Actually, the two variances are different in the work of (Amit and Meir 2018).\n   - Following the PAC-Bayesian theory and Line 154-156, the prior must be independent from the samples but could be selected from the distribution. However, in this paper, the prior w^P_i is determined from the samples S_i via LCC. I expect the theoretical analysis is not valid. \n   - From Equation (10), I do not expect the task complexity term to be reduced necessarily. Could the authors elaborate any guarantee of this algorithm to learn an effective v^Q so that this term is reduced?\n   - What are the implications of c1 and c2?\n   - In fact, there are many gradient-based meta-learning algorithms working on adapting the global meta-knowledge to specific tasks, including [1] and [2]. The authors did not compare any of them. Moreover, I am not convinced by the results in E.4. Figure 5. I am curious about LML's performance on standard mini-ImageNet and Omniglot datasets. \n    - This algorithm lacks flexibility in handling regression problems. Could the authors provide an explanation and discussion on this?\n    - Could you illustrate the anchor points u empirically?\n\t\t\n[1] Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Toward multimodal model-agnostic meta-learning. arXiv preprint arXiv:1812.07172, 2018.\n[2] Yao, H., Wei, Y., Huang, J., & Li, Z. (2019, May). Hierarchically Structured Meta-learning. In International Conference on Machine Learning (pp. 7045-7054).", "extra_scores": {}, "extra_text": {}}, {"review_id": "C2r23uj7e2t", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Update: I appreciate the response to address the major concerns. The proposed approach doesn't follow the episodic training, so there exists a clear difference from advanced MAML approaches, which update the task-specific parameters in the episodic training. I still believe that more empirical justifications should be required to decide which approach performs better than the others. On a positive side, I completely agree that it is non-trivial to develop a localized meta-learning framework from a theoretical perspective. So, I increased my score by one point. \n\n**Summary**\nThis work proposes a localized meta-learning framework that adaptively determines a hyperprior for a specific task, derived from a PAC-Bayesian analysis. \n\n**Detailed comments**\nThe main idea of this approach is to adjust the global hyperposterior distribution by the information of the input task. Conceptually, this approach is very similar to previous ones:\n* Meta-learning with latent embedding optimization, ICLR’19.\n* Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace, ICML’18.\n\nBoth propose data-dependent parameter generative processes, in which a subset of parameters is used to solve the input task. But, the proposed approach has not been compared to them.\n\nThe proposed method has been validated in very limited few-shot classification scenarios (only 5-way 50-shot tasks). It’s necessary to consider more standard experiments settings. I guess 1-shot / 5-shot on miniImagenet and tieredImagenet tasks should be included in experiments. Otherwise, it’s very difficult to see that this approach works well in general settings. \n\nFor few-shot classification tasks, there exist many attempts to improve the performance. But, none of these approaches has not been compared to the proposed method. More empirical justification is needed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5eiSF3WID3j", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "----------Updates after rebuttal------------\n\nThe author did not provide a revised version and additional experiments to address my questions. \n\nI keep my original score.\n\n------------------------------------------\n\n\nSummary: \n\nThis paper proposed a new localized prior PAC-Bayes meta-learning. Specifically, they adopted local coordinate coding (LCC) to replace the previous global hyper-posterior predictive distribution.  Empirical results on few-shot tasks verified its practical benefits.\n\n------------------------------------------------------\n\nOverall review \n\nPros:\n\n[1] This problem is well-motivated. I think such a direction is meaningful. \n\n[2] The high level of proof is technically sound.\n\nCons:\n\n[1] The paper organization indeed requires a better refinement. I can follow the main proof but too many non-important details make the paper rather difficult to follow. \n\n[2] (main concern) The theoretical and practical benefits are not significant. It seems like a plug-in approach for LCC.\n\n[3] Some technical details need better justifications and discussions.\n\nBased on these, I recommend a weak rejection but encourage a major revision for resubmission.\n\n-----------------------------------------------------------\nDetailed explanations\n\n[1] Paper organization\n\nThe main contribution of this paper is Theorem 2 (and an improved Theorem 1). I think more intuitions/insights/discussions about these are expected. In contrast, the definition of Lipschitz and Smooth and Lemma 1,2 are not critical in understanding your main idea. \n\nThis is particularly important in Sec 3, where the main contributions occur. But it is quite difficult to follow the insights in this part. And this also influences the significance of the paper.\n\n[2] The significance of the results\n\nTheoretical perspective:\n\n[a] The author claimed they reach a tighter bound with $\\mathcal{O}(\\frac{1}{m})$ (Line 259). I respectfully disagree with this claim. In fact, the author chooses a specific function (smooth property) for deriving a tighter bound. But in the deep network due to the non-linear activation function, this condition can hardly be achieved in practice. Therefore the reason for improving the performance is not the proposed good theory.\n\n[b] I cannot understand why using LCC in meta-Pac-Bayes.  Why not using other clustering-based approaches to set anchor points such as the core-set approach? If there is a **particular** benefit to use LCC rather than other localized approaches, the explanations are highly expected. \n\nSince the whole proof idea is quite similar to Amit (2018) and Pentina (2014) with two-level Pac-Bayes theorems except the LCC part. From the current version, it seems a simple LCC plug-in approach, which hurts its significance.  \n\nPractical Perspective:\n\n[c] The empirical results indicate LCC based PAC-Bayes heavily depend on the number of anchor numbers. i.e if we do not choose the best anchor number, the results can be even worse than the simple baseline ML_AM. \n\n[d] The computational complexity of LML is not reported. I think it can be much higher than ML_AM/ML_PL, which restricts its practical utility.\n\n[e] The analysis of the result can be much better improved. For example, the author claimed better local information extraction. I would like to see clear evidence/analysis  (e.g. TSNE, visualization of anchor points, the relation of different tasks) rather than prediction accuracy curves. This also enhances the significance of LCC.\n\n\n[3] Other minor details\n\n[a] The loss in PAC-Bayes in $[0,1]$, the cross-entropy is surely not.\n\n[b] The Catoni bound (eq 25) the $\\pi$ must not depend on $x_1,\\dots x_n$. This may contradict your data-dependent bound?\n\n[c] Lemma 1 and Lemma 2 are straightforward. I do not think it does not provide strong practical insights into the proposed approach.  \n\n\n--------------------------------------------\n\nSuggestions \n\nI suggest a major revision on Sec 3, theoretical discussions, and empirical analysis (not only accuracy) on the benefits of LCC.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a PAC-Bayesian approach for meta-learning that utilizes information of the task distribution in the prior. The presented localized approach allows the authors to derive an algorithm directly from the bound - this is a worthwhile contribution. Nevertheless there are several concerns that were raised by the reviewers and in its current form the work is not ready to appear in ICLR.  


Decision: Reject