Meta Gradient Boosting Neural Networks
Authors: Manqing Dong
Lina Yao
Xianzhi Wang
Xiwei Xu
Liming Zhu
Venue:  (2021)
Keywords: meta learning
deep learning

Abstract: Meta-optimization is an effective approach that learns a shared set of parameters across tasks for parameter initialization in meta-learning.
A key challenge for meta-optimization based approaches is to determine whether an initialization condition can be generalized to tasks with diverse distributions to accelerate learning. 
To address this issue, we design a meta-gradient boosting framework that uses a base learner to learn shared information across tasks and a series of gradient-boosted modules to capture task-specific information to fit diverse distributions.
We evaluate the proposed model on both regression and classification tasks with multi-mode distributions. 
The results demonstrate both the effectiveness of our model in modulating task-specific meta-learned priors and its advantages on multi-mode distributions.

Under review as a conference paper at ICLR 2021
META GRADIENT BOOSTING NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Meta-optimization is an effective approach that learns a shared set of parameters across tasks for parameter initialization in meta-leamingg A key challenge for meta- optimization based approaches is to determine whether an initialization condition can be generalized to tasks with diverse distributions to accelerate learning. To address this issue, we design a meta-gradient boosting framework that uses a base learner to learn shared information across tasks and series of gradient-boosted modules to capture task-specific information to fit diverse distributions. We evaluate the proposed model both regression and classification tasks with multi-mode distributions. The results demonstrate both the effectiveness of our model in modulating task-specific meta-learned priors and its advantages on multi-mode distributions.
INTRODUCTION
While humans can learn quickly with few samples with prior knowledge and experiences, artificial intelligent algorithms face challenges in dealing with such situations. Learning to learn (or meta- learning) Villata & Drissi 2002) emerges as the common practice to address the challenge by leveraging transferable knowledge learned from previous tasks to improve learning on new tasks (Hospedales et al. 2020).
An important direction in meta-learning research is meta-optimization frameworks (Lee & Choi 2018
Nichol & Schulman, 2018 Rusu al. 2019), a.k.a., model-agnostic meta-learning (MAML) (Finn et 2017). Such frameworks learn initial model parameters from similar tasks and commit to achieving superior performance on new tasks that conform the same distribution through fast adaptation. They offer excellent flexibility in model choices and demonstrate appealing performance various domains, such image classification (Li et al. 2017 Finn et al. 2017), language modeling (Vinyals et al. 2016), and reinforcement learning (Fernando et al. 2018 Jaderberg et al. (011)).
Generally, such frameworks define target model Fe and meta-learner M. The learning tasks T T= (Tirann Theat] are divided into training and testing tasks, where T are generated from the meta-dataset D, i.e., T P(D). Each task contains a support set DS and a query set D인 for training and evaluating local model. The initialization of the model parameter 0 is learned by the meta learner, i.e., + M(Trra))) We denote the meta-learned parameter as Φ so that +- Φ. For each task, the model obtains locally optimal parameter by minimizing the loss ((Fo(D))) The meta parameter ó will be updated across all training tasks by minimizing the loss MTeTrran (C(F, (Dº)) ))). Generally, it takes only small number of epochs to learn locally optimal parameters across training tasks so that meta-learned parameter á can quickly converge an optimal parameter for new tasks,
Most methods assume some transferable knowledge across all tasks and rely on a single shared meta parameter. However, the success of the meta-learners are limited within similar task families. and the single shared meta parameter cannot well support fast learning on diverse tasks (c.g.. large meta-dataset) or task distributions (e.g., are generated from multiple meta-datasets) due to conflicting gradients for those tasks (Hospedales et al.) 2020). Recent efforts have studied multiple initial conditions to solve the above challenges. Some employ probabilistic models (Rusu et al.) 2019 Finn et al. 2018 Yoon et al. 2018) while others incorporate task-specific information (Lee & Choi 2018; Vuorio et al. 2019 A et al. 2018). The former learns to obtain an approximate posterior of unseen task yet needs sufficient samples get reliable data distributions: the latter conducts task-specific parameter initialization using multiple meta-learners yet requires expensive computation and cannot transfer knowledge across different modes of task distributions.

---

Under review as a conference paper at ICLR 2021
In this work, we aim to resolve the above challenges from a novel perspective by proposing meta gradient boosting framework. Gradient boosting (Friedman 2001) aims to build a new learner towards the residuals of the previous prediction result for each step. We call the learner for each step as weak learner and make predictions based on summing up the weak learners. Recent research (Badirli et al. 2020; Olson et al. 2018) has demonstrated the potential of decomposing deep neural nets into an ensemble of sub-networks with each achieving low training errors. We propose to use the first or first few weak learners as the base learner, followed by series of gradient boosting modules cope with diverse array of tasks the base learner is responsible for inferring transferable knowledge by learning across all tasks; the crudient-boosting the base learner. Compare with existing work, which uses multiple initial conditions, our approach does not require specifying a set of initialization conditions and thus has better flexibility in dealing with mudti-mode tasks. Our proposed framework is also more efficient than its counterparts as it does not require a large number of gradient boosting modules. We evaluate the proposed framework on few-shot learning scenarios for both regression and classification tasks. The experimental results show the well performance of the proposed framework, which demonstrates the model's 's ability in learning with very few cases.
modules are designed to make task task-specific updates to
2 RELATED WORK
Meta-learning has the potential of replicating human ability to learn new concepts from one or very few instances, It has recently drawn increasing attention, given its broad applicability to different fields (Hospedales et al. (020)). Pioneers (Finn et al. 2017 Nichol & Schulman 2018) in this topic propose optimization algorithms with learned parameters to automate the exploitation to the structures of learning problems. However, most of them initialize the same set of model parameters for all tasks, which may have different distributions, thus resulting in over-fitting.
Recent studies either model the mixture of multiple initial conditions via probabilistic modeling Yoon et al. 2018) or incorporate task-specific knowledge (Lee & Choil 2018
(Finn et al.)
2018
Alet et al. 2018). to address the above issues. Yoon et al.] (2018) and [Finn et al. (2018) use
variational approximation to enable probabilistic extensions to MAML. But it is unclear how to extend MAML for wide range of task distributions. Rusu et al. (2019) consider multiple conditions
by borrowing the idea of variational autoencoders (Kingma & Welling 2014), which encodes inputs to a low-dimensional latent embedding space and then decodes the learned latent code to generalize task- specific parameters. Another line of research defines a set of initialization modules and incorporate task-specific information to select task-specific modules; this way, it can identify the mode of tasks sampled from a multimodal task distribution and adapt quickly through gradient updates (Vuorio et al.) 2019). Yao et al. (2019) propose a Hicrarchically Structured Meta-Learning (HSML) framework to perform soft clustering tasks. HSML first learns the inputs and then obtains clustering results by the hierarchical clustering structure. HSML tailors the globally shared parameter initialization for each cluster via parameter gate to initialize all tasks within the clusters. The above approaches have common limitations in 1) requiring sufficient data samples to generalize task distribution thus may fail in few-shot cases; 2) being computationally expensive, due the globally stored initialization modules; 3) facing challenges in exhaustively listing every possible initial condition.
Two closely-related topics to meta-learning are modular approaches (Andreas et al. (016)) and multi- task learning (Zhang & Yang 2017). Modular approaches are similar to meta-learning in that the input signal gives relatively direct information about good structural decomposition of the problem. For example, Alet et al. (018)) adopt the modular structure and parameter adaptation method for learning reusable modules. Multi-task learning aims learn a good shared- parrmmeee or make the parameter for each task as similar as possible (Wang et al. 2020). For example, Zhang et al. (2018) propose two task networks that share the first few layers for the generic information before applying different prediction layers to different tasks. These approaches differ from meta-learning in requiring fine-tuning the models over all training samples and thus cannot adapt well to new tasks.
Our framework Meta Gradient Booting (MGB) neural network is based on the idea of gradient boosting (Friedman 2001), which aims to build a new learner towards the residuals of the previous prediction result for each step. The learner for each step is called weak learner, and the prediction is based on the summation of weak learners. Weak learners may vary from traditional decision trees (Chen& & Guestrin 2016) to neural networks Tannor & Rokach 2019 Badirli al. 2020).
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Training of MGB
Toak
1: Randomly initialize global parameter
Feature
2: while Not done do
3: for T E T do
4:
for (z, y) E D do Initialize fo. by 80 for 1 E range (K) K do
5:
OO O
6:
7:
8 O BL,
Local Update
8:
end for end for Get updated parameter
9:
Target
Oulbuts
Locile
10:
11:
for (x. y) E D2 do
Taok
Feature
Feature
12:
Calculate predictions (x) Calculate task loss Ly.
Base
Gradiont Boost Modsle
Vl.c0
13:
14:
end for end for
15:
Figure 1: Example of the model with only one
Update Φ by t 7Lmeta
module. Green lines are for local update and red lines are for global update.
17:
end while
A recent study (Badirli et al., 2020) proposes a general framework for gradient boosting on neural networks, which work for both regression and classification tasks. It uses the deep layers of neural nets bagging mechanism in similar spirit to random forest classifier (Veit et al. 2016). After only slight tuning, deep neural nets can perform well on a wide range of small real-world datasets (Olson) al., 2018). These findings demonstrate the potential of decomposing deep neural nets into ensemble of sub-networks each achieving low training errors. our framework, we use the first weak learner the first few weak learners as the base learner for learning the shared initialization parameter across tasks. The output for each weak learner then aggregated to the inputs for the next step for constructing end-to-end learning strategy until the last gradient boosting module. This way, the base learner serves as transferable knowledge, and the gradient boosting modules following it are trained for task-specific predictions.
3 METHOD
We explore the problem in the context of supervised learning, where input-output pairs available in both training and validation sets. Similar to previous meta-optimization based approaches (Finn et al.)
Nichol & Schulman, 2018), assume the tasks generated from underlying distribution P(D), where is the meta-dataset, which is either a uni-mode dataset or multi-mode datasets. traain Trest) }, each task T E T contains support dataset D and query dataset DR, both representing input-output pairs (x, y). We aim to learn a meta-learner M to guide the initialization for target model Fa so that the target model can quickly adapt and perform well on given new task. We propose Meta Gradient Boosting (MGB) framework as the target model Fe. which consists of several weak learners and can be represented Fg ΣK o The first weak learner So, or the first few weak learners are regarded as the base learner for learning the shared information across tasks; the weak learners are gradient boosting modules for capturing task-specific information. The meta learner aims to learn transferable knowledge and provides initialization details for the base learner so that the model can quickly adapt to task-specific predictions with a few gradient-boostee steps. Figure I shows an example of our MGB framework under K = 1, where update the model locally for task-specific predictions and update the meta-learner globally for all tasks.
2017
Given a of tasks
3.1 LOCAL LEARNING:
UPDATING VIA BADDIENT-BOOSTIN
Gradient boosting machines hold out optimization in the function space (Friedman 2001 In particular, the target model is addition Fo K o where K is the number of adaptions
3

---

Under review as a conference paper at ICLR 2021
(gradient boosts), fep is the first weak learner, which provides initial prediction of the inputs, for are the function increments (gradient boosting modules), and ag is the boosting rate. In each step, the new gradient boosting module is formulated in greedy way. To start, the base-learner fo, minimizes a prediction loss L(y, fo,) to predict the outputs ÿ + Jo, (2)) Then, at gradient boosting step k, the gradient boost module So, minimizes the loss [eg,. where Ihe = FCCy.F2-'(on
OF. ((x)
F E -,=00=11 fo. denotes the ensemble of functions gradient step k, and 9t is the negative gradient along with the observed data. Traditional boosting frameworks learn each weak learner greedily; therefore, only parameters of k-th weak learner are updated at boosting step k while all the parameters of previous k- - -1 weak learners remain unchanged. This together with the single shared meta parameter make easy for the model to stuck in a local minimum. The fixed boosting rate ay further aggravates the issue. In response, we construct the gradient boosting neural network cascading structure (He et al. 2016). Similar to Badirli al. (2020), at each step k. we take the concatenation of the inputs and the hidden layer of the previous weak learner hg-1 = 00.-1 (x) as the inputs for the current step gradient boost module, i.e., 9k + fo, ( [ht-1,.)] æI). But our approach differs in optimizing the gradient boosting neural networks in a macroscopic for each step k. we learn the ensemble of the weak learners Fh by minimizing the loss function
arg min C(y, + arg min Cly, fo (x) + k = (hx-1,
(I)
#
We update parameters of both weak learners and gradient boost module via back-propagation. Generally, the parameter 0 of Fo is locally updated via <- BL Fo), where , is the task learning rate. The boosting rate ag can be set in various forms in the simplest case, can use an increasing or decreasing boosting rate, e.g. Og = 0k-1/c (c is a constant), to decrease or increase the contribution of the base learner. We will discuss model performance under different settings of the boosting rate later. Both the boosting rate and the number of gradient boost modules affect the sharing ability and prediction performance of the base learner. Hochreiter et al. (2001) found that the gradient for the earlier weak learners decays with the increasing number of gradient boost modules. On balance,
the base learner of our proposed gradient boosting framework weak learner and series of gradient boosting modules as strong learners for specific task.
3.2 GLOBAL LEARNING: META-OPTIMIZATTIN FOR TRANSFERABLA KNOWLEDGE LEARNING
The learning and updating strategy of the gradient boosting framework ensure a weak base learner. Since the base learner could be the first weak learner or the first few weak learners, we use fo, to represent for both conditions for ease of illustration. We take the meta-optimization approach for initializing the base learner so that the model can provide an initial guess for the prediction based on the transferable knowledge over a wide arrange of tasks. Specifically, we learn global sharing parameter s.t. 0g + é. Similar to other MAML-based approaches (Finn et al., 2017 Lee & Choi 2018), we minimize the expected loss on the local query set D for tasks T E T FrOw in meta optimization. Since the meta- peta-ggrddeet may involve higher-order derivatives, which are computationally expensive for deep neural nets, MAML et al. (2017) takes one-step gradient descent for meta-optimization. Following the above, obtain updated model parameters 0 after updating the target model Fg for K steps on the local support set DS. Global learning aims at minimizing the loss
arg min Limeta + argmin min EreTiman (e Fg)
The global sharing parameter @ updated via ø + Φ - Tmmetas where 7 is the global learning rate. The pseudocode of the training process is described in Algorithm 1.
4 EXPERIMENTS
We test proposed framework few-shot learning scenarios and compare it with three other meta-learning approaches: MAML (Finn et al. 2017). Multimodal Model-Agnostic Meta-Learning (MMAML) Vuooio et al. 2019), and Meta-learning with Latent Embedding Optimization (LEO) Rusu et al. (211)) Both MMAML and LEO model a mixture of multiple initial conditions. MMAML modulates its meta-leamed prior parameters according to the identified mode to enable more efficient
4

---

Under review as a conference paper at ICLR 2021
(a) Sinusoid function
(b) Linear function
(c) Quadratic function
(d) Absolute value function
Figure 2: Performance on 4-mode regression tasks with one step of global learning. Blue lines show the real distribution. Red nodes stand for the real values of the support set samples.
adaptation; LEO solves the problem via probabilistic modeling that learns a stochastic latent space with an information bottleneck, conditioned on the input data, from which the high-dimensional parameters generated. We compare the results on both regression and classification tasks.
4.1 REGRESSION TASKS
Setups We adopt the simple regression task with similar settings to Finn et al. 2018). In the ID regression problem, we assume different tasks correspond to different underlying functions, and we aim to predict the outputs for new tasks by learning on very few samples. To construct a multi-mode task distribution, we set up four functions (sinusoidal, linear, quadratic, and abstract value functions) and treat them discrete task modes. The function settings are detailed in Appendix [ Uni-mode training tasks are generated from a single function. For each task, input-output pairs are generated from the given function with randomly initialized parameters (e.g.. g. the slope for the linear function). We generate multi-moue training tasks by first selecting the function type for each task and then initializing random parameters for the function generate samples. Similar the settings in Pinn et al. 2017), we use two hidden layers of size 40, followed by ReLU as the activation function.
Learning from multi-mode distributions (e.g., the above four distributions) is challenging. First, the function values may vary across those distributions. For example, the output of the quadratic function can range from zero to dozens (see Figure 2 while the other three functions are more likely to produce outputs within [-10,10]. Second, the lew-shot samples that sit on line might be generated from non- linerr functions, which make it difficult to learn the real modality from such samples. Updating more steps for task-specift models could solve the first challenge yet may cause over-fitting The second challenge can be mitigated by producing a set models for task learning. Our proposed MGB can well handle the two challenges by a series of task-specific gradient boost modules and providing flcxibility in updating the framework.
Results We use Mean Absolute Error (MAE) the evaluation metric to evaluate models' per- formance in training on one-mode (sinusoidal function), two-mode (sinusoidal function and linear function) four-mode (all the four functions) tasks. To ensure fair comparison, the settings of the network structure are configured for all the compared methods, and the results are learned within certain training epochs. Detailed settings are in A.3 Overall, the proposed MGB framework shows competitive and stable performance on multi-mode regression tasks. The results (shown in Table M reveal that it is more difficult to capture task-specific patterns on multi-mod tasks-tha MAE IS larger when more modes are considered. This makes sense considering the randomness choosing functions and function parameters for all models. The results also show that incorporating task identities can significantly improve the performance of multi-mode learning. MAML has the highest error in all settings, which suggests that MAML does not perform as well on multi-mode tasks as on uni-mode tasks. Our model shows more stable performance in all settings when compared with LEO and MMAML. Performance comparison of our model with one, two, and five gradient boosting modules (corresponding to MGB-1, MGB-2, and MGB-5, respectively) suggests that the performance improves when more gradient boosting modules are used; however, the effect decreases as the number gradient boosting modules increases.
basic
5

---

Under review as a conference paper at ICLR 2021
Table I: Results Regression Tasks (MAR)
Mode
2 Modes
4 Modes
Method
5-shot
10-shot
5-shot
10-shot
5-shot
10-shot
MAML
1.234=0.174
1.054:0.077
0.548
1.356 109
2.044:10.472
1.698:10.266
LEO
0.957=0.123 123
0.789=0.042
1.127#0.177 175
0.899:100084
1.234++0224
1.095+00166
MMAML
0.638=t0.053
0.526=+0.027
0.783+00.096
0.709:
1.016++0.181
0.920=t0.099
MGB-1
0.674=0.009
0.5241:.013
0,999:10 103
0,906:100038
1.213++0.173
0,928:10.08
MGB-2 2
0.629=0.004
0.512=0.005
0.822:00032
0.73410.021
1.046:0.110 106
0.899:00.027
MGB-5
0.615=0.005
0.406=0.005
0.704++.002
0.672++0.077
0.985++0.089
0.825++0.043
Table 2: Results on Classification Tasks (Accuracy)
Modes
Modes
4 Modes
Method
5-way
20-way
5-way
20-way
5-way
20-way
1-shot
5-shot
1-shot
1-shot
5-shot
1-shot
1-shot
5-shot
1-shot
MAML
0.6381
0.7524
0.4296
0.5235
0.6481
0.2986
0,4223
0.5172
0.2415
LEO
0.6676
0.7689
0.4318
0.5129
0.6413
0.3106
0.3948
0.4757
0.2301
MMAML
0.6797
0.7738
0.4521
0.5536
0.6728
0.3534
0.4812
0.5528
0.3149
MGB-1
0.6394
0.7579
0.4228
0.5241
0.6435
0.3002
0.4277
0.5209
0.2533
MGB-2
0.6501
0.7633
0.4270
0.5503
0.6750
0.3459
0,4531
0.5374
0.2682
MGB-5
0.6834
0.7830
0.4426
0.5611
0.6897
0.3568
0.4725
0.5530
0.2956
4.2
CLASSIFICATION TASKS
Setups We evaluate the proposed framework (Finn et al. 2017) on n-way few-shot image clas- sification tasks. We use four datasets to constitute multi-mode tasks: Omniglot Lake et al. (201)
minilmageNet Ravi & Larochelle (2016), FC100 Oreshkin et al. (2018). and CUB Wah et al. (2011)
Details about those datasets can be found in the supplementary material A.2 We follow train-test splitting methods as described in Finn et al. (0017); Vuorio et al. (2019) and train models on tasks with two modes (Omniglot and minilmageNet), three modes (Omniglot, minilmageNet, and FC100), and four modes (all four datasets) tasks. The weak-learner uses CNN modules for learning the image embedding and fully-connected layers for classification. Similar to previous work Finn et al. (2017) 7 Vuorio al. (2019), we configure each CNN module with 3 x convolutions, followed by 2 x max-pon maax-pooling pooling and batch normalization (Ioffe & Szegedy 2015). Since the embedding module can significantly affect classification results (Sun et a 2019). , ensure fair comparison,
the same embedding module for all compared methods. The detailed settings are described in A.3
Results We consider 1-shot and 5-shot learning for 5-way classification and 1-shot learning for 20-way classification. We evaluate the performance of our MGB framework with one (MGB-1), two (MGB-2), or five (MGB-5) gradient boosting modules. The resulting performance is shown in table 2 Overall, the proposed MGB performs well on multi-mode tasks. Compared with MMAML, our method works better on most scenarios except on 1-shot 20-way classifications, where MMAML can store more parameters. Similar to the regression tasks, MGB with more gradient boosting modules shows better performance while MGB-I can make only slight improvement over MAML because images contain more complex information than real numbers. More modes of tasks increase the performance gap between MAML and the other methods, which suggest the other methods (which consider multiple conditions) can handle multi-mode tasks better than MAML. Under the same experimental settings, i.e., with the same image embedding modules, LEO does not perform well on tasks with more modes, partially because it is largely impacted by the quality of the learned image embedding- first, LEO's 's learning strategy (Rusu et al. 2019) pretrains the dataset-specine image embedding (Qiao et al., 2018) before meta-learning; then, LEO uses an encoder-decoder structure to generate parameters for the classifier from the learned embedding.

---

Under review as a conference paper at ICLR 2021
5 DISCUSSION
Our experimental results on both regression and classification tasks suggest our method can adapt to the optimal results with few gradient boosting modules. this section, we take further step to discuss i) the configuration of gradient boosting modules and ii) the sharing ability of the base learner during the back- propaagaiinnn through meta- eeta-gradient boosting modules. The results are presented for 5-way 1-shot image classification with 4-mode tasks.
5.1 CONFIGURATION OF GRADIENT BOOSTING MODULES
Settings for the weak learner Our MGB framework consists of series of weak learners, where the first or the first few weak learners serve as the base learner to be shared among tasks. Type and dimension of the weak learner are two key factors that may affect the final results. Vuorio al.) (2019) find that LSTM models perform better than linear models in regression tasks. [Sun et al. (2019) show the choice of feature extractor for images has strong correlation with the final results, and using pre-trained network network structure can improve the results significantly. For example, it improves the accuracy by about 6% (Mishra et al. 2018: Oreshkin et al. 2018)) to use ResNet-12 as the feature extractor. Here, we use four convolutional layers for learning from images to ensure fair comparison. The dimension of the weak learner includes the number of hidden layers and the number of neurons in each layer. Figure 3] (a) shows the performance of the model trained on 4-modes classification dataset under different settings of the two parameters above. The results show deeper model or larger neuron size gives better performance, while the network may need more time learn with a larger neuron size.
aa)
(b)
(c)
Figure 3: Model performance under different settings on (a) the dimension of the weak learner, (b) the number of gradient boosting modules, and (c) the updating strategy for gradient boosting modules on 4-modes image classification tasks.
Settings for gradient boosting modules The number of gradient boosting modules and the updat- ing strategy for the gradient boosting modules are two vital factors that may affect the final results. Since the prediction is based on the summation of a series of weak learners, more gradient boosting modules will reduce the contribution of each weak learner. According to the results shown in Figure (b), the model with only one gradient boost module cannot well handle multi-mode tasks; more gradi- ent boost modules can help improve the results while taking more time to learn the task information. The traditional gradient boost module greedily updates the whole framework at each step, but we can let the added weak learner grasp more information from the inputs by updating several times in each step. We can further adjust the contribution of the base learner and gradient boosting modules to the final results by allowing them to update for different times each step. Intuitively, if we fine-tune on the base learner (i.c., updating multiple times on the base learner), the model may get stuck in local optimal and decrease the impact of gradient boosting modules on the model performance; conversely, if we conduct more updates on the gradient boosting modules, the model may need more training epochs from all training tasks to grasp the general sharing information that helps the model in fast-adaptation. The results under different settings of the updating times for base learner and gradient boosting modules are shown in Figure (c). We can see the updating strategy significantly affects model performance, and updating more steps gradient modules improves the model's 's robustness.
The performance tends to become unstable if we conduct more updates for the base learner (i.c., the yellow line in Figure 3 (c)). which observation aligns with analysis above.
7

---

Under review as a conference paper at ICLR 2021
5.2 SHARING ABILITY OF THE BASE LEARNER
The base learner of our MGB framework is shared across tasks for capturing the general sharing knowledge. Three factors may affect the sharing ability of the base learner: 1) which part share; 2) how to share; 3) how the shared base learner contributes to MGB. We discuss these three components as follows.
Single weak learner v.s. Multiple weak learners Instead of choosing single weak learner as the base learner, MGB with one, two, or three weak learners as the base learner and one gradient boosting module in Figure (a). Generally, when more weak learners are used as the base learner (which is more than the number of gradient boost modules), MGB faces difficulties in capturing multi-mode patterns and thus achieves degraded generalization performance.
choose the first few weak learners the base learner. We present the results of
(a)
(b)
Figure 4: Model performance under different settings for the base learner with respect to (a) the number of weak learners, (b) the sharing strategy of the base learner, and (c) the choice for the boosting rate a on 4-modes image classification tasks.
Static base learner V.S. Dynamic base learner The base learner is initialized using the global sharing parameter @ can be either static (if we keep its parameter unchanged) or dynamic (if we update the base learner during the training of gradient boost modules). We compare the versions of MGB that use a static base learner and a dynamic base learner, respectively. In both versions, we append one gradient boost module to the base learner and update it multiple times during each step. The results (shown in Figure 4 (b)) reveal that keeping the shared information (i.e. using static base learner) improve the stability of the model.
Boosting rate a The boosting rate (a) probably the most vital component for the MGB frame- work because it directly indicates the contribution of each weak learner to the final prediction. We test the performance of MGB under various settings of the boosting rate a, where the rate is either decayed (i.e. Olk 0/- where is constant), automatically learned, or equally contributed (i.e. a. = for base learners), respectively. The result suggests that using the automatically learned a equally contributed leads to more stable performance, while decayed results in more time task learning. This supports our analysis that our gradient boosting modules help learn task information.
CONCLUSION
In this work, we propose a novel direction for solving the problem faced by previous meta- optimization approaches, i.c., using the same initialization for diverse tasks. We present a meta gradient boosting framework that contains a series of weak learners to make predictions, using base learner to grasp shared information across all tasks. We have conducted extensive experiments to evaluate the model's ability to capture meta information and the task-specific information on regression and classification tasks. Our experimental results show the effectiveness our framework in learning multi-mode tasks. Our results reveal the necessity of selecting the weak learner carefully according to task types. An example is that CNNs outperform simple fully connected (FC) layers on image classification problems while FC layers can perform better regression tasks. In future work, we will extend our framework by considering multi- modal problems, e.g., learning from images, texts and numerical values, and study how to choose appropriate weak learners for specific applications.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ferran Alet, Tomás Jozano-Pérez. and Leslie P Kaclbling, Modular meta-learning. arXiv preprint arXiv:1806. 10166, 2018.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 39-48, 2016.
Sarkhan Badirli, Xuanqing Liu, Zhengming Xing, Avradeep Bhowmik, and Sathiya S Keerthi. Gradient boosting neural networks: Grownet. arXiv preprint arXiv:2002.07971 2020.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785-794, 2016.
Chrisantha Fernando, Jakub Sygnowski, Simon Osindero, Jane Wang, Tom Schaul, Denis Teplyashin, Pablo Sprechmann, Alexander Pritzel, and Andrei Rusu. Meta-learning by the baldwin effect. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1313-1320, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- -oouue 70, pp. 1126-113: JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 9516-9527, 2018.
Jerome H Friedman. Greedy function approximation: gradient boosting machine. Annals of statistics, pp. 1189-1232, 2001.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter, Yoshua Bengio, Paolo Frasconi, Jürgen Schmidhuber, et Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: survey, arXiv preprint arXiv:2004.05439, 2020.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Max Jaderberg, Wojciech M Czarnecki, lain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human- level performance in 3d multiplayer games with population- based reinforcement learning. Science, 364(6443):8599 855, 2019.
Diederik Kingma and Max Welling, Auto-encoding variational bayes. stat, 1050:1, 2014.
Brenden Lake, Ruslan Salakhutdinov, Jason Gross, and Joshua Tenenbaum. One shot learning of simple visual concepts. In Proceedings of the annual meeting of the cognitive science society, volume 33, 2011.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. International Conference on Machine Learning, pp. 2927-2936, 2018.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint 10X.vv/777009333 2017.
Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural network acoustic models. In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.

---

Under review as a conference paper at ICLR 2021
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-
learner. In International Conference on Learning Representations, 2018.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv: 1803.02999, 2:2, 2018.
Matthew Olson, Abraham Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In Advances in Neural Information Processing Systems, pp. 3619-3628, 2018.
Boris Oreshkin, Pau Rodriguez Lopez, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. In Advances in Neural Information Processing Systems, pp. 721-731, 2018.
Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan I. Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7229-7238, 2018.
Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. ICLR, 2019.
Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 403-412, 2019.
Philip Tannor and Lior Rokach. Augboost: gradient boosting enhanced with step-wise feature augmentation. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pp. 3555-3561. AAAI Press, 2019.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. In Advances in neural information processing systems, pp. 550-558, 2016.
Ricardo Vilalta and Youssef Drissi. A perspective view and survey of meta-learning, Artificial intelligence review, 18(2):77-95, 2002.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Risto Vuorio, Shao-Hua Sun, Hexiang Hu, and Joseph J Lim. Multimodal model-agnostic meta- learning via task-aware modulation. Advances in Neural Information Processing Systems, pp. 1-12, 2019.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM Computing Surveys (CSUR), 53(3):1334, 2020.
Huaxiu Yao, Ying Wei, Junzhou Huang, and Zhenhui Li. Hicrarchically structured meta-learning. International Conference Machine Learning, pp. 7045-7054, 2019.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian agdel-agnostic meta-learning. In Advances in Neural Information Processing Systems, pp. 7332-7342, 2018.
Yabin Zhang, Hui Tang, and Kui Jia. Fine-grained visual categorization using meta-learning opti- mization with sample selection of auxiliary data. In Proceedings of the european conference computer vision (ECCV), pp. 233-248, 2018
Yu Zhang and Qiang Yang. A survey on multi-task learning. arXiv preprint arXiv: 081 14, 2017.
10

---

Under review as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
A.1 SETTINGS FOR REGRESSION TASKS
Dataset details We use four functions for generating toy regression tasks: sinusoidal function, linear function, quadratic function, and absolute value function. For each task T' E T, the selected function generates input-output pairs with randomly initialized parameters. The tasks are generated from an underlying distribution T ~ P(D), where the meta-dataset D is the function set. Depending on the choice of the function set, we evaluate our model on uni-mode training (i.e. the tasks are generated from single function with different random initialized parameters) and multi-mode training. To ensure each function can generate a variety of tasks, we define the four functions with three variables follows:
Sinusoidal function: 02 sin(2 + v1) + 03
Linear function: 0/2 (r + 01) + i/3
Quadratic function: of -I + U2 x x U3
A bsolute value function: 02 |æ + 01 +
where V1, 222 and V/ are sampled uniformly from ranges [0. 7], [-3,3], and [-3,3], respectively. The inputs are within [-5,5], and the outputs are perturbed with Gaussian noise with the standard deviation of 0.3. We sample 10 batches of 50 tasks for uni-mode and multi-mode training tasks, respectively.
Model details Following Finn al.) (2017). we use two fully-connected layers of size 40 as the weak learner for regression tasks. The activation function is Leaky ReIUU (Maas et al. (2013), which provides good and stable results over the four functions. By default, the MGB framework includes one weak learner as the base learner and three gradient boosting modules. We update ten times for both the base learner and gradient boosting modules. Both the task learning rate 3 and the meta learning rate y set to 0.01. We apply an automatic learning strategy to the boosting rate (a:), which is initialized to be 0.5. When constructing the gradient boosting neural network, we use a cascading structure (He et al., 2016) to make the network faster and stable. We use batch normalization Szegedy 2015) for the input for gradient boosting modules. Recall that at each step k: we take the concatenation of the inputs and the hidden layer of the previous weak learner hg- = 08. (x)) as the inputs of gradient boost module at the current step, i.c., 9% + fe, (f4x-1, x)). Specifically, we latent layer h2 with same shape as the hidden layer of the
&
use shortcut layers to map the input previous weak learner; then, we concatenate ht and h which are both batch normalized, as the input for the current gradient boost module. We also test the impact of forwarding [ hg-2] to gradient boost modules and find it only has a slight impact model performance, In the default settings, we use [ x as the input for each gradient boost module. Generally, it takes around one hundred epochs before getting acceptable results.
A.2 SETTINGS FOR CLASSIFICATION TASKS
Dataset details We use four datasets to evaluate multi-mode few-shot learning tasks: Om- niglot (Lake et al., 201T), minilmageNet (Ravi & Larochelle (016), FC100 (Oreshkin et al., 20188) and CUB (Wah et al. (20T)) Details about the datasets are listed in Table 3 We divide tasks in each dataset into train, validation or test tasks. Omniglot is a dataset of 1623 handwritten characters from 50 different alphabets, where use the characters from the first 30 alphabets, the following 10 alphabets, and the last 10 alphabets as training, validating, and testing tasks, respectively. The minilmageNer dataset is subset of the Im Following Finn et al. (2017), we divide 100 classes into 64, 16, and 20 classes for meta- meta-training, meta- meta-validation, and meta-testing, respectively, Fewshot-CIFAR 100 (FC100) is based the widely applied CIFAR 100 datase Similar to (Ore- shkin et al. 2018). we divide 100 classes into 60 training classes, 20 validation classes, and 20 testing classes. As for the Caltech- UCSD Birds 200 dataset, i.c., the CUB dataset, we follow the settings in Vuorio et al. (2019) and use 140, 30 and 30 classes for training, validation, and testing, respectively. We generate uni-mode training tasks from one dataset. For each n-way m-shot learning task, we
https://www.t /ww. www.sttorooooooooooo
11

---

Under review as a conference paper at ICLR 2021
Table 3: Basic information of the image datasets
Dataset
Number of classes
Samples per class
Image channel
Image size
Omniglot
1623
28x28
minilmageNet
100
600
84x84
FC100
100
600
3
x2x32
CUB
200
:40
84x84
randomly draw n. classes from the dataset and m samples from each class to form the support set. Then, we select from the unselected samples as the query set. generate multi-mode training tasks by repeatedly run random selection of dataset and drawing task samples for this dataset. In general, we use 10 batches of 50 tasks to train the model for both uni-mode tasks and multi-mode tasks. We resize the image for each dataset into 84 x 84 so that the model can share across the four datasets. Besides, we map images in Omniglot dataset to layer with three channels so that they can be learned by the sharing model.
Model details The weak-learner uses CNN modules for learning the image embedding and fully- connected layers for classification. Similar to Finn et al. (2017). we configure each CNN module as 4 layers with 3 x convolutions, followed by batch normalization (loffe & Szegedy 2015). Leaky ReLU nonlinearity (Maas et al.) 2013), and 2 x 2 max-pooling. The results of the CNN module are first mapped to latent layer with 200 neurons and then predicted by classifier with two fully-connected layers. The MGB framework, by default, includes one weak learner as the base learner and two gradient boosting modules. We update five times for both the base learner and the gradient boosting modules. The boosting rate (a:) for each gradient boost module is automatically learned and is initialized as 0.5. The task learning rate / and the meta learning rate 7 are set to 0.001 and 0.005, respectively. Similar the settings for regression tasks, we use shortcut layers for the input æ and a hidden layer of the previous weak learner as the input for the current gradient boost module. The model needs thousands of training epochs obtain fair results. The results we presented for discussion are averaged for every tens of epochs.
A.3 SETTINGS FOR COMPARED METHODS
We compare our work with three compatitve baselines: MAML (Finn et al. 2017). LEO (Rusu et al., 2019 and MMAMI. (Vuorio et al. 2019).
MAML one of the pioneer frameworks for meta optimization. For each task, the parameter 0 of the task model Fo is initialized by the global sharing parameter and is updated by learning from the support set DS. The meta parameter á will be updated by learning from the loss on the query set L(DR).
LEO use an external encoder-decoder initialize the parameter 0 of the task model Fg. The external encoder-decoder framework has three parts: an encoder Jø., relation network Jø.- and a decoder Ja.- For each task, the inputs are mapped to the latent code 2 by 9o. and go. 0 is initialized based on Pou and the latent code When learning on task, the latent code z is first updated by the loss on support set (i.e., obtain the parameter Parameters of the encoder-decoder, i.e. &e. di, and dr. are updated globally by the query set loss on all tasks.
VV ) and then decoded to
MMAML first uses modulation network to obtain the mode information of a sampled task and then uses this mode information to initialize the task model Fg. The modulation network uses task encoder Is., to obtain a task embedding vector is then used to provide the modulation vectors for the parameters of the task model Fa (via r = - (v)). Suppose we have modules the task model Fo. and each module has parameter 0', then each module will be updated as 0³ + 0³ о O-i For each task, the task model is first initialized by a meta parameter &m. i.e., + dm: then, each module of the task learner is adapted by 0¹ t- 8¹ @ r', according to the outputs of the modulation network. The parameter 9 is locally updated for each task, and the meta- mara-eeasmeters (øm, On. and (by)) are updated across all tasks.
< 9ø.. (x)), which
12

---

Under review as a conference paper at ICLR 2021
We choose the task learner Fø and configure the settings (including activation function, task learning rate 3, and meta learning rate 7) for the compared methods in the same way as we configure our method both regression and classification tasks. In particular, LEO uses pretrained embedding as input to statistically generate means and variances for the predictor in its encoder-decoder structure. The predictor
fully-connected layer that takes the pretrained embedding (Qiao et al., (018)) as input. The weights of the predictor are sampled from a distribution specified by the means and variances generated by LEO. Since LEO cannot be updated via classic pack-propagation we use similar measures presented by Rusu et al. 2019). But we additionally use our strategy for generating the embeddings.
be regarded
13


Reviews:
[{"review_id": "fMtoqy3yqt", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed a method to incorporate neural networks into gradient boosting for meta-learning with a limited number of samples in each task. \n\nThe simulated experiments, where the generated tasks contain four continuous function, demonstrated that the proposed model outperformed MAML and its related variants, and on the classification task, it performed similarity to a previously proposed method MMAML.\n\nIn general, I am very interested in seeing neural networks being combined with boosting algorithms, but I do have quite a few questions.\n\n1. The paper claimed that the learning and updating strategy proposed in the method ensured a weak base learner. I have to say that I am not satisfied by the claim. One can simply construct a linearly-separable dataset, which doesn't contribute to the overall tasks, and it only takes few updates for a two-layer neural network to converge, then the learner itself is not weak anymore. In addition, it adds noise to other tasks in the task set.\n\n2. Normally, when a weak learner is learnt, a 1-D line search is conducted to find the optimal contribution of this particular learner. Although, shrinkage is usually applied in practice to anneal the contribution of new learners as a regularisation method to avoid overfitting. The paper seemed to have ignored the line-search part which can be done efficiently in 1D, whilst only the shrinkage was applied in the modelling, and I was wondering if there was a specific reason for that. \n\n3. In terms of the application of the proposed algorithm, I think it is rather limited. My understanding is that, although the proposed algorithm worked better than MAML and its variants on simulated data, the algorithm at the same time involves many hyperparameters, including (1) the design of the base feature extractor, (2) the design of gradient boost modules, (3) the boosting rates, and the annealing of them, (4) number of local update steps,  (5) number of global update steps, and many other hyperparams regarding to the training of neural networks. The first two hyperparameters or designs are critical in the success of the gradient boosting algorithm as weak learners are needed for learning so that the gradient is informative for the subsequent learners. Tuning those hyperparameters could be a time-consuming and also non-trivial task itself already, compared to the MAML algorithm itself, or generally multi-task learning approaches for meta learning, this algorithm doesn't seem to be outstanding. \n\n4. A fair comparison, IMO, could be to use the same feature extractor for producing vector representations of samples, and then directly apply gradient boosting with trees for meta learning. As mentioned in the paper, the authors also agreed that the feature extractors themselves have a huge impact on the final performance, therefore, it might be a good practice to check how well gradient boosting is able to handle meta-learning on top of extracted features.", "extra_scores": {}, "extra_text": {}}, {"review_id": "dgdgiZjtZZH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a meta-gradient boosting framework that uses a base learner to learn shared information across tasks and gradient-boosted modules to capture task-specific information. The proposed approach is applied to various regression and classification tasks.\n\nTo me the motivation of using a gradient boosting framework remained unclear throughout the paper. It is stated in the last paragraph of the Introduction that ‘Recent research.. with achieving low training errors’. However, why this is important to the meta-learning problem remains unclear.\n\n\nIn Section 3, the model is not explained clearly:\n-\tWhat is a weak learner? In the introduction it is mentioned that ‘Gradient boosting aims to build a new learner towards…We call the learner for each step as weak learner.’ What is a step here?\n-\t‘..the first few learners are regarded as the base learner for learning the shared information across tasks..’ This was also not evident how it is designed to achieve this. Given that I have no prior on working with gradient boosting directly, I found the motivation and the method quite hard to follow.\n-\t\nSection 3.1. Second sentence: ‘..where K is the number of adaptions’-what is adaptions?\nWhat is the definition of a gradient boost?\nHow is $\\theta_k$ defined in equation (1)?\n\nSec 3.2:\nThe first sentence- what does it mean?\nIt is mentioned here that a first-order MAML type approach is taken to learn the ‘base learner’. But how the two goal of using gradient boost modules to ensure task-specific  information is achieved is not clear.\n\nFigure 2,3,  and 4: The axes and fonts are unreadable.\n\nIn Results, the authors state that ‘ the results show that incorporating task identities can significantly improve the performance of multi-mode learning’. It was not clear to me how this is true. First, how is task identity even abstracted or assigned here? What features of the approach are able to get this information is not clear.\n\nThe gain classification tasks also seems marginal compared to the other approaches. The key emphasis of the authors is on the importance of the approach on multi-mode distributions. It is not made clear why the approach would be suited to this setting. This is supported numerically only in the case of regression experiments.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "RtheyDjxFrf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This study is presented clearly, and the core idea is interesting. However, the presented novelty is limited to a globally (for all tasks) and locally (task-specific)  learning paradigm using a framework inspired by (Badirli et al., 2020). The authors have presented experimental results for both regression and classification setups, which are interesting.\nIn my opinion, the paper has relatively high quality and can be interesting for the ICLR community. \nOne question regarding Figure 4 (a), you have mentioned that adding more weak learners causes difficulties in capturing multi-mode patterns, but from this figure, one can see that it is not completely true for the early epochs (before 400), comparing the two weak learners versus one weak learner case and it seems more like fluctuations, how would you explain it. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "3DhIID3GIlO", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Paper summary\nThis paper addresses a problem of model-agnostic meta-learning (MAML) and most of its variations/extensions - learning only a single parameter initialization for the entire task distribution, which might not be effective when the task distribution is too diverse. Inspired by gradient boosting, which aims to train a new learner to predict the residuals of the previously predicted result for each step, this paper proposes a meta gradient boosting framework. Specifically, the proposed framework represents the meta-learned global initialization as a base learner, consisting of the first few weak learners, which is responsible for acquiring sharable transferable knowledge by learning across all tasks (or task modes). Then, many gradient-boosting modules learn to capture task-specific information to fit diverse distributions more effectively. This paper evaluates the proposed framework and the baselines, including MAML, Multimodal MAML, and LEO, on both the few-shot regression task and the few-shot image classification task. To create diverse task distributions, it combines different function families (linear, sinusoidal, etc.) for the regression and merges multiple datasets (Omniglot, miniImageNet, etc.) for the classification. The experiments show that the proposed meta-gradient boosting framework (with 5 gradient boosting modules) achieves better or competitive results compared to the baselines. Ablation studies justify some design choices of the proposed framework, including the architecture of the weak learner, the number of gradient-boosting modules, the updating strategy for the gradient boosting modules, the boosting rate, etc. I believe this work studies an important problem and proposes an interesting framework. Yet, I have a few concerns regarding experimental setup and results which prevent me from accepting this paper (see below for details).\n\n### Paper strengths\n**motivation**\nThe motivation for addressing the issue of model-agnostic meta-learning (MAML) - learning only a single parameter initialization for the entire task distribution, which might not be effective when the task distribution is too diverse, is convincing.\n\n**novelty**\nAs far as I am concerned, the idea of utilizing the intuition of gradient boosting is novel and interesting. This paper presents a reasonable way to implement this idea.\n\n**technical contribution & ablation study**\nAblation studies provide insights that help understand the proposed framework and justify many design choices. This includes the architecture of the weak learner, the number of gradient boosting modules, the updating strategy for the gradient boosting modules, the boosting rate, etc. \n\n**clarity**\nThe writing is very clear and the figures illustrate the ideas well. Also, the organization of the paper is easy to follow.\n\n**experimental results**\n- The description of the experimental setup is comprehensive. The presentation of the experimental results is clear. \n- The proposed meta-gradient boosting framework outperforms or at least performs competitively compared to the representative baselines.\n- This paper studies a variety (i.e. different numbers of task modes) of settings, which provides great insights.\n\n### Paper weaknesses\n**baselines**\nI believe this paper ignores many important baselines. As a result, the experimental conclusions are less convincing. I list some of the baselines and brief reasons why I believe they should be included as below:\n- Hierarchically Structured Meta-Learning (HSML): is designed to perform soft clustering on tasks. It would be interesting to see if HSML can handle the task distributions considered in this paper. Also, it shows superior performance compared to MAML and a workshop version of Multimodal MAML. So, it is a stronger baseline to be included.\n- Probalistic MAML / Bayesian MAML: both of these two methods consider , outperforming MAML. Intuitively, this probabilistic schema should deal with multimodal task distributions better since it inherently learns to handle uncertainty. It would be great to compare against at least one of them.\n- Proto-MAML: presented in the meta-dataset paper (Triantafillou et al. in ICLR 2020), is designed to deal with multiple datasets combined and therefore should be able to perform well on the setup considered in this paper. Proto-MAML shows the strongest performance compared to MAML based methods and even outperforms some metric-based meta-learning methods. Showing the proposed framework can outperform or perform competitively compared to Proto-MAML would make this paper much stronger.\n- Metric-based meta-learning methods: this paper does not include comparisons against any metric-based meta-learning methods such as matching networks, prototypical networks, relation networks, TADAM, etc. Yet, the state-of-the-art results on the few-shot image classification are mostly achieved by metric-based meta-learning methods. Therefore, I believe it would be essential to include representative metric-based meta-learning methods.\n\n**MAML with a comparable number of parameters**\nSince the proposed meta-gradient boosting framework has more parameters than the vanilla MAML, it is possible that the performance gain comes from the larger capacity. It would important to include a MAML baseline that has a comparable number of parameters to justify this. \n\n**RL experiments**\nAs far as I am concerned, the few-shot regression task is more a task for detailed analysis for research rather than a task with a wide range of applications, and the state-of-the-art results of the few-shot image classification task have been mostly achieved by metric-based meta-learning methods. Therefore, I am mainly interested in the model-agnostic meta-learning  line of work because of its potential in reinforcement learning where the ability to adapt to unseen scenarios is crucial. Yet, this paper does not include any experiments on RL without any reasons, which makes the paper less convincing to me.\n\n**MGB-1 vs. MAML**\nIt seems that the proposed meta-gradient boosting model with one gradient-boosting module outperforms the vanilla MAML by a significant margin on the regression task yet only performs similarly to the vanilla MAML on the classification. Can the authors give some intuition about why this is the case?\n\n**meta-dataset**\nTo evaluate if the proposed framework and the baselines can deal with diverse task distributions on the few-shot image classification task, this paper combines four different few-shot learning datasets to produce a 4-mode classification task. Yet, the meta-dataset has been proposed for this purpose. Also, the meta-dataset paper provides a comprehensive comparison of recent meta-learning methods. Therefore, I believe this paper should evaluate the proposed framework on the meta-dataset and see if it outperforms the baselines.\n\n**stddev of the classification task**\nIt seems that the performance gap between the baselines and the proposed framework is insignificant on the image classification task. In this case, it would be important to also provide the standard deviation of each task.\n\n=== After rebuttal ===\n\nI am not satisfied with the response from the authors. I can only hardly recognize the effort made by the authors during the rebuttal - most of my points were only briefly discussed in the response without revising the paper. \n\n- The suggested baselines were merely briefly discussed but not added to the comparison.\n- The results of the meta-dataset, which in my opinion is the most suitable dataset for the purpose, are still not included in the revised paper.\n- The stddev of the classification task is not still not provided, making it hard to justify the performance gain.\n- Why is RL left to future work?\n\nI have read the reviews from other reviewers. With the little revision from the authors, I have decided to keep my original rating and would not recommend this paper to be accepted.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a meta-gradient boosting framework to tackle the model-agnostic meta-learning problem. The idea is to use a base learn that learns shared information across tasks, and gradient boosted modules to capture task-specific modules. The experiments show that the proposed meta-gradient boosting framework (with 5 gradient boosting modules) achieves better or competitive results compared to the baselines. However, there were several issues that the author feedback did not addressed properly. For instance, R2 were not satisfied by discussing briefly the suggested baselines without adding the comparison, or R1 pointed out that the claim “the learning and updating strategy proposed in the method ensured a weak base learner” because clear separable datasets could convergence quickly and weak is not anymore applicable. Besides these two specific concerns, the reviewers expected a large revision of the paper due to several cons about the paper. All reviewers agreed a mayor revision is needed before acceptance. Therefore I recommend rejection.
Decision: Reject