Meta Gradient Boosting Neural Networks

1 INTRODUCTION . While humans can learn quickly with a few samples with prior knowledge and experiences , artificial intelligent algorithms face challenges in dealing with such situations . Learning to learn ( or metalearning ) ( Vilalta & Drissi , 2002 ) emerges as the common practice to address the challenge by leveraging transferable knowledge learned from previous tasks to improve learning on new tasks ( Hospedales et al. , 2020 ) . An important direction in meta-learning research is meta-optimization frameworks ( Lee & Choi , 2018 ; Nichol & Schulman , 2018 ; Rusu et al. , 2019 ) , a.k.a. , model-agnostic meta-learning ( MAML ) ( Finn et al. , 2017 ) . Such frameworks learn initial model parameters from similar tasks and commit to achieving superior performance on new tasks that conform to the same distribution through fast adaptation . They offer excellent flexibility in model choices and demonstrate appealing performance in various domains , such as image classification ( Li et al. , 2017 ; Finn et al. , 2017 ) , language modeling ( Vinyals et al. , 2016 ) , and reinforcement learning ( Fernando et al. , 2018 ; Jaderberg et al. , 2019 ) . Generally , such frameworks define a target model Fθ and a meta-learnerM . The learning tasks T = { T train , T test } are divided into training and testing tasks , where T are generated from the meta-datasetD , i.e. , T ∼ P ( D ) . Each task contains a support set DS and a query set DQ for training and evaluating a local model . The initialization of the model parameter θ is learned by the meta learner , i.e. , θ ←M ( T train ) . We denote the meta-learned parameter as φ so that θ ← φ . For each task , the model obtains locally optimal parameter θ̂ by minimizing the loss L ( Fθ ( DS ) ) . The meta parameter φ will be updated across all training tasks by minimizing the loss ΣT∈T train ( L ( Fθ̂ ( D Q ) ) ) . Generally , it takes only a small number of epochs to learn locally optimal parameters across training tasks so that meta-learned parameter φ can quickly converge to an optimal parameter for new tasks . Most methods assume some transferable knowledge across all tasks and rely on a single shared meta parameter . However , the success of the meta-learners are limited within similar task families , and the single shared meta parameter can not well support fast learning on diverse tasks ( e.g. , a large meta-dataset ) or task distributions ( e.g. , T are generated from multiple meta-datasets ) due to conflicting gradients for those tasks ( Hospedales et al. , 2020 ) . Recent efforts have studied multiple initial conditions to solve the above challenges . Some employ probabilistic models ( Rusu et al. , 2019 ; Finn et al. , 2018 ; Yoon et al. , 2018 ) while others incorporate task-specific information ( Lee & Choi , 2018 ; Vuorio et al. , 2019 ; Alet et al. , 2018 ) . The former learns to obtain an approximate posterior of an unseen task yet needs sufficient samples to get reliable data distributions ; the latter conducts task-specific parameter initialization using multiple meta-learners yet requires expensive computation and can not transfer knowledge across different modes of task distributions . In this work , we aim to resolve the above challenges from a novel perspective by proposing a meta gradient boosting framework . Gradient boosting ( Friedman , 2001 ) aims to build a new learner towards the residuals of the previous prediction result for each step . We call the learner for each step as weak learner and make predictions based on summing up the weak learners . Recent research ( Badirli et al. , 2020 ; Olson et al. , 2018 ) has demonstrated the potential of decomposing deep neural nets into an ensemble of sub-networks with each achieving low training errors . We propose to use the first or first few weak learners as the base learner , followed by a series of gradient boosting modules to cope with a diverse array of tasks—the base learner is responsible for inferring transferable knowledge by learning across all tasks ; the gradient-boosting modules are designed to make task-specific updates to the base learner . Compare with existing work , which uses multiple initial conditions , our approach does not require specifying a set of initialization conditions and thus has better flexibility in dealing with multi-mode tasks . Our proposed framework is also more efficient than its counterparts as it does not require a large number of gradient boosting modules . We evaluate the proposed framework on few-shot learning scenarios for both regression and classification tasks . The experimental results show the well performance of the proposed framework , which demonstrates the model ’ s ability in learning with very few cases . 2 RELATED WORK . Meta-learning has the potential of replicating human ability to learn new concepts from one or very few instances . It has recently drawn increasing attention , given its broad applicability to different fields ( Hospedales et al. , 2020 ) . Pioneers ( Finn et al. , 2017 ; Nichol & Schulman , 2018 ) in this topic propose optimization algorithms with learned parameters to automate the exploitation to the structures of learning problems . However , most of them initialize the same set of model parameters for all tasks , which may have different distributions , thus resulting in over-fitting . Recent studies either model the mixture of multiple initial conditions via probabilistic modeling ( Finn et al. , 2018 ; Yoon et al. , 2018 ) or incorporate task-specific knowledge ( Lee & Choi , 2018 ; Alet et al. , 2018 ) , to address the above issues . Yoon et al . ( 2018 ) and Finn et al . ( 2018 ) use variational approximation to enable probabilistic extensions to MAML . But it is unclear how to extend MAML for a wide range of task distributions . Rusu et al . ( 2019 ) consider multiple conditions by borrowing the idea of variational autoencoders ( Kingma & Welling , 2014 ) , which encodes inputs to a low-dimensional latent embedding space and then decodes the learned latent code to generalize taskspecific parameters . Another line of research defines a set of initialization modules and incorporate task-specific information to select task-specific modules ; this way , it can identify the mode of tasks sampled from a multimodal task distribution and adapt quickly through gradient updates ( Vuorio et al. , 2019 ) . Yao et al . ( 2019 ) propose a Hierarchically Structured Meta-Learning ( HSML ) framework to perform soft clustering on tasks . HSML first learns the inputs and then obtains clustering results by the hierarchical clustering structure . HSML tailors the globally shared parameter initialization for each cluster via a parameter gate to initialize all tasks within the clusters . The above approaches have common limitations in 1 ) requiring sufficient data samples to generalize task distribution thus may fail in few-shot cases ; 2 ) being computationally expensive , due to the globally stored initialization modules ; 3 ) facing challenges in exhaustively listing every possible initial condition . Two closely-related topics to meta-learning are modular approaches ( Andreas et al. , 2016 ) and multitask learning ( Zhang & Yang , 2017 ) . Modular approaches are similar to meta-learning in that the input signal gives relatively direct information about a good structural decomposition of the problem . For example , Alet et al . ( 2018 ) adopt the modular structure and parameter adaptation method for learning reusable modules . Multi-task learning aims to learn a good shared-parameter or make the parameter for each task as similar as possible ( Wang et al. , 2020 ) . For example , Zhang et al . ( 2018 ) propose two task networks that share the first few layers for the generic information before applying different prediction layers to different tasks . These approaches differ from meta-learning in requiring fine-tuning the models over all training samples and thus can not adapt well to new tasks . Our framework Meta Gradient Booting ( MGB ) neural network is based on the idea of gradient boosting ( Friedman , 2001 ) , which aims to build a new learner towards the residuals of the previous prediction result for each step . The learner for each step is called weak learner , and the prediction is based on the summation of weak learners . Weak learners may vary from traditional decision trees ( Chen & Guestrin , 2016 ) to neural networks ( Tannor & Rokach , 2019 ; Badirli et al. , 2020 ) . Algorithm 1 Training of MGB 1 : Randomly initialize global parameter φ 2 : while Not done do 3 : for T ∈ T do 4 : for ( x , y ) ∈ DS do 5 : Initialize fθ0 by θ0 ← φ 6 : for k ∈ range ( K ) do 7 : θ ← θ − βL ( y , Fθ ) 8 : end for 9 : end for 10 : Get updated parameter θ̂ 11 : for ( x , y ) ∈ DQ do 12 : Calculate predictions Fθ̂ ( x ) 13 : Calculate task loss L ( y , Fθ̂ ) 14 : end for 15 : end for 16 : Update φ by φ← φ− γLmeta 17 : end while ... Feature Target φ θ0 ∇Loss ... Feature θ1 ∇Loss Outputs Hid Task 1 Task T Base Learner Gradient BoostModule Feature ∇Loss Feature Local Update G lo ba l U pd at e ... Figure 1 : Example of the model with only one gradient-boosting module . Green lines are for local update and red lines are for global update . A recent study ( Badirli et al. , 2020 ) proposes a general framework for gradient boosting on neural networks , which work for both regression and classification tasks . It uses the deep layers of neural nets as a bagging mechanism in a similar spirit to random forest classifier ( Veit et al. , 2016 ) . After only slight tuning , deep neural nets can perform well on a wide range of small real-world datasets ( Olson et al. , 2018 ) . These findings demonstrate the potential of decomposing deep neural nets into an ensemble of sub-networks each achieving low training errors . In our framework , we use the first weak learner or the first few weak learners as the base learner for learning the shared initialization parameter across tasks . The output for each weak learner is then aggregated to the inputs for the next step for constructing an end-to-end learning strategy until the last gradient boosting module . This way , the base learner serves as transferable knowledge , and the gradient boosting modules following it are trained for task-specific predictions . 3 METHOD . We explore the problem in the context of supervised learning , where input-output pairs are available in both training and validation sets . Similar to previous meta-optimization based approaches ( Finn et al. , 2017 ; Nichol & Schulman , 2018 ) , we assume the tasks are generated from an underlying distribution T ∼ P ( D ) , where D is the meta-dataset , which is either a uni-mode dataset or multi-mode datasets . Given a set of tasks T = { T train , T test } , each task T ∈ T contains a support dataset DS and a query dataset DQ , both representing input-output pairs ( x , y ) . We aim to learn a meta-learnerM to guide the initialization for a target model Fθ so that the target model can quickly adapt and perform well on a given new task . We propose a Meta Gradient Boosting ( MGB ) framework as the target modelFθ , which consists of several weak learners and can be represented asFθ ∼ ΣKk=0fθk . The first weak learner fθ0 or the first few weak learners are regarded as the base learner for learning the shared information across tasks ; the weak learners are gradient boosting modules for capturing task-specific information . The meta learner aims to learn transferable knowledge and provides initialization details for the base learner so that the model can quickly adapt to task-specific predictions with a few gradient-boosting steps . Figure 1 shows an example of our MGB framework under K = 1 , where we update the model locally for task-specific predictions and update the meta-learner globally for all tasks .

Summary: This study is presented clearly, and the core idea is interesting. However, the presented novelty is limited to a globally (for all tasks) and locally (task-specific)  learning paradigm using a framework inspired by (Badirli et al., 2020). The authors have presented experimental results for both regression and classification setups, which are interesting.