Continuous Meta-Learning without Tasks

1 INTRODUCTION . Meta-learning methods have recently shown promise as an effective strategy for enabling efficient few-shot learning in complex domains from image classification to nonlinear regression ( Finn et al. , 2017 ; Snell et al. , 2017 ) . These methods leverage an offline meta-training phase , in which they use data from a distribution of tasks to optimize learning performance on new tasks . These algorithms have focused on settings with task segmentation , where the learning agent knows when tasks change . At meta-train time , these algorithms assume access to a meta-dataset of datasets from individual tasks , and at meta-test time , the learner is evaluated on a single task . However , there are many applications where task segmentation is unavailable , which have thus far been under-addressed in the meta-learning literature . For example , consider a robot which must learn to adapt to a changing environment . The robot may switch from one environment to another during the course of deployment , and these task switches may not be directly observed . Furthermore , using an existing time series from interaction to craft a meta-dataset may require a difficult or expensive process of detecting switches in task . In this work , we aim to enable meta-learning in task-unsegmented settings , operating directly on time series in which the latent task undergoes discrete , unobserved switches , rather than requiring a pre-segmented meta-dataset . Equivalently , this problem can be viewed from the perspective of continual learning , in that we apply the meta-learning approach to the standard online learning problem statement wherein an agent must sequentially make predictions and learn with a potentially varying latent data generating process . To accomplish this , we integrate a Bayesian changepoint estimation scheme with existing meta-learning approaches , allowing the algorithm to reason about whether or not the task has changed in a time series . Thus , we enable a standard meta-learning algorithm , which is designed for the task segmented setting , to be both trained and tested directly on time series data without the need for task segmentation . Contributions . The primary contribution of this work is an algorithmic framework for task unsegmented meta-learning which we refer to as meta-learning via online changepoint analysis ( MOCA ) . MOCA wraps arbitrary meta-learning algorithms in a differentiable changepoint estimation algorithm , enabling application of meta-learning algorithms directly to problems in the continuous learning setting . By backpropagating through the changepoint estimation framework , MOCA learns both a rapidly adaptive underlying predictive model ( in the form of the meta-learning model ) , as well as an effective changepoint detection algorithm . MOCA is a generic framework which can be paired with many existing meta-learning algorithms . We demonstrate the performance of MOCA on both regression and classification settings with unobserved task switches . 2 PROBLEM STATEMENT . Our goal is to apply meta-learning tools to the problem of task-unsegmented continual learning , in which an agent is presented sequentially with input xt , asked to make a ( probabilistic ) prediction p ( ŷt | xt ) , and is then given the true label yt , and can thus ideally improve its predictions by learning from the labeled examples . Following the terminology of meta-learning , we assume that these data are drawn from a distribution according to some latent task Tt , p ( xt , yt | Tt ) = p ( xt | Tt ) p ( yt | xt , Tt ) . We will write x , y ∼ Tt as shorthand for x , y ∼ p ( x , y | Tt ) . We assume a distribution over tasks , which we write p ( T ) , and that the initial task T1 ∼ p ( T ) . At each timestep , the task is re-sampled from p ( T ) with some probability λ ( which we refer to as the hazard rate ) , or remains the same . Our goal is to optimize a learning agent to perform well in this setting . Let pθ ( ŷt | x1 : t , y1 : t−1 ) by the agent ’ s prediction for yt given input xt and the past labeled examples . We will evaluate the learner ’ s performance through a negative log likelihood loss , and our objective is as follows : min θ E [ ∞∑ t=1 − log pθ ( yt | x1 : t , y1 : t−1 ) ] subject to xt , yt ∼ Tt , Tt = { Tt−1 w.p . 1− λ Tt , new w.p . λ T1 ∼ p ( T ) , Tt , new ∼ p ( T ) ( 1 ) We assume that we have access to a representative time series generated in the same manner from the same distribution of tasks , and use this time series to optimize θ in an offline , meta-training phase . Critically , however , in stark contrast to standard meta-learning approaches , we do not assume access to task segmentation , i.e . that this offline data is pre-grouped by latent parameter T . Moreover , we highlight that we consider the case of individual data points provided sequentially , in contrast to the common “ k-shot , n-way ” problem setting prevalent in few-shot learning ( especially classification ) . Our setting may easily be extended to the setting in which multiple data points are observed simultaneously . 3 PRELIMINARIES . 3.1 META-LEARNING . We begin by presenting a unified perspective on meta-learning in the task-segmented setting , which allows straightforward presentation of the algorithms used in this work as well as generalization to the task-unsegmented case . The core idea of meta-learning is to directly optimize the few-shot learning performance of a machine learning model over a distribution of learning tasks , rather than just a single task , with the goal of this learning performance generalizing to other tasks from this distribution . A meta-learning method consists of two phases : meta-training and online adaptation . Let θ be the parameters of this model learned via meta-training . During online adaptation , the model uses context data Dt = ( x1 : t , y1 : t ) from within one task to compute statistics ηt = fθ ( Dt ) ( 2 ) where f is a function parameterized by θ . For example , in MAML ( Finn et al. , 2017 ) , the statistics are the neural network weights after gradient updates computed using Dt . In neural processes ( Garnelo et al. , 2018 ) , the statistics are the aggregated context parameters computed via encoding and aggregating the context data . For recurrent network-based meta-learning algorithms , these statistics correspond to the hidden state of the network . For a simple nearest-neighbors model , η may simply be the context data . The model then performs predictions by using these statistics to define a conditional distribution on y given new inputs x , y | x , Dt ∼ pθ ( y | x , ηt ) . Adopting a Bayesian perspective , we refer to pθ ( y | x , ηt ) as the posterior predictive distribution . The performance of this model on this task can be evaluated by considering how well this posterior predictive distribution matches the true task data distribution , L ( Dt , θ ) = D ( p ( y | x , Ti ) ||pθ ( y | x , fθ ( Dt ) ) ) where D is a measure of the dissimilarity of the two distributions , e.g . the KL divergence , for which this objective becomes standard negative log likelihood minimization . Meta-learning optimizes the parameters θ such that the model performs well across a distribution of tasks , min θ ETi∼p ( T ) [ EDt∼Ti [ L ( Dt , θ ) ] ] . Across most meta-learning algorithms , including all of those referenced above , both the update rule and the prediction function are chosen to be differentiable operations , such that the parameters can be optimized via stochastic gradient descent . Given a dataset pre-segmented into groups of data from individual tasks , standard meta-learning algorithms operate via first sampling a group for which T is fixed , treating part of that group as the context dataDt and sampling from the remainder to obtain test points ( x , y ) from the same task . While this strategy can be very effective and produce expressive models that are capable of few-shot learning in complex domains , it relies on task segmentation which in many settings , especially continual learning , is not easily available . 3.2 BAYESIAN ONLINE CHANGEPOINT DETECTION . To enable meta-learning without task segmentation , we extend prior work in changepoint detection . Specifically , we build on Bayesian online changepoint detection ( Adams & MacKay , 2007 ) , an approach for detecting changepoints ( i.e . task switches ) originally presented in a streaming unconditional density estimation context , which we review here . BOCPD operates by maintaining a belief distribution over run lengths , i.e . how many of the past data points yt correspond to the current task . At time t , run length of rt = τ indicates that the task has switched τ timesteps ago , i.e . D−τ = yt−τ : t are all drawn from a shared task T . A belief that rt = 0 implies that there has been a task switch , and that the current datapoint yt was drawn from a new task T ′ ∼ p ( T ) . We denote this belief distribution at time t as bt ( rt ) = p ( rt | y1 : t−1 ) . Given rt , we know the past rt data points all correspond to the same task , and thus the density p ( yt | y1 : t−1 , rt ) corresponds to the posterior predictive density after conditioning on the past rt data points . We can reason about the overall posterior predictive by marginalizing over the run length rt according to bt ( rt ) , p ( yt | y1 : t−1 ) = t−1∑ rt=0 p ( yt | y1 : t−1 , rt ) bt ( rt ) , where p ( yt | y1 : t−1 , rt ) is referred to as the underlying predictive model ( UPM ) . BOCPD recursively computes posterior predictive densities for each value of rt ∈ { 0 , . . . , t− 1 } , and then evaluates new datapoints yt+1 under these posterior predictive densities to update the belief distribution b ( rt ) . In this work , we extend this approach of Adams & MacKay ( 2007 ) beyond Bayesian unconditional density estimation to apply to general meta-learning models operating in the conditional density estimation setting , and derive these update rules in more detail for our context . 4 META-LEARNING VIA ONLINE CHANGEPOINT ANALYSIS . MOCA uses Bayesian changepoint detection to enable the application of meta-learning algorithms to settings without task segmentation , both at train and test time . Specifically , we extend BOCPD to derive a recursive Bayesian filtering algorithm for run length in the conditional and joint density estimation setting , and leverage a base meta-learning algorithm with parameters θ to provide an underlying predictive model when conditioned on a run length . In the following subsections , we first derive MOCA ’ s Bayesian filtering updates , and then outline how the full framework can be used to both train and evaluate meta-learning models on time series without task segmentation . 4.1 BAYESIAN RUN-LENGTH FILTERING . As in BOCPD , MOCA maintains a belief over possible run lengths rt . Throughout this paper , we use bt to refer to the updated belief before observing data at that timestep , ( xt , yt ) . Note that bt is a discrete distribution with support over rt ∈ { 0 , ... , t− 1 } . At time t , the agent first observes the input xt , then makes a prediction p ( yt | x1 : t , y1 : t−1 ) , and subsequently observes yt . Generally , the latent task can influence both the marginal distribution of the input , p ( xt | x1 : t−1 , y1 : t−1 ) as well as the conditional distribution p ( yt | x1 : t , y1 : t−1 ) . Thus , the agent can update its belief over run lengths once after observing the input xt , and again after observing the label yt . We will use bt ( rt | xt ) = p ( rt | x1 : t , y1 : t−1 ) to represent the updated belief over run length after observing only xt , and bt ( rt | xt , yt ) = p ( rt | x1 : t , y1 : t ) to represent the fully updated belief over rt after observing yt . Finally , we will propagate this forward in time according to our assumptions on task dynamics to compute bt+1 ( rt+1 ) , which is used in the subsequent timestep . To derive the Bayesian update rules , we start by noting that the updated posterior is proportional to the joint density , bt ( rt | xt ) = p ( rt | x1 : t , y1 : t−1 ) = Z−1p ( rt , xt | x1 : t−1 , y1 : t−1 ) = Z−1p ( xt | x1 : t−1 , y1 : t−1 , rt ) p ( rt | x1 : t−1 , y1 : t−1 ) = Z−1pθ ( xt | ηt−1 [ rt ] ) bt ( rt ) ( 3 ) where the normalization constantZ can be computed by summing over the finite support of bt−1 ( rt ) . Importantly , this update requires pθ ( xt | ηt−1 [ rt ] ) , the base meta-learning algorithm ’ s posterior predictive density over the inputs . Within classification , this density is available for generative models , and thus a generative approach is favorable to a discriminative approach within MOCA . In regression , it is uncommon to estimate the distribution of the independent variable . We take the same approach in this work and assume that xt is independent of the task for regression problems , in which case bt ( rt | xt ) = bt ( rt ) . Next , upon observing yt , we can similarly factor the belief over run lengths for the next timestep , bt ( rt | xt , yt ) = Z−1pθ ( yt | xt , ηt−1 [ rt ] ) bt ( rt | xt ) . ( 4 ) Again , the normalization constant can be computed via a sum over the support of rt . Finally , we must propagate this belief forward in time to obtain bt+1 ( rt+1 ) : bt+1 ( rt+1 ) = p ( rt+1 | x1 : t , y1 : t ) = ∑ rt p ( rt+1 , rt | x1 : t , y1 : t ) = ∑ rt p ( rt+1 | rt , x1 : t , y1 : t ) p ( rt | x1 : t , y1 : t ) = ∑ rt p ( rt+1 | rt ) bt ( rt | xt , yt ) . where we have exploited the assumption that the changes in task , and hence the evolution of run length rt , happen independently of the data generation process . The conditional run-length distribution p ( rt+1 | rt ) is defined by our model of task evolution . Recall that we assume that the task switches with fixed probability λ , the hazard rate . Thus , for all rt , p ( rt+1 = 0 | rt ) = λ , implying bt+1 ( rt+1 = 0 ) = ∑ rt λbt ( rt | xt , yt ) = λ . ( 5 ) Conditioned on the task remaining the same , rt+1 = k > 0 and rt = k − 1 . Thus , p ( rt+1 = k | rt ) = ( 1− λ ) 1 { rt = k − 1 } implying bt+1 ( rt+1 = k ) = ( 1− λ ) bt ( rt = k − 1 | xt , yt ) . ( 6 ) Equations ( 5 ) and ( 6 ) together define bt+1 over its support rt+1 ∈ { 0 , . . . , t }

Summary: This paper pushes meta-learning towards task-unsegmented settings. Different from the traditional offline meta-learning phase with explicit task segmentation, MOCA adopts a Bayesian changepoint estimation scheme for task change detection. The setting is novel and deserves research in-depth, and the idea is easy to understand. The proposed method can learn the meta-learning model and changepoint detection model simultaneously. Besides, the MOCA framework is not designed specifically for one algorithm and can be easily combined with other meta-learning models.