itself is uh 10 pages and then there is this immune repertoire classification which uh i will like spend one minute looking at it so you have you have these set classifications so for each human you obtain a set of immune receptors and you simply obtain one label whether that human is immune to a particular disease or not and your task is kind and then a different


human has a different set you have no idea which one of these things is responsible for it being for the human being um for the human being immune or not in fact there is a it you can't even decide based on these you can only decide based on like sub sequences uh of these and they might be in combination with each other so there might not be a single one responsible

Hopfield Networks is All You Need (Paper Explained)


but like a combination but you don't have labels for the individual ones and you have different ones per human and they are different lengths all of this is just a giant um giant task and you have many of them you have tens of thousands per human right so they build a system here where first they do these 1d convolutions to process the inside sequences and then they do this


hopfield um attention mechanism all with with learned queries uh over these things and then they train on the output label and surprisingly uh that actually works even with tens of thousands of inside sequences and only one label for all of them and so they they achieve i guess uh favorable results compared to other baselines on this task using these hopfield network which is pretty


interesting but i'll let you look at that paper yourself so i hope this somehow uh made it a bit clear what happens here and it would actually be pretty interesting um if we you know to see what happens if we just do maybe two rounds of these updates is this even desirable right uh is it desirable to run this to convergence is there something good about not running into convergence or does it


thank you for the introduction Riya we're excited to be giving this tutorial on meta learning throughout the tutorial we encourage you to be thinking about questions that you might have about the content that we're presenting and as you go through if you have a question you may come up to one of the four microphones but also if you want to also ask a question from from your own seat

Learning to learn: An Introduction to Meta Learning


without having to get up we have a link for that will allow you to post questions and to also upload other questions and we'll be monitoring that and also asking questions throughout from that link so that's slide okama sauce meta but we'll also have the link on the future slides also we posted all of a pdf version of these slides at tinyurl.com sasha ICML - meta slides and


so if you don't want to be putting your phone up to take pictures of the slides and you can look at the slides there as well great okay I mean additionally we'll also be taking questions at the break and at the end of the tutorial so let's get started so a lot of the motivation for meta learning comes from being able to learn from small amounts of data and in particular what we've


seen is that meta learning thrives with large data sets if there's one thing to take away from the last few years of machine learning research I think it's that large diverse data sets plus large models leads to broad generalization we've seen this time and time again from systems trained on imagenet to transform our models trained on large machine translation systems to GPT to trained


for large-scale language modeling and all this falls under the paradigm of deep supervised learning but what if you don't have a large data set what if you're in domains such as medical imaging or robotics or translation of rare languages or recommendation systems in each of these situations we don't have a large data set for every possible task every possible switch situation or


every possible person we want to personalize our machine learning system to or what if you want a more general-purpose AI system that can do many different things that you want to be able to continuously adapt and learn on the job if so it's impractical to learn each and everything from scratch for doing this and so instead we want to be able to very quickly learn new things based on


our previous experience and finally what if your data has a long tail for example what if the number of data points starts going down significantly as you encounter more objects or as you interact with new people here new words and encountering new driving situations in these situations your standard machine learning systems will do well in in the kind of the big data regime but


as as you move towards having fewer examples these systems will start to break down they're prettier it not only in the long tail something but in all three of these situations these settings start to break the standard supervised learning paradigm ok so what I'd like to try out next is is it actually give you guys a test so supervised learning breaks down here but actually humans are


pretty good at these situations and I want to give you a future learning test and your goal is I'll give you six training data points which are shown on the left the three on the the first column are from the painter Brock and the the middle three are from Cezanne and your goal is to be able to classify the painter for the for the paintings shown on the right who painted that


painting and this is a future learning problem because you only get six data points in order to do this how so you get six label data points for this binary classification problem okay so raise your hand if you think that the painting on the right was painted by Cezanne okay and raise your hand if you think that the painter the painting was drawn by Brock okay great


so most of you got the right answer so this is indeed by Brock and so and in the way that you could recognize this is that there's kind of some some more straight lines and more kind of high contrast lines in the painting and so how did you accomplish this so this sort of thing trading from learning from only six examples but have to be extremely hard for a lot of modern machine


learning systems and yet all of you eight guys were able to do it or most of you guys were able to do it quite well so the way that you were able to accomplish this was because you have previous experience you weren't trying to learn from these six examples from scratch and many of you probably haven't seen these particular paintings before or maybe you haven't even


scene paintings from these particular artists before but you have experienced different shapes different textures you've probably seen other paintings before and do that previous experience you're able to figure out how to solve this task from only six examples okay so now how about we get a machine learning system to solve this task depending on what era you're in you


would probably answer it differently you might try to model the image formation process you might try them all the geometry of different objects in the image if you were using slightly more sophisticated techniques you might use something like histah features or Haga features with a support vector machine or more recently maybe you try to fine tune from imagenet features or try to do


domain adaptation from other painters for example and and maybe in the future we'll be doing something even more sophisticated so these different approaches may seem very distinct in that kind of the approach that they're taking but they all share one thing in common which is all of them are different ways to inject previous knowledge or previous experience into


the system and as you move down these these prior knowledge you get few engineered human engineered priors and more data-driven priors and also as you move down you get systems that work undoubtedly better and so in this tutorial we want to try to take this one step further and in particular we want to be able to learn priors explicitly from previous experience that lead to


efficient downstream learning an entirely data Duren approach to acquiring these priors that is can we how these systems learn how to learn to solve tasks and this is what is known as meta learning in the rest of this tutorial Sergey will first talk about the problem statement and overview the general meta learning problem then we'll be talking about different meta learning algorithms


ranging from black box out updation approaches to autumn tape optimization based approaches to nonparametric methods then we'll discuss how we can develop bayesian variants of each of these methods then we'll talk about how meta learning has a plot been applied to different application areas we'll take a short five-minute break and also allows additional questions Sergey will then


talk about meta reinforcement learning and we'll conclude by discussing challenges on frontiers ok next circuit will be talking about the problem statement and overview chelsey and those of you that were trying to find the slides we did actually there was somebody who actually posted the link again if you go to the thing on the slide here the slide Oh / meta the first question is actually a


link to the slide deck so if you want the slide deck please check that out there alright so let's start with a discussion of how we can actually formulate the meta learning problem and there are really kind of two distinct viewpoints on meta learning there's kind of a mechanistic view and a probabilistic view let me explain what I mean by these the mechanistic view looks


at meta learning as a setting where there's a deep neural network model that can read in an entire data set and then make predictions for new data points training this network use a metadata set which itself consists of many data sets each for a different task and this view of meta learning makes it easier to implement meta learning algorithms so if you're actually coding something up in


tensorflow or PI torch the mechanistic view is probably the one that makes it clearest the probabilistic view treats meta learning as the problem of extracting prior information from a set of meta training tasks that allows for efficient learning of new tasks this view says that learning a new task basically used this prior plus a small amount of training data to infer the


most likely posterior parameters that will allow you to solve this task and this view of meta learning makes it easier to understand meta learning algorithms these are not two views that result in different methods they're actually two viewpoints that can be taken to understand the same methods so in this part of the tutorial I'll actually focus on the second view on the


probabilistic view because our aim is really to help everybody to understand meta learning algorithms but we'll see the more mechanistic view emerge when we talk about particular practical instantiation of these methods okay so just to work towards a problem definition for meta learning let's first start with a problem definition for regular supervised learning and cast it


in a probabilistic framework so a lot of what I'm gonna say some of you might have already seen may be in a course on machine learning or a textbook but I just want to walk through it step by step because the meta learning problem definition will build on this so if we're doing supervised learning what we're really doing is we're finding the most likely parameters Phi given our


data D so Phi denotes the parameters of your model so if your training for example a deep neural network model Phi literally refers to the weights the D refers to your training data so it's a set of tuples of input-output pairs where the input might be something like an image and the output is maybe the label corresponding to the class of the object in that image now when we actually want


to do this kind of maximum like that estimation problem we typically apply Bayes rule and rewrite it as the sum of log P of D given Phi plus log P of Phi and the first term is typically referred to as the likelihood of your data and the second term is the prior or the regularizer so if you're using weight decay that corresponds to a Gaussian prior for example and if we factorize


the likelihood if we assume independent and identically distributed data points then we get the familiar form shown here it's a sum over all of your data points of the log probability of the label Y I given the input X I and your parameters Phi so this is essentially supervised learning now of course there are some things that are a little bit problematic about this as Chelsea alluded to in the


motivation the models that will work best the most powerful models will typically require a large amounts of data so if your data is very limited it might be very difficult to get a very accurate posterior or very accurate estimate of Phi so the problem at its core that we're going to be dealing with a meta learning is how do you do a good job of estimating Phi when your data is


limited and the way you're going to do that is by incorporating more data that is not exactly for the tasks that you want but somehow structurally related so the question is how can we incorporate additional data and I should say as an aside this is very much the same kind of challenge that things like semi supervised learning and unsupervised learning deal with so in semi-supervised


learning you incorporate additional data that doesn't have wise and and so forth in metal learning you incorporate additional data that we're going to call D meta terrain which is labeled data it's just labeled data for different tasks so D meta trained is actually a data set of data sets so it's a set of data sets d1 through DN where each of those data sets di itself consists of a


set of tuples X I and why I wear those why's our labels for a different task so we assume the tasks are somehow structurally similar but not actually the same so you can't just directly incorporate Dee Mehta trained as trainee dated or supervised learning let me give a little example this is based on a popular benchmark for meta learning called the mini image in a data set


let's say that your few shot classification task requires you to do five-way classification between cats dogs lions worms and stacks of bowls I don't know why you would want to do this task but let's say this is a few shot tasks and you have only a few examples of each image now those few examples are not enough to solve the tasks by themselves so we're gonna use


the meta train which is a collection of data sets for other five-way classification problems so maybe one of them classifies you know birds mushrooms dogs singers and pianos for instance different tasks but some structural similarity because all the more visual recognition tasks another example maybe the tasks you want to solve is a few shot regression problem you have a few


examples of input-output pairs but then your meta training tasks consist of other curve fitting problems so other curves with a few sample input-output pairs or maybe you have some kind of speech recognition task or some kind of language translation task and so on so in all these cases you can formulate a set of meta training tasks that are not the same as the tasks you want to solve


but somehow structurally similar and typically these would come from your prior data now we could simply stop right there and treat meta learning as a nonparametric learning problem so you want to basically use D and D meta train together but oftentimes we want to use high capacity models we don't want to store all of our data and keep it around forever we'd like to somehow to SIL it


into a model into a parametric model with learn model parameters so in meta learning we don't want to keep D meta train around forever what we'd like to do instead is learn some meta parameters theta so we're going to use D meta train to learn theta and theta will basically contain all the information that we need for solving new tasks that we've extracted from D meta train so whatever


we need to know about D meta train is going to be baked into theta via a meta learning process and that's essentially the essence of the meta learning problem now if we want to treat this probabilistically what we can do now is we can say well we can write out our p of phi given d common Mehta trained as an equation where we're integrating out the these sufficient


statistics of our meta training data called theta and this implies the assumption that Phi is conditionally independent of D meta train given theta which is very reasonable because we just said the theta should be whatever extracts all the necessary sufficient statistics from D meta train now in reality integrating out theta is computationally very very expensive so


we wouldn't want to do this what we would want to do in practice typically is use a maximum a posteriori estimate which is which means that we're going to approximate this integral with just a point estimate for theta star where theta star is whatever actually maximizes the log probability of theta given D meta train which again is a very standard thing to do in machine learning


so this Arg max that I have written on the right side here this is the meta learning problem the meta learning problem is to pull out the right theta from your meta training data so that that theta contains everything you need to know to efficiently solve new tasks and efficiently solve new tasks means figure out Phi so once you have theta the problem of getting Phi can be


written as the Arg max of log P Phi given D comma theta star because you don't need the D mail train anymore that's all been baked into theta star okay so that's the basic problem formulation if anybody has any questions feel free to come up to the microphones and ask me otherwise I'm going to move on to a simple example yeah that's an excellent question so meta learning is


conceptually quite related to a number of other problem settings including transfer learning multitask learning you know even things like semi-supervised learning in that all of these problem settings deal with incorporating additional data that is not quite from your task but is going to help you solve your task more efficiently the main difference is that meta


learning deals with a setting where you still have to do some amount of adaptation on your new task transfer learning formula in a certain ways can I should be viewed as a type of metal learning as well I'll describe related problem settings a little bit more at the end of this section and maybe then things will be a little clearer okay let's continue so let's work through a little example


of how we can design a cartoony version of a meta learning algorithm Chelsea will talk about much more practical algorithms this is just meant to be an illustration so first let's talk about the the adaptation let's say that we already have this theta star we don't care how we learn it and now we just want to classify new data points using a small data set D so classifying new data


points means your test data point X test goes in your label Y test comes out and this function that does this is somehow parameterize by theta star so theta star determines the mapping between X test and y test Waiters theta star come from well it comes from using your data set D which might be a small data set for your new task together with your theta star


so D is going to be read in by some function and that function that reads in D is parametrized by theta star sorry about that so that function is parametrized by theta star now you would also like to of course be able to learn this theta star using large amounts of metal training data which I'll come to in a second but if you can somehow use that Mediterranea to get theta star then


that will process your data set up with Phi star and allow you to turn your test inputs into test labels so theta star is what parametrize is this function okay so now how do we actually train this thing well as I alluded to before it's going to involve this meta training data and the key idea behind setting up metal learning algorithms I think is best summarized by this sentence from a paper


by venules at all called matching networks which says that our training procedure is based on a simple machine learning principle which is that tests and train conditions must match now let's unpack this a little bit what are the test conditions well test here refers to meta test right so meta test time is adaptation the test condition is that a model parametrized by theta star


reads in d outputs Phi star and five stars then used to classify new data points for your task so the training time conditions need to match so it met a training time you also need to have a model that reads in a data set which data set well a data set di from your met training set it is going to be parametrized by theta it's going to output Phi star and that Phi star needs


to be good for classifying points which points well that's maybe the puzzle so what is it that we're actually going to classify here what we need to do in order to complete the meta learning problem definition is we need to reserve a little test set for each task so it's not enough to just have a training set the training set is what the model needs to read in but then needs to be trained


on something and what it actually going to be trained on is a little test set for each task so for every one of our few shot tasks we're gonna assume that we have K training points but done also some number L of little test points and those test points are was going to supervise the metal learning they are not used for adaptation they're just used for metal learning so D test is


where X tests and Y tests will be sampled from so the game that you're playing then is read in D train output Phi and make sure that Phi is good for classifying points from D test for that same task so now we can actually complete the meta learning problem definition so the adaptation step we can write more compactly as some function f theta star of D train so f theta star


reads in D train and outputs Phi star so now all we have to do is learn theta such that Phi equals F theta D train is good for D test so for every task I you want to read in D train I and be good for D test I which means that we can write down the meta learning problem formulation like this theta star is the Arg max for the sum over all of your tasks of log P Phi I given D test I where Phi I


is equal to F theta apply to D train so notice that we get 5 from D train but we met a train on D test we can also represent this with a graphical model so if you're into graphical models here's a graphical model that represents this relationship so you have theta which are your global metal earned parameters for every task you have a Phi I and X train together with Phii


determines y train and X tests together with Phii determines Y test and Y test is observed during meta training but not observed during that a testing that's why it's kind of half shaded there okay so this basically defines the meta learning problem but let's kind of round out this explanation with a little bit of an overview of terminology because we'll see a bunch of this terminology


pop up again again during the tutorial so we make a distinction between meta training meta testing training and testing so you're learning the parameters theta during a meta training phase that meta training phase trains on a collection of data sets each of which is separate into a training set and a test set so when we say training set we mean that small few shots set for a


particular task when we say test set we mean the corresponding test images when we say meta training we mean the whole set of tasks so meta training consists of multiple tasks each one with a training set of a test set and meta testing is what happens once you're done meta training and you want to adapt to a training set for a new task so the set of day of data sets is called D meta


train that's what we refer to it as these are meta training tasks we're gonna use Ti to refer to meta training tasks so these are all of our key is this is our meta test tasks I'm sorry and then sometimes you hear people say support set and quarry set so support refer is basically synonymous with training set and sometimes people use supports that just to avoid the


confusion between meta training and training so if someone says support they mean the inner training set and when someone says quarry they're referring to the test the the inner test not the meta test just the test so the quarry is the thing that you actually want to classify correctly after reading in the support and if someone says like oh I have a case shot classification problem what


they're referring to is the number of examples if someone says I have a five-way classification problem they're referring to the number of classes so if you say have a five-shot five-way classification problem that means I have five classes each of which has five examples there's a little bit of confusion about the word shot sometimes it means the number of images per class and sometimes it means


the total number of images usually we'll use it to mean the number of images per class so five shot five way means twenty-five data points okay now just to wrap up a few closely related problem settings that are good to be aware of and this is coming back to that question about transfer learning from before so middle learning is closely related to a few other things that we can actually


cast as you know within the same terminology so multitask learning deals with the problem of learning a model with parameters theta star that immediately solve multiple tasks so you can think of multitask learning as sort of zero shot metal learning so that corresponds to defining parameters that immediately solve all the tasks at the same time this is usually not possible


in metal learning problems you can't have one model that classifies you know that does the five Way classification with dogs and lions and also with you know the pianos and the cats but you can view multitask learning is a special case where Phi is just equal to theta another very closely related problem setting is type of parameter optimization and auto ml these can be


cast as metal learning they're actually they are essentially metal learning problems they're outside of the scope of this tutorial but I'll just mention briefly how they can be related so in hyper parameter optimization you can say that theta refers to your hyper parameters that's where you're going to get out of your meta training set and Phi is the network weights so you'll


learn your hyper parameters from d mehta train and then you'll use them to get Phi architecture search for same deal theta refers to the parameters of your architecture and Phi is the actual weights in the model this is a very active area of research unfortunately outside the scope of this tutorial but hopefully this will tell you a little bit about how they relate


okay and next Chelsea will discuss a number of actual metal learning algorithms that we can use based on this problem setting oh yes and we'd be happy to take any questions right now - yeah so one question from can you elaborate more on the structural similarity that's required between the Mediterranean tasks yeah so in regard to the structural similarity between the metal training


tasks this is we can actually make that notion formal the way we make it formal as we say there's a lucien over tasks there's a distribution p task and you assume that all of your mediterranean tasks are drawn from that distribution and you assume that all of your meta test tasks are drawn from the same distribution so this is the meta learning analog of the standard


supervised learning assumption now what does a distribution over tasks really mean well that's sometimes ends up being a much more subjective notion if you have a piece of code that generates your tasks and you can say well these it needs to be generated by the same code but of course in reality those tasks are probably produced by nature and there it becomes a much fuzzier line so Chelsea


will also discuss a little bit about extrapolation and generalization the perhaps pertains to this great so before we actually want to start going about evaluating meta learning algorithms are going about designing my lorry algorithms we need to figure out how to actually evaluate a meta learning algorithm once we have one and so it's worth mentioning that there a lot of the


the modern metal learning advances and and techniques were motivated by some some work done by Brendan Lake in 2015 and Brendan introduced the Omniglot dataset which is a is much more simple than the mini imagenet dataset that the circuit was showing on the previous slides but allows us to really study some of the basics of mendler nning so the Omniglot dataset it has six hundred


twenty three sixteen hundred twenty three characters from 50 different alphabets that are had many classes and few examples per class a few classes per class and what I find really appealing about these kinds of datasets is that they're more reflective of the statistics of the real world in the real world we have tremendous diversity in terms of the number of objects and


number of items and people that we encounter and we don't encounter them over and over again we often encounter many new things constantly throughout our lifetime okay so proposes both discriminative and generative problems for example an initial approaches for this data set and for other data sets for a few shot learning we're based off of Bayesian models and on parametric's


and similar to what Sergey was mentioning before in addition to this which in many ways actually methods are doing quite well on these days you've also been using things like mini image net C far cub celeb a and other data sets for for evaluating meta training algorithms and many of these were not necessarily initially purposed for for medal learning but we're able to


kind of put them in to the purpose that we would like okay so this is similar to what was discussing earlier where we have some n way K shot classification problem such as image where we wanted to be able to perform learning from very small data sets so we might want to be able to learn from one example of five different classes to classify new examples or new images as being among


one of the five classes shown on the left and the way that we can do this is we can take data from other image classes structure it in the same way is what we're gonna be seeing about a test time for example taking images of mushrooms and dogs and singer is structuring it into the likewise these these five way Wangchuk classification problems doing this for many different


other image classes training a training or networked in order to perform these types of things across these training classes such that an evaluation is able to solve the problem on the top with held out classes and this is an example that's specific to image classification and we're gonna be coming back to this example a number of times because it's useful for comparing different


approaches but the same sorts of ideas are also applicable to things like regression to language generation and and and prediction to skill learning really any machine learning problem you can construct in this way where you're training it on a number of machine learning problems and you want it to be able to generalize to learning a new problem with a small amount of data ok


so now that we know how to evaluate a meta learning algorithm was actually dig into how we actually design these meta learning so the general recipe and the general principle behind these algorithms is that we need to choose some form of inferring the parameters of a model Phi given our chain data set and our meta parameters theta and then once we choose choose the form of this we can then


optimize it optimize the meta parameters fight theta with respect to a maximum likelihood objective using our meta training data okay and many of the different algorithms that were gonna be looking at today really only differ in step one choosing how we want to represent this inference problem essentially and so I you can ask well can we just treat this as an inference


problem and and pretty clear neural networks are actually are quite good at inference so maybe we can just use a neural network to to represent this function itself and that's exactly what the first approach will be so this is what we'll refer to as blackbox adaptation approaches and the key idea is for a neural network to represent this function that outputs a set of


parameters given a data set and a set of meta parameters and for now we're going to be using deterministic or point estimate of this function key we told to note as f theta and of course we'll get back to Bayesian methods later and so we'll see Bayes a bit later okay so how do you actually try to design a neural network to do this well one thing you could do is you could use a recurrent


neural network that takes in data points sequentially and produces a set of parameters Phi and so this recurrent neural network in this case will be representing F theta and then we'll take the outputted parameters use those parameters for another neural network that's gonna make predictions about test data points and so these are gonna be the data points from D test okay and


then once we have this model we can train it with standard supervised learning this is just a standard recurrent neural network so we can train it to maximize a log probability of the labels of the test data points given the test inputs and we can do this optimization across all of the tasks in our meta training data set we can rewrite this loss function of performing


a bag of evaluating predictions of a model as simply a loss function operating over the parameters Phi and the test data points so we're gonna write this right here and this this will be used mostly for convenience later on and then with this form we can write the full optimization problem as an optimization of the over the the parameters outputted by the neural


network F theta and the test data set okay so now that we have this this optimization objective how do actually what is the algorithm that's used to optimize this so what the algorithm looks like is we first sample a task from our meditating data set or a mini batches of tasks then for that task we have a data set di and will sample disjoint data sets D tre and I and D


test I from that data set and then once we have so I guess what this looks like I say these are the images corresponding to tasks i we want to be able to partition this or basically sample this sample D train and sample t-tests from from this data set and so we'll assign like so and then once we have D train and D test will compute the parameters using D train and then evaluate those


predicted parameters using the test data points and it's quite important that d'être and D tests are disjoint so that we're not training for memorization of the labels but instead training for generalization and then once we update our meta parameters of them were of course going to repeat this process for new tasks and if we use a mini batch of tasks the gradient in step four is gonna


be averaged across that mini batch great okay so there's the algorithm now how to actually represent the form of F data so the the form that I have written here is a recurrent neural network you could use something like an Ellis TM you could also use something that another memory augmented neural network like a neural Turing machine which has done it has been done in past work you could also


use something like self attention or 1d convolutions or even really just a feed-forward network that then averages in some embedding space the key thing is that you want this you want these networks to be able to take in sets of data points and often times you want it to be able to take in variable numbers of data points and so that's where I will be using these types of


architectures that have the capability to take in take in sets and variable numbers of data points okay great so I know that we've gone over kind of this type of approach and how it works and and the different architectures what are some of the challenges that come up so one thing that you might ask is well if our neural network is outputting all of the parameters of of another neural network


this doesn't really seem scalable but if the the the neural network that's making inferences about test data points has millions of parameters then we need a neural network that outputs millions of a million million dimensional output and one idea that we can use to remedy this is we don't actually need to output all of the parameters of another neural network we really just need to output


the sufficient statistics of that data set of the training tasks that allow us to make predictions about new data points and so what we can do is we can take this take this architecture instead of outputting Phi we can output something like H where H is representing a low dimensional vector and then and this will be essentially representing information about the task everything


that's needed about the task in order to make predictions and then you can combine this sufficient statistic H with another set of parameters theta G that are also metal learned along with the parameters of F such that with these with both H and theta we can make predictions about new data points and then theta G can be something very high dimensional and with the combination of


the two we'll be able to make predictions for a new task okay so the general form of what this looks like is you can kind of abstract oh wait is this notion of H and just write out the ability to make predictions given a training data set and a new test input outputting the corresponding label okay before I move on to the next type of approach are there any questions so the


one question from the audience was does the network need to be recurrent or are there other architectures that could work well absolutely so as I mentioned on the previous slide this could be something that's recurrent like Ellis gems and you're all Turing machines you could use something like self attention or work recursive models but you also don't need to have something that


actually is like a sequence the sequence model could also simply being be something that has a feed-forward model and then average if you assume that you're trained is that has fixed-length than you could of course also concatenate and use a fully connected network although these are the approaches on this slide tend to be a bit more scalable I think it's on hello hi um so this


might be related to the few previous slide so I I'm trying to understand the difference between meta learning and they're learning to learn framework by Jonatan vector of the T / posts around 1990s so I think if I understand correctly the the basic idea of learning to learn is that you're trying to learn the inductive bias of the learning problem which in this case is data start


in your notation could you elaborate a little bit more in this similarity or the difference Thanks yeah so I'm not sure if I'm familiar with the particular work that you mentioned but in general many of the ideas that we're presenting are inspired by work that was done initially in the late 80s and early 90s with with older types of neural network approaches many


of those approaches didn't specifically look at the few shot learning setting which were focusing on this tutorial but looked at in general learning from from relatively large data sets and it's worth mentioning that actually this particular approach up here was actually done was was one of the approaches that was used in the 90s and and also by hawk radar at all in 2001 but some of the


approaches that we'll be discussing in later parts of the tutorial are more new another question from the audience was can we use meta learning approaches to solve classical supervised learning problems and are there any benefits to doing so so I think that we'll get to this question a bit at the end there are situations where I guess the main thing the main type of problem that you want


to be able to address with meta learning techniques is settings where you want to be able to take in information about a task that has that takes on the form of some data whether it be fully supervised or weekly supervised and so if your supervisor problems setting doesn't have that sort of structure where you want to learn from data to solve new problems then I think it would be quite


challenging to apply some of these techniques but in other situations maybe your meta learning maybe your supervised learning problem does have that structure and these approaches would definitely but definitely do well ok so let's move on to the optimization based approaches so now that we just kind of talked about one way to kind of make this approach more scalable is there a


way to infer all of the parameters of the neural network in a way that's scalable without having to train a neural network to output all of the parameters in a particular as Sergei mentioned before you can view the problem of supervised learning as an inference problem where I goes in for a set of parameters using data and the way that we solve supervised learning


problems is through optimization and these optimization perches are quite scalable so what if we treat the problem of inferring a set of parameters from data and using meta parameters as exactly an optimization problem and and this is what optimization based approaches do and so the key idea here is that we're going to acquire our task specific parameters Phii through an


optimization procedure that depends both on the training data and our meta parameters theta and the optimization will look something like this where we are optimizing objective that looks like the likelihood of the data given our toss parameters as well as the likely of our task parameters given our meta parameters we're essentially the meta parameters are serving as a prior now


you might ask well what should the form of the prior be for our meta parameters well there's a lot of different approaches a lot of different things that we could do here but one very successful form of prior knowledge that we've seen in deep learning for example is using is training from an initialization provided by another data set and in particular what we seen is if


we train on things like image and then fine-tune that model on other data sets were able to capture a lot of the rich information and supervision that's exists in the image that data set and use it for new tasks so this is a very successful form of prior knowledge and of course the way that it works is you have a set of pre trading parameters data and you run gradient descent using


training data for your new tasks okay so this is this works really well in a number of different situations but what if your train data for your new task has only a few data points like the six data points that I showed in the example at the beginning well in this case things like fine tuning are gonna break down a bit because though they weren't actually trained for the ability to adopt very


quickly and as a result you'll either over fit to those six examples or you won't be able to adapt quickly enough and move far enough from your initialization so this is what we want to be able to do a test I may be quite nice if we can just run fine tuning on on our six examples and get a get it up get some answer some function so this what we wanna be able to do at test time


the key idea behind this approach is to explicitly optimize for us how to pre train parameters such that fine tuning with a very small data set works very well and so what this looks like is we're going to take the fine tuning process written here this is just one step of gradient descent but you could also use a few steps or up to like ten steps of green descent for example then


we are going to take where we were after got after fine tuning this can be like Phii for example evaluate how well that generalizes to new data points for that task this is measuring how successful fine-tuning was and then we can optimize this objective with regard to the initial set of parameters so we're gonna optimize for set of pre train parameters such that fine tuning gives us a


generalizable function for that task and of course we don't want to just do this over one task but we'll do this over all of the tasks that are meta training data set so that we can learn an initialization that's amenable to fine tuning for many different types of tasks ok so the key ideas is to learn this parameter vector that transfers effectively via fine tuning ok so what


does this look like at a somewhat more intuitive level say theta is the the parameter vector that we're meta-learning and Phi I star is the optimal parameter vector for task I am then you can view the meta learning process as a thick black line where when we're at this point during Mediterranean we take a gradient step with respect to task 3 we're quite far from the optimum


for task 3 whereas at the end of meta learning if we take a gradient up with respect to toss 3 or quite close to the optimum and likewise for a number of other tasks we refer to this procedure as model agnostic medal earning in the sense that is agnostic to the model that you use and loss function that you use as long as both of them are amenable to gradient based adaptation okay so now


that we've gone through the objective again let's let's go through what actually that algorithm looks like so we can take the algorithm that we showed before for the standard for the the black box adaptation approach and instead we want to derive the algorithm for an optimization based approach and what we can do is we can simply just replace step 3 that's inferring the parameters with the


neural network with a step that's actually optimizing for the parameters in this case through gradient descent so we'll sample a task sample destroy its data sets for that and for parameters with gradient descent on the training data and then update our meta parameters using the test data points note that this does bring up a second-order optimization problem because you have to


compute the gradient in step four you have to differentiate through a gradient step and test in in step three in practice a number of standard auto differentiation libraries like tensor flow and PI torch can handle this quite gracefully and and really you don't have to worry about it too much at all and it also isn't particularly computationally expensive but we will talk a bit more


about a ways to mitigate this in a few slides okay so now how does this approach compared to the black box out of potations that we mentioned before so let's bring up the general form that we talked about before where you have some neural network that's taking a training data point or training data set and a test data point and it's producing a prediction for the test data point it


turns out that you can view the optimization based approach in the same general form so before we were using like a recurrent neural network to represent this function but now we're using what I'll denote as F mammal to represent this function and that is the function with parameters Phi that takes an X tests and produces a prediction where Phi is defined as the initial meta


parameters plus gradient descent on the data point and so essentially you can view the mammal algorithm as a computation graph just with this funny embedded gradient operator within it and so I really you can just view it as a very similar approach for doing it but one that has a lot more structure the structure namely of optimization within it and also with this view that means


that we can very naturally mix and match components of these different approaches so for example one of the things we could do is learn the initialization that that mammal is doing but also learn learn how to make gradient updates the initialization and that's exactly what ravi and la rochelle did in 2017 which actually preceded the mammal work and the and this computation graph view will


come back again in as we discussed the third type of approach great so questions how is theta G learned this is coming back to the black box adaptation yeah so great question so theta G this is going back to the black box and in that case we had a we had a very like submission statistics age that are produced by the neural network and we also had theta G that was using to use


to make predictions about the test data points and in that case theta G is optimized with all the other meta parameters of F so it's optimized just like all the other meta parameters another question about the black box setup tation so we'll Hib trivial meaning like why wouldn't a chai just learn to recognize which task is fed in and just up what's sort of like a one


hot indicator for one of the N tasks that's a good question I think that in practice it doesn't do you have an answer for that yeah maybe one way to think about this is it's kind of the same problem as memorizing labels for regular supervised learning so in the same way that supervised learning can overfit meta learning can net over fit so if you start seeing that your that


your model just outputs a task indicator that can happen if you have a very small number of meta training tasks that's just an instance of overfitting we'll talk about metal or filling towards the end of the tutorial great oh one more another one I don't know if you know this but what's oh gee from memory augmented neural networks yeah so that's a great question


so the memory augmented neural networks question by santoro at all basically that just used a standard RNN to take in data points including the test data point and so in that case theta G was actually the same exact parameters as theta in F so H the sufficient statistic is simply the hidden state of an RNN and there is weight sharing across across theta G and theta enough where it's


represented by the the weights that are shared across time and it recurrent neural network so this one is a crowd favorite apparently how was oldest related to hyper networks where we were interested in giving parameters of a model s output great yeah so the the first blackbox annotation approach is also what is done in hyper networks I think that I'm not completely sure about


this but I think that the hyper networks paper wasn't particularly focused on better learning problems and look at you looking at other problems where you're gonna be outputting parameters of neural networks but the approach the algorithm used is is exactly the same as the black box on updation approaches that I mentioned before this question I can answer myself can we adopt mammal in a


framework that we don't have a batch of tasks ahead of them instead we get them sequentially to hear the answer this question you'll have to wait until the very end of the tutorial on the last slide and she'll see we'll answer it okay one quick audience question okay so this is a question about FML so if I understand it correctly that MMO essentially learns the base model which


kind of in the middle of like the optimal parameter for different tasks but it's it is working analyzing an assumption that those optimal parameters for those different tasks are now too far away so if depth in some kind of if you are choosing a model space that the optimal like parameter or far away so that maybe you're the middle point of the base below although it is not too far away


from each of the optimal parameter it might be not optimal for any of those I noticed it in the original paper that you are basically using a pretty simple commercial neural net which have fewer neurons I'm just wondering will the MMO works keeps its performance like if you are using a more complex model where its parameter space is more complex and speaker yeah


so first it's mentioning that this diagram we like to use for an illustration purposes in terms of understanding the algorithm but it can also be a bit misleading which is that in many cases particularly with heavily over parametrized neural networks there isn't just a single optimum for the correct solution there's actually an entire space of optimum and and with


with those types of problems we see that the it is actually a little bit easier to find something where you're simply one or a few gradient steps away and in fact in a minute I'll talk about the expressive power of of the Manuel algorithm and its ability to adapt even when your tasks are extremely different with regard to architectures I'll talk a bit about that but in practice we do


find that it scales well to to larger larger architectures but it may require a bit more tuning than other meta learning methods okay thank you okay so great leaving off where you left off so mammal exhibits this kind of structure unlike the black box on updation approaches which is it has this gradient operator inside of it and so it's actually performing an optimization


problem both within the better training process as well as it met a test time and so one thing that might be quite natural to ask is using that structure does that mean that we can generalize better to two tasks that are slightly out of distribution and this is of course an empirical question that we're gonna study more so than a theoretical question and so what we're gonna do is


we're going to compare mammal with with blackbox annotation approaches such as snail and meta networks and we looked at image Dominika image classification problem and we tried to plot the task variability versus performance and what we found consistently across the board is as we move away from the the Mediterranean tasks with either zero zero shear or a scale of 1 we see that


of course performance drops for all approaches but that mammal is able to perform better because it has the structure of being able to run in gradient descent at test time and at the very least you are still running grained descent so you'll you won't be doing significantly worse than then what you might be doing with with a neural work that you really don't can't really


say anything about if it's just outputting parameters for example okay so you might say well we get this nice structure but does this come at a cost like as the question alluded to before do the did you need to assume that the task parameters are very close to each other for different tasks and so we studied this question by studying the expressive power of a single gradient


step basically the update that's used in the mammal function and what we can say is that actually for sufficiently deep neural network function f the mammal function on the right can represent anything that the recurrent neural network can represent on the Left which is that it can represent any function of the training data and the test input and we can show this under a few assumptions


that are relatively mild such as the non zero learning rate as well as unique data points in the training data set and the reason why this is interesting is that it means that mammal has the inductive bias of optimization of precede procedures being embedded within it but without losing the expressive power of gradient descent or without losing the expressive power of deep


recurrent neural networks ok great so let's go back to some of the motivation that we talked about a little bit with optimization based approaches where we're saying that the meta parameters serve as a prior and we talked about how one form of prior knowledge is is initialization for fine-tuning can we make this a bit more formal and actually better characterize what's work sort of


prior that is it mammal is imposing on the learning process and to do so we're going to look at Bayesian meta learning approaches that use graphical models so this is a kind of a graphical model similar to the one that Sergei showed before where Phii is the top specific parameters and theta is the meta parameters so if you want to do do meta learning or learn a prior theta in this


graphical model it's gonna look like the following optimization where we're optimizing the logs likelihood of the data given given the parameters you can write this out similar to the equations that Surya showed earlier as an integration over the top specific parameters Phi which are not observed and the and this is simply empirical Bayes and the we can approximate is is


it's quite intractable and so one thing we could do is we could approximate this integral with you maximum a posteriori estimate of the Taos Pacific parameters Phi I and this is a fairly crude approximation but it but one thing interestingly that we can show is if you compute the map estimate basically gradient descent with early stopping corresponds to map inference


under a Gaussian prior with mean theta and a variance that's a function of the number of gradient descent steps and the learning rate and this is exact in the in the linear case and approximate approximate in the non-linear case and so what we can see is that through this approximate approximate equivalence as well as the appraoch the approximation of the integral with the map estimate is


that mammal is approximating inference in this hierarchical bayesian model which i think is useful for providing some intuition for the types of priors that we're learning in the meta learning process okay so mammal is a form of implicit prior are there other forms of priors that we can impose on the optimization procedure one thing we could do is do grain descent with an


explicit Gaussian prior the log likelihood of a Gaussian shown here and this is what was done by Reuters Warren at all in the implicit mammal paper we could also have a prior used in Bayesian linear regression in this case we can't input and post a prior on all weights of the neural network that would be intractable but we can impose it on the last layer of the neural network and do


that on top of metal earned features this was done in an alpaca another type of mammal and we can also this is moving more away from Beijing about this but we can also do other forms of optimization on the last layer of the neural network such as doing Ridge regression logistic regression or support vector machines and this is uh this forward prior is essentially that


we want features that are useful for linear classification that can be performed with these methods and to my knowledge is this last approach meta objet as the current state of the art on P shot image recognition benchmarks okay so now that we've talked about optimization based approaches let's go through a couple challenges with them so one challenge is how do you choose an


architecture that's effective for embedding this integrated in descent procedure and one way to do this is to do architecture search and the interesting things that was found in this paper is it found that highly non-standard architectures that were very very deep and very narrow we're quite effective for use with mammal and this is a bit different from standard


architectures that work well for standard supervised learning problems and I'm particularly many images at five wave five shot classification proceed benchmark mammal with the basic architecture I see achieves around sixty three percent accuracy while mammal with the architecture search is able to achieve seventy four percent accuracy a pretty substantial boost by actually


tuning the architecture that works well for it lastly one other challenge that you come up with with the medal the mammal algorithm is that you run into the second order optimization procedure and this can exhibit different and different instabilities one idea for trying to mitigate this is really the dumbest idea you can come up with is to assume that the Jacobian of Phi with respect to


theta is identity and simply copy the gradient with this rectify to be the gradient with respect to theta and this actually works somewhat surprisingly well oddly enough on relatively simple problems although anecdotally we found it not to work well as you try to move towards more complex problems like imitation learning and reinforcement learning I another thing you can do is


automatically learn the inner and outer learning rates you can also optimize only a subset of parameters in the inner loop such as the last layer or affine transformations at each layer you could also try to decouple the the learning rate in the back term statistics that each gradient step to have fewer decoupled parameters that might cause instabilities and finally you could also


introduced introduced additional context variables into the architecture to allow for multiplicative interactions between parameters and allow for a more expressive gradient and so come my takeaway here is that there are a range of simple tricks that can help the optimization significantly great so before we move on to nonparametric methods let's take one question from the


audience and potentially some subtitle questions do you know how Mammon compares to other first-order method learning algorithms particularly reptile by Jones shown many tail so you questions how does it compare to first-order algorithms like I mean can it get over some of these problems you just presented in this fight and you're asking about the de mammal no like if you see reptile which is a


first-order method for meta learning instead of mammal can it get over some of the second-order gradient problems you just presented here yeah so as I mentioned on the on the first idea I listed here both first-order mammal and reptile do by using this crude approximation you can have a faster optimization procedure and and it potentially can be less stable but the


the main benefit that you get from it is this is faster and lower in memory but we have found that there are a number of problems where these types of first-order methods don't work at all and you need to use the the second-order methods in order to optimization optimize them well thank you why is theta minus Phi in the Gaussian prior this is a few slides ago the


answer to this question though is that the the interpretation as a Gaussian prior it basically says that the prior is on Phi and Phi is normally distributed with a mean of theta the variance of that prior depends on the number of breeding steps you take which is actually a very natural thing bei so the more grading steps you take the further away you get from theta that


corresponds to a prior with the wider variance next question is there any guarantee or test that Phi is not multimodal as map will assume unimodality yeah so it's certainly yeah this this tribution certainly could be multimodal and this approximation is ignoring that in practice we have found that that mammal can work quite well on multi modal problems where you have like


quietly quite different functions that you're that are represented in the task variables although you do need a deeper neural network for that and there are also approaches such as multi modal mammal I believe that try to tackle this problem head-on and enable you to get more efficient use of your neural network parameters and by by allowing it to represent multimodal distributions


over five data a couple more quick ones in the case of orthogonal tasks with mammal just memorize so if the tasks are if a single function can represent both tasks without relying on the data then it will just memorize the function and ignore the data in many cases can we do gradient descent for multiple steps to get Phi in mammal yeah absolutely so you can use a variable number of gradient


steps and practice we found that up to five gradients that works well but in practice you can use you can use more than that if if you find it helpful for your algorithm I it does not introduce higher order terms than a second order optimization it still remains the second order optimization if you go through if you go through the mouth okay great so let's move on to nonparametric


approaches and so far we've talked about parametric methods and that we're gonna be learning a model that's parameterize by five and and what about using methods that don't have parameters five and the motivation here is that in low danger regimes nonparametric methods are quite simple and work quite well and during meta test time we're in a future learning setting and so we are in elite


low data regime however during but a training we still want to be parametric because we have large amounts of data across all of the meta training tasks so the key idea behind these approaches is can we use a parametric meta learner in order to produce a nonparametric learner okay and note that some of these methods that I'll be presenting do precede parametric approaches but we're


presenting them in this in this group setting to aid in understanding okay so the key idea here is is here's a few shell learning problem and one of the things you might ask is well what one thing you could once very simple thing you could do in this approach is just take your test data point and compare it to each of the the data points in your training data basically do nearest


neighbors by by comparing to each of the images the this is a quite a simple opportunity quite valid for these types of problems the key question is in what space do you compare these images and with what distance metric I for example you could do pixel space or l2 distance but that probably wouldn't give you an effective metric over the similarity between these


images and so the key idea behind these on parametric methods is to learn a metric space that leads to effective comparisons learn a more semantic metric space that leads to effective predictions on the test data points and then we learn how to compare these images to make effective predictions and so the first very simple approach for doing this is to train a Siamese Network


to predict whether or not two images are the same so we can train a neural network that takes in two images and is trained to output whether or not they're the same we're not so zero correspond so them not being the same one corresponds to them being the same class and repeat this through all of the data in your meta training data set and so once you've trained this neural network to be


able to compare pairs of images about a test time you can then take your test data point compare it to each of the images in your training dataset see and then output the corresponding label to the image that is the closest okay so in this case meta training is give me a two-way classification problem and then meta testing is going to be an anyway classification problem by doing all of


these pairwise comparisons so now to improve upon this approach you might ask well can we make Mediterranean meta testing match can we train it such as it can actually perform effective anyway classification and this was kind of a key idea that Sergey alluded to in the problem definition which is to try to match Mediterranean meta testing in this case putting a an NBA classification


problem with nearest neighbors into a neural network and so we're gonna be feeding the train data set as shown on the left into an into the neural network comparing this to our test input shown on the bottom two and outputting these similarities and then training it such that the predictions that it gets out which corresponds to weighted nearest neighbors are correct with respect to


each of the tasks in our meta training set cool so this is basically embedding nearest neighbors into a neural network the the metric that we're using can be correspond to a convolutional encoder or some other architecture to get the embeddings of the training data points this method uses a bi-directional ostium and then we get a model that can do anyway classification is actually better


trained for anyway classification now if you have more than one example per class what this will do is it will independently compare the test image with each of those examples per class and one of the things that might be nice to do is actually to better integrate the information across different examples for our class to aggregate cost information into a sort of prototypical


embedding of that class such that when we do comparisons we're comparing at a class level rather than at the example level and this is exactly what prototypical networks does so they embed each of the data points for a given class such as the class corresponding to green blue or orange I average these to compute a prototype shown as CK and then make predictions based off of embedding


the test data point X test and comparing it to each of the prototypes and what this looks like is we simply measure the distance acts between each of the prototypes this is denoted as D and then perform a softmax operation to decide which class it corresponds to and in this case D can correspond to Euclidean distance or cosine distance great so this is a simple approach that works


quite well and there are a couple other extensions that we can make upon it so so far we've looked at Siamese networks about two networks they're prototypical that works at all correspond to some sort of embedding in the nearest neighbors either two examples or two prototypes when tanta comes up is what if you need a reason about more complex relationships between data points we


don't simply want to average to create a single prototypical example pour per per class to handle this we could learn a nonlinear relation module on the embeddings so we can essentially learn that D function by embedding each of the pairwise examples and and producing predictions for each of them we could also learn a different prototype more than one type for each class learning an infinite


mixture of prototypes or perform message passing on the different classes and the different examples in our data set okay so stepping up a bit let's actually try to think about how these different approaches compare so you can go back to the computation graph perspective and bring up the kind of computation graph view of the black box approaches as well as the optimization based approaches and


the nonparametric approach can be viewed in the same exact way except here I we're gonna be having a new function which I'll denote as PN for prototypical networks where the function corresponds to the softmax of the the negative distance between the embedded query tests data point and the prototypes where the prototypes are defined as the average embedding of each


of the class for that example and so with this view again we can mix and match components of this computation graph to create hybrid approaches among these different three and this includes things like camel which can both conditions on the data with a black box approach and run gradients and runs grain to send all the parameters it's also another hybrid approach is to run


gradient descent on an embedding that produces a set of parameters so this sort of combines all three approaches and then finally we can do something like mammal but initialize the last layer to be a linear classifier equivalent to the prototypical networks classifier okay before we move on to the takeaways are there any questions about nonparametric approaches or any


questions from slide oh no there are roughly 200 questions and I can't sort through them all but there are a couple of questions that are kind of crowd favorites that are maybe good to take not not to do that primary methods necessarily but one common theme is well can we do meta training where we have different numbers of classes for different tasks or different numbers of


data points for different classes yeah absolutely so and actually this is a this is a good question to talk about now too because um these different approaches can handle variable numbers of training data points and numbers of number of data points per class and varied number variable numbers of classes the this is something that optimization based approaches can handle


very gracefully because you can simply compete your loss function over different batches of data points very very naturally blackbox approaches you can do it but you need to train it for to be able to do that you need to be able to train it with variable data set sizes and variable numbers of examples per class and nonparametric approaches can also handle it fairly gracefully but


in practice we found them not to always work well to to variable numbers of data points per class so another question which is kind of a summary of multiple questions but I think it applies to all three of the methods as well to what degree to meta learning methods actually learn to use the entirety of kind of a test batch meaning that they classify multiple


images at once versus are they classifying individual images one at a time this is all the questions about inductive versus transductive are also getting at this point yes I think it depends on your application so if you think that you are going to have just a single example that you need to classify it test time then then all of these approaches can handle that setting and


then if you're in the transductive setting and you think that you're gonna actually have a test dataset that you need a label that corresponds to two multiple classes then you could also you could either independently classify them of course or try to actually combine information between them different approaches can handle this in different ways for example you can use the batch


storm statistics to I basically transmit information across different tasks or across different examples in your meta test set and with this you can actually transmit a fair amount of information through those about statistics and there are also more recent approaches that try to learn lost functions that can effectively transmit information across unlabeled examples


another common theme in regard to mammal and a few questions here is well what are the relative benefits of simply doing pre training followed by fine-tuning versus doing a mammal and also can mammal itself be used as just a pre training stage without any fine-tuning so in practice mammal is certainly training for like fine-tuned ability to a new task and so I'm


practice you really do need to actually fine-tune it to a test task although perhaps if you only care about getting good features and maybe the features learned by the method are something that is reasonable okay so some intermediate takeaways of these different approaches one of the benefits of things like blackbox approaches is that they are easy to combine with a variety of


learning problems such as supervised learning and as you'll see after the break and reinforcement learning they do involve a challenging optimization problem and this is a bit of a subtle point which is that initialization because you're not embedding a known learning procedure into the meta learning process there's no inductive bias that points you in the right direction for how to


learn from the data and as a result you have to learn how to learn from the data completely from scratch and as a result these methods can often be data inefficient because that optimization process requires time to actually learn how to learn from data and lastly these methods do have an intertwined model and architecture essentially the the model that you're using to make predictions


about data points is inherently intertwined with the model that you're using to to take in the training data points as input okay as I mentioned before optimization based approaches can very nicely handle varying K and large K the structure also lends well to out of distribution tasks and one of the downsides is that does it involve the second order optimization although there


are a couple of approaches towards trying to mitigate this nonparametric pressures are quite simple and they're entirely feed-forward and because they're entirely feed-forward they're computationally fast and easy to optimize because you don't need to worry about by propagating through gradient steps or through recurrent neural networks for example they're harder to


generalize to varying K this is more of an empirical observation than then a theoretical one and of course because they're using non parametric approaches they're harder to scale to very large K because the the computation grows with more and more data points okay and the mostly to the approaches so far have been limited to classification approaches because you're inherently


making making decisions based off of the based on the data points that you've seen so far okay and lastly it's worth pointing out that well tune versions of each of these approaches tend to perform compare ibly on existing few shot learning benchmarks so at the performance level they all perform quite well okay so now let's talk about Bayesian meta learning approaches so we


had this really nice motivation for for Bayesian methods and then we kind of threw that all away in the last thirty minutes or so and so and particularly what we did is we assume that this PFI given the data set and the meta parameters is going to be a point estimate or a deterministic function of the training data point and and the meta parameters you might as well maybe this


is all fine like we don't we don't really need Bayesian things but there are many situations where it actually is useful to be representing actually a distribution over the potential functions that we might be encountering and in particular because future learning problems only have a small number of data points they may be ambiguous even with when you do have a


prior for example if your goal is to classify between all the images on the left and all the images on the right all the folks on the left are smiling wearing hats and young all the folks on the right or not and so if you get an image that has someone that smiling and wearing hot and not young or not smiling and wearing a hat and young that it's inherently ambiguous on what the correct


label is for these classes because you don't know if you're supposed to classifying on the attribute of smiling on the attribute of wearing a hat or on the attribute of being young and so what you might ask this can me generate hypotheses about the underlying function I like the three hypotheses that I mentioned before essentially can you sample from this function PFI and this


is gonna be important for for ambiguous problems but also for safety critical few shot learning such as medical imaging for learning to actively learn because if you care about actually reducing your uncertainty about a given function and getting labels for new data points and you want to be able to reason about your uncertainty over your function space right now and it's also and there have


been approaches for using active learning with meta learning and it's also useful for learning to explore and meta reinforcement learning because again we can try to explicitly reduce our uncertainty by collecting more data okay so how do we go about trying to medal learn in a way that allows us to generate hypotheses and reason about distributions so we can bring up the


graphical model that sorry I mentioned before where we're going to have a distribution of our prior parameters theta we will have a distribution over our TAS specific parameters by I given theta and then we'll also have probability distributions correspondent the probability of our training dataset giving or given our toss parameters and our test dataset given our task


parameters and then our goal will be can we sample task specific parameters Phi I given our training data points and ex-house essentially given all of the observed variables and because of conditional independence a test time we we don't really need to worry too much about X test we just want to be able to sample Phi I given x training in a white rain so blackbox adaptation approaches


are quite easy to to extend to this Bayesian setting so what we can do is we can use amortized variational inference we can train a neural network to represent a distribution over sufficient statistics we can also represent a distribution over full parameter vectors Phi I although that might get a bit unwieldy at some point and so we could train a neural network to represent a


Gaussian distribution over H and then feed this H into another neural network and use ideas from from amortized variational inference such as the reprioritization trick to train this model in order to effectively produce this distribution indeed various approaches have used this type of approach for example Gordon at all I use this approach where H correspond to the


the weights of the last layer of a neural network okay so now now that we've gone over how you can a very naturally apply this with blackbox approaches what about optimization based metal learning approaches can we further extend these to the Bayesian setting there's a few different ways that you go about doing this well this is this figure than the simple black box


approach one thing we could do is model the distribution of Phi given theta as Gaussian and use the same sort of variational inference for training as I mentioned before but have her inference that work you perform an optimization over Phi and this is what was done in Ravi and pizza in 2019 another thing that we want to do is if you don't want to model p5 theta I give PFI given theta


as Gaussian we can use stein variational gradient to on the last layer of the neural network and do gradient based inference on the last layer and only have a single set of parameters similar to theta G for all the other layers and also we could use something like an ensemble of mammals ensemble of méditerranée all networks that allows us to represent basically particles of the


distribution and both of these were proposed by Kim at all in 2018 know one thing you might ask is well can we can we get some kind of the benefits of both of these can we model both a non Gaussian posterior and do so over all of the parameters and there is a way that we can do this in fact and so let's go back to our graphical model say that we want to sample from this this


distribution you can write out as the integral where we're integrating out p theta of course this integral is completely intractable if theta are the parameters of a neural network but one thing you might ask is well what if we knew p if i given theta and the training data set if we knew this distribution our graphical model would be transformed in this way switching the arrows from


the training dataset to 5 and then we could simply sample with ancestral sampling we could sample at theta from the prior sample 5 from this distribution and and and get get a get a sampled 5 and so the key idea is what we can do here is we can approximate this distribution Phi given theta and D train as a a point estimate using map of course this is extremely crude


approximation similar to the one that we use in mammal but it's also extremely convenient and in particular if we do this approximation we can still get samples from PFI given the training data and in fact we can to do this approximation we can use the same gradient based map approximation as shown before but we don't require we don't require doing the intractable integral at the top okay so


at test time what this corresponds to is we'll have initial set of parameters theta will add noise to sample will add noise so that that parameter vector to basically sample a new theta and then we will run gradient descent from there so it's essentially a way to make mammal they give you estimates and sample from PFI given your training data but I'm doing so in a way that actually will


lead to the correct posterior distribution I didn't go through actually how we train this that's a bit harder and for details you can look at the probabilistic mammal paper on the bottom we refer to this algorithm as platypus or a probabilistic latent model for incorporating priors and uncertainty in future learning which is a bit of a tortured acronym but we can then get


another type of male and what this gives us is it gives us the ability to sample different functions with only a few data points such as linear functions sinusoid functions but they were inherently ambiguous because there's some noise in the data as well as represent different classification problems for different classifiers with different decision boundaries as again shown in the colored


dashed lines and perhaps most importantly it's also it's better able to model ambiguous future learning problems like I showed on the first slide okay for further reading on Bayesian by learning approaches you can look at the papers shown here and next we can go to a couple questions before moving on to applications so one question is if we want to do a Bayesian


metal learning approach can we just use drop out at a test time right so if you just do drop out at test time you will certainly get a distribution there's no guarantee if you don't train for that distribution to actually represent the posterior then you won't actually get the correct posterior over functions for probabilistic mammal you assume that there's no relationship


between the training X and the training Y the graphical model assumes that the that there is that the labels Y are dependent on the inputs the the labels Y train are dependent on the inputs X train and the the parameter is Phi okay so let's go through a couple applications this will be quite quick meta learning has been used in a variety of computer vision applications such as


image recognition modeling the motion and the pose of humans uses for domain adaptation for example when you want to adapt to new domains and you have a variety of domains to train on as well as for a few shots segmentation problems or you want to segment images given only a few labeled pixels beyond some of those kind of typical supervised computer vision problems people have


also looked at tired of modeling with with meta learning methods this includes a few shot image generation if you shot image the image translation when you where you want to translate between different types of images where you want to translate things to a new type of animal that that you haven't seen before also generation of novel viewpoints given a single viewpoint as well as


generating videos of people from just a single image of that person okay and then another application that people have looked at is imitation learning where the goal is given one demonstration of a task can you learn a policy for that task and this is quite a fairly simple extension of the supervised meta learning approaches that we showed before because you can treat


imitation learning as a supervised learning problem this has been looked at for stocking blocks on top of each other for extended step temporarily extended tasks also for other kind of robotics control tasks in the real world as well as for high fidelity imitation where you want to very closely match the the training demonstration and also it has also been used with optimization based


patient techniques such as it showed here so here the goal is to after seeing a single demonstration of placing an apple into a bowl through teleoperation throughout what is able to figure out how to place the Apple onto the ball in new situations in this case I the the robot had never seen any of these objects before and so it's adapting to being able to manipulate objects in new


objects in in this setting okay and also for more advanced topics you can look at approaches for one time inverse reinforcement learning and one-shot hierarchical imitation learning now one thing you might say is can we take this one step further can we do imitation from a video of a human per se where our goal is given a video of a human performing a task such as placing into


the red bowl can the robot figure out a policy for placing for performing the toss turn in the video such as placing the peach into the red bowl and in this case you need to learn how to learn from weak supervision and what I mean my weak supervision is that the video of the human has all the information about the task but is not accessible in a way of kind of a standard machine learning data


set and so we need to do something a bit different here and in particular what we're going to do is we can take the the mammal objective or really any meta learning objective and the the the meta objective the outer objective is going to be fully supervised and the inner objective is going to be weekly supervised so we're be learning how to learn from weekly supervised data using


fully supervised data and then a test time you'll run gradient descent using the weekly supervised data now you one question you might ask is well what if the weekly supervised loss function isn't available like from a video of a human in that case you could actually learn a loss function for performing adaptation such as I know all Network loss function that outputs a scalar


value and you train this loss function such that the gradients it provides are effective for adaptation okay and then lastly it's worth mentioning that meta learning for languages been explored in a variety of contexts this includes adapting link adapting models to modeling new programs such as program induction and program synthesis this also includes adapting to new languages


such as translating between two language pairs that you only have a small amount of data for by meditating on language pairs that you have a lot of data for this also includes learning new words so this is actually done in the original matching networks paper that learns how to use a new word in the new context from a single example usage of that word and lastly


adapting to new personas so training dialog agents to be able to generate dialogue from a particular persona from only a few examples of that persona okay great so next will you will take a short five-minute break we will be taking questions during the break and then after that Sergei we will talk about meta reinforcement learning so feel free to stand up and and and rest your legs


hello everyone today we're looking at bird pre-training of deep bi-directional transformers for language understanding by Jacob Devlin and then why Chung Kenton Lee Cristina Tata Nova these are people from Google AI language so you're about to see the most hyped model currently so basically Bert is a model that takes as an input language sub token sequences and outputs various

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding


things so it can it can be made to do various things almost any NLP tasks with basically little training because the Bert model comes pre trained on a very large corpus and we're gonna see how that's done all right so the paper introduces basically the kind of current state of the art of language models and they say okay what they want to do new is they want to do


bi-directional training I'm going to go down here and see their comparison right so here they compare three models and these are representative of three types of models so first here is for example the the open AI transformer so this is a this is the classic or one of the classic transformer models we've talked about transformers before and the attention is all you need video so what


a transformer does is it uses attention and if for those who forgot what attention is if you have like a token sequence ABCDE then a classic model to use that would be an LST M so the other stem would go here it would like have a vector representation a hidden state and then it would take this a it would take this hidden state and compute a new hidden state and then it will go on and


take be and incorporate this into the hidden state the hidden state kind of always stays the same size but the the recurrent model will update the hidden state as it goes over the input sequence so this is one way of dealing with language but people have kind of done another way and that's the attention based mechanism is where basically for each of these you compute a compute a


vector independently of each other so each one has a vector representation and then you have a vector representation of what you of what you want which is called an attention head and you can have multiple of these but in the simplest case let's just say we are looking for the subject in this sentence ABCDE is a sentence and one of the words is the subject of the sentence then we


could have a vector here that's called a query rec vector so these are called these are called values V and this is called a query Q and then these vectors are the same size report at this you're gonna compute the inner product with each of these so the inner product you wanna you wanna do okay I already screwed this up you're actually computing two vectors for each token so honesty but this is


this is not too important for this step one is the key and one is the value all right value and this is called the key and you have your query Q and you compute the inner products actually with the key sorry values aren't too important for what I want to demonstrate but you compute key with query all right and that gives you basically for each key it's gonna give you an output and so


you're gonna have you're gonna have a for this ABCDE you're gonna have like this much inner product this much inner product this much this much this much inner product so after maybe the softmax you have like a nice distribution and then you can say AHA here this is the the biggest the biggest alignment of the particular key with my query and that query is which one is the subject of


course you're gonna train all these queries and keys producing procedures so this is this is a tension mechanism and if you then want that that's where the value comes in you can if if your queries not only which ones the subject but it's actually generically read that okay I'm gonna extract some information from some token that I'm going to use later then you would actually take B say


aa B is the best one okay I'm gonna take the value of B you're basically gonna get a cab it a weighted average of the values according to these values here right so this is very shortly what attention is if you if you want a lengthy explanation go to the attention is all you need video right so open that I cheap T uses attention here and it's a it's a left to right transformer that's


what it says here and what that means is because also step-by-step but in each step it uses attention so here is the input tokens and as you can see it goes in this direction so each one of these and these are multiple layers of attention so you can also layer these of course so each one of the attention intermediate steps can only attend to whatever is to the left of it right you


can see this here so it goes step by step and it goes left to right basically so it can it can kind of take the sequence in as a left to right input basically what that means is whenever you interpret a particular token you can your context is only to the left of that token you don't know what's coming in it's like what do you when you read a sentence from left to right but then as humans


unconsciously we probably go and at the end of the sentence kind of make sense of the thing as a whole but here the model is forced to make sense of what the thing only from whatever is to the left of it so that's a basic limitation of these left to right models then there is another approach which is called Elmo which has been popular recently as a substitute for word vectors so if you


know word vectors were vectors are basically the kind of first stage in most language processing tasks where for each word say the cat sat something for each word you have you had a big giant table and for each word you associate a vector of fixed size dimension right so you place every word in a vector space and these vectors you pre compute for something like word Tyvek or glove and


that gives you a nice way to basically deal with these words in a canonical way and you can pre train the word vectors that's all really nice but people have realized ok words kind of multiple meanings and words can kind of slightly change meaning depending on words around them and so on so what Elmo does is Elmo uses to Alice Tian's one Elliston goes into this direction


when I Austin goes into this direction and basically a single lsdm as we saw the four it takes in the input sequence one by one so here e1 and e2 if the en it produces hidden states at each step produces a hidden state that is a result of previous hidden state and the current token and and then what it says is ok now these hidden States here basically these are now the embeddings


of the token here on e and so on right these are the embeddings so the world vectors as to say Oran are no longer just one vector per word so they're not in isolation anymore but basically you need the entire sequence to compute the word vectors as a result of this of this Alice T and this is more powerful because it can give individual words multiple or each it basically each word


has kind of a unique and then depending on the surrounding words you would still hope that a given word would have similar similar embedding or similar word vector all across the language but you can kind of fine tune it to the particular sentence it is in and also you can completely change its meaning if it's if it's kind of a word that has a completely new meaning in that sentence


so basically uses two LST M's one as as I said here forward one backward these also have multiple layers and so and each of these produce one such hidden vector per token and he would simply concatenate the two from the from so from here is this leather stain on the left produces one disaster the right produces maybe here another one and you simply concatenate the two to get the


the final embedding the final word vector for each token so the fundamental limitation here is that this is kind of you have information from the left end you have entered information from the right so other than here the original transformer you actually have you actually can condition on the left context and the right context but it's very it's it's very shallow because it's simply a


concatenation of the left facing Alice TM and the concatenation of the right facing Alice TM and and these ultimately intrinsically they have nothing to do with each other they so you simply concatenate the two things that the left facing Ostiense still can only see to the left and the right facing Alice team still can only see to the right so you basically have to