have big inputs or big outputs they actually do use a smaller m what you can see right here is that i wasn't really that wasn't really correct of me to say that it's now o of m because you you still have the entire query space so you multiply query by by keys now even if you make the keys to be 1 by m yes you reduce definitely you reduce this from height times width to times height times width to this but then you can see on this thing right here if you take it and let's say we have this kind of row pattern and we replace m


by the width then we have width squared so again the square appears however it's smaller than the original attention the original attention was h squared w squared right because hw is the image and you need that squared in order to do the attention mechanism now we've basically reduced one of the factors it is still an attention mechanism so there's still a tension going but we've basically transformed the the image we've reduced it to one column now the one column is still a tension so this is still a tension like here so this

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (Paper Explained)


now reduces to the attention that you see in a in a single sequence okay if you see the image as a long stretch of pixels what this does is basically it's up it simply subdivides that into neighborhoods so we're back to neighborhoods basically um but we shift the neighborhoods from layer to layer so in the next layer the neighborhoods are going to be just alternating right the neighborhoods is going to be this is one neighborhood connected to this neighborhood connected to this neighborhood i hope this makes sense


so it's going to be it's basically a mix between if you if you were to do this in convolution you could do one layer where it's neighborhood convolution and then one layer where it's like convolution with holes in it i think they're called actress convolutions or something like this with like giant holes in it that are exact is exactly the anti-pattern of the neighborhood convolution from before that's what this is so you see their axial attention block right here their axial attention block replaces the resnet block so if you know


resnet i've done a paper on resnet resnet basically takes the input pipes it through straight and adds to it whatever comes out of this operation okay that's a residual block now usually this thing here would be convolutions and convolutions and they are now replaced by these multi-head axial attention you can see there is a multi-headed tension in the height and there is a multi-head attention in the width and that gives us the property that every node can send around information to every other node in two steps


i don't like the fact that there is only two because um well this i guess this gives a significant bias to one or the other direction depending on the order that you do them in if if i had done this i maybe would have used three of them because it depends on how you want to aggregate information right like here you train the network specifically to aggregate information first in this direction and then in this direction which might work and it will give you that sending around information anywhere so maybe they've actually tried and it


it just performed the same so i i just might have a dumb suggestion right here in any case they simply replace in we've come a long way right we've gone to like neighborhoods and blah blah blah ultimately take a res net replace the convolutions with the height axis attention and the width access attention and we're good and then we come to results so that's it you have these positional embeddings you have the axial attention and it turns out that on imagenet they perform fairly fairly well so you can see that models


like a resnet 50 model will get a 76.9 on imagenet which is not state of the art but it's also not it's not bad right the resnet 50 is pretty good model um you can see the full axial attention right here uh achieves a 78.1 also not state-of-the-art but still pretty good and as they say it's the best fully attentional model on imagenet as our standalone attention model on imagenet so where this model really shines is where you really have to make long-range connections between pixels and that's these kind of segmentation


tasks and i want to skip the tables right here yeah their best and everything and go to the appendix where they have some examples of this so here you can see specifically uh this is the original image you have a ground truth and you have the differences between their model this axial deep lab and the panoptic deep lab um that is a baseline for them and you can see that the the failure cases here are are pretty you know show how show how the axial deep lab is better i don't know if they are cherry picked or not but


at least you can see that at some point so it handles occlusions better it handles instances better so here you see that the ground truth separates the person from the tie and the axial attention is able to do this but the the baseline is not able to do this correctly because it labels part of that white shirt also as and you can see why there's kind of a delimiter line here here here here but if you have long range dependencies right if you have long range dependencies in the model the model will recognize wait wait that's that must be the same


thing as this thing here and this thing here and this thing here so that must be the same object um it's simply that the shirt was occluded by the tie and goes beneath it and now appears again it's not a different it's not part of the tie and it's not part of the um of a different object it's actually part of the shirt so the long range attention you can see at these examples sometimes here okay this might not be an instance of super duper long range dependencies this is simply where the model performs better so you can see


here the ground roof has that surfboard segmented and the baseline does not um that this can also just be you know there are a lot of tricks to make this work of course and you throw a lot of compute at it and sometimes you just get better numbers or part of the better numbers because of the additional compute right here what do we have so you can see occlusions it appears to handle occlusions uh in a better way and this might be due to this axial attention it might be due to the positional embeddings but you can see that the


ground truth here has the laptop between the person's hands segmented the baseline cannot do that but the axial tension does do that and i don't know what this is honestly this is um uh you can you can see though the axial attention also misses the fact that it should segment this in the background and if this occlusion handling you can see best in this example where the person in the back reappears on both sides of that person so you can see that the axial attention manages to segment that where that is just a mutant person right


here though the ground truth is equally shaky i think there is might be some ambiguity of how you can segment these images obviously but you can see the fact that there are long range dependencies probably helped with this saying that wait in this image there's this white stuff right here and there's this white stuff right here and um connecting these two regions with attention probably helped in segmenting uh these to be the same object even though you can see there is a break in the object so there is a break


no at no point is the object on the left uh touching or the segment on the left touching the segment on the right and still the model manages to put those into the same label category there is the last um last thing where they they want to research what their heads learn and usually you can do this right you can kind of visualize what the attention has learned so in this case right here in the column heads the way you have to read this is that this particular head right here um aggregates information from its column so everywhere where it lights


up it there's a lot of information being routed you can see specifically in this here uh the heads of the people or the heads of the persons in the picture light up fairly well so for example this head right here is probably aggregating information a lot from this position right here and this head here is aggregating information from this position so you can deduce that that particular attention head probably deals with people's faces uh whereas that particular attention head probably deals you can see the attention is mostly on the grass right


here and you can see the same with the for the row heads now their description here is that we notice that column head one corresponds to human heads while calm at four course correlates with the field only which you know you can interpret it as this this seemed pretty clear but then they say something like row head six focuses on relatively large relatively local regions where column head five pools all over the image so row head six which is this thing right here you can see that okay it maybe focuses on small regions


though you can see okay what like here you can get it that's a person but in other places um i don't know where column head five pools over the whole image and this i don't know maybe they just needed something more to say because they put these pictures here they were like oh okay the the column heads are really nice because we couldn't like these this one's really nice because it you know just pays attention to the people and this one looks really nice because it pays attention to the field but we can't really put the column head


attention without putting the row head attention but then none of the row heads really are like super distinctive on the particular thing in the image so we need to come up with something that we can say and then he's like ah this one this is there's not a lot of attention so we need to contrast this with something then you would think that they contrast it with another row head but then there's no row head that does this whole image so there's like ah column at five yeah i'm i'm not sure if there's there's a


bit of there's a bit of uh tactical writing going on here i suspect i mean it's still you know it's doing something uh cool but yeah there's there's a definitely an element of sales in when you do when you write research papers and just um not to this data but just props to the lines in front of the histograms makes it so much easier to read how big the stupid bars are why does everyone put the lines behind the histogram i probably do that myself and now i'm just i'm realizing how much easier that is all right there is a big


big big experimental section right here and there's a big appendix where you can read up all of the different numbers comparisons ablations what not um ultimately i just wanted to go over the method basically putting this into context with other things like putting this into context with stuff like big bird axial attention other positional encodings uh how it co how it relates to convolutions how it relates to feed forward networks and what convolutions did to feed forward networks and so on i hope you at least


Thailand GNN learning to tile with self supervised graph neural network many problems in computer graphics face combinatorial optimizations which are typically solved by approximation algorithms are heuristic search methods in this work we explore whether a learning-based approach can solve a classical combinatorial geometric problem tiling our problem specifically we focus on tiling the interior of an arbitrary 2d shape using a given tile set while avoiding holes in tile overlaps our trained network can help produce

TilinGNN: Learning to Tile with Self-Supervised Graph Neural Network (SIGGRAPH 2020)


tilings in time roughly linear to the number of candidate tile locations significantly outperforming traditional combinatorial search our learned to tile approach given an input tileset we first enumerate candidate tile locations then we generate random shapes to crop and locate candidate tile locations after that we create a graph to describe each tile placement and train our network to predict tile placements with our self supervised loss at test time we applied the Train Network to predict tile locations for


arbitrary shapes where we first locate tile placements and progressively fill the shape with the health of our network overall this work has three technical contributions first we model this tiling problem as an instance of graph learning second we design a graph convolutional neural network to predict tile placement to be a graph convolution here we call our network tile and GNN third we define loss terms directly on the network output so Thailand GNN can be trained with self supervision interactive design interface we provide


everything we as human beings have created on this planet was essentially first created in our minds all that you see which is human work on this planet first found expression in the mind then it got manifested in the outside world so one thing we need to understand is the wonderful things that we have done on this planet and the horrible things that we have done on this planet both have come from the human mind so if we are concerned as to what we create in this world it's extremely important that first of all we learn to create the right things

Law of Attraction simplified by Sadhguru


in our mind how we keep our minds if we do not have the power to keep our minds the way we want it what we create in the world is also going to be very accidental and haphazard so learning to create our minds the way we want is the basis of creating the world the way we want there is a wonderful story in the yogic lore on a certain day a man took a walk he went for a long walk accidentally unaware he walked into paradise fortunate isn't he he just took a walk and he landed up in paradise after this long walk he felt


a little tired so he thought oh i'm tired i wish i could rest somewhere he looked around there there was a nice tree underneath which there was very cushiony grass so it was inviting he went and put his head down there and slept after a few hours he woke up well rested and he thought oh i'm well rested but i'm feeling hungry i wish i had something to eat and he thought about all the nice things that he ever wanted to eat in his life and instantly all those things appeared in front of him you need to understand there are the


services like that hungry people don't ask questions food came and he ate stomach became full then he thought oh my stomach is full i wish i had something to drink all the nice things that he ever wanted to drink he thought about it and all of them just appeared in front of him drinking people also don't ask questions so he drank now with a little bit of alcohol in him you know charles darwin told you all of you were monkeys your tail fell away not me charles darwin told you that you were all monkeys and your tail fell away and then you


became human yes definitely the tail fell away but the monkey in yoga we always refer to an unestablished mind as murkata which means a monkey why we are referring to the mind as a monkey is what are the qualities of a monkey one thing about a monkey is it's unnecessary moment and another thing about the monkey is if i say you're monkeying somebody what does it mean imitation monkey and imitation have become synonymous so these two essential qualities of a monkey are very much the qualities of an unestablished mind


unnecessary moment you don't have to learn it from the monkey you can teach it to the monkey and imitation is full-time job of the mind so when these two qualities are on a mind is referred to as a monkey so this monkey became active within him he just looked around thought what the hell is happening here i asked for food food came i asked for drink drink came there must be ghosts around here and ghosts came oh the ghosts have come they're going to surround me and torture me he thought immediately the ghost surrounded him and started torturing him


then he started screaming in pain and said oh they're going to kill me and he died just now he said he's a fortunate being the problem is he was sitting under a kalpaviksa or a wishing tree he asked for food food came he asked for drink drink came he asked for ghost ghost came he asked for torture torture came he asked for death death happened now don't go looking for these kalpa rukshas in the forest you can barely find a tree these days a well established mind a mind which is in a state of samyukti is referred to as a culpa victim


if you organize your mind to a certain level of organization it in turn organizes the whole system your body your emotion your energies everything gets organized in the direction once all these four dimensions of you your physical body your mind your emotion and the fundamental life energies are organized in one direction once you are like this anything that you wish happens without even lifting a little finger actually it would help to assist it with activity but even without doing any activity you can still manifest what you want


if you organize these four dimensions in one direction and keep it unwavering in that direction for a certain period of time right now the problem with your mind is every moment it is changing its direction it is like you want to travel somewhere and every two steps if you keep changing your direction the question of you reaching the destination is very remote unless it happens by chance so organizing our minds and in turn organizing the whole system and these four basic dimensions of who you are right now in one direction if you do this you are


a kalpa ruksha yourself anything that you wish will happen but right now if you look at your lives everything that you have wished for till now if it happens you're finished everything and everybody that you have desired for if all of that lands up in your house today could you live with that so if you want to become empowered it is also important that you become responsible as to what you ask for and what you don't right now the world situation is just this we are hugely empowered with technology today it doesn't take six six billion


people to destroy this planet one man by pressing the wrong button can destroy the whole planet when we are empowered like this it's very important that our physical action emotional action mental action and energy actions are controlled and properly directed if it is not so we become destructive self-destructive right now that is our problem the technology which is supposed to make our life beautiful and easy has become the source of all the problem that we are destroying the very basis of our life which is the planet


so what should have been a boon we are making a curse out of it what has brought incredible levels of comfort and convenience to us in the last 100 years or so has also become a threat to our life simply because we are not conscious action we are in a compulsive state of action so organizing our minds fundamentally means moving from a compulsive state of activity to a conscious state of activity you might have heard of people for whom they asked for something and beyond all expectations it came true to true for them generally this happens to


people who are in faith now let's say you want to build a house if you start thinking oh i want to build a house to build a house i need 50 lakhs but i have only 50 rupees in my pocket not possible not possible not possible the moment you say not possible you are also saying i don't want it so on one level you're creating a desire that you want something on another level you are saying i don't want it so in this conflict it may not happen someone who has some faith in a god or in a temple or whatever who is how simple-minded faith works


only for those people who are simple-minded thinking people people who are too much thinking for them it never works a child-like person who has a simple faith in his god or his temple or whatever he goes to the temple and says shiva i want a house i don't know how you must make it for me now in his mind there are no negative thoughts will it happen will it not happen is it possible is it not possible these things are completely removed by the simple act of faith now he believes shiva will do it for him and it will happen


so is shiva going to come and build your house no i want you to understand god will not lift his little finger for you what you refer to as god is a source of creation as a creator he has done a phenomenal job there's no question about it could you think of a better creation than this is it in anybody's imagination to think anything better than what is there right now so as a creator he has done his job wonderfully well but if you want life to happen the way you want it because right now the very crux of your happiness and your


well-being is this if at all if you're unhappy the only and only reason why you're unhappy is life is not happening the way you think it should happen that's all it is so if life is not happening the way you think it should happen you're unhappy if life happens the way you think it should happen you are happy it's as simple as that so if life has to happen the way you think it should happen first of all how you think with how much focus you think how much stability is there in your thought and how much reverberance is there in the


thought process will determine whether your thought will become a reality or is it just an empty thought or how you do not create any impediments for your thought by creating negative thought process is something possible or not possible is destroying humanity what is possible and not possible is not your business it's nature's business your business is just to strive but what you want right now you're sitting here if i ask you two simple questions i want you to just look at this and answer this right now from where you're


sitting can you just fly off you say no right now from where you're sitting can you get up and walk you'll say yes what is the basis of this why you say no to flying and yes to walking because past experience of life many times you've gotten up and walked never did you fly off or in other words you're using the past experience of life as a basis for deciding whether something is possible or not possible or in other words you have decided that what has not happened till now cannot happen in your life in future


this is a disgrace to humanity and the human spirit what has not happened till now on this planet can happen tomorrow human beings are capable of making it happen tomorrow so what is possible and what is not possible is not your business that is nature's business nature will decide that you just see what is it that you really want and strive for that and if your thought is created in a powerful way without any negativity without any negative thoughts bringing down the intensity of the thought process it will definitely manifest the whole


existence today modern science is proving is just a reverberation of energy it is a vibration similarly your thought is also a vibration if you generate a powerful thought and let it out it will always manifest itself so generally people are using faith as a means to remove the negative thought today once you have become thinking human beings your faith is not too deep it doesn't matter how much faith you think you have somewhere doubts always crop up right now the way your minds are made this moment if god appears right here


you will not surrender to him you will want an investigation whether he is really god or not with this kind of mind you should not waste your time on faith so there is an alternative which is commitment if you simply commit yourself to creating what you really care for now once again your thought gets organized in such a way there is no such thing as whether it's possible or not possible there is no hurdle in your thought process your thought flows freely towards what you want once this happens making it happen will


also naturally follow so to create what you really care for first and foremost thing is that what you want must be well manifested in your mind that this is what i want is that what you really want you must look at it because any number of things in your life you have thought this is it the moment you reach there you realize that's not it it's the next one and the next one and the next one so what is it that one really wants is one thing first of all we must explore once that is clear and we are committed to creating it now


there is a continuous process of thought in that direction once you can maintain a steady stream of thought without changing direction definitely this is going to happen in your life or it will definitely manifest as a reality in your life so either you make this human form into your kalpa riksha or you make it into one big mess which is happening all over one reason why we have not created the kind of world that all of us would want to live in is [Music] too many people are busy looking up too many people are interested in other


planets they are not interested in this planet but every aspect of their life they are checking out with other planets and further too many people are in service of heaven they are not on service of this earth i think right now we need people who are in service of this earth not heaven i believe if heaven is closer to divine than where we are if that is so i believe they are little more organized than us they don't need help from us this has been a major problem on the planet anything that is valuable for a human being


anything that is good the highest aspects of human life have unfortunately been exported to heaven for example if you say love people say god is loving we do not know whether god is loving or not human beings are capable of love it is very very important that people understand human beings are capable of love human beings are capable of compassion human beings are capable of joy and blissfulness all the good things that are possible for a human being unfortunately have been exported to heaven if we want to create the kind of world


that we want to we need to understand whatever you refer to as god the idea of god has entered our mind only because we have seen creation around us because there is creation we have assumed a creator god is a great creator what you refer to as god is a source of creation that source of creation has not failed us has done a fantastic job but now the question is about management if you want to leave the management in the hands of the creator he will manage it in his own way according to his agenda but that's not


what you want you want life to happen the way you want it now for example let's say all of you here are the india soccer team for the next world cup and i am the coach so these next four years everything that you need to know about football is taught to you everything that i know about football is poured into you in so many ways now the time to play the match has come you're on the field and the ball comes near your foot but you look at me and it's no good you've seen those coaches sitting there and boiling nothing happens


because now once you're on the field it's your job so this is the same thing the creator has done a fantastic job now you're here it is for you and me to see how to manage this how we want it how to keep this world how in what condition would all of us enjoy best is something that we have to look at so at every stage in our life we tend to think this is it if this one thing happens everything will be fine with my life you reach there and you realize that's not it and you postpone it to something else and something else this is going on


the first and foremost thing is you must be clear what is it that you really want if you do not know what you want the question of creating it doesn't arise if you look at what you really want what every human being wants is he wants to live joyfully he wants to live peacefully in terms of his relationships he wants it to be loving and affectionate or in other words all that any human being is seeking for is pleasantness within himself pleasantness around him if this pleasantness if it happens in our body we call this health and pleasure if it


happens in our mind we call this peace and joy if it happens in our emotion we call this love and compassion if it happens in our energy we call this blissfulness and ecstasy this is all that a human being is looking for whether he is going to his office to work he wants to make money build a career build a family he sits in the bar sits in the temple he's still looking for the same thing pleasantness within pleasantness around if this is what we want to create i think it's time we addressed it directly and commit ourselves to


creating it so you want to create yourself as a peaceful human being joyful human being loving human being a pleasant human being on all levels and do you also want a world like this a peaceful world a loving world a joyful world no no i want greenery i want food when we say a joyful world that means everything that you want has happened so this is all that you're looking for so all that you need to do is commit yourself to creating it to create a peaceful joyful and loving world both for yourself and everybody around


you every day in the morning if you start your day with this simple thought in your mind that today wherever i go i will create a peaceful loving and joyful world if you fall down 100 times in the day what does it matter for a committed man there is no such thing as failure if you fall down 100 times 100 lessons to be learned if you commit yourself like this to creating what you really care for now your mind gets organized once your mind gets organized the way you think is the way you feel your emotion will get organized once


your thought and emotion is organized your energies will get organized in the same direction once your thought emotion and energies are organized your very body will get organized once all these four are organized in one direction your ability to create and manifest what you want is phenomenal you are the creator in many ways why i am saying you are the creator is i want you to look at the nature of your life right now if you eat a banana in four hours time this banana becomes a human being there is something within you a life


creating process a process which builds this body the manufacturer of this body is within you give him a banana he makes a human being out of that banana transforming a banana into a human being is not a small thing it is a phenomena it is just that this phenomena is happening within you unconsciously if you could only consciously manifest this making a banana into a human being you are the creator you are nothing less than that as the theory of evolution goes to make a monkey into a human being it took millions of years over an


afternoon you can make a banana into your human being or whatever else a piece of bread that you eat into a human being so the very source of creation is functioning within you if you organize these four dimensions of mind emotion body and energy in one direction the source of creation is with you you are the creator what you want to create will happen to you effortlessly once you're organized like this now you are not a mess you are a culprit you have the power to create what you want there are tools and technologies as to how to


organize the system in such a way that instead of being a psychological mess you can make yourself into your kalpa ruksha this culture these traditions the whole technology of yoga is just about this transforming yourself from being just a piece of creation to the creator himself this is not in search of god this is in search of becoming a god this is not in search of divine this is in search of becoming divine because that which you call as divine that which is the source of existence is throbbing within you every moment of


your life otherwise a piece of bread cannot become a human being in the course of an afternoon so shifting from being just a piece of flesh and blood to becoming a creator there is a whole science and technology for this there are tools to make this happen that which is the source of creation is functioning within you every moment of your life it is just that have you kept access to that dimension or not organizing the four basic elements of your life will give you that excess there are tools and technologies to do this


the whole signs of yoga the whole technology that we refer to as yoga is just about this transforming yourself from being just a piece of creation to become a creator for example 100 years ago if i picked up something like this and i start speaking speaking to someone who is across in another part of the world you would think it's some kind of a miracle either i must be a messenger or a son or maybe god himself but today this is just another gadget that every one of us carry and use today sitting here without


using this instrument if i speak to some someone in another part of the world it is still a miracle for you so this instrument happened or because of human mind wanting it to happen 100 years ago nobody thought this was possible but today it is just a common thing similarly many many many things which are not in our perception yet can be brought into our perception and our ability to create our lives can be greatly enhanced so first and foremost thing is to organize the mind and to organize your emotions body and


energy in that line once this happens you're in touch with the fundamental life creating process within you once you're in touch with it once you're in excess of that power you have the power to create you have the power to create your life and your surroundings the way you want it because we have lost our power to create we are making a mess out of ourselves and the world around us if we operated as the true creator as it is operating within us creating this body for us if we could create our lives with the


all right today we're going to find out why ai is much better at governing people why poor people really should pay more taxes and how donald trump is just a normal human all right we'll dive into it we're looking at the ai economist by salesforce research now salesforce research has kind of created a simulated world environment where they can place agents in it and the agents they can move around they can collect resources they can trade those resources and they can use those resources to build houses and that will earn them coins

The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies (Paper Explained)


and each agent wants to maximize its own coins but also there's the government and the government can set taxes so they collect money from everyone and they redistribute it and the goal now is to going to be that the ai handles both the agent and the taxes and we want to maximize the social welfare of the entire population all right that's the goal so the paper here is called the ai economist improving equality and productivity with ai driven tax policies by stefan cheng and alexander trott and other people from salesforce


research and harvard university so as i said this is a simulated environment and the simulated environment works like this there is a 2d plane kind of like a game playing field and in this game there are agents here you can see the agents there are always four agents where oh down here what are you what are you doing in the corner come on be productive um the the the agents are in this world and they can do certain things they have certain actions at their disposal so first of all they can move around they can move down left right and so on


whenever they walk past a resource tile they collect the resource this is stone and this is wood so there are two kinds of resources and then the last actions the agents have is building a house one wood and one stone will create one house and the house gives you coins so this is a house and that will give you coins but how much coins you get is different from agent to agent and this represents the agent's different skill levels this is an abstraction and the kind of economic theory behind it is that the income inequality in people


one of the main drivers of it is that they are skilled differently and therefore are able to they are able to convert one unit of labor into more money than another lower skilled worker so this is here represented by the fact that maybe if this agent here builds the house they'll get 50 coins but if this agent here would build the same house they'll only get 10 coins so we'll call this here a high skilled worker and this here a low skilled worker now the last thing sorry i thought last thing before but the very last thing the


agents can do is they can trade so if one agent has too many resources and the other one has not enough they can trade those resources among each other for those coins so once you build a house you collect some coins you can then either go and collect more resources or you can use those coins in order to buy resources off of other people this guy this is unlucky no coins no houses and no resources look at them oh yeah so you also can't move across the water here um you can only move on the grass you can also not


move through a house which gives you some interesting abilities because you can just build a house right here and um yes so and you can't move over other players but these are so the rules are pretty simple and the goal here is for the agents to maximize the number of coins they get in a thousand steps so the number h here is one thousand which is the number of steps that the agents can take before the game is over and it restarts again so each agent is using reinforcement learning in order to learn how to achieve the maximum number of


coins now the policies of course going to be different depending on whether that is a high or a low skilled worker the catch here is that outside of this there is the government the government here let's draw this big house with the flag of our fictitious nation which is like this that's the flag and the government will observe what's happening here and they will issue a tax um taxes so it will issue a tax distribution now how do you imagine that so if you imagine the government says something like this for the first ten coins you own


you owe us five percent of that um for the next 10 coins so from 10 to 20 you earn you owe us 10 and so on so if you earn even more you owe us more and more percent of those extra coins this is what you might know as a progressive tax schedule the more you earn the more percentage-wise you pay on that extra earned money this is what you might be used to but there are other tax schedules and the exact histogram you see or the exact how many percent for which amount of coins that is the action of the government so


the government decides on the taxes and the taxes are just collected from the income so if you if an agent earns these coins then it has to pay taxes to the government and the government will redistribute all the taxes it has collected equally among the population so if you pay a lot you might lose through this process and if you just pay a little taxes you might gain through this process so that's it that is the basic premise of the game the agents are using reinforcement learning and i believe the newness of this paper is also that the


government now is using reinforcement learning in order to determine the optimal tax policy there is kind of this inner loop here and there is this outer game where the government also tries to maximize the rl and what does the government try to maximize good question it is a measure that's called social welfare now social welfare consists of two things and they have this here way down in the paper social welfare in this paper consists of two things first of all economic productivity which basically just means


how many coins have has anyone produced it doesn't matter who but just the total amount of coins produced the second one is income equality and this is related to the the genie index so if you plot the cumulative distribution of wealth a fully equal society would be a straight line because 50 percent of the people would have 50 of the money and so on but a almost all true societies have something like this where fifty percent of the people might have ten percent of the money and the rest fifty percent of the people has the


other ninety percent and the measure of inequality is this area here um this is called the genie index and one minus this area is what this paper has as an equality measure so the higher this number the more equal is the society in terms of their income distribution now what is actually optimized for here is this thing equality times productivity so you want both to be high your income equality and your productivity there's a trade-off here of course but you can you can have multiple ways to trade that off and that will give you


the different uh thing they call this the social welfare function and that's the thing that the government rl agent optimizes for so you can see here already the free market even though it's the most productive produces the most coins because if you haven't free market means no taxes if you have no taxes then people are basically encouraged to uh earn more money because they don't have to pay taxes on them right as soon as you tax them they're less encouraged to earn more money and therefore if you have no taxes the


most coins will be earned in total but the equality suffers so the equality is the lowest among these things considered if you compare that to the ai economist the ai economist achieves the highest social welfare it achieves the highest equality but it doesn't suffer as much in productivity as other systems here and the baseline systems are first of all the u.s federal system this is not particularly tight to the u.s this is basically every system uh or most of the systems that you have currently in the world


is the progressive tax system and the size formula which i believe is an economically theory-based system which is a regressive tax schedule you can see them down here where the u.s federal will be progressive means the more you earn the more percentage-wise you pay while the says formula will be regressive which generally means the more you earn the less you pay i believe this was derived under some assumptions to be the optimal tax distribution and the ai economist will come will come to will come to this in in a second


let's actually just look at one of these things first one of these games how this plays out the cool thing here is that they have pretty flashy animations so you can look how does one of these games turn out now this is a free market game and you can see the agents moving around collecting things building houses and you might notice that one of the agents namely agent one is just building all of the houses and generally just kind of being a dick being in everyone's face and kind of building things everywhere


and the other ones don't and or or just very few like the light blue on the on the bottom left build some houses on the right you can see how the distribution of wealth is is structured and you see agent one ends up with most of the wealth now the size of the circle i think is the total productivity so you can see this grows over time mainly because agent one becomes so rich and if you analyze this if you analyze what's happening here then you'll see that agent one and i might be yeah they have a graph up here so so it


is very interesting what happens this is kind of the same game so agent one here is this orange dot and agents two three and four are these dots here and this graph here is coin from trading so how much money they win or lose from trading now you the green bars are trading wood and the the brown bars are trading stone so you see agent number four which is the lowest skilled um the skill is just determined at the beginning of the episode it will just make all of its coins basically by selling wood and agent 3 will make all of its


coins by selling stone and agent 2 will collect both and sell both and agent one will just spend money in trading so you'll have a specialization here agent one which is the highest skill one right here will buy resources in order to build more houses because it clearly profits from building lots and lots and lots and lots of houses so it will use that money to buy more resources rather than go in collecting them while all the other ones basically forego building houses in favor of they just collect the resources and they just trade them way


to the agent one that's more profitable for them than building houses themselves so you see this kind of specialization emerging in these games which i find i find this to be pretty cool that you see something like this like a really stark division of labor emerging just from these very very uh small set of rules and you can analyze this game in different ways they have a few more plots where this becomes quite apparent that um sorry that that these agents specialize so you see here resources collected sorry about that resources collected


uh if you have the lowest skill and the highest skill labors the reas the lowest skills they mainly about this this should be a pen they mainly collect resources while the highest skill labor mainly goes for building things it doesn't collect resources but net income from building is really high while everyone else just doesn't build at all all right so we have a division of labor emerging now this was a free market let's actually compare the different algorithms so if you look at social welfare this is this thing


here equality times productivity you can see that the ai economist will outperform over time over the training progress it will outperform all of the other systems so it will outperform the free market the u.s federal tax system and the sas formula um if trained for long enough which is to be expected right if you put rl onto a cost function it will then optimize that cost function but it's pretty cool to see that it had there's a lot of lot of headroom here over what we currently have now let's look at some of these


strategies it comes up with so what do these games look like where the ai has imposed different tax strategies so this is with the size strategy you see that here again you you see this inequality emerging with the yellow player here building most of the houses with the ai economist again there is inequality but you can see at the distribution that agent one only ends up with about half of the wealth where if you compare this to the free market here then agent one ends up with like two-thirds of the wealth right this


is the game we saw before um but there is not qualitatively that much of a difference uh but there is in the end result all right let's look at what the these policies actually come up with so what is the tax policy that the ai comes up with so this tax policy outperforms on this social welfare metric and this is very interesting right so first of all you see that it's right zigzag it's like down up down up uh which is already weird so the first very weird thing is the the spike at the very bottom so that thing here what's that thing


here those are the poorest people in your society and you're taxing them the highest right so just imagine this you're here uh downtrodden by life abandoned by society you have no money no house no nothing and you're just trying to get a job you're just getting like a little bit of money and you can buy a cheeseburger and then the government comes give us that us that money come on so basically this these are the poor and the poor in this system is just fu fu the poor now the reason why this happens is pretty clear


right the reason why this happens is because you want to encourage people to go here to earn more money right so so it's not like the government makes any money from the poor people independently of how it how high it taxes them but it is a basically an incentive structure to make them move over to the somewhat more productive population because here it's assumed kinda that even the lowest skilled ones can move over a bit if you just tax them enough at the low brackets right so um this this is what i find to be


you just have to realize that it is so hard i believe it is almost impossible to encapsulate what we really want in a system into a formula to be into a cost function to be optimized it is so incredibly hard and you see that here of course it is going to result in a better social outcome but it just doesn't feel right to tax the poor at what 60 okay so f the poor right and then you get to to this to this level right here and interestingly if you earn even more you'll be taxed high again right so this this um


this we're kind of used to that you earn little you pay little you earn more you er you pay more but then comes this entire valley here what's up with that right like wtf doesn't matter and this can be this this is now of course the same reasoning as you have with this science formula here is where the rich people you want to tax them less so that they are more productive such that they generate more coins and even though you tax them less percentage-wise they will end up paying more uh money in absolute terms because


because you basically encourage them to produce more so that is that is can that is the i guess the reasoning behind this but what you have to re you have to recognize what's happening here right what are we optimizing we're optimizing this productivity times equality right and what do we get you see you get two big values of attraction one here and one here and that means that this algorithm favors a two-class society right and i believe this is this is partially the limitations of this simulation here the fact that you're only a f4 agent the


fact that you can only do two things either collect or build right it encourages a two-class society this specialization that you saw right so you say these here are the money makers right and these here are the collectors and it is very hard to move from one group to the other because if you you earn more coins as a collector you're here and you're really discouraged here if you move there you want to move all the way over here right now the people that are are already over here if they earn an extra coin that doesn't bother them too much so


they're very encouraged to earn more money but the very the poorer people on this side they're basically discouraged from earning more money because the system needs them to stay at that collector level right so the system encourages the two-class society because we have not built social mobility into the into the into the equation we have not built a measure for social social mobility into the cost function and therefore the ai doesn't care that the poor people will stay poor and the rich people will stay rich


uh it just knows that this is the best outcome for society overall given the cost function that we had again this just doesn't seem like fair to us like what we want we want someone to be able to make it over here right even if they start out from the bottom and so we'd have to we have to build that in so we have a system that is effing f the poor right no social mobility mobility no and then look at what happening at the end what's happening at the end this is beautiful very rich people these are the money maker right this is


the this is the monopoly guy top hat monocle wearing scrooge mcduck bathing in coins this is where the the government makes their money and um the discrepancy is really stunning because you could also argue hey why don't we apply the same reasoning as we applied here and here right it's not is it not like the case that if the rich people if if you tax them lower they'll pay more money and so on i believe again this might be just a result of this how the simulation is set up so we'll move away quickly and we'll come


back to this here is what i find particularly interesting about this paper which just confuses the heck out of me it is a double periodic game so it's an inner outer loop game what do i mean by that they have these episodes right here is the start and here is the end and they subdivide this into as we said 1 000 steps so an agent is here and they can do step step step step step and it can perform these actions this is the agent there are 1 000 steps here and the agent just tries to collect as much coin so this is your classic rl


problem but also they divide this into 10 what they call periods and i'm just going to draw maybe four periods right so this thing here they call one period where the whole thing is an episode now the purpose of the period is that at the beginning of each period the government the government can impose a new tax schedule so the government doesn't only fix the taxes once but it can change the taxes over the course of the episode right now this is what i find i i just don't see why so now you're formulating the


tax giving objective as a sequential decision making it's like the government saying well today we have high taxes but tomorrow we have low taxes and the day after that we have high taxes again and it just doesn't make sense to to for any government to do this um what you should do is you should set taxes once at the beginning of the episode and then see how that turns out and then try to maximize uh your tax schedule because all we're looking at um we're only ever looking at how the taxes are at the end right the things


that we've examined are just the last taxes that the ai has issued we don't know the dynamic of what happens in between this might be super wild actually what the ai does in between and i just don't see the framing as a as a as a sequential decision problem and i believe this is just an over engineered thing because someone wanted a reason and here is the architecture right you see someone wanted a reason to put an lstm in there someone is thinking like well rl that means like sequential decisions and so on and rl


in this outer loop the way i propose it would just be a one step per episode decision which is a banded problem and as we all know bandits are boring so they didn't want this to be a bandit problem they wanted to be a sequential problem and that's why they made this period thing which i find dumb um so another factor here and i'm going to tell you how this relates to the to the weird rich people are taxed high another factor here is look at this it's a cnn an mlp an lstm and an mlp and the agent as well and i can tell you right now the cnn has


two layers two and the lstm has like 128 units in its hidden state so these are tiny tiny models and it is not a model based rl it's model free or else proximal policy optimization and the the um the ability of these agents or planner to learn anything substantial here i believe is just not uh super duper uh well right so the i i believe that these are rather dumb agents and you can see the tax rates given by the planner is fed into the agent model but i don't think that the agent given such a small model can actually adjust to these inputs


because you have to do some pretty good logic in order to from these tax brackets to determine uh how you should act right now what i think is happening is the agent just kind of is aware of its skill level and through its rewards it's trying to maximize its in future rewards and then when the government changes the tax rate it will not i am almost positive it will not directly change its response to that but it will kind of observe that something's happening in the world and then adjust maybe a little bit its overall


strategy uh but not in that particular instance and it will be delayed or it will be like an overall strategy and this might be one of the reasons why the tax brackets here might be screwed up because who says who says if i were this ai what i could do is in period one through nine i make the taxes really low for the rich people so i just encourage everyone to make more money right like come on become more productive and i get the benefits of that and then in the last episode and last period right i just freaking jack up that final tax bracket


it's like you you have lots of money give it to me right and then you just redistribute what you got there to the poor people in the very last period and thereby you achieve your goal of this social welfare function but of course this is not sustainable because all the rich people would just be kind of screwed through that and move down again but it's the end of the episode so what are they going to do so i think the fact how this is framed that there are just two different ways to get coins uh the fact that this is this


periodical nature of the outer loop all might lead to something that becomes slowly more and more and more uninterpretable uh still cool though all right so the final thing they do this with humans yes real humans so they let humans try it and they have this interface here and the humans they behave quite differently from the ai so there are a few different things where the humans act but look at that here ai economist this is what the agents do right so this ai economist is the tax strategy so just take these


developed tax strategies and let the humans be the agents so that the you you just want to observe how the agents act and whether or not the tax strategies also work when it's real humans acting in this environment and not rl agents so compare this to how the humans act the humans they just build their houses in like neat little packets or straight lines or stuff like this i just i just find it to be very funny now there are some things lacking in the human environment which i find really important so first of all they have no cost for


moving which i guess is minor but um second of all they have no trade and i think that is that just kills the whole experiment because now of course what you're gonna get is the wealth is just going to be proportional to how much you get coins per house which is different for each agent right so to me that that is now a pointless experiment if you can't uh trade because the outcome is just predictable and i don't think that the human behavior changes in response to the different tax brackets i think they'll just


do and however they can make money they'll make money they'll build more houses until it becomes unprofitable and that's it so i don't see the i don't see the value of these experiments even though they show that again the ai economist outperforms the other tax strategies in this equality times productivity metric and also in another metric that they measure um the second problem i have is for the human experiments they take this distribution here they say well the a this is one of the distributions that the ai came up with


but you notice the lack of the fu poor people and the lack of this big spike here for the rich people which i find um are one of the two features of the other distribution so i think there's quite a bit of variance in what this ai comes up with or maybe it's just because this is periodical but this is really confusing because they show and discuss that other distribution and now all of a sudden they say well we use this distribution that was also created by our ai and it seems to be qualitatively quite different


in any case um let's look at how the humans behave under the um under the different strategies so in the size formula you'll see that yeah the light blue person here is kind of spreading out a bit probably playing correctly everyone else is just neatly building their houses look at humans are so territorial and most of them they kind of they kind of stay in their little corner and they're like this is my corridor i'm gonna build my houses here in a nice thing and under the ai economist again you don't really see a different thing


just because the taxes are different uh the qualitative behavior is quite the same it's just building straight lines and here i think the difference is more between the humans so i think it's not always the same humans and um the difference might be more between the humans and you kind of see that the humans clearly don't haven't really trained or discovered the optimal strategy they're just doing something and you what you're seeing is just a result of the taxation uh it's not different behavior and this here this this


is the best okay watch the on the bottom right the human they're just first they do something and they're just walling off walling up the other players and this is this is the best i'm going to build a big beautiful wall and i'm going to have the orange guy pay for it it's donald trump in the game amazing and look at the end they actually managed to lock in the other players so they can't move anymore donald trump wins amazing though actually the yellow player appears to win economy-wise but what do you want with lots of money if


thank you for the introduction Riya we're excited to be giving this tutorial on meta learning throughout the tutorial we encourage you to be thinking about questions that you might have about the content that we're presenting and as you go through if you have a question you may come up to one of the four microphones but also if you want to also ask a question from from your own seat without having to get up we have a link for that will allow you to post questions and to also upload other questions and we'll be monitoring that

Learning to learn: An Introduction to Meta Learning


and also asking questions throughout from that link so that's slide okama sauce meta but we'll also have the link on the future slides also we posted all of a pdf version of these slides at tinyurl.com sasha ICML - meta slides and so if you don't want to be putting your phone up to take pictures of the slides and you can look at the slides there as well great okay I mean additionally we'll also be taking questions at the break and at the end of the tutorial so let's get started so a lot of the motivation for meta learning comes from


being able to learn from small amounts of data and in particular what we've seen is that meta learning thrives with large data sets if there's one thing to take away from the last few years of machine learning research I think it's that large diverse data sets plus large models leads to broad generalization we've seen this time and time again from systems trained on imagenet to transform our models trained on large machine translation systems to GPT to trained for large-scale language modeling and all this falls under the paradigm of


deep supervised learning but what if you don't have a large data set what if you're in domains such as medical imaging or robotics or translation of rare languages or recommendation systems in each of these situations we don't have a large data set for every possible task every possible switch situation or every possible person we want to personalize our machine learning system to or what if you want a more general-purpose AI system that can do many different things that you want to be able to continuously adapt and learn


on the job if so it's impractical to learn each and everything from scratch for doing this and so instead we want to be able to very quickly learn new things based on our previous experience and finally what if your data has a long tail for example what if the number of data points starts going down significantly as you encounter more objects or as you interact with new people here new words and encountering new driving situations in these situations your standard machine learning systems will do well in in the kind of the big data regime but


as as you move towards having fewer examples these systems will start to break down they're prettier it not only in the long tail something but in all three of these situations these settings start to break the standard supervised learning paradigm ok so what I'd like to try out next is is it actually give you guys a test so supervised learning breaks down here but actually humans are pretty good at these situations and I want to give you a future learning test and your goal is I'll give you six training data points which are shown on


the left the three on the the first column are from the painter Brock and the the middle three are from Cezanne and your goal is to be able to classify the painter for the for the paintings shown on the right who painted that painting and this is a future learning problem because you only get six data points in order to do this how so you get six label data points for this binary classification problem okay so raise your hand if you think that the painting on the right was painted by Cezanne okay and raise your hand if you


think that the painter the painting was drawn by Brock okay great so most of you got the right answer so this is indeed by Brock and so and in the way that you could recognize this is that there's kind of some some more straight lines and more kind of high contrast lines in the painting and so how did you accomplish this so this sort of thing trading from learning from only six examples but have to be extremely hard for a lot of modern machine learning systems and yet all of you eight guys were able to do it or most of


you guys were able to do it quite well so the way that you were able to accomplish this was because you have previous experience you weren't trying to learn from these six examples from scratch and many of you probably haven't seen these particular paintings before or maybe you haven't even scene paintings from these particular artists before but you have experienced different shapes different textures you've probably seen other paintings before and do that previous experience you're able to figure out how to solve


this task from only six examples okay so now how about we get a machine learning system to solve this task depending on what era you're in you would probably answer it differently you might try to model the image formation process you might try them all the geometry of different objects in the image if you were using slightly more sophisticated techniques you might use something like histah features or Haga features with a support vector machine or more recently maybe you try to fine tune from imagenet features or try to do


domain adaptation from other painters for example and and maybe in the future we'll be doing something even more sophisticated so these different approaches may seem very distinct in that kind of the approach that they're taking but they all share one thing in common which is all of them are different ways to inject previous knowledge or previous experience into the system and as you move down these these prior knowledge you get few engineered human engineered priors and more data-driven priors and also as you


move down you get systems that work undoubtedly better and so in this tutorial we want to try to take this one step further and in particular we want to be able to learn priors explicitly from previous experience that lead to efficient downstream learning an entirely data Duren approach to acquiring these priors that is can we how these systems learn how to learn to solve tasks and this is what is known as meta learning in the rest of this tutorial Sergey will first talk about the problem statement and overview the general meta


learning problem then we'll be talking about different meta learning algorithms ranging from black box out updation approaches to autumn tape optimization based approaches to nonparametric methods then we'll discuss how we can develop bayesian variants of each of these methods then we'll talk about how meta learning has a plot been applied to different application areas we'll take a short five-minute break and also allows additional questions Sergey will then talk about meta reinforcement learning and we'll conclude by discussing


challenges on frontiers ok next circuit will be talking about the problem statement and overview chelsey and those of you that were trying to find the slides we did actually there was somebody who actually posted the link again if you go to the thing on the slide here the slide Oh / meta the first question is actually a link to the slide deck so if you want the slide deck please check that out there alright so let's start with a discussion of how we can actually formulate the meta learning problem and there are really kind of two distinct


viewpoints on meta learning there's kind of a mechanistic view and a probabilistic view let me explain what I mean by these the mechanistic view looks at meta learning as a setting where there's a deep neural network model that can read in an entire data set and then make predictions for new data points training this network use a metadata set which itself consists of many data sets each for a different task and this view of meta learning makes it easier to implement meta learning algorithms so if you're actually coding something up in


tensorflow or PI torch the mechanistic view is probably the one that makes it clearest the probabilistic view treats meta learning as the problem of extracting prior information from a set of meta training tasks that allows for efficient learning of new tasks this view says that learning a new task basically used this prior plus a small amount of training data to infer the most likely posterior parameters that will allow you to solve this task and this view of meta learning makes it easier to understand meta learning


algorithms these are not two views that result in different methods they're actually two viewpoints that can be taken to understand the same methods so in this part of the tutorial I'll actually focus on the second view on the probabilistic view because our aim is really to help everybody to understand meta learning algorithms but we'll see the more mechanistic view emerge when we talk about particular practical instantiation of these methods okay so just to work towards a problem definition for meta learning let's first


start with a problem definition for regular supervised learning and cast it in a probabilistic framework so a lot of what I'm gonna say some of you might have already seen may be in a course on machine learning or a textbook but I just want to walk through it step by step because the meta learning problem definition will build on this so if we're doing supervised learning what we're really doing is we're finding the most likely parameters Phi given our data D so Phi denotes the parameters of your model so if your training for example a


deep neural network model Phi literally refers to the weights the D refers to your training data so it's a set of tuples of input-output pairs where the input might be something like an image and the output is maybe the label corresponding to the class of the object in that image now when we actually want to do this kind of maximum like that estimation problem we typically apply Bayes rule and rewrite it as the sum of log P of D given Phi plus log P of Phi and the first term is typically referred to as the likelihood of your data and


the second term is the prior or the regularizer so if you're using weight decay that corresponds to a Gaussian prior for example and if we factorize the likelihood if we assume independent and identically distributed data points then we get the familiar form shown here it's a sum over all of your data points of the log probability of the label Y I given the input X I and your parameters Phi so this is essentially supervised learning now of course there are some things that are a little bit problematic about this as Chelsea alluded to in the


motivation the models that will work best the most powerful models will typically require a large amounts of data so if your data is very limited it might be very difficult to get a very accurate posterior or very accurate estimate of Phi so the problem at its core that we're going to be dealing with a meta learning is how do you do a good job of estimating Phi when your data is limited and the way you're going to do that is by incorporating more data that is not exactly for the tasks that you want but somehow structurally related so


the question is how can we incorporate additional data and I should say as an aside this is very much the same kind of challenge that things like semi supervised learning and unsupervised learning deal with so in semi-supervised learning you incorporate additional data that doesn't have wise and and so forth in metal learning you incorporate additional data that we're going to call D meta terrain which is labeled data it's just labeled data for different tasks so D meta trained is actually a data set of data sets so it's a set of


data sets d1 through DN where each of those data sets di itself consists of a set of tuples X I and why I wear those why's our labels for a different task so we assume the tasks are somehow structurally similar but not actually the same so you can't just directly incorporate Dee Mehta trained as trainee dated or supervised learning let me give a little example this is based on a popular benchmark for meta learning called the mini image in a data set let's say that your few shot classification task requires you to do


five-way classification between cats dogs lions worms and stacks of bowls I don't know why you would want to do this task but let's say this is a few shot tasks and you have only a few examples of each image now those few examples are not enough to solve the tasks by themselves so we're gonna use the meta train which is a collection of data sets for other five-way classification problems so maybe one of them classifies you know birds mushrooms dogs singers and pianos for instance different tasks but some structural


similarity because all the more visual recognition tasks another example maybe the tasks you want to solve is a few shot regression problem you have a few examples of input-output pairs but then your meta training tasks consist of other curve fitting problems so other curves with a few sample input-output pairs or maybe you have some kind of speech recognition task or some kind of language translation task and so on so in all these cases you can formulate a set of meta training tasks that are not the same as the tasks you want to solve


but somehow structurally similar and typically these would come from your prior data now we could simply stop right there and treat meta learning as a nonparametric learning problem so you want to basically use D and D meta train together but oftentimes we want to use high capacity models we don't want to store all of our data and keep it around forever we'd like to somehow to SIL it into a model into a parametric model with learn model parameters so in meta learning we don't want to keep D meta train around forever what we'd like to


do instead is learn some meta parameters theta so we're going to use D meta train to learn theta and theta will basically contain all the information that we need for solving new tasks that we've extracted from D meta train so whatever we need to know about D meta train is going to be baked into theta via a meta learning process and that's essentially the essence of the meta learning problem now if we want to treat this probabilistically what we can do now is we can say well we can write out our p of phi given d common


Mehta trained as an equation where we're integrating out the these sufficient statistics of our meta training data called theta and this implies the assumption that Phi is conditionally independent of D meta train given theta which is very reasonable because we just said the theta should be whatever extracts all the necessary sufficient statistics from D meta train now in reality integrating out theta is computationally very very expensive so we wouldn't want to do this what we would want to do in practice typically


is use a maximum a posteriori estimate which is which means that we're going to approximate this integral with just a point estimate for theta star where theta star is whatever actually maximizes the log probability of theta given D meta train which again is a very standard thing to do in machine learning so this Arg max that I have written on the right side here this is the meta learning problem the meta learning problem is to pull out the right theta from your meta training data so that that theta contains everything you need


to know to efficiently solve new tasks and efficiently solve new tasks means figure out Phi so once you have theta the problem of getting Phi can be written as the Arg max of log P Phi given D comma theta star because you don't need the D mail train anymore that's all been baked into theta star okay so that's the basic problem formulation if anybody has any questions feel free to come up to the microphones and ask me otherwise I'm going to move on to a simple example yeah that's an excellent question so meta learning is


conceptually quite related to a number of other problem settings including transfer learning multitask learning you know even things like semi-supervised learning in that all of these problem settings deal with incorporating additional data that is not quite from your task but is going to help you solve your task more efficiently the main difference is that meta learning deals with a setting where you still have to do some amount of adaptation on your new task transfer learning formula in a certain ways can I


should be viewed as a type of metal learning as well I'll describe related problem settings a little bit more at the end of this section and maybe then things will be a little clearer okay let's continue so let's work through a little example of how we can design a cartoony version of a meta learning algorithm Chelsea will talk about much more practical algorithms this is just meant to be an illustration so first let's talk about the the adaptation let's say that we already have this theta star we don't care how we learn it and now we just


want to classify new data points using a small data set D so classifying new data points means your test data point X test goes in your label Y test comes out and this function that does this is somehow parameterize by theta star so theta star determines the mapping between X test and y test Waiters theta star come from well it comes from using your data set D which might be a small data set for your new task together with your theta star so D is going to be read in by some function and that function that reads in


D is parametrized by theta star sorry about that so that function is parametrized by theta star now you would also like to of course be able to learn this theta star using large amounts of metal training data which I'll come to in a second but if you can somehow use that Mediterranea to get theta star then that will process your data set up with Phi star and allow you to turn your test inputs into test labels so theta star is what parametrize is this function okay so now how do we actually train this thing well as I alluded to before it's


going to involve this meta training data and the key idea behind setting up metal learning algorithms I think is best summarized by this sentence from a paper by venules at all called matching networks which says that our training procedure is based on a simple machine learning principle which is that tests and train conditions must match now let's unpack this a little bit what are the test conditions well test here refers to meta test right so meta test time is adaptation the test condition is that a model parametrized by theta star


reads in d outputs Phi star and five stars then used to classify new data points for your task so the training time conditions need to match so it met a training time you also need to have a model that reads in a data set which data set well a data set di from your met training set it is going to be parametrized by theta it's going to output Phi star and that Phi star needs to be good for classifying points which points well that's maybe the puzzle so what is it that we're actually going to classify here what we need to do in


order to complete the meta learning problem definition is we need to reserve a little test set for each task so it's not enough to just have a training set the training set is what the model needs to read in but then needs to be trained on something and what it actually going to be trained on is a little test set for each task so for every one of our few shot tasks we're gonna assume that we have K training points but done also some number L of little test points and those test points are was going to supervise the metal learning they are


not used for adaptation they're just used for metal learning so D test is where X tests and Y tests will be sampled from so the game that you're playing then is read in D train output Phi and make sure that Phi is good for classifying points from D test for that same task so now we can actually complete the meta learning problem definition so the adaptation step we can write more compactly as some function f theta star of D train so f theta star reads in D train and outputs Phi star so now all we have to do is learn theta


such that Phi equals F theta D train is good for D test so for every task I you want to read in D train I and be good for D test I which means that we can write down the meta learning problem formulation like this theta star is the Arg max for the sum over all of your tasks of log P Phi I given D test I where Phi I is equal to F theta apply to D train so notice that we get 5 from D train but we met a train on D test we can also represent this with a graphical model so if you're into graphical models here's a


graphical model that represents this relationship so you have theta which are your global metal earned parameters for every task you have a Phi I and X train together with Phii determines y train and X tests together with Phii determines Y test and Y test is observed during meta training but not observed during that a testing that's why it's kind of half shaded there okay so this basically defines the meta learning problem but let's kind of round out this explanation with a little bit of an overview of terminology because


we'll see a bunch of this terminology pop up again again during the tutorial so we make a distinction between meta training meta testing training and testing so you're learning the parameters theta during a meta training phase that meta training phase trains on a collection of data sets each of which is separate into a training set and a test set so when we say training set we mean that small few shots set for a particular task when we say test set we mean the corresponding test images when we say meta training we mean the whole


set of tasks so meta training consists of multiple tasks each one with a training set of a test set and meta testing is what happens once you're done meta training and you want to adapt to a training set for a new task so the set of day of data sets is called D meta train that's what we refer to it as these are meta training tasks we're gonna use Ti to refer to meta training tasks so these are all of our key is this is our meta test tasks I'm sorry and then sometimes you hear people say support set and quarry set so support


refer is basically synonymous with training set and sometimes people use supports that just to avoid the confusion between meta training and training so if someone says support they mean the inner training set and when someone says quarry they're referring to the test the the inner test not the meta test just the test so the quarry is the thing that you actually want to classify correctly after reading in the support and if someone says like oh I have a case shot classification problem what they're referring to is the number of


examples if someone says I have a five-way classification problem they're referring to the number of classes so if you say have a five-shot five-way classification problem that means I have five classes each of which has five examples there's a little bit of confusion about the word shot sometimes it means the number of images per class and sometimes it means the total number of images usually we'll use it to mean the number of images per class so five shot five way means twenty-five data points okay now just to


wrap up a few closely related problem settings that are good to be aware of and this is coming back to that question about transfer learning from before so middle learning is closely related to a few other things that we can actually cast as you know within the same terminology so multitask learning deals with the problem of learning a model with parameters theta star that immediately solve multiple tasks so you can think of multitask learning as sort of zero shot metal learning so that corresponds to defining parameters that


immediately solve all the tasks at the same time this is usually not possible in metal learning problems you can't have one model that classifies you know that does the five Way classification with dogs and lions and also with you know the pianos and the cats but you can view multitask learning is a special case where Phi is just equal to theta another very closely related problem setting is type of parameter optimization and auto ml these can be cast as metal learning they're actually they are essentially metal learning


problems they're outside of the scope of this tutorial but I'll just mention briefly how they can be related so in hyper parameter optimization you can say that theta refers to your hyper parameters that's where you're going to get out of your meta training set and Phi is the network weights so you'll learn your hyper parameters from d mehta train and then you'll use them to get Phi architecture search for same deal theta refers to the parameters of your architecture and Phi is the actual weights in the model this is a very


active area of research unfortunately outside the scope of this tutorial but hopefully this will tell you a little bit about how they relate okay and next Chelsea will discuss a number of actual metal learning algorithms that we can use based on this problem setting oh yes and we'd be happy to take any questions right now - yeah so one question from can you elaborate more on the structural similarity that's required between the Mediterranean tasks yeah so in regard to the structural similarity between the metal training


tasks this is we can actually make that notion formal the way we make it formal as we say there's a lucien over tasks there's a distribution p task and you assume that all of your mediterranean tasks are drawn from that distribution and you assume that all of your meta test tasks are drawn from the same distribution so this is the meta learning analog of the standard supervised learning assumption now what does a distribution over tasks really mean well that's sometimes ends up being a much more subjective notion if you


have a piece of code that generates your tasks and you can say well these it needs to be generated by the same code but of course in reality those tasks are probably produced by nature and there it becomes a much fuzzier line so Chelsea will also discuss a little bit about extrapolation and generalization the perhaps pertains to this great so before we actually want to start going about evaluating meta learning algorithms are going about designing my lorry algorithms we need to figure out how to actually evaluate a meta learning


algorithm once we have one and so it's worth mentioning that there a lot of the the modern metal learning advances and and techniques were motivated by some some work done by Brendan Lake in 2015 and Brendan introduced the Omniglot dataset which is a is much more simple than the mini imagenet dataset that the circuit was showing on the previous slides but allows us to really study some of the basics of mendler nning so the Omniglot dataset it has six hundred twenty three sixteen hundred twenty three characters from 50 different


alphabets that are had many classes and few examples per class a few classes per class and what I find really appealing about these kinds of datasets is that they're more reflective of the statistics of the real world in the real world we have tremendous diversity in terms of the number of objects and number of items and people that we encounter and we don't encounter them over and over again we often encounter many new things constantly throughout our lifetime okay so proposes both discriminative and generative problems


for example an initial approaches for this data set and for other data sets for a few shot learning we're based off of Bayesian models and on parametric's and similar to what Sergey was mentioning before in addition to this which in many ways actually methods are doing quite well on these days you've also been using things like mini image net C far cub celeb a and other data sets for for evaluating meta training algorithms and many of these were not necessarily initially purposed for for medal learning but we're able to


kind of put them in to the purpose that we would like okay so this is similar to what was discussing earlier where we have some n way K shot classification problem such as image where we wanted to be able to perform learning from very small data sets so we might want to be able to learn from one example of five different classes to classify new examples or new images as being among one of the five classes shown on the left and the way that we can do this is we can take data from other image classes structure it in the same way is


what we're gonna be seeing about a test time for example taking images of mushrooms and dogs and singer is structuring it into the likewise these these five way Wangchuk classification problems doing this for many different other image classes training a training or networked in order to perform these types of things across these training classes such that an evaluation is able to solve the problem on the top with held out classes and this is an example that's specific to image classification and we're gonna be coming back to this


example a number of times because it's useful for comparing different approaches but the same sorts of ideas are also applicable to things like regression to language generation and and and prediction to skill learning really any machine learning problem you can construct in this way where you're training it on a number of machine learning problems and you want it to be able to generalize to learning a new problem with a small amount of data ok so now that we know how to evaluate a meta learning algorithm was actually dig


into how we actually design these meta learning so the general recipe and the general principle behind these algorithms is that we need to choose some form of inferring the parameters of a model Phi given our chain data set and our meta parameters theta and then once we choose choose the form of this we can then optimize it optimize the meta parameters fight theta with respect to a maximum likelihood objective using our meta training data okay and many of the different algorithms that were gonna be looking at today really only differ in


step one choosing how we want to represent this inference problem essentially and so I you can ask well can we just treat this as an inference problem and and pretty clear neural networks are actually are quite good at inference so maybe we can just use a neural network to to represent this function itself and that's exactly what the first approach will be so this is what we'll refer to as blackbox adaptation approaches and the key idea is for a neural network to represent this function that outputs a set of


parameters given a data set and a set of meta parameters and for now we're going to be using deterministic or point estimate of this function key we told to note as f theta and of course we'll get back to Bayesian methods later and so we'll see Bayes a bit later okay so how do you actually try to design a neural network to do this well one thing you could do is you could use a recurrent neural network that takes in data points sequentially and produces a set of parameters Phi and so this recurrent neural network in this case will be


representing F theta and then we'll take the outputted parameters use those parameters for another neural network that's gonna make predictions about test data points and so these are gonna be the data points from D test okay and then once we have this model we can train it with standard supervised learning this is just a standard recurrent neural network so we can train it to maximize a log probability of the labels of the test data points given the test inputs and we can do this optimization across all of the tasks in


our meta training data set we can rewrite this loss function of performing a bag of evaluating predictions of a model as simply a loss function operating over the parameters Phi and the test data points so we're gonna write this right here and this this will be used mostly for convenience later on and then with this form we can write the full optimization problem as an optimization of the over the the parameters outputted by the neural network F theta and the test data set okay so now that we have this this optimization objective how do actually


what is the algorithm that's used to optimize this so what the algorithm looks like is we first sample a task from our meditating data set or a mini batches of tasks then for that task we have a data set di and will sample disjoint data sets D tre and I and D test I from that data set and then once we have so I guess what this looks like I say these are the images corresponding to tasks i we want to be able to partition this or basically sample this sample D train and sample t-tests from from this data set and so we'll assign


like so and then once we have D train and D test will compute the parameters using D train and then evaluate those predicted parameters using the test data points and it's quite important that d'être and D tests are disjoint so that we're not training for memorization of the labels but instead training for generalization and then once we update our meta parameters of them were of course going to repeat this process for new tasks and if we use a mini batch of tasks the gradient in step four is gonna be averaged across that mini batch great


okay so there's the algorithm now how to actually represent the form of F data so the the form that I have written here is a recurrent neural network you could use something like an Ellis TM you could also use something that another memory augmented neural network like a neural Turing machine which has done it has been done in past work you could also use something like self attention or 1d convolutions or even really just a feed-forward network that then averages in some embedding space the key thing is that you want this you want these


networks to be able to take in sets of data points and often times you want it to be able to take in variable numbers of data points and so that's where I will be using these types of architectures that have the capability to take in take in sets and variable numbers of data points okay great so I know that we've gone over kind of this type of approach and how it works and and the different architectures what are some of the challenges that come up so one thing that you might ask is well if our neural network is outputting all of the


parameters of of another neural network this doesn't really seem scalable but if the the the neural network that's making inferences about test data points has millions of parameters then we need a neural network that outputs millions of a million million dimensional output and one idea that we can use to remedy this is we don't actually need to output all of the parameters of another neural network we really just need to output the sufficient statistics of that data set of the training tasks that allow us to make predictions about new data


points and so what we can do is we can take this take this architecture instead of outputting Phi we can output something like H where H is representing a low dimensional vector and then and this will be essentially representing information about the task everything that's needed about the task in order to make predictions and then you can combine this sufficient statistic H with another set of parameters theta G that are also metal learned along with the parameters of F such that with these with both H and theta we can make


predictions about new data points and then theta G can be something very high dimensional and with the combination of the two we'll be able to make predictions for a new task okay so the general form of what this looks like is you can kind of abstract oh wait is this notion of H and just write out the ability to make predictions given a training data set and a new test input outputting the corresponding label okay before I move on to the next type of approach are there any questions so the one question from the audience was does


the network need to be recurrent or are there other architectures that could work well absolutely so as I mentioned on the previous slide this could be something that's recurrent like Ellis gems and you're all Turing machines you could use something like self attention or work recursive models but you also don't need to have something that actually is like a sequence the sequence model could also simply being be something that has a feed-forward model and then average if you assume that you're trained is that has fixed-length than you could of


course also concatenate and use a fully connected network although these are the approaches on this slide tend to be a bit more scalable I think it's on hello hi um so this might be related to the few previous slide so I I'm trying to understand the difference between meta learning and they're learning to learn framework by Jonatan vector of the T / posts around 1990s so I think if I understand correctly the the basic idea of learning to learn is that you're trying to learn the inductive bias of the learning


problem which in this case is data start in your notation could you elaborate a little bit more in this similarity or the difference Thanks yeah so I'm not sure if I'm familiar with the particular work that you mentioned but in general many of the ideas that we're presenting are inspired by work that was done initially in the late 80s and early 90s with with older types of neural network approaches many of those approaches didn't specifically look at the few shot learning setting which were focusing on this tutorial but