the income so if you if an agent earns these coins then it has to pay taxes to the government and the government will redistribute all the taxes it has collected equally among the population so if you pay a lot you might lose through this process and if you just pay a little taxes you might gain through this process so that's it that is the basic premise of the game the agents are using


reinforcement learning and i believe the newness of this paper is also that the government now is using reinforcement learning in order to determine the optimal tax policy there is kind of this inner loop here and there is this outer game where the government also tries to maximize the rl and what does the government try to maximize good question it is a measure that's

The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policies (Paper Explained)


called social welfare now social welfare consists of two things and they have this here way down in the paper social welfare in this paper consists of two things first of all economic productivity which basically just means how many coins have has anyone produced it doesn't matter who but just the total amount of coins produced the second one is income equality and


this is related to the the genie index so if you plot the cumulative distribution of wealth a fully equal society would be a straight line because 50 percent of the people would have 50 of the money and so on but a almost all true societies have something like this where fifty percent of the people might have ten percent of the money and the rest fifty percent of the people has the


other ninety percent and the measure of inequality is this area here um this is called the genie index and one minus this area is what this paper has as an equality measure so the higher this number the more equal is the society in terms of their income distribution now what is actually optimized for here is this thing equality times productivity so you want


both to be high your income equality and your productivity there's a trade-off here of course but you can you can have multiple ways to trade that off and that will give you the different uh thing they call this the social welfare function and that's the thing that the government rl agent optimizes for so you can see here already the free market even though it's


the most productive produces the most coins because if you haven't free market means no taxes if you have no taxes then people are basically encouraged to uh earn more money because they don't have to pay taxes on them right as soon as you tax them they're less encouraged to earn more money and therefore if you have no taxes the most coins will be earned in total but the equality suffers


so the equality is the lowest among these things considered if you compare that to the ai economist the ai economist achieves the highest social welfare it achieves the highest equality but it doesn't suffer as much in productivity as other systems here and the baseline systems are first of all the u.s federal system this is not particularly tight to the u.s this is


basically every system uh or most of the systems that you have currently in the world is the progressive tax system and the size formula which i believe is an economically theory-based system which is a regressive tax schedule you can see them down here where the u.s federal will be progressive means the more you earn the more percentage-wise you pay while the says formula will be


regressive which generally means the more you earn the less you pay i believe this was derived under some assumptions to be the optimal tax distribution and the ai economist will come will come to will come to this in in a second let's actually just look at one of these things first one of these games how this plays out the cool thing here is that they have pretty flashy


animations so you can look how does one of these games turn out now this is a free market game and you can see the agents moving around collecting things building houses and you might notice that one of the agents namely agent one is just building all of the houses and generally just kind of being a dick being in everyone's face and kind of building things everywhere


and the other ones don't and or or just very few like the light blue on the on the bottom left build some houses on the right you can see how the distribution of wealth is is structured and you see agent one ends up with most of the wealth now the size of the circle i think is the total productivity so you can see this grows over time mainly because agent one becomes so


rich and if you analyze this if you analyze what's happening here then you'll see that agent one and i might be yeah they have a graph up here so so it is very interesting what happens this is kind of the same game so agent one here is this orange dot and agents two three and four are these dots here and this graph here is coin from trading so how much money they


win or lose from trading now you the green bars are trading wood and the the brown bars are trading stone so you see agent number four which is the lowest skilled um the skill is just determined at the beginning of the episode it will just make all of its coins basically by selling wood and agent 3 will make all of its coins by selling stone and agent 2 will collect both and sell


both and agent one will just spend money in trading so you'll have a specialization here agent one which is the highest skill one right here will buy resources in order to build more houses because it clearly profits from building lots and lots and lots and lots of houses so it will use that money to buy more resources rather than go in collecting them while all the other ones basically


forego building houses in favor of they just collect the resources and they just trade them way to the agent one that's more profitable for them than building houses themselves so you see this kind of specialization emerging in these games which i find i find this to be pretty cool that you see something like this like a really stark division of labor emerging just from


these very very uh small set of rules and you can analyze this game in different ways they have a few more plots where this becomes quite apparent that um sorry that that these agents specialize so you see here resources collected sorry about that resources collected uh if you have the lowest skill and the highest skill labors the reas the lowest skills they mainly about this this should be a


pen they mainly collect resources while the highest skill labor mainly goes for building things it doesn't collect resources but net income from building is really high while everyone else just doesn't build at all all right so we have a division of labor emerging now this was a free market let's actually compare the different algorithms so if you look at social welfare this is this thing


here equality times productivity you can see that the ai economist will outperform over time over the training progress it will outperform all of the other systems so it will outperform the free market the u.s federal tax system and the sas formula um if trained for long enough which is to be expected right if you put rl onto a cost function it will then optimize that cost function but it's


pretty cool to see that it had there's a lot of lot of headroom here over what we currently have now let's look at some of these strategies it comes up with so what do these games look like where the ai has imposed different tax strategies so this is with the size strategy you see that here again you you see this inequality emerging with the yellow player here


building most of the houses with the ai economist again there is inequality but you can see at the distribution that agent one only ends up with about half of the wealth where if you compare this to the free market here then agent one ends up with like two-thirds of the wealth right this is the game we saw before um but there is not qualitatively that much of a difference


uh but there is in the end result all right let's look at what the these policies actually come up with so what is the tax policy that the ai comes up with so this tax policy outperforms on this social welfare metric and this is very interesting right so first of all you see that it's right zigzag it's like down up down up uh which is already weird so the first very weird thing


is the the spike at the very bottom so that thing here what's that thing here those are the poorest people in your society and you're taxing them the highest right so just imagine this you're here uh downtrodden by life abandoned by society you have no money no house no nothing and you're just trying to get a job you're just getting like a little bit of money and


you can buy a cheeseburger and then the government comes give us that us that money come on so basically this these are the poor and the poor in this system is just fu fu the poor now the reason why this happens is pretty clear right the reason why this happens is because you want to encourage people to go here to earn more money right so so it's not like the government makes any money from


the poor people independently of how it how high it taxes them but it is a basically an incentive structure to make them move over to the somewhat more productive population because here it's assumed kinda that even the lowest skilled ones can move over a bit if you just tax them enough at the low brackets right so um this this is what i find to be you just have to realize that it is


so hard i believe it is almost impossible to encapsulate what we really want in a system into a formula to be into a cost function to be optimized it is so incredibly hard and you see that here of course it is going to result in a better social outcome but it just doesn't feel right to tax the poor at what 60 okay so f the poor right and then you get to to this to this level right here and


interestingly if you earn even more you'll be taxed high again right so this this um this we're kind of used to that you earn little you pay little you earn more you er you pay more but then comes this entire valley here what's up with that right like wtf doesn't matter and this can be this this is now of course the same reasoning as you have with this science formula here is where the


rich people you want to tax them less so that they are more productive such that they generate more coins and even though you tax them less percentage-wise they will end up paying more uh money in absolute terms because because you basically encourage them to produce more so that is that is can that is the i guess the reasoning behind this but what you have to re you have to


recognize what's happening here right what are we optimizing we're optimizing this productivity times equality right and what do we get you see you get two big values of attraction one here and one here and that means that this algorithm favors a two-class society right and i believe this is this is partially the limitations of this simulation here the fact that you're only a f4 agent the


fact that you can only do two things either collect or build right it encourages a two-class society this specialization that you saw right so you say these here are the money makers right and these here are the collectors and it is very hard to move from one group to the other because if you you earn more coins as a collector you're here and you're really discouraged here


if you move there you want to move all the way over here right now the people that are are already over here if they earn an extra coin that doesn't bother them too much so they're very encouraged to earn more money but the very the poorer people on this side they're basically discouraged from earning more money because the system needs them to stay at that collector level right


so the system encourages the two-class society because we have not built social mobility into the into the into the equation we have not built a measure for social social mobility into the cost function and therefore the ai doesn't care that the poor people will stay poor and the rich people will stay rich uh it just knows that this is the best outcome for society overall


given the cost function that we had again this just doesn't seem like fair to us like what we want we want someone to be able to make it over here right even if they start out from the bottom and so we'd have to we have to build that in so we have a system that is effing f the poor right no social mobility mobility no and then look at what happening at the end


what's happening at the end this is beautiful very rich people these are the money maker right this is the this is the monopoly guy top hat monocle wearing scrooge mcduck bathing in coins this is where the the government makes their money and um the discrepancy is really stunning because you could also argue hey why don't we apply the same reasoning as we applied here and here right


it's not is it not like the case that if the rich people if if you tax them lower they'll pay more money and so on i believe again this might be just a result of this how the simulation is set up so we'll move away quickly and we'll come back to this here is what i find particularly interesting about this paper which just confuses the heck out of me it is a double periodic game so it's an


inner outer loop game what do i mean by that they have these episodes right here is the start and here is the end and they subdivide this into as we said 1 000 steps so an agent is here and they can do step step step step step and it can perform these actions this is the agent there are 1 000 steps here and the agent just tries to collect as much coin so this is your classic rl


problem but also they divide this into 10 what they call periods and i'm just going to draw maybe four periods right so this thing here they call one period where the whole thing is an episode now the purpose of the period is that at the beginning of each period the government the government can impose a new tax schedule so the government doesn't only fix the taxes once but it


can change the taxes over the course of the episode right now this is what i find i i just don't see why so now you're formulating the tax giving objective as a sequential decision making it's like the government saying well today we have high taxes but tomorrow we have low taxes and the day after that we have high taxes again and it just doesn't make sense to to


for any government to do this um what you should do is you should set taxes once at the beginning of the episode and then see how that turns out and then try to maximize uh your tax schedule because all we're looking at um we're only ever looking at how the taxes are at the end right the things that we've examined are just the last taxes that the ai has issued


we don't know the dynamic of what happens in between this might be super wild actually what the ai does in between and i just don't see the framing as a as a as a sequential decision problem and i believe this is just an over engineered thing because someone wanted a reason and here is the architecture right you see someone wanted a reason to put an lstm in there someone is thinking


like well rl that means like sequential decisions and so on and rl in this outer loop the way i propose it would just be a one step per episode decision which is a banded problem and as we all know bandits are boring so they didn't want this to be a bandit problem they wanted to be a sequential problem and that's why they made this period thing which i find dumb um


so another factor here and i'm going to tell you how this relates to the to the weird rich people are taxed high another factor here is look at this it's a cnn an mlp an lstm and an mlp and the agent as well and i can tell you right now the cnn has two layers two and the lstm has like 128 units in its hidden state so these are tiny tiny models and it is not a model based rl it's


model free or else proximal policy optimization and the the um the ability of these agents or planner to learn anything substantial here i believe is just not uh super duper uh well right so the i i believe that these are rather dumb agents and you can see the tax rates given by the planner is fed into the agent model but i don't think that the agent given such a small


model can actually adjust to these inputs because you have to do some pretty good logic in order to from these tax brackets to determine uh how you should act right now what i think is happening is the agent just kind of is aware of its skill level and through its rewards it's trying to maximize its in future rewards and then when the government changes the tax rate


it will not i am almost positive it will not directly change its response to that but it will kind of observe that something's happening in the world and then adjust maybe a little bit its overall strategy uh but not in that particular instance and it will be delayed or it will be like an overall strategy and this might be one of the reasons why the tax brackets


here might be screwed up because who says who says if i were this ai what i could do is in period one through nine i make the taxes really low for the rich people so i just encourage everyone to make more money right like come on become more productive and i get the benefits of that and then in the last episode and last period right i just freaking jack up that final tax bracket


it's like you you have lots of money give it to me right and then you just redistribute what you got there to the poor people in the very last period and thereby you achieve your goal of this social welfare function but of course this is not sustainable because all the rich people would just be kind of screwed through that and move down again but it's the end of the


episode so what are they going to do so i think the fact how this is framed that there are just two different ways to get coins uh the fact that this is this periodical nature of the outer loop all might lead to something that becomes slowly more and more and more uninterpretable uh still cool though all right so the final thing they do this with humans yes real humans


so they let humans try it and they have this interface here and the humans they behave quite differently from the ai so there are a few different things where the humans act but look at that here ai economist this is what the agents do right so this ai economist is the tax strategy so just take these developed tax strategies and let the humans be the agents so that


the you you just want to observe how the agents act and whether or not the tax strategies also work when it's real humans acting in this environment and not rl agents so compare this to how the humans act the humans they just build their houses in like neat little packets or straight lines or stuff like this i just i just find it to be very funny now there are some things


lacking in the human environment which i find really important so first of all they have no cost for moving which i guess is minor but um second of all they have no trade and i think that is that just kills the whole experiment because now of course what you're gonna get is the wealth is just going to be proportional to how much you get coins per house which is


different for each agent right so to me that that is now a pointless experiment if you can't uh trade because the outcome is just predictable and i don't think that the human behavior changes in response to the different tax brackets i think they'll just do and however they can make money they'll make money they'll build more houses until it becomes unprofitable and


that's it so i don't see the i don't see the value of these experiments even though they show that again the ai economist outperforms the other tax strategies in this equality times productivity metric and also in another metric that they measure um the second problem i have is for the human experiments they take this distribution here they say well the a this is one of the


distributions that the ai came up with but you notice the lack of the fu poor people and the lack of this big spike here for the rich people which i find um are one of the two features of the other distribution so i think there's quite a bit of variance in what this ai comes up with or maybe it's just because this is periodical but this is really confusing because they show and


discuss that other distribution and now all of a sudden they say well we use this distribution that was also created by our ai and it seems to be qualitatively quite different in any case um let's look at how the humans behave under the um under the different strategies so in the size formula you'll see that yeah the light blue person here is kind of spreading out a bit


probably playing correctly everyone else is just neatly building their houses look at humans are so territorial and most of them they kind of they kind of stay in their little corner and they're like this is my corridor i'm gonna build my houses here in a nice thing and under the ai economist again you don't really see a different thing just because the taxes are different uh


the qualitative behavior is quite the same it's just building straight lines and here i think the difference is more between the humans so i think it's not always the same humans and um the difference might be more between the humans and you kind of see that the humans clearly don't haven't really trained or discovered the optimal strategy they're just doing


something and you what you're seeing is just a result of the taxation uh it's not different behavior and this here this this is the best okay watch the on the bottom right the human they're just first they do something and they're just walling off walling up the other players and this is this is the best i'm going to build a big beautiful wall and i'm going to


have the orange guy pay for it it's donald trump in the game amazing and look at the end they actually managed to lock in the other players so they can't move anymore donald trump wins amazing though actually the yellow player appears to win economy-wise but what do you want with lots of money if you can't move so i again i find these human experiments to be rather pointless here because you


disable trade and you don't train the humans to find a good strategy all right but in that i find the entire paper to be pretty cool code is going to be released they promise and they have checked that they have no ethical problems of course i invite you to check out the paper if you like content like this please uh subscribe share and leave a comment of what you think


so yeah it's a pleasure to be here and today we'll be cooking through the labels a couple of words about me I'm a kegel competition Smith master located in Minsk Belarus my name is Kegel is V dot e dot s dot and currently I work as a data scientist at h21 so today we'll be talking about some distinctions between labeled and unlabeled data then we will talk about

How to cook pseudo-labels | by Yauhen Babakhin | Kaggle Days Dubai | Kaggle


what is said at Lackland actually is some use cases and some recipes of how to cook so the label to suit the labels and some example on the real okaycome competitions where to the labels were applied and achieved good results so here is the general supervisor on problem train and test data and for train data we are given a label so have some kind of wave of data and our goal


is to build a model to predict the test labels so test is kind of unlabeled data and generally it is a kind of usual keiko competition scheme where we are given some kind of label data and you need to make a model to predict what the unlabeled [Music] and the problem we'll be talking about today is the kind of situation when a lot of unlabeled data and label data is


very small so this could happen to both in Europe some kind of usual motion journal projects and as well as on Kegel competitions and here probably you could some kind of create some tricky models to build them on this small label data and apply some different techniques but probably the better approach is to somehow use this huge unlabeled data and the reasons why like what will have such


situations when the label data is very small the first one is it is expensive so to label your data you need to acquire some special people or you need to acquire domain experts or use some special software in order to label your data and obtain label data so consequently it is also time consuming so you need for example one month to label one more potential for your data


and of course your management will not be satisfied with with this approach that you need - too much time and you need to put the model right away there are some other reasons for example seconds it could be some sophisticated experiments you need for example build some I don't know very very hard experimented start to establish and there are lots of lots of stress this is


a need and it is hard to repeat it frequently and that is why it is also could be expensive a time consuming so it's kind of basic reasons why why we have such deviations with a small label data and here is a quote by entry in there it is not who has the best algorithms that wins it has the most data and probably it is much more crucial nowadays when there are lots of


machine learning models that work in out of box so it would apply and you could solve almost any problem type are just using some some some predefined models and the problem is that you can't apply models when you don't don't have data or this is the reliable data is too small so probably it could all it could also be extended for personal data so if I have small label data's and it's also


hard to do some kind of huge supervisors to supervise or model and in case if you're unable to get more label data and if it is impossible or it is hard to acquire to reduce the message codes same supervised chlorine is introduced in a simple example here on the left you have a classic supervised subra specification a problem where we have for example two classes triangles and squares and our


goal is to build a classifier so a decision boundary that would distinguish these two classes so here on the image pad on the image B we have some kind of sample decision boundary between these two classes what same service were allowed to do it allows you to utilize all cells and label data so this is red dots some kind of way observe this data but we don't we don't have the real


labels and actually from this red dots with some kind of get the structure of the data and it allows to sometimes change our decision boundary itself uses this knowledge and we see that this is and right now is more more reliable and more generalizable probably yeah and what a sexual thrill delivers so the labeling it's kind of the simplest semi-supervised learning so simple


superest Waldron have has lots of different approaches but see the labeling is like the most petition and so it's the most easy to use and the idea is pretty straightforward so we have the label data we train Air model like some supermodel one when this label data and afterwards we just make the prediction spawns and label data and actually these predictions are already


through the labels so we treat all the test observations is that we have predicted by our model as to the labels and then we could some kind of concatenate these two data sets so initial data and our predictions made by our models and treat all of this data as a kind of extended version of our label data and users to the labels in our subsequent training yeah before before


speaking of how to with how could we utilize to the labels I'll talk about some kind of couple of ingredients the first one is confidence so instead of taking all the predictions from the code test set we're interested only in the content predictions the reason for that is if we add to power of two the labels some observations that are hard to predict or some special cases some


corner cases then it would like been late when been detect our our subsequent training because it introduces some noise and some bias in now in our model and we need which is only the confidence intervals in order to like I select only the reservations that our model is confident in there are different different definitions of what is obscure or what is a confidence


prediction for different types of of the problems so for example falsification problem the easiest way is to kind of take the probabilities for each process that were predicted and for example if at least one one plus have probability over 0.9 then okay it is reliable reliable observation we could edit in the content through the labels for image segmentation problems for example we


could use some kind of percentage of motivated or pixels away we'll just find in 110 pixels on the image and then we're treating like the person of course two pixels for example over eighty percent those senses alteration is confident and of course aggregation type of process it's a little bit tricky because it's hard to tell what is kind of confident prediction for the


regression problem but one approach what you could use for example in different problems is that you can look at your predictions from one epic to another unit year training of the neural network and if you see like huge jumps for one equal to another during the nest neural network training it means that probably this observation is unreliable and it is not not a good idea to include it in the


to the labels and overall pseudo labels is kind of the message that is widely used in the in the deployment context as wrist posture is that no metals allows to treacherous terrain online and you can add some new data to either intentional training so but probably not probably so for sure it could also be used for some classic machine learning problems but it is not


not so popular so my talk will be more about about some neural networks and the deployment context yeah ii ii degraded in isn't semblance so instead of creating one one model and predicting what one set of students who could train multiple models sending samples Emmet's in some way and obtain a new set of to the labels the reason for using an semblance observed two


different reasons the first reason is that worse it is better to use in similar photos instead of one model if you talking about the quality so single model is almost always be the first versus it's an example of model and the second reason for that is that in several photos allows to get diversity to up to the labels so we for example using only single model from one page to


another and Catina training with a single model sinu put some kind of propagate the errors or homes on this model and if you're using example since the diverse models could just somehow eliminate this effect and we obtain a more generalizable to the labels all right so the first recipe how how we could utilize to the labels about change multi nia sleaze so it consists of two steps


the first step is that we just Union two datasets label the data and see the labels obtained and treat it as a mutable data so of course he could from to deliver secret select one with continent predictions were some kind of only subsets Anderson dish but which is also the labels as well as a column as a label and afterwards this new label data set could be used to train a new model


so now instead of using original we use both pseudo inverse sine theta and trainer model and it occurs is it such approach allows to get better results and change with a single model I mean the model is a zero strain on unlabeled data so identity so what is the process it is to concatenate train data and to the labels all right another approach is called pre-trained it's a kind of a little


different in the previous approach was based on some kind of on a data level so we can created our data in the city in the interested single set of labels now we are solving some kind of a model level problem so we take on the schudle labels change the model only opens up to the labels and we have change some kind of weight in civilization so we will save the way so this one is a bit we


have obtained and this weights could be used as a starting point for subsequent train on the label data on the initial table data and as the reason why it is working is that afterwards after you obtain your model or some pseudo labels now you wait wait no information about you your data set about your domain you're working with and this installation works better than for


example if a man in image net installation so the pipeline works in the following way we have our pertained models of Australian condoms with to the labels we initialize the weights and it allows to train faster and obtain better results if a fan changes this model or going to label data on the initial label data and after we look ups to point to you is this model at NU model you could


make predictions on general data and again this approach allows to get better results compared if we have started for from here and just insulin insulin ensures the weight with the image net okay so each recipe has some herbs and spices and here we'll talk about the validation when we are talking about pseudo labels case absolutely labels is kind of agreeing a great way to - or


that kind of over senior model and yoke and abused or fits into the leaderboard or fit into a validation data and you need to establish the proper way how to compare the models before you can apply to the labels and after the second version of will be applied and for example have this for four volts away using the basically k-fold cross-validation and the first approach


could be we just train four different models for each of the poles and samples them and obtain through the labels and since these ends the specific labels are used in the first or second recipe in order to continue the training however this approach is weak because we now see the label dataset contains the information about also also targeted target Morocco's also late labels is a


train data and if for example after work limited to the labels we want to measure the quality ones there for example for sport then it may acure that our our quality is too too too too optimistic because so the label school reticle since the right labels also causes this of this first hole so the better approach is to use some kind of out of focus to the to the labels so for each


fault the train has separate a separate model and predict and create a separate suitable data change independently and afterwards it kind of provides a reliable reliable schema validation so then we can compare the models before the first ago that was Jack have been applied the only drawback here is that we need to train for different models for each mode and obtain only one set of CB


labels and they're not really reliable because we need this kind of an example ingredient when we are developing multiple diverse models and actually in practice everyone uses these schemes just like a skin so it is it is letter Western model I focus as I said in the kit but the reason for the usage of this scheme is that we are already obtained an assembler model so they don't have


four models and if for example for each move with well-developed three different architectures narrow Network center cap over 8 to 12 models and it is really a great world chain of two the labels and divorce so yes the take to to this rotation scheme and now I will talk about a couple of examples where the labels showed pretty pretty good results and one of their competitions is Canada


model and education it was posted by keiko last year and the problem was to kind of quantify photos made by some camera into the camera it was taken by so it was a kind of multi conservation problem with ten classes and the classes are kind of Apple some soon some some kind of other devices and for example this particular image was taken by a HTC One and yeah it's kind of impossible to


say just looking but by eyes so it's another nano net network approach have been applied here for this competition and actually in this competition train that is data we're about the same size so so what about I guess 3/7 of images and we remember that do the labels show great great value when it have some kind of small small label data sets and large unlabeled data but the reason why


syllabus are working here on this problem is this train and and test we'd actually also train images were taken by a single physical device so for example this is green form but the test images for example might may be taken by the same model for example HTC or iPhone 4 but they were taken by different physical devices is this orange one and so the goal here was to some kind of use


assisted predictions in terms of silver labels and probably if it could allow find as some particular features some particular ste facts that are specific for this country to first particle of polish model and the said that is ins ik handle learned for flow from from the green model okay here is kind of a recipe what has been applied and they want what would


have worked or for this particular competition so firstly we're just making a kind of classic classic approach we train multiple models and sample them and sample them so Lucas for example tried different architectures different training procedures and so on so welcome to the labels names this approach allows us to get 66 place on the quanta private leaderboard


the next step is could be the kind of route would take take this to the labels pretend pretend on them and printing on train data and it gives like a huge boost and we have receiving a top 20 position however as we've discussed there like distribution between train and test data it is probably the better idea to train on the unpure through the labels so the source step is kind of


minute eliminates the second one and an instead of points Union on train data we train our model on pure silver labels so at this step we don't use as a the initial train data at all and in such case if it was to get even better results and it achieves the same place or on the private leaderboard alright the next the next example is salt identification challenge it is the


problem was in in semantic image segmentation so we were given some kind of images obviously some kind of or surface under earth and each pixel of the image was classified in the two classes whether it is sold on unsold and the goal was to build a model that predicts the salt deposits like some kind of mask masks represented all cones right yeah in this case in this competition as


a trained data contains only four four thousand images and the test data contains eighteen thousand images so it's really a perfect candidate for pop pseudo employment when I have this Assessor says what difference between labeled and another available data and again we start with a simple approach will train multiple models obtain filter labels and it gives in this case is a


forty six position it's pretty pretty good I guess it was around three thousand participants in this competition and on the second stage we apply our second recipe so we retain the model on CD labels twenty non-trained and obtain citizenship is a place in top ten and in order to achieve better results we just repeat this this is this these two steps multiple times so what


does it mean so after second second step of ten you set up to open up a new set of see the labels we again train multiple models obtain new pursue the labels and also I can pretend which allows a French Union trade so repeaters just look multiple times allows to give the better results and a ship to touch top one position for this competition and actually the skin looks likely cut


off in this way so initially we have only label data between a model in this first round send a predictive see the labels on the unlabeled data select on the confident ones and retrain the model general repeats is this test for example K times and at each iteration which this improvement is a sports oh yes the sports improvement are kind of degrade so it's a smaller and smaller but it


should iteration gives more and more information about syllabus improves the calls up see the labels and achieve the better results also either board and actually one more dish that we are using here is that where a train is a model so at each stage we train the model from scratch so what what does it mean if we're trains we forgot to change model from with the


first stage to the second and sort and so on we could finish with the situation when our like error propagates through each iteration so if I have made an error in the first round and out to the label I'll quit inaccurate then this error will propagate through each round so with each round we're just starting from imagenet weights and start start start a model from scratch okay so yeah


basically there is no kind of universal universal recipe of how to cook soup without the labels and how they could be applied because it's really well specific to data uuuu you're using ends it's a problem type you're dressing but you have some kind of building box that could be a temple night together with some kind of your sub some spices and and some ideas that could improve your


performance patron performance of your models yes this approach could be applied in both competitions and in real machine learning projects and it really performs very well when it have very small label did the data the data set but unlabeled data is available kind of in fact in in in yeah so unlevel did this is a lots of unlabeled data and when when what is a good idea what when


to use this substitute the labels it could be also quite in some kind of final stage of the competition when you kind of stuck with the ideas and your score does not approve you can try to the labels idea and probably it adds some kind of minor improvement is the final session of the competition so yeah here is a kind of screenshots of where with the winners of the


I got you know basically my spacian is to kind of get to the next step in AI machine running etc and the what we what we see today is is a huge amount of success in machine learning but the sample efficiency of all of the techniques that we use today are much much worse than everything we observe in humans and animals in other words it take many more samples or many more

Yann LeCun: "Energy-Based Self-Supervised Learning"


trials in the case reinforcement learning for a machine to learn anything compared to humans and animals so a lot of people are very quick to draw conclusions from this but you know humans draw on an animal's drawn evolution and innate behavior but I think it's just more efficient learning and another kind of reaction to this is we draw our background knowledge about


about the world and that's true the big question I'm asking here is what does that come from how do we acquire all the background knowledge we have about the world that allows us to learn a new task very quickly so so all the success that we you see in practical machine learning today almost all of it is due to supervised running and we all know what what that means right to give


let's say you want to do image recognition you give an image to the Machine and if the machine doesn't give you the right answer you tell you what a right a right answer is and you adjust its internal parameters using stochastic gradient descent or something like that a gradient based method to get the output closer to the one you want the amount of information you give to the


machine at every trial is relatively small even in the case of something like image net which has 1,000 categories you tell it with the correct categories and that's less than 10 bits of information so you're asking efficient to predict a very small amount of information every time as a result you need a lot of samples to try need to do anything reinforcement learning is even worse so


reinforcement learning a situation where you don't tell the Machine the correct answer you only tell it whether the answer it produced was good or bad ok now there is like a harder form of reinforcement learning where what the Machine sees next types type depends on the answer we got and then there is a problem of exploration exploitation etc but even without talking about this if


you look at how long it takes even to learn to for a machine to learn to place the condition that it just runs by reinforcements come into play an Atari game very simple Atari game from the 1980s it takes the equivalent on average of 80 hours of training to reach the performance that any human can reach in about 15 minutes those machines actually get to superhuman performance but it takes them


a long time the the go system that was produced by deep mine and the one that was produced by Facebook a little bit later I can I know the numbers for Facebook because they published and also they're my friends this takes about 20 million of self play games to to reach super human performance running on 2003 P news for two weeks this is a lot of games mortal


games that any humans can play go yes go is complicated and Starcraft so this is a recent the paper actually just appeared last week but the results have been known for a while from deepmind the Alpha star system takes about 200 years of equivalent real time to learn to play on a single map for every single player a single type of player if you want and that's an enormous amount of computation


there's rumors that just to train this for a week or two that team took more computational resources at Google that auto research okay and you know similar for there is a you know recent demo by open AI and of course when he paper are using the first month on internal manipulation from simulation and then you can sort of transfer this to a real real robot and it takes the equivalent


of ten thousand years of real time in simulation so you can run simulation fast or you can run it in parallel it just costs money or power or co2 emissions but but it doesn't work in the real world so if you want to train a car to drive itself and you don't have accurately no simulation to turn on this in simulation it's not gonna work you will have the car you'll need a car to drive itself or you


know millions of hours caused thousands of accidents destroyed South multiple times it will have to run out forklift multiple times before it realizes it's a bad idea to another cliff when it starts it doesn't know anything about gravity or anything like that and so it's not practical for the real world although it may be practical in simulation if you can do an accurate enough simulation but


it's gonna cost you a lot in terms of computation so how is it that humans can learn to drive a car in about 20 hours of training for most of us without causing any accidents also for most of us right that's a big question how do humans and animals run so quickly and what happens is there is that it's not supervised running it's not refunds my drowning it's something else and so when


you look at babies you talk to a cognitive scientist development to a cognitive scientist and and you ask them you know when do baby learn basic things like gravity you know when do they learn that objects are supposed to fall I'll tell you around nine months so before nine month old you should be the scenario here but there's a little car on the platform you push it off and the


car appears to float in the air it's a trick of course babies barely pay attention that's another thing they see how is your Kendall stuff doing there that day they learn from every single one of them but that doesn't surprise them after nine months old they've run it by gravity and they look at this like a little girl here they're really really surprised because in the meantime


they've learned that objects are not supposed to you know kind of float in the air they're supposed to Foley's are not supported and so that's actually a trick that a method we just I would say that psychosis is used to identify when babies learn new concepts so you know babies run face-tracking very quickly and you know this is could be you know there are computational models that learn kind of


face detection based on emotion you know self supervise being on motion that learning in minutes so like that could be learn but really quickly the notion of object permanence the fact that when an object is hidden behind another one is still there we don't seem to be born with this but we learn this quite quickly as well so many moles are born with this the distinction between animate and


inanimate objects that's run around three months they're objects whose trajectory are completely predictable and others that are not animate objects and you go gravity inertia a conservation of momentum basically what we call intuitive physics that that comes much later around nine months and it looks as if or maybe that's our hypothesis but you know babies kind of


learn sort of basic concepts kind of a you know step survive stage pieces sort of building more abstract concept on top of simpler ones so for example you know are we born with the concept that the world is three-dimensional or do we learn this I think it's a good hypothesis to think that we learned this a lot of psychologists will tell you we're born with it but I don't see like how the


cortex could be wired to sort of you know you know tell us how to compute depth right although there is certainly some bias in the in the wiring that makes this favorable in the sense that you know connections from the left eye and the right eye actually go to the same place in the cortex so if the if the cortex wants to compute disparity it's easy for it the work the wires are


there okay but the function not not really and so here is how you could compute how you could learn that the world is two-dimensional if you train your visual system to predict what the world is going to look like when you move your head the best explanation for how the world changes is the fact that every pixel every location in the world has a depth right because then you get


parallax motion so implicitly if you want to predict what the wall is going to look like when you move your head you're gonna have to learn that implicitly even if you have no idea that the world is two-dimensional that's the best way to explain how the world changes okay so that's not that's an idea that suggests how we can learn very simple concepts just by learning to


predict essentially and that's going to be the general theme of this talk which is learning to predict prediction is the essence of intelligence in my opinion and so we build models of the world that allows us to learn to drive into in 20 hours to know kinds of stuff but animals do that too so I really love this video of this little baby or wrong song here is being


shown a magic trick where you put a cherry in a cup and and then the cherries removed but he doesn't see that and then the cup is empty and he was on the floor laughing okay so he's his model of the world is obviously being violated and he finds that funny Hey I mean there's the these two things that happen when your model of the world is violated either you find that funny or


you find out or you find it scary because here is something you didn't predict it could kill you in both cases you pay attention okay so that brings us to this this audio cell supervised running this idea of running by prediction okay so not learning a task not running to classify objects in categories that you know come to you from a deus ex machina but learning the


structure of the world by just observing the world essentially so the basic hypothesis of this is or the principle that you can base this on is predict everything from everything else what do I mean by this so let's say you have a piece of data for the sake of concreteness here let's let's think about a video clip for example there's going to be a piece of that data that


you're gonna tell the Machine you can look at it and there's another piece that the Machine pretends it doesn't know it doesn't see here it's the the future frames of the video okay so it looks at the video up to a point and then it tries to predict the rest of the video but it pretends it doesn't know it yet and then it trends itself to predict it of course it can just wait and observe


what's going to happen in the world and it trains and predicts it by just observing what happened another form of this is this it's called a mast cell supervised running you give a piece of data it's very popular in the context of natural language processing these days take a window of text a bunch of words you remove some of the words and you ask the machine to predict the words that


are missing in the process of doing so the machine has to basically develop a representation of language that this allows it to make those predictions and basically in the process of doing this it kind of understands language not completely not deeply but but still but really more generally is the idea of taking a piece of data and asking the machine to predict a piece of it from


the piece that it sees so as I just mentioned this type of learning is extremely has become in the last year has become extremely popular in natural my which processing is actually part about a huge improvement in performance of all natural language processing systems including translation search to Google there's been a series of ideas you know going back to the 90s on this


but really sort of a paper that that convinced everyone that's you know decide to be the thing came up on archive in October last year from from Google from Google Google a I was on Google brain actually and they use a particular type of neural net a gigantic one called a transformer architecture so transformer architecture is kind of a funny kind of neural net where groups of


neurons basically implement some sort of memory module differentiable memory module so they know they don't just compute weight it's on the complete weighted sums but then they compare those weighted sum with with vectors that called keys and then that gives them scores that you normalized to one and then you can compute a linear combination of other vectors and sort of


complicated but it's kind of an associative memory every module in there is an associative memory and you put 40 layers of those with hundreds of millions of parameters you train this on billions of words of text and you train it in the following way you take a window of a few hundred words you take out 15% of the words and you train the machine to just predict the missing


words now the machine cannot do a perfect job at this so what it outputs is for each word there are missing it that puts a probability vector whose size is the size of the of the dictionary and it gives you a probability for every every word right so that's the way it handles uncertainty in the prediction it produces a large probability vector this has completely


revolutionized NLP everybody does this now it works so well that Google deployed this within like in in the last few weeks they basically deployed this as a way of for example if you ask a question to Google it will produce an answer the answer is computed by anything like this Facebook has deployed as a develop things like this for translation and content filtering hate speech detection all


kinds of stuff yes right well yeah here you know what those words are right it's just you you it's not right it's not it's supervised running with two differences what is you don't have a extraneous piece of data us submission to predict so basically you're not asking you to perform any task although that understanding the input data the internal structure of input data second


thing is that prediction cannot be known exactly because you know you can't predict exactly which word is going to go here and so you need to deal with uncertainty and those are crucial key points that so distinguish this from sort of regular supervisor on human okay it doesn't work so well for yes yes it produces a probability vector over all words yeah it's a separate one for every


word by the way so yeah so there's no consistency between if you pick one where you can put another word independently format from that distribution vector yeah so of course people try to do this for images so the equivalent of this would be take an image you know blank out some of the areas of this image and then train a neural net the coefficient net or something to predict the missing parts


the problem with this is that now the distribution of outputs is all is over a high dimensional continuous space and we don't know how to parameterize good distributions over those so those so far have not been very successful not to the extent that those have been successful so the way you use those things is you train this network and then you take the internal representation of language that


those things learn and use it as input to a supervised task his fish detection answering questions you know whatever this group to students at Facebook in Paris who have used this for training your translator so you give a sentence a sentence in English and a sentence in French you remove different words randomly from the two sentences and then you ask the system to translate


and the magic thing is that because some of the words that are removed from the French version are present in English version it learns to produce a representation that is independent of language so what you get in the end is a meaning representation that works for English in French and you have two encoders went for engagement from French Google has a version of this has you


know a hundred languages Facebook now has a version of this that has compiled languages those are massive networks the latest biggest ones are tens of billions of parameters it's just ridiculously large yeah it's inventing on steroids exactly yeah yeah so because you can't you can't pick it you know you can't pick pixels independent of each other so this is a trick that deepmind has


proposed if a former postdoc of my daddy find Carl Gregor which is you make you make the prediction of pixels sequential and you turn it into a classification problem for the great scale where it's you know one among 256 its it just strikes me as wrong you know it kind of work surprisingly well but you know it can be the ultimate answer no I think we'll find something better so there is


actually yeah there is there is a second studies about this actually people who have tried to study what the representation inside so Chris Manning at Stanford has done some some work on this and his various groups it seems that those things actually represent meeting to some extent right it's not a deep understanding of text you know but its shadow because those are words that


are not connected with the real world right i mean the thing only sees text this big question so it has the linguistics community up in arms because it basically you know breaks the entire universe okay of like you know what about grammar what about you know semantics what about all those things is all statistics you know what about symbol manipulation right those things


basically just represent everything by vectors they embed everything in vector spaces and so the the chomskyan linguistics say oh my god they write books against this okay so supervised running you you you train a system with sort of a pretext task which is not really a task it just reconstruction or prediction and as I said as we said I said it works with you well for texts


and symbols and people use this now for like you know DNA sequences all kinds of stuff it's very new images yeah not so much video not so much either signal audio not so much either there is some results in other words so improves the steadily out a little bit but they're not as successful as in NLP NLP they're incredibly successful okay there's another reason why we might want to use


our supervised learning which is and it goes back to this idea of training a car to drive itself the reason why we can learn to drive a car in 20 hours with that crashing most of the time is that we have this model of the world that allows us to predict the consequences the consequences of our actions so we know that if we drive next to a cliff and we turn the wheel to the right


you know the car with we're off to the right it will run off the cliff and crashing on the bottom because we know about gravity and nothing good is going to come out of it so we don't even try right because we had this predictive model we can predict the consequences of some of our actions at least so the way this works is actually a very standard thing for optimal control theory which


is if you have a predictive forward model of the world they gives you the state of the world at time T plus 1 as a function of state the world time T your action and perhaps some latent variable that represents all those stuff you don't know about the world that you can you can sort of run this in your head you can you know run your world model in your head with a proposed sequence of


action and see what the result will be and you can measure the cost of it you know you can have an eternal cost for a good things are you know and I don't want to cry right and so you can you can sort of run this model forward and perhaps infer a action sequence that will minimize your costs right that model will have to be learned with self supervised running


basically here is a state of the world let me take an action and see what the result is or not taking action just because the world is being the world and that's the same problem we need to solve here the self supervised learning problem and the main part the main issue is that the world is not deterministic okay so that it be to this picture of the three paradigms of running if you


want reinforcement learning supervised running and soft supervising the difference is how much feedback information you give to the system at every trial or every sample here is just one scalar here it's just a few bits for example and here it's basically a whole video right it's a huge amount of information you give to the machine so the hope is that you can try and


gigantic networks without them being you know ridiculously overpriced and it will learn a lot of the structure about the world just by observation without actually taking any risk and without you spending money correcting labels that's probably how humans and animals learn so much that might be how common sense emerges right the accumulation of all the background knowledge we have about


the world by that we accumulate by observation that's how that's the basis of common sense essentially so we need to get machines to do this and for I mean I've been sort of advocating for this for a while and you know what Middle's obnoxious you know slides without reinforcement learning being a cherry on the cake of machine learning and supervised reading being the dark matter of AI we don't


know what it is there's actually the dark energy you've more you know it's most of the energy okay so the next revolution is I will not be supervised you will not be reinforced either I saw this from Alyosha of course my colleagues Eterna bamileke who's from Berkeley also in Facebook here says labels are the opium of the machine learning researcher so you know


it's all like revolutionary statements and some dude actually produces a t-shirt that you can buy okay so that gives me two inertia base models which you really really is a proposal for how we approach this problem so the main problem is how do we predict with uncertainty so if I if I do an experiment I'll come back to this if I do an experiment which is I put a pen


on the table and a lady go and in a minute and if I I repeat the experiment all table times and it's not the same video clip every time the pen will fall in the different direction and if I ask you to predict what is the state what is going to be the set of the world in two seconds you can tell that the pen is going to fall but you can't tell really in which direction right most of the


time so if you train a deterministic function to make one prediction the best thing you can do is predict the average of all the possible futures which would be a transparent pen in all possible configurations and if you actually do this you train a system to predict video to pointing video frames you're the fourth frame or the the first four frames of the videos are observed the


last two are predicted you get blurry predictions they're basically the average of all the stuff that could happen and then machines can't decide which one it has to make one prediction yes yes No okay so address all that works ten try to solve the same problem in a different way from the one I'm going to explain but I'll come back to that analogy later okay so the point is if you have an


input a deterministic function that produces one output and some cost function here that measures the discrepancy if this cost function is only zero when when the prediction and the observation are the same then this guy can only predict the average now to me to come back to your point if you make this cost function complicated in such a way that it doesn't compare


points but it compares then you know distribution so example then yes but then that becomes complicated that's what adversarial training is about you have to train this thing to do the to copy all distributions which is hard okay so we're not going to use a deterministic function so here is the the crux of energy based models and it's very connected to things like factor


graphs that the people were talking about in the context of graphical models and Bayesian networks and stuff like that so basically you have an input and observation you have an observed or hypothesis for prediction and you have an energy function here that measures the compatibility between the two if the two are compatible if Y is a good prediction for X then the energy


produced by this is Rho if they are not compatible it's high okay and the way you do inference is that I give you an X you find a why that might be multiple that produces a low energy okay so for example here this is x and y if I give you this X there are two possible answers for why I have low energy right this is the manifold of data that's why they does that our sample form and you


know you'll you'll have to answer so I'm not telling you how to do this minimization and how to produce multiple answers but that's the inference mechanism so the energy here is not used for training is used for inference it's very different so you could say well alright you know that's an energy function but you know you can take the exponential and normalize with a gives


distribution and it gives you a probability yes except that I don't actually want the log of a probability I don't want the energy to be the log of a probability here the probability is a set of measure zero its peak like if I know anything about math right you know it's a thin plate and so the the distribution here would be you know you know infinity on that point on that on


that manifold and 0 just epsilon that side of it which means this energy function or ever you parameterize it will have to have infinite parameters infinite weights something you can set your own net and it's not very useful because you can't do inference with this it becomes a golf course you can do inference what you want is a function that is smooth so that at any point here


the gradient of that function might tell you where to go to get a point from the manifold so this is very emphatic you do not want to learn distributions they're bad for you right maximum rank liquid sucks it just doesn't do the right thing this is big mistake that actually gets to the original formulation insists that you know this should be you know one here zero outside


and that's just it's just a bad idea so that's an example where you know applying probability theory blindly actually is bad for you and I've been trying to snap people out of it for twenty years without success so far okay so this is the what I just showed you is the conditional version and there's an unconditional version where you don't actually have an observation the only


thing you want to know is like model the internal consistency of why and really those two problems are not that different from each other in the first case you know a priori which set of variables are observable in the second case you don't know which part of why it's going to be observed and so here the what the model gives you is kind of a dependency you know a function that


gives you the dependency between y1 and y2 in this case so things like auto encoders or this type yes okay it's akin to negate you look like a hood but you don't want to train you to with maximum maximum okay good or not at least without heavily regularizing it and you don't need the normalization because it's not like you're gonna sample from it anyway so in the end it's just an


energy that's the more elementary concept that you can derive from and you know physicists in the audience know that energy is more fundamental than probabilities probability is going to derive from from it or Hamiltonian if you're a quantum physicist but it's not probability this amplitude well whatever okay so how do we trade on energy base model so of course we're gonna primate


Francis in some way right it's going to be some sort of neural net with a particular architecture and it's gonna have parameters in it and we need to train it in such a way that it takes to shape so that the data the training data we observe take low energy and everything else has higher energy and not specifying that it should be the you know that the difference of energy


should be akin to wish you know difference of love of our abilities insist on that and there are two classes of methods for doing this without contrastive methods and architectural methods so quadratic methods basically consists in pushing down on the energy of points of data points right so give a give a pair XY to the model and twist the parameters so that the energy coming


out of it goes down easy and then the contrastive term which prevents this thing from just collapse into zero everywhere picks points intelligently outside and pushed their energy up okay and the problem is how intelligently intelligent you have to be - can I pick those points and by the way guns are an example of this so again for example where the discriminator is this


energy function and the generator is the smart system that picks out the points whose energy going to push up that's called energy B's gets any paper on this a few years ago and then there is architectural methods and those architectural methods consist in building the energy function in such a way that the volume of stuff they can take low energy is is limited or


minimized okay so either about construction or through some regularization term and I'll come to how you do this in a minute but let's start with okay so there are all kinds of you know traditional and supervised learning methods that you can caste that in that language and I said you know basically the construct the basic idea contrastive methods is push down the energy of data


point push up everywhere else or which what is maximum likelihood does if they have if you have tractable partition function approximation put down the energy of data points and push up on chosen location and maximum likelihood with Monte Carlo Markov chain Monte Carlo an alternate Monte Carlo contrasted divergence metric you learn English for matching all the stuff


basically are different versions of this including gas and the third one is trying to function at most points of the data manifold two points on the data manifold that's called denoising auto-encoder and that's why those NLP it was largely a pea model i was telling you about do that's called a master to encode particular case of denoising auto-encoder I'm gonna mention a little bit metric


learning because it's one of the few cases where it works and it's in fact the only case we know it works in the context of images so it's kind of important and it's those results are reason like last week and then there's architectural methods some of them some of you I'm sure know so I think like PCA so PCA you make sure the whole space is not reconstructed because the


representation is constrained to be a low dimensional k-means just make sure model square is here etcetera these are the ones I'm going to talk about because that's where my money is right now so I think like sparse coding sparse or two encoders which some of you of course I've heard of and then the other ones I'm not gonna mention okay so how does it work in the context of PCA k-means so


in PCA the the the region of the space that is perfectly reconstructed is the principal subspace right in this case here if the data points are sampled from this from the sparrow the principal subspace here dimension 1 is this so this has become our reconstruction error 0 and everything else grows could racket quadratically right because you take a point you project it on this and so if


it's already there reconstruction here at zero if it's here reconstruction error scores that the square of the Euclidean distance not a good model of a spiral as you can tell k-means says so k-means interesting because it has a latent variable in it so the energy function is not directly a function is the minimum of some other more elementary energy function okay so it's


the the min over a vector Z of the squared distance between the data point and the this Z vector multiplied by a matrix whose columns are the prototypes of the the k-means model and you constrain this vector to be a one heart vector okay so only one one only one component can be one the other ones have to be 0 and so you have to do this search exhaustively which is


akin to a nearest neighbor and what you see here those of you black areas are the minima centered on the prototypes ok so KB's just put you know prototypes more or less equally distant over the many fold it looks great in two dimensions and how dimension k-means really doesn't work that well but what's interesting about those both of those cases is that they work because the


capacity the volume of the white space that can take low energy is limited okay that's a key concept or you talked about the maximum likelihood so I'm going to keep that okay so that leads us to this idea of latent variable models right so so energy models f of x and y that are actually defined by minimizing a more elementary energy function e of X Y Z with respect to Z or by marginalizing


over Z which is equivalent to defining f of X Y as minus 1 over beta which you can think of as a new verse temperature log integral over Z exponential minus beta the energy ok this is a log partition function for those of you know and this is a free energy which is like Oh F for physicists in the room so there's the conditional version and the unconditional version which only


consists in taking X out ok so that's that's what the model looks like you have observe variable variable you need to predict and some latent variable you have to minimize over now why is that interesting latent variables are interesting because they are an essential tool for making a system be able to predict multiple outputs instead of just one so if I build a system out


of deterministic functions here I have you know a neural net with a few layers it produces some representation of the observed the observed variables and then I feed this to another neuron that I call the decoder together with a little variable by varying this latent variable over a set I can make the output very over set may be something complicated manifold of this if this network is


complicated and basically that allows me to solve this problem of not really take the average right I can just you know predict the actual thing I'm observing by finding the latent variable that will make my model pretty the best thing so here is how you train an image based model there you show it an x and y you find the Z that minimizes the the reconstruction error okay and if that's


not perfect and we're one step O's to caste gradient descent you update the parameters of whatever functions you're using to make this small okay this works great except there's a there's a slight problem with this which is that imagine that Z has the same dimension as Y okay so in C are the same dimension as Y and the decoder is not you know the generate function then it's kind of a powerful


parameterize function then for any Y you show the Machine there's always going to be a Z that's going to reconstruct it perfectly which means your energy surface can be completely flat it's not a good model dependency of Y on X because your energy function doesn't tell you you know which Y is good so again what we're going to have to do is limit the information capacity of Z like


we did with camis basically we're going to have to limit the volume of white space they can take low energy and that volume will have to be commensurate with the you know the volume of our or data manifold essentially okay but let's start with contrastive embedding so church embedding is following idea to handle the fact that multiple Y's are compatible with X you feed both x and y


to neural nets and those neural nets will have invariances and so you're going to be able to modify Y for a given X without changing the output okay because of the embarrasses built into the system and that's a way of handling the fact that there are multiple ways are compatible with the next but now the way you need to train this is that you need to tell it ok here are two images


they are actually the same you know conceptually so whatever representation you extract from this image should be similar to the representation you extract from this image so basically I want HNH fine here to be as close to each other as possible because really they represent the same thing but if you if you only do this you see you collapse you see basically those Network


completely ignoring the inputs and producing constant vectors right so you need a contrastive term the contractive term which is kind of a subclass so this side you are pushing up the energy of so if you don't want is you show pairs of examples that are dissimilar and then you train those networks to produce outputs that are different from each other and these various loss functions


to do this so this is in the business it's called Siamese neural net and it's another idea but it's been kind of revived more recently it's been successful for training of face recognition system or there's a the paper that just came out that actually beats to actually use some supervised learning in vision to improve performance of a super purely supervised


learning is this one moko and it's using a trick to kind of slow down one of those the main issue here is the difficulty of finding hard- so here you have to mine your entire data set for images that the system thinks are similar to this one but really aren't and that's really where where things become complicated but that idea this paper I just mentioned actually improves


the performance of image recognition systems over purely supervised yes/no nights you know whatever your favorite primary classes but in these cases large compositional nets in fact the name of the architecture is right here this means resonate 50 with four times the size of the future maps resonate 50 is going to a standard architecture for image recognition


so the the bird system that's used for an LP those those mashed auto-encoder or denoising auto-encoder the the diagram looks like this you start with a piece of data you corrupt it which means you remove some pieces you run it through a few layers of neuron that there's a latent variable which is implicit in those models which is like which of the outputs is picked as a function of the


of the probability distribution on the output and then you compare this with the the actual data that you observed and you train the entire system to minimize the reconstruction error and in continued space the conceptually what that does is that if you imagine that your data manifold is this okay those points you take a point you corrupt it so you you add noise to it for example


in this case and then you train the your parameterize neural net to map this input to the output okay you feed this as input and you tell it you should map it here once the system is trained you can actually plot the vector field of you know those are little vectors that point in the direction where the neural net if you feed with this input would would take you I mean you have to


lengthen the the thing but they almost all take you to the to the manifold here and the color here indicates the energy so the energy is low on the manifold which is what you want it's high outside except there's a problem right here there's a ridge here and it's a it's a kind of a flat Ridge which is not good so here the reconstruction error is actually zero because the system when


this train doesn't can't decide whether to go this way or that way so there's a flaw with this with this thing there ways to fix it but no clear the main issue with this is that it doesn't scale with well in high dimension because in high dimension there are many many many ways to be different from a sample and you're never going to explore the entire dimension of the all the space


it's right it could be either but the part I'm abusing the reconstruction error is that here the reconstruction error is zero so which means the energy is zero spent so it's a phantom Low Energy I think there are ways to fix it there are no cheap you sure I'm not gonna go too okay so prediction with latent variables as I told you before I give you an X I'll give you Y you find the Z that minimizes


the reconstruction error and unfortunately because if Z is high-capacity this is gonna give you a flat energy surface so the solution to this is your regular Y Z you basically add a term in the energy along that x RZ where our Z measures you know basically tells you if you are at a particular region of space that you're happy with and so basically you pay a price for


making Z go outside of that region a good example of this which is familiar to many of you is our Z could be the l1 norm of Z so if you put the other one normal Z the sum of the absolute values of the components of Z to make this small you have to basically make many of the components of Z zero as many as possible and so you end up with sparse sparse representation and that actually


limits the the volume of space that is the has low energy essentially this is this is what what you get so this is sort of the unconditional version of it where there is no X is you just modeling Y and here I give you a Y Z the regular as before Z is the other one norm Z is multiplied by a matrix color item matrix he won't decoding decoding matrix it produces a reconstruction you measure


the square Euclidean distance between the two and that's your energy function so a classical in the applied math community at least you could generalize this and so what you get is when you turn this on this little spiral here yeah you get that the low-energy regions are basically piecewise linear approximations of with sort of low dimensionality in your subspaces of the


entire system that works really well in high dimension that's the cool thing and it's been studied you know a lot one thing that is not yet studied is what if you make the decoder nonlinear let's say instead of having a matrix here that you multiply Z by you have an entire neural net what happens I'll tell you about this little bit later now here is the problem though finding you know finding


I'm sorry it's not what I wanted to show finding Z for a given X a pair X Y finding the Z that minimizes the sum of those two terms can be expensive you have to back popping equation today is to go in and decide this could be a non smooth functions you have to do those l1 l2 optimization you know Easter or whatever it can be expensive so one idea is you actually train and you're on that


to predict the optimal solution to that optimization problem okay so ignore the the great part for now I'll give you an X on you I find the Z that minimizes the sum of this and that and then use this as a target to train a neural net which from x and y is going to predict this guy and then if this guy is well trained that I don't need to run the optimization algorithm for inference


anymore I just need to run through the encoder so that becomes very clear that is very important to limit the information content of Z because the system can cheat here it actually has access to the answer and you can just you know copy the answer on the output and so unless you have a way of restricting information content of this asset ml children completely ignore X


okay so if you have the unconditional version of this we don't have this part this is called a regularized auto encoder or sparse auto encoder this works really well in the sense that if you train as possible to encoder like this weather color is linear and the encoder is a few layers of a neural net on the at least the columns of the decoding matrix end up being little parts of characters


which means you can reconstruct any character with a linear combination of a small number of those things people call this call this these things atoms if you try our natural image patches this is a running algorithm running you end up with oriented edge detectors which is great you have to do a little bit of whitening for the images if you do this in a compositional mode where the the


decoder is actually conditional so Z is not a vector it's a bunch of feature maps and then you run them through collisions and you compute the sum and that's how you decode you get beautiful so these are the the business functions in the decoder the kernels that I used to reconstruct the the outputs and these are weights in the first layer of the uncoated encoder on only has two layers


in this case and they're basically mirror images of of the decoder and this is four one two four eight 16 32 64 filters you get a very high diversity of filter centers around gratings oriented edges at various frequencies it's really nice this is these are ten year old results more recently we can revive this technology because it's very interesting so this is again filters that are


learned on natural image patches from the c4 dataset 9x9 kernels and those are corresponding feature maps that are extremely sparse that can reconstruct basically any image on so far with real chip very good accuracy some other work we're doing along those lines is having multi-layer decoders so basically here is an image and then take a bunch of feature mats here run them through


completions and value and commissions on value when we construct and then you can sort of stack multiple layers of those on train this carefully if you know how to do it she's not good not easy but it kind of works these are reconstructions this is the original and these are kind of reconstructions you obtained with sparse representation so if you only reconstruct from here you ignore the


rest you get you get sort of high frequency information if you only reconstruct from here ignore this you just run through this network you can reconstruct a low-resolution version so this is you can think of this as like nonlinear wavelets if you want write the system so naturally learns to represent this let me skip this ok talk about this really quickly so something that's become very


popular in the in the business is something called version of tiny coders and variation over 20 coders are basically auto-encoder models they could be made conditional if you want but I cannot grade this out and they are an example of a model where you also limit the capacity of the of the the representation here in the middle and the way you limit the information


capacity of this vector is that you add noise okay so basically here's a why you run to an encoder you produce a prediction for what the code should be and then you add additive Gaussian noise to it and you run to the decoder and there's a constraint here that it's a penalty really used during learning that the the norm of the outputs of the encoder need needs to be as small as


possible okay so it's l2 regularization if you want during learning now what how does that give me the information content of the code well let's say that you train without noise right so you train your two encoder is going to assign a code this is in code space it's going to assign a code vector to every training sample these are all of the training samples now I'd know as to


those guys you turn them into fuzzy balls okay and those fuzzy balls might overlap so for example this this sample and that sample mind up my end up being confused with each other because when you add noise you can turn one into the other and so the reconstruction error will probably increase so what is the system going to do very easy it's gonna make those fuzzy ball fly away from each


other right so that they don't overlap and that really is not that interesting you know it just makes the norm of the output of the encoder just larger but it you know it doesn't do anything for you so what you do is you play a trick you attach each of those little fuzzy balls to the origin with a spring okay you tell them okay you know you can fly away but not too far so you cannot have to


overlap with each and construct some sort of data manifold if you want and two bubbles will overlap to the extent that the reconstruction error is not dramatic on the output okay so there's a trade-off between the strength of that spring the size of those bubbles which in the case of rational coders autoencoders are maximized actually and and and things like that and if you if you read all the


papers and rational tone colors it's never formulated like this it's formulated as you know some version will lower bound or some probability distribution but it's a mechanical analogy I mean it makes it completely clear this is just a way of reducing the information content to keep the information capacity of the code okay I'm gonna end with an application of of all this which is the problem of


predicting what the world around you is going to do for things like avoiding to bump into other cars for example right so I already talked about this idea that if you have a forward model of the world that gives you the state of the world at time T plus one of the state the function of the state at time T and the action you're gonna take you can sort of roll out a an action in your head with


using this model and there has planned a sequence of action that will minimize your cost here the cost being I want to stay in my lane I don't want to bump into other car I don't want to get too close to any other cars okay and that's a different triple cost so I'm not talking about reinforcement running everything is differentiable everything is computable I don't need to try


anything I mean I don't need to like estimate gradients of stuff by by trial and error everything is differentiable so the part name of course is that this this model of what cars around you are going to do is not deterministic right there's a lot of things that cars around you are going to do that you know you may not predict and so there is a latent variable in the


model that you're going to need to sample which is which is going to parameterize the set of all stupid things that cars are on you can do and and non stupid things as well okay so you start from a state which you observe this is your current state this is what you wear the cars around you are and you sample the certain variable you take an action of your action and then the


system gives you a prediction for where the car and you are gonna be proud chiptune you okay if you decide to turn the wheel the world around you is gonna reel is gonna rotate okay so this is predicting what the world around you is going to look like and then what you could do is you can back propagate gradient from the cost to a network here that is supposed


to predict the correct action from the state so should I turn the wheel should I break sure that accelerate and by sampling multiple samples and running this on different initial conditions you you might have a car that trains itself to drive without actually driving just by just thinking about it having trained is forward modelled by observing all the cars driving so the way we do this is


that there is a camera that looks at a highway from the top and then you track every car and you extract a little rectangle around every car centered on every car and it turns with a car and so that's the world around every car and then you you can record sequences of those little things by tracking every car and that constitutes a training set the set of videos centered on every car


and so you give a few frames of this thing observe frames and you train a system that has written variables and all that stuff to predict the next frame so Z Z we represent all the stuff you can predict that the other cars are gonna do essentially right oh I see it's that's a good question I think it's I think is 256 dimension vector of 206 dimensions so this is so for inference you need to


kind of sample z4 for training Z is given to you by by an encoder basically right you train but you need one of those information capacity reduction here which in our case is done by a combination of adding noise and what we call dropout but it's basically set Z to zero it forces it to be zero half the time and so it tells the system like you know even if you don't have a latent


variable do a good job at predicting whatever you can and then half the time it lets the system use Z and the the latent variable is combined additively with the representation extracted from the predictor so that zero has kind of a special meaning if you want so this is what it produces so this is recording of the real world this is a prediction when you set Z to zero all the time and so


you get blurry predictions and what you see here are I'm gonna restart if it wants to restart so what you see here are four different predictions you know run kind of recursively for different samplings of the Z variables and you see they predict different features and it's indicated by the square on the circle here the indicate cars that do different things


for the different simply the cost function for training this thing is very simple it's you know whether the car is in lane or whether it's and how far it is from its neighbors and so you can train this policy by just by popping in gradient of the cost through the entire system all the way down to the policy Network if you do this it doesn't work because what happens is the system gets


into regions of the space where the foreign model does a really bad job at predicting what happens to have low cost okay so the car can it goes off the road or something like that and this can be due also to flaws in the in a cost function but but basically it doesn't do what you want so what you have to do is regularize it by forcing the system to stay within regions of the forward model


that where the foreign model is pretty sure of its predictions so that the system doesn't try to drive crazy stuff in crazy ways that are not present in the training set and and where its forward model can't really predict what's going to happen accurately and you do this by estimating the uncertainty in the prediction to the forward model by sampling the output of


the forward model with these random variables you can sample like the the drop out of in the network computing the variance of it and then using this as a term in the cost function so it forces the system to stay within a region of space where predictions are fairly reliable with low variance and this is where the system does so so this is the car being driven the green cars are


recorded videos and the white dot indicates whether the car wants to turn accelerate brake etc and they it's perhaps more visible in this example so the yellow car is the car that is in the recorded video the blue car is the one that we are driving and it didn't change lane the problem is that the blue car invisible to the other ones and so it get squeezed and it has to escape


because the other cars will just record it right so they don't they don't see the blue car is another example they're you know they are there is gonna less issues it's trying to stay sort of halfway between the cars in front in the back okay so that's slide I think this whole idea of supervised running is associated machine learning this don't necessarily believe me but that's where


everybody is I think we can learn complex hierarchical feature for low resource tasks which is which is becoming really important using supervised running actually in natural language it works is very important for natural language for example it's important for Facebook to be able to translate Burmese into English or to more precisely to actually train a classification system that detects his


speech in Bernie's because there is a sneek conflicts in in in Myanmar and so you want to be able to detect a speech to prevent bad things from happening but how much data how much training data we have in Burmese so one way to do this is to kind of turn text into a language independent representation and then train a speech detector independently of language it's very important for low


resource languages like Burmese or whatever I mean there is 2,000 language is something that people use on Facebook the advantage of that is that we can train massive networks it can accumulate a lot of background knowledge about the world in an on task dependent way and then we can use several techniques that handle uncertainty by to learn forward models for model-based control and


reinforcement learning model base reinforcement random so my money currently is on energy based approaches latent variable models so that it can handle multi modality regularize rates and viable models to prevent this collapse problem in particular sparse latent variable models although the precise way how to make that sparse is now clear and then latent variable prediction through a


yeah so I think since I'm one of the organizers I'll actually take the opportunity to thank all the speakers and all of you for attending it's been a lot of fun to hear the wide range of perspectives and topics this week and of course also thanks to the Simons Institute for for hosting us the date I noticed is wrong here actually obviously it's November 20th not October 20th

Dynamic Pricing in Ride-Sharing Platforms