But I would say that the original hypothesis would in fact predict that you could train those things because you've pruned away these entire modules, which is very coarse grained and that leaves still good weights in the bad parts.


When BERT Plays the Lottery, All Tickets Are Winning (Paper Explained)

Okay, so they conclude, however, we can see that both good and bad networks can be retrained with comparable performance for many tasks. The inverted bad networks perform worse than the sampled ones, but that could be due to them being smaller in size.


Performance of all inverted bad networks on call is almost zero. Okay, very little remains when that mask is inverted. That's the task we looked at because they claim that's so small, which makes sense, right?


So discussion, say does BERT have bad subnetworks? The key result of this study is that as far as fine tuning is concerned, BERT does not seem to have bad subnetworks that cannot be retrained to relatively good performance level, suggesting that the weight that do not survive pruning are not just inactive.


However, it is important to remember that we consider elements of BERT architecture as atomic units, while the original lottery ticket work relied on magnitude pruning of individual weights. So they're well aware here of these differences, and they concede to that right here.


So that's good. On that level, BERT probably does have bad subnetworks, and they show that can be found in the transformer model with global iterative pruning.


We leave it to future research to find out to what extent the effective subnetworks overlap with the effective architectural blocks, and what that says about the architecture of BERT and the other transformers.


So as you see, they're well aware that all of what I said is the case. So it's not like I'm criticizing and saying they're wrong.


It's just that if you read it, you sort of get the impression that this is what they're saying.


And I think the light of which a reader goes through it is just a bit such that you come off, if you don't read until here, you come off thinking something else.


Our results suggest that most architecture blocks of BERT are potentially usable in fine tuning. This should not be interpreted as proof that they all encode potentially irrelevant linguistic information.


That's absolutely true. It is also possible that pre-training somehow simply made them more amenable to optimization, which is another question for future research.


And I go into what do different BERT components do in the different things. So again, I think this work here is actually most relevant for investigating this question.


What do BERT components, the different BERT components do for the different tasks to look which tasks use which things.


And the actual recognition that none of these modules is useless, I would consider pretty cool finding.


Okay, so in conclusion, they say prior work shows that it was possible to prune most self-attention ads. We extend this to the fully connected layers.


We show fine tuned BERTs as good and bad top networks where the good heads and MLPs alone reach performance comparable with the full network and the bad ones do not perform well.


However, this pattern does not quite conform to lottery ticket hypothesis. Both good and bad networks can be fine tuned separately to reach comparable performance.


We also show that 86% of heads and 57% of MLPs and good subnetworks are not universally useful across glue tasks and overlap between good and subnetworks do not necessarily correspond to task types.


So that's where we didn't go into.


This raises questions about the degree to which fine tuned BERT relies on task specific or general linguistic knowledge and opens up the possibility of studying the good subnetworks to see what types of knowledge BERT actually relies on at inference time.


So this is sort of future research direction.


And with that, I think we've gone through the paper. I hope you got something useful out of this.


I think it's a pretty cool paper. It's a pretty cool methodology. And I think a lot of work can build upon this to do interesting analysis of these language models.


Descending through a Crowded Valley -- Benchmarking Deep Learning Optimizers (Paper Explained)

Hi there, today we'll look at descending through a crowded valley benchmarking deep learning


optimizers by Robin Schmidt, Frank Schneider and Philip Henning of the University of Tübingen.


So this paper is an empirical investigation a benchmark into optimization algorithms for


deep learning. The short story of the paper is use Adam, it's fine. The long story is


a bit more complicated. And the resulting answer is basically we still don't know even


after this paper, if there is a single good recipe for optimizing deep learning. And if


so, which one it is, and where it works and where it doesn't work. A lot of things are


still unclear. And I think the biggest lesson from this paper is that probably the best


thing you can do is pick Adam or SGD with momentum, tune it a little bit, and whatever


comes out of that is probably doing okay. So let's dive into the abstract here. But


first, as always, if you like content like this, don't hesitate to share it out. And


also tell me what you think in the comments. With this paper, we're going to see that there


is a big room for interpretation here. So you're going to see experimental results.


And the experimental results, they can always be interpreted in the light of different hypotheses


that you have what's going on. And very often, you have to pay careful attention that something


like Occam's razor, that you obey something like Occam's razor, sometimes people try to


read a lot into their experimental results, when a much simpler explanation would actually


be sufficient. Not that much with this paper, but you're going to see a lot of results,


they can be interpreted in a lot of ways. So yeah, tell me what you think in the comments,


happy to have a discussion about this and hear your thoughts. So they say choosing the


optimizer is considered to be among the most crucial design decisions in deep learning.


And it's not an easy one. The growing literature now lists hundreds of optimization methods.


In the absence of clear theoretical guidelines, guidance and conclusive empirical evidence,


the decision is often made based on anecdotes. So I'm just going to show you they have actually


a list in the appendix, they are tracking this optimization algorithm, you already see


this is massive, right? So you have things in here like, you know, Nesterov and Polyak,


which are very, very senior in the field. But as you can see, a lot of algorithms popping


up in 2016, 2018, 2019, 2020. And it's poly atom power SGD. And all of them have their


respective paper here SGD, look at that going strong 70 years. So you can see that this


is almost an impossible list of things to consider when you choose when you choose your


optimization algorithm. And it seems like it's just getting it's just getting worse.


They have this graph over here, where they count how many times each of the major optimization


algorithms has been cited. 2020 is shorter because the year is not over yet. I was kind


of surprised as well. Like, wait a minute. It can't be that our field is shrinking, this


will never happen, surely. But it's just because I think the year isn't over yet, or wasn't


at the point where this paper was written. But you can see the popular optimization algorithms


are mentioned more and more and more. And also the non popular optimization algorithms,


they seem to multiply over the years, as we've seen from the list. So choosing one is hard.


What this paper does is it doesn't compare all of them. So they choose a list of 14 different


optimization algorithms. Oh, they also attract these learning rate schedules, which is also


ridiculous. It's like, oh, no, but we don't we don't do a constant factor decay, we do


multi step decay. And all of this makes all the difference. Remember that each of these


papers that some Okay, sometimes it's just been suggested in a paper, but especially


for the optimization methods. Most of these papers are about the optimization methods,


right? They are saying this is a new optimization method. It's good for either all of deep learning


or as particular subset particular algorithms or settings. And it's better than than everything


that came before either it's faster or uses less memory or something like this. So all


of these, all of these are papers that suggest some kind of new algorithm and show that it's


better in their paper, you'll always find that their algorithm is better. And having


read and try to reimplement and so on a bunch of these paper, I can tell you that not a


lot of the papers are, let's say all of them in their experiments is of course better.


But that's not a recipe for for taking the optimizer and applying it to other problems.


It always looks good in the papers. And that's why independent benchmarks like this are valuable.


You see the decay decay rates for the learning rate or learning rate schedule, it's not always


decaying. So here are the things that they actually consider. These are what they consider


the popular algorithms. So you have things like add a delta, add a grad, add them. You


have things like look ahead momentum, which is SGD plus momentum. You have RMS prop, just


plain SGD, and so on. And you can see each of those comes with its set of hyper parameters.


So for example, in pretty much all the methods, you have a learning rate, which here they


call alpha. And in the momentum, you additionally have the momentum term, which is here called


what's that row. Of course, in other methods, like in look ahead random, you have a slew


of hyper parameters that you can all tune all these hyper parameters come with their


default setting. And the authors here, additionally define a tuning distribution over which they


search. So I'm going to criticize this work here quite a bit. Remember, most of what I


say in the criticism is actually acknowledged by the paper itself in their limitations,


which is much to their credit, right? It's so just because I criticize it, it's very


easy to criticize empirical studies, investigations, especially benchmarks, especially comparisons.


Most of it is addressed by the paper, which is a very, very good. It's very, very nice


for a paper to be honest about its shortcomings. And yeah, just keep that in mind. So the first


criticism I have is what they're going to do is for each of those things, they're going


to compare three settings. So in the first setting, wow, that's a big pen. In the first


setting, it's one shot. So they just say, we are going to take the optimizer, let's


say Adam, and we're just going to plug in the default parameters for it. And we just


let it run and see how well that does. Okay. And the second is with tuning a little to


the call this, I think that the small budget tuning small budget, and then the third one


is the tuning with a large budget. And the difference is simply that you try more things


in the large, in the large, in the large budget. And you take the best one according to your


validation metric, and then you let it evaluate it on the test metric. We'll get to that in


a second. My point here is that there's two things. So first of all, they do a lot of


experiments with in this setting one, and they make a lot of claims about it. And this


setting one is entirely dependent on the default parameters given either by the authors, or


by let's say popular frameworks, which often take them from the authors, which it's okay,


like most people are going to use it and put some like use the default parameters. But


I would argue investigating the default parameters in this kind of setting where you compare


optimizers is kind of useless. What I would expect from a benchmark like this is to determine


its own default parameters like to determine okay, what are what parameters are the best


maybe you take you have your what you're going to see is they do a benchmark over different


deep learning problems, you take half of them, and you determine what single set of parameters


works best on half of them. And then you evaluate, say, that's the default parameters for the


other half or something like this comparing just out of the box default parameters, it


might just mean that the default parameters, the authors haven't really spent time worrying


about it and simply released a bunch of code. And by simple simply changing the default


parameters, you can improve it, you're going to see that the second one is here over the


tuning ranges. So for each of these, the authors define tuning ranges. So ranges where these


tuning algorithms are going to search over, they are going to do random search. And here,


for example, this is a log uniform distribution, the L U. So it's going to search from 10 to


the negative four to one, which of course is 10 to the zero in log space. So it means


it samples, it's a it kind of samples the exponent on a uniform scale, and then it plugs


that in, which is, you know, good. That's how we do it in research. However, look at


compare, for example, you have something like Adam, where the default parameters tend to


the negative three. And you have something like momentum, where the default learning


rate is 10 to the negative two, yet, the range here is the same. And that's they make this


clear, they say, when the authors don't give a range to search over, we simply take over


the range from a different from what is commonly done for that parameter from a different method,


which you can see that 10 to the negative two is exactly in the middle of this log uniform


range, however, 10 to the negative three isn't. So when you already make the case that you


use the default parameters, you really, I think, have to make sure that the range you


search over the default parameter is kind of in the middle of that range. Otherwise,


your range is kind of kind of not according to you know, the default parameter. So that's,


that's kind of already slight criticisms of this paper. And you can already see I'm not


telling you that to trash the paper, I'm telling you this to this is extremely hard, like to


benchmark optimization algorithms with hyper parameters with different hyper parameters


with different amounts of hyper parameters is super duper duper duper hard. Okay? Like


everything influences the results here, what the default parameters are, what the ranges


here are, how big the ranges are, right? If you make them too big, your search is going


to spend a lot of time in in regions where nothing's happening, how how often you search


in them. So let's say, what you what a lot of people do in Adam is they keep these constant,


but they just tune the learning rate a lot. So how much you tune each parameter is important.


How many parameters are there are is important. All of these things like if you have to search


over four parameters, it's going to be much noisier results than if you just have to search


over two parameters and so on. So this already, as you can see, is a is a hard, hard, hard


task. And this says nothing yet about the learning rate schedules that they also try.


Where is it? Okay, they they try four different learning rate schedules, which, again, can


be tuned, though I think they don't tune them here. And they do so on 14. No, sorry, on


eight different on eight different problems. So there are eight different problems. Where


are they listed right here, there are eight different problems. So you have what they


call small models over here. These are like artificial data quadratic noisy quadratic,


a small mnist VAE, small convnets, as I understand it. And then you have what they call large


problems, which is a CIFAR 100, CNN, SVHN, character, RNN, and so on. You might already


notice that also in this department in the problems department that they search over,


these are very particular kinds of problem and that they acknowledge this as well. There's


like no reinforcement learning, no GANs and so on. And they are not that big, even the


even the large ones, they are kind of small. And of course, they are doing grid search,


you know how much compute they spend doing this benchmarking stuff, you can't benchmark


models like GPT-3. On the other hand, we know we know for a fact that there are effects


of scale that quality make, there is a qualitative difference between large models and small


models and ever larger models, you can't simply extrapolate from small models because they


have very different properties. It's also a relation to how big your data is in relation


to your model. So my kind of criticism here is that we are searching, oh, here are the


problems. Yeah, you see that there are eight problems, the bottom ones they call large,


the top ones they call small. We are searching over a very small set subset of deep learning


problems, namely, and this is something I pointed out already, I think, a few videos


ago. If, like, let's consider all of these things small models compared to something


like ImageNet model or a big, big translation model or something like this. Let's consider


these small. If I have a small model, I can do grid search, no problem, I can tune, I


can try out all my optimizers. If I have a, sorry, if I have a large problem, I can't.


Yet these studies, they only tell me something about small models. And we already know it's


very difficult to extrapolate from small models to large models. We know that there are effects


in batch sizes, new transformer models on TPUs train with batch sizes of 4000 or something


like this. The epochs, we know that, for example, self supervised pre training train with much,


much, much higher epoch counts than classic supervised learning and so on. This is, so


this tells you something about a very tiny subset of problems, about a tiny subset of


optimizers on these particular problems. And it is highly dependent on how you exactly


set up these experiments. So we finally go to how they combine this, we've seen what


optimizers they choose. And we've seen what problems they apply them to. So they here,


how do you select an optimizer? Now, where was the thing that I was going to? Yeah, so


when they when they tune after so the one shot setting is they just take the default


parameters, which I already said I criticize you should determine good default parameters


overall problem and that be the default parameters and then yeah, but I guess they they go after


what people do people just plug it in. And first thing they try is the default parameters.


So what they do is they when they tune, they tune over these ranges that we've seen, they


say we only use a single seed for tuning. Okay, so they set the random seed of an experiment


to a particular point. And then they tune, for example, the learning rate, always starting


with the same random seed. And they look at the validation loss for that random seed.


And then once they have the best learning rate, they repeat the best setting 10 times


using different seeds. So they train they tune, tuning is done in a single seed, but


testing is done. Testing is done using different seeds. Okay. They say right here, that progressing


this way has the feature that our tuning process can sometimes pick lucky seeds, which do not


perform as well when averaging over multiple runs. This is arguably a good reflection of


reality, which is true, right. But the inherent problem here is that. So what's the danger?


The danger is that you have a lost landscape, whatever, and you start maybe here, okay,


that's your random seed where you start, and you tune the different learning rates like


going down, down more down, that's too much, and so on. Okay. So when you start there,


one algorithm might look very good, an algorithm that is suited to starting at the edge of


like a cliff, but only there, like that algorithm might perform very poorly anywhere else in


the landscape. So you this this is your tuning seed, and you tune that and and the learning


rate and algorithm you determine are performing fairly well. And then you take that same setting


that learning rate you determined, and you started from different places, right from


here from here from here from here, and all of a sudden, this performs very, very crappy.


However, a different learning rate might have done, or a different algorithm might have


done very, very well. So maybe for the red one, you determined a small learning rate


is actually pretty good, because I'm right at this edge of a cliff. And the small learning


rate, you know, prevents me from going there. And this small learning rate looks pretty


good in the validation loss, but then you start from here from here from here. And the


small learning rate, it does nothing from here. It just blows. And so you get you get


what I mean, you can get very unlucky in this tuning seed. And while it's true, that this


is correct, this is happening in the real world, this is not suitable for a benchmark,


right. So keep in mind that these benchmark results, it could just be the entirety of


it of a test outcome for a given algorithm could just be due to the fact that the tuning


seed was crap. Because even though the test runs are averaged, the tuning is done on one


particular seed. Okay, I would argue that they say yes, if we used all 10 random seeds


for tuning as well, it would drastically increase cost not only for this benchmark rendering


practically infeasible, but also as an approach for the practical user. Look, look, I agree,


I agree. But this is not like it's really necessary in something like this to to to use


different randoms is because what you want to show in the benchmark is how this algorithm


is doing on average, right? Because the benchmark is supposed to inform future users. However,


right now the benchmark is like a single user that can be lucky or unlucky, right? It's not


informative. And I see the point what they're saying is that it would make this benchmark


infeasible. However, it doesn't change the fact that it's necessary in the benchmark,


any experiment that you do is like a fraction. Okay, the fraction down here is cost. And


it's like dollars spent or time spent or whatever. And the fraction and the and in the the numerator


is going to be maybe something like information. Information, the information that you gain


from an experiment. Now what what there are, it not all experiments are the same, right?


You can't you can't just say, Well, we use as much we use as much cost in our experiments


as the people who invented resnets, right? Maybe maybe you do that. Maybe it's actually


true. Maybe they actually use more because they do this giant grid search, like our experiments


cost more than resnets. So therefore, they should be respected even more than the experiments


who figured out resnets, which is not true, because you have to pay attention to the numerator


right here, which is the information that you gain from an experiment. And if you do


it like this, yes, your cost is lower, but your information like goes to towards zero,


in my opinion, not to know, it's not zero, but it is very small, because you have this


one seed per algorithm that you bind everything to. So the entire benchmark can just get lucky


or unlucky with a particular algorithm. Okay, so that is, that is kind of my biggest criticism


with the tuning right here. So let's go into the results. I think enough me brabbling about


the setup right here, they have these deep learning problems, they have these 14 algorithms,


the learning rate schedules, they come in later, but they're not really prominent in


the benchmark. What they do is they compare the algorithms with the default parameters


with a small amount of tuning, and with a large amount of tuning. And this is one of


the main results right here. Let's actually look at this particular thing here a bit more.


So what you see as the read, the way you read this is these numbers represent algorithms,


you can see it beside them. But you know, you can't see it down here, but they represent


the same algorithm. So one here is AMS bound is also one here. On the left on the y axis,


you have the one shot performing algorithms. And on the x axis, you have the same algorithms,


if they are given a small budget to tune. So if we analyze one of those, for example,


number let's call let's go numbers, number four and five, so number four and five, number


four and five. So four is Ada delta and five is AdaGrad. What we can say if we look at,


for example, let's look at this number right here. We see that what's this five, number


five, so AdaGrad, AdaGrad is 40% better than Ada delta when it is allowed when it is given


a small budget to tune. So when AdaGrad is given a small budget to tune itself, it is


40% 44% better than Ada delta when it is not given a budget to tune itself. All right,


I hope that that kind of so we compare having tuning budget to not having tuning budget.


And this is the absolute test set performance improvement after switching from any untuned


also you don't see that from any untuned optimizer to any tuned optimizer. So the y axis are


the untuned and the x axis are the tuned and you already see a lot of kind of different


effects right here. So you see that sometimes which is interesting in in the red right here,


these are negative numbers. So sometimes an algorithm, even given a small budget to tune


is actually worse than a different algorithm when doing the default parameters and this


is on one of these small problems on one of these small c410 problems. Okay, you so that's


one interesting thing. But I would argue it's it's actually not that meaningful for reasons


for which I'll, I'll get to in a second. The most prominent thing probably you'll see is


that there are rows that are kind of colored very uniformly. So you have, for example,


this row, which is solid green, and then you have other rows which are, you know, very


either light or even red and so on. So what's going on here? What does a solid green row


mean, especially look at these high numbers like 4543 4344. So there, this is performance


improvement. It means that add a delta is when not tuned, is this much worse than any


of the algorithms with if given a small budget. So it's default parameters suck, suck badly.


Okay, that's, that's the message right here. If you see like a solid green row, the default


parameters of this method suck badly. Now, I'm, as I said, what the value of this is,


it actually maybe this is the most valuable thing that comes out of this comes out of


this benchmark, honestly, because everything else is so noisy, right? In theory, I would


say this is the least valuable thing, because let's just, you know, get good default parameters


for all this stuff. And then we're done. But apparently, this is not done yet. So add a


delta as default parameters, at least given in the paper, apparently, they suck. So does


momentum, though, does polyac give or Nesterov, whoever invented it, give momentum, default


parameters, maybe, maybe those were different times certainly didn't give default parameters


for deep learning. But you see, again, they like the default parameters suck. What is


also interesting is to look at the diagonal, okay, so the diagonal shows you how much the


same algorithm improves if given a budget. Again, you can make an inference about the


default parameters when you say, okay, add a delta improves over itself by 40%. If just


given a little bit of budget to tune, while add a grad is only improving 2.3%. There are


situations in other graphs where there's actually a negative, negative values, you can see,


for example, right here, there is a negative value in a different problem in the C for


100. And they can show in the appendix that this is due to not enough tuning. So basically,


the tuning is just a random search. And the random search is, again, this is the random


search is so bad, that it doesn't even hit the the the any, any sort of setting where


the default parameters are present. So all its search space is basically bad parameters,


which again, is you can say that the algorithm is not really robust to parameter change,


but you can also say that this is entirely due to the choice of search space to search


over. So you can see that the algorithms five, seven, eight, and 13 are particularly bad


at this. Here we see that's add a grad LA 13. What RMS prop? Yeah. But then if you look


at other problems, you see that different algorithms, okay, the number seven here is


also kind of kind of shady. So look ahead seems to be kind of shady in general. But


this also switches from problem to problem, which is something I already introduced, there's


a lot of noise here, a lot of noise. And therefore, yeah, what is a bit harder to parse out is


how the algorithms compared to each other. So in order to determine that what you have


to do is you just have to look at relative performance. So for example, take a any column,


any column, for example, this column right here, you see that no matter how high the


number is, it's always a bit smaller than the rest of the row. Okay, so in every row,


this is smaller than the rest of the row, which means that number four, what's number


four, add a delta, when you tune at a delta, it compares less favorably to all the other


algorithms than when you tune other algorithms. Okay, that's so in order to really compare


optimizers to each other in this graph, you have to kind of do this relative math in your


head. And that's why I'm saying the red, the negative numbers aren't even that important,


as long as they're not on the diagonal, right? If they're on the diagonal, they mean, if


you tune the same algorithm, it's worse than when you just run the default parameters,


which is just means that your search sucked. Or your random seed is is is somehow lucky


or unlucky. What do I know? But the negative numbers off diagonal don't mean anything that


the fact that they're negative, because what you would expect is that the small budget


always increases at least in expectation over the one shot. Okay, the question is then how


much would you expect it to increase? So even though a number like 0.3, here is a positive


number, which means that the small budget number two improves over the one shot number


11. This could still be a bad thing, because you'd say, well, if I give you a small budget,


I expect any algorithm to improve like 2% or 3% or 5%, something like this, right? That's


why you have to look at the at the relatives with respect to the other algorithms, you


can't really look at the absolute numbers right here. So even the negative numbers don't


mean anything, because zero has no meaning here, except on the diagonal. Because you


would always act even like even on the diagonal, you always expect some kind of improvement


from tuning. And we need to know kind of this average expected improvement before we can


make judgments about the numbers in here. What you can see is that some algorithms clearly


underperform with respect to the others, at least in this particular problem. Again, this


is highly problem dependent. So I'll add a delta pretty bad. Then what's this right here?


This is four, five, six, seven. Again, look ahead with momentum, look ahead momentum pretty


bad. And you can find others. And this again, varies from problem to problem, though, numbers


four and seven are pretty bad here. Numbers four and seven here also five. Yeah, so you


kind of see that you can make some conclusions about these problems. But here, look at that.


So here they now include the they now include the schedules. And here you start out one


shot with a constant schedule. If you add some of these schedules, it goes up a little


bit. This is the median, right? And this orange stuff is the what is it the 25th to 75th percentile.


Look at the amount of noise right here. So when you see these plots, it's just I feel


it's quite, quite helpless. Okay. Again, when you look at these plots, so what they give


you right here is the red bars are whatever Adam does when it's tuned. So when you tune


Adam, and then let it run over these 10 different test seeds, this is the range it gets. And


this the other lines are simply the mean across the other optimizers when you tune them, you


can see just from the spread of Adam, that the order in which these lines appear mean


almost nothing except here when they like crash like horribly. It just probably means


that these optimizers, some optimizers just aren't made for some problems. But other than


that, the order here is kind of useless. And you see the downward facing triangle is always


untuned Adam, which in most cases performs fairly, fairly well compared to the others


and compared to the noise you have over the different over the different tuning outcomes.


So that's why I said at the beginning, use Adam, it's probably fine, tune it a little


bit, if you realize it doesn't work at all, then switch to something like SGD with momentum,


or the other way around, use SGD with momentum, if you realize it just screws up, maybe try


Adam. And that's actually a thing they say as well. So one of their conclusions is one


of their conclusions is that instead of tuning a single optimizer tuning helps about as much


as trying other optimizers. And they repeat this point throughout the paper, it's instead


of trying a different settings for a single optimizer, it you can get the same kind of


outcome by simply trying a bunch of different optimizers in their default settings, and


then picking the best one of those, which it's, you know, the, the entire literature


seems to point to, whatever you do, it's probably fine if you take one of these generic algorithms


and kind of do whatever it whatever to select a good thing. Let's assume for a minute that


all of these algorithms are the same, and you simply change the algorithm instead of


tuning the learning rate. Well, these algorithms come with different default learning rates,


right? These all these algorithms come with different default learning rates. And the


learning rate goes into the algorithm in a different way. So the effective learning rate,


even if I put in the same number, the effective learning rate is going to be different for


each algorithm. So maybe what their their effect here, when they say, it's the same


when you tune the parameters, or when you simply pick a different default parameterized


optimization algorithm, maybe what you're doing is the same thing, maybe all these algorithms


are actually kind of the same. And overall, right for a particular problem, it's different,


but overall, they're kind of the same. And when you pick a different algorithm, you simply


pick a different learning rate for the same algorithm in disguise, because the learning


rate, the default learning rate for that algorithm goes into its formula a bit different. And


ultimately, you're simply tuning as well. So the the benchmark is extensive. Again,


I don't want to rag on this paper, the benchmark is super extensive, they also do rerun stability


and so on. But it this paper shows that it is possible to do an extensive, extensive


search extensive benchmark that is still largely useless. And I don't I don't want to say that,


because they because they, what I don't want to say is, they didn't determine a clear winner,


therefore it's useless. That's not what I'm saying. I'm saying the information content


that I can get out of these experiments, especially for situations where it would help me like


for where I can't do grid search is close close to zero. I think the two big things


that the community can learn from these papers is one, the default settings for some of these


things are crap in the papers and maybe maybe in our frameworks. So maybe we'll go over


that once more. And two, as like, at least on these small kind of problems, it seems


not that important which algorithm you pick, pick one that you like, tune it a little bit,


and you're probably good to go. If it doesn't work, pick another one. So that was it for


this paper. Again, tell me what you think. What worked for you. If you have horror stories


with optimization algorithm, they used to be much more, much more prevalent. I think


also our, our advances in architectures have made it easier for optimization algorithms.


So like something like ResNet, giving you really nice gradient flow has made it much


more easy to optimize the network as a whole and therefore the optimization algorithms aren't


as important. And the other the last comment I want to make here is that a lot of a lot


of these papers, as I said, they deal with specific situations like oh, if you have low


memory or or if you have that or they say our algorithm is really good, but only only


if you add like a bit of Gaussian noise on the input or only if you use this very exotic


learning rate scheduler or something like this, which this paper of course, hasn't done.


This is still a very small subset. So yeah, these are these are common criticisms for


benchmarks. I think we'll take from it what it is. It is a cool paper. It is extensive.


Building our Neural Network - Deep Learning and Neural Networks with Python and Pytorch p.3

What is going on everybody and welcome back to part three of our deep learning with?


Python and Pytorch tutorials in this tutorial


We are going to be actually building our neural network and at least really basically explaining how we pass data


And then in the next tutorial we'll actually train the neural network


So in the previous tutorial we actually got our data kind of explained things like normalizing and balancing and stuff like that


And now we're going to actually build the model itself so in order for us to do that


Let me make a new cell here, and we're gonna call this class or actually we're gonna make a couple new imports


So we want to import torch import torch dot nn as nn


And then we're going to import torch dot nn dot nn dot


functional as


F so the way I understand this is most things are kind of in both of these these


versions of Pytorch torch dot nn is more of like your object-oriented programming and then torch dot nn dot


functional is more of just like functions and


In general we're going to be writing object-oriented programming Pytorch


But there's going to be times where you actually just want to run a specific function


So you'll see what I mean, but in general these two libraries are interchangeable


So if it exists in nn it exists in functional and so on and just keep in mind that like with functional


You have to always like pass parameters whereas like with nn it would just you would initialize things right and so


Some things to my knowledge don't actually exist in both but anyways


Most code that I've seen ends up using both in theory


I think you could write everything as a function in Pytorch, but I'm not sure why you would but anyway


I'm not a Pytorch expert, but basically these two things are very very similar and


Depending on exactly what you're trying to build you might use


Something from just nn or something from the functional, but anyways carrying on


so now


Now we'll build our neural network, so we're gonna say class capital n


Net it is going to inherit from nn module, and then we're gonna say define in it


Self and we are going to it first initialize nn dot module, so we're gonna say


super in it and


If you don't know what this does basically all this does is like super


Corresponds to nn dot module and all this is doing is running the initialization for nn dot module as well as whatever else we happen to


Put in in it so when you inherit


You inherit like the methods and attributes


so long as they weren't set and in it I suppose anyways you inherit at least the methods and


But the thing is the initialization method is not run


So if you want the initialization method to run of your parent class that you're inheriting from you run super in it if that's fuzzy


To you for any reason go to the text based version of this tutorial


I show a couple examples with classes and importing or not importing inheriting so it should clear it up there


But if you have further questions feel free to make a comment or ask in the discord


so


Let me do


I


guess


Trying to decide what I think we'll just start defining layers. There's something else. I want to talk about here in general, but


First we'll define our layers so in our init method. This is where we're gonna just define


we're just going to define the


The fully connected layers to the neural network and then in another method we actually define how data passes through so what I'm gonna say


Is self dot FC one so FC stands for fully connected?


And then the one is just our first fully connected layer


So we're gonna say self dot fully connected one equals n n dot and then we're gonna say linear


And this is going to take basically you've got input and then output to each layer


So the input is quite literally our images now. I happen to know the images when flattened or 784


But you might not so where does this number come from and this is kind of one of my?


complaints with both


TensorFlow and PyTorch docs is this sometimes they like throw these numbers, and it's like whoa


Where did that come from and they like half explained, but then they miss certain things so especially when we get to the convolutional neural network


I will point that out again, but anyways


784 is 28 times 28 because our images are 28 by 28


images and we can't pass the image instead we pass the


Flattened image so you can think of pixels. They're like rows right so you've got rows of pixels and when we flatten those


What we're doing is you got your first row and then your second rows down here


Well, we actually take that second row and we put it over here


So so eventually all of the rows just make a flat row of pixels, so that's what we refer to generally as flatten


So the input will be a flattened 28 by 28. What's the output well our target here is to make a


Three layers of 64 neurons for our hidden layers


So the output will be 64 so for every fully connected layer


We're going to have input and the input is whatever is coming into this to this


Layer so in this case the input is is the actual input layer which happens to be the flattened image the output can be


Whatever we want so we could say this if we wanted, but we're we're going with 64 so


Finally the next thing is nn linear this just means fully connected


It's just a flat simple you might also when we do a convolutional neural network


It'll be nn.com and so on so


So cool. I think I've described everything there so now we're going to say pasta pasta


And then we'll do one more pasta, so then we're going to say two three four


so keep in mind though that certain things have changed when we go from image to


Actual code because this kind of this is this is the extent of our input layer, right?


And this number here is kind of the extent of our hidden layer the hidden layers are almost in between these calls or something


but anyways


fully connected to


Takes in 64 because the previous layer output 64 so this one has to take in 64


But but now what it outputs we could pick anything we want. I'm going to go with 64 though, so that means


Layer 3 has to take in 64 again. We could output anything we want


I'm gonna go with 64 again, so this final layer 4 takes in 64


But we said this was a 3 by 64 so you got 64 64 64


And then this is our output layer, so I'm gonna say 10 where does 10 come from we have 10 classes, right?


We've got 0 1 2 3 4 all the way to 9 so this is our our output layer only has 10 neurons


Cool, so that is our init method. It's done. We actually could define our network now


So we could say net equals capital N. Net and then we can print net


Cool, so we get to see this because we get some special methods from


module so


What might happen though if we forgot super in it right?


So part of learning to work with any of the deep learning libraries is getting used to seeing some of the errors and working


Through them so if we ran this we would get the following attribute error cannot assign module before module that init call


This is not the most helpful error in the world you would have no idea what the heck that means


But it means you forgot you're super in it now normally


It wouldn't be commented out right you would you wouldn't have even had that line there, so it wouldn't be totally obvious now


I'm sure if you google this you might find the answer


But yeah, so as I think about common errors that you might hit


I'll try to show them but for the most part you can google them or you can post a comment or


Join the discord and ask


Cool, so continuing along so we've defined our layers


But what we don't have is any path for data to take through these layers


So a simple neural network is what's referred to as a feed forward neural network


It means data passes in one direction. It just goes one side to the other side


nothing fancy so


For our for or for feed forward


It's a lot of f-words for feed forward. We're gonna say define forward and


I'm gonna make some space here


And what we're gonna say here is self, and then the input is X so the input is the data


This is the the actual method that defines how the data will flow through our network


You know we could have called these layers anything we wanted we just said FC one two three four


But if we wanted we could actually attempt it would fail because this is 64


But we could attempt to actually start by passing data through fully connected for again


It's not gonna work because the size that fully connected for is expecting is a 28 by 28


So it wouldn't work out


But we could if we change this to be 28 by 28, but the order that we've defined these layers is


irrelevant


Just for the record, so how do we want data to pass through here?


This is the cool thing about PyTorch that if you're just starting out. You've never worked with any other library


Is going to be probably lost on you, but you get to choose anything you want so we're gonna define something pretty simple


We're just gonna say X as it passes through we're gonna say first X passes through fully connected one


So we're gonna say self dot FC one


X


That's it, and then we're gonna do the same thing


We're just gonna say hey X is gonna pass through all the layers right so we're gonna say pop up up


And then we'll say two three four so X equals fully connected X so it redefines X


And then we pass it again, and then we pass it again, and then we pass it again


In theory you could embed like all this stuff. It would be like one really long line


It'd be so ugly, but you could do it and then return X


So in theory I mean this would work this would run you could pass data


It probably won't learn why won't it learn because data won't be scaled properly most likely why won't it be scaled properly?


We forgot our activation function


But you wouldn't I don't think it would error to be honest with you


I'm pretty sure we could run this and we could attempt to adjust weights and biases and stuff like that


But it's not gonna work out well because we don't have an activation function so what we're gonna say instead is


We're gonna pass


capital F


dot


Rel U which stands for rectified linear?


This is our activation function, so we're gonna run literally an activation function over that entire layer


but but keep just remember this is part of the helper miss of


pi torch


I've never seen anybody do it, but in theory you you could have I think different activation functions per neuron


I've never seen it done. I can't imagine why one would do that, but that's possible


especially if you made such a large because


I'm just getting way ahead of things never mind anyways rectified linear is


The activation function if you forget what the activation function is this is whether or not the neuron is


Firing like the human brain right so or you know the neural networks kind of came from trying to model brains


The brain however as far as I as far as science knows is really more of a stepper function the neuron is either


Firing or it's not firing


In the case of most


Most activation functions, it's not a stepper function


It's a sigmoid so it's a range between zero and one but regardless of what activation function


It is it keeps the outputs of these layers from just just exploding into these like crazy numbers it can still happen


Especially when we go to like compute gradients and loss and stuff like that these explosions can still occur


But we do everything we can to try to not have these


magnificent explosions


So and if that doesn't make sense to you don't worry


We will continue ratcheting up, and you'll soon learn about loss explosion


So so okay, so we want to actually run this on every single layer so in fact


I'm just going to copy and paste again, so plus plus plus and


Two three and actually we don't want to run that on four


We just actually want to pass the data. We'll just pass it just so everything's nice and explicit


so


four


So for these like three layers. We're and keep in mind the activation function kind of runs on on this side, right?


It doesn't run on the input data, so that's being run on this 64 this 64 this 64, but then on this 10


We're not actually running anything on those neurons yet. We don't want to run rectified linear here


Because that's not the actual task again the neural network


Doesn't really know anything about anything so when we talk about optimizers and back propagation


And all this stuff you'll I'll bring this up, and I'll show you price and beautiful imagery


But the thing to keep in mind is the neural network really is pretty stupid like it's not actually the smartest thing ever


It's just when you combine all these things it just can do amazing things


It's like ants or something right like a lot of little basic stuff can do really complicated things


But on this fully this thick output layer. We don't really want to have


rectified linear because that doesn't really make sense instead what we want is some sort of function that


Constrains to what we actually want and what we actually want is only one of these neurons to be fully fired


Right our goal like let's say we've got ten output


I don't really want to write it, but let's say we've got cats dogs and humans right we want app outputs like


Cat dog human let's say zero zero one we would want our output layer if we passed a human


We would want it to say zero zero one right that is our goal these two are off this one is fully on


But we would also say this would be acceptable right so if it was like 78 percent human and then zero point two two


Dog I don't remember what order I said things were but anyways


We want a


Distribution like that so when you're working with something that's like a multi-class


So if it it's not a regression algorithm where we're trying to like predict some sort of range you can do regression with neural networks


That's just not what we're talking about right now


What we're talking about is we've got multiple classes and what we really want is a probability distribution on the output


So for a probability distribution, we're going to use log softmax so kind of like


When we talk later about our loss, there's different losses that you might use optimizer


Generally everyone uses the same optimizer like we do with rect like with your hidden layers pretty much everybody uses rectified linear


For the optimizer pretty much everyone uses the same optimizer


But it's when you have different types of what you want is the output


This is where things kind of change so with multi-class usually softmax works


So what we're going to do instead is we're going to apply a softmax to this X


So we'll just throw that into our return statement, so we'll return f dot log underscore softmax


X and then we have this other thing that we have to throw in here


Which is the dimension on which we want to apply softmax now for an output layer?


That is of the shape that our output layer is


Which will always be flat if we're doing things this way with linear layers


You just you just assume. It's always going to be dim one, but what is this so dimension equals one is I believe?


similar to


Axis so if you're coming from like numpy or pandas you can kind of think of it like axes


I think I could be totally mistaken, but dim one just me


It's like what which thing is the the probability distribution that we want to sum to one?


so in this case we're going to be passing batches of data, so the output layer will be a


Batch of tensor distributions now. What do we want the probability?


What do we want this probability distribution to be a distribution of?


I don't know anyways. What do we want to sum to one basically?


It's not all of the batches right so like all the batches aren't the probability distribution


We care about we care about the classes themselves, so this will be a batch of tensors


so if we said dim zero we would be like distributing across the


Batches so dim one is we're distributing across the actual output layer


the output layer


the output layer tensors, so anyways dim one I probably did a terrible job explaining that


but like I said you can think of it like an axes or just say dim one


And don't worry about it to be honest because I can't imagine you're going to change that


I don't even know what the like the default value might be dim one. I really don't know


anyway


Cool, so that's our neural network


And let's go ahead and run that make sure oops good that would have been a problem


Let's go ahead and run that okay cool, so it looks like everything sort of works


Let's actually pass some data through the neural network though


So what we're going to do is first we need some data, so we're going to say x equals


x equals


Let's say let's try to mimic an actual image that we might have


So we're gonna say capital X equals torch dot brand


And then we're going to say it's a 28 by


Cool, let's just look at it real quick. Okay, so man my nose is super itchy


Okay, so that's pretty meaningless to us, but yeah those are some random values


then we could in theory pass this to our neural network, but


recall


Actually, let's pass it. Let's see what happens because it's I think it will yell at us


Let's just say output equals


Net and then we're literally going to pass X


This is probably going to error yes


So we come down here, and we see here that we have it some sort of size


Mismatch okay, so it says 28 by 28


So and then the input is a


784 so again it has to be flattened first


So what is one thing that we could say well we could say


X equals


X dot view


And then we'll just say 28 times 28 run that run this again, and we're like dude. What the heck?


right and again


We get some sort of error that if you're new to this you're gonna not understand


What the hey this thing is talking about so so instead what we have to do and in fact?


I don't think I'll get away with this, but let's try this


And just pass it. I'm just curious if I get away with it. I don't think I would but no so you have to


format things exactly how


These libraries want them in this case. We're going to say we actually want to view as a negative 1


28 by 28 let me run that make sure that works it sure does


The negative one just specifies


That this is a these will be this this input will be of an unknown shape


We also should be able to get away with 1 by 28 by 28 we do


And in fact let me just output. Let me just paste it real quick


cool, so


So either way whether you do negative 1 or negative 1


So either way whether you do negative 1 or 1


You still get the exact same answer. Why is this?


Because this actually is a 1 by 28 by 28, but in theory this could be many more right so


You could have a 3 by 28 by 28 or something like that right like so you're this will be whatever your batch size is


So we can say negative 1 because we're basically just saying it's of any size


Just be prepared for any amount of data to be passed through don't worry about it. It doesn't matter and


So so this just is an array that is of any size or a tensor rather


But


Then we just pass it through and then what is this these are the actual predictions, right?


So this is the prediction for a 0 1 2 3 4 and so on it just goes on


Now there's a couple of things that we didn't quite do we haven't like initialized weights randomly


We're also


The first few passes of information through your neural network are going to be worthless like they're not actually meaningful in any way


But now you've seen we passed data through the neural network


And we got a return now what we have to talk about is what do we do with it right?


So you see gradient or grad FN?


that stands for gradient function and when we pass information through what we're gonna do next is


Calculate how far off were we so what was the number that we passed who knows we don't I don't actually know but


We're gonna say is the goal. Let's say it was a one it would be this this here though


So hopefully this was the arg max and humans we think about things in terms of whether or not you were right or wrong


But there's degrees right if you were like slight if you think it's a one


But you're not really sure right that's different than if you're like a hundred percent positive. That's a one


Well if it's not a one you're way more wrong in the second case than the first case


And that's important if we want to slowly adjust and figure out how we can be right more often


so


Cool, so I think that's probably all I really want to talk about


Up to this point. I don't really think there's anything else


That I want to show here so a


Quick shout out to my channel members who have been with me now for six months half a year


That's a long time Howard Liang fidgets Kirill Groshenkov


Cheeky buns Frank Lloyd jr. And Mohammed Al Moore, thank you guys very much for your support


It allows me to do what I love to do, so thank you guys very much


Okay, so that's it for now. I feel like there was one more thing. I wanted to talk about and I honestly don't remember


Hopefully I remember at some point, but yeah


There was something really important that I wanted to tell you guys and I just it's eating at me excited


There's something I want to say can't remember. I remembered what it was


Now I got to edit things okay, so


This will be way more important like down the line


But one thing to think about is this here this forward function that we've done especially if you're coming from any other learning library


You can do fancy things here, so you could you know you could say


if


Weather equals sunny right or something like that


Do these things at this at this point else?


Right do some other things right so you can actually you can throw logic into this forward method to do


Whatever you want so you could actually come up with some really advanced models and yes


you can do this in other languages you can branch out and do fancy things and


You can do it, but it's way more challenging with Pytorch


It's just that simple and the gradients are automatically calc like what you actually ran through and what it needs to adjust


That's really cool. I don't know if we'll ever see a


I don't know if we'll actually I'd love to come up with like a really cool example where we actually do that


One example that I always think about or at least that I thought about when we when I figured out you could do that


Is like the python plays Grand Theft Auto so this is an agent that needs to be able to drive really well


But also do things like if it's on foot figure out how to steal a car or maybe do some shooting or something like that


But the primary task of that in real life is to figure out how to do that


But the primary task of that neural network at first is just analyze and process imagery


So the first few layers could be image processing layers that are going to be the same whether you're driving


You're on foot or whatever and then the subsequent layers could be very specific to the task that thing is doing so to think like


Robotics or you know other things in general where the first few layers that the general task is always the same


And then the next few layers could be very specific to exactly what the task is so anyway


That was the thing that I meant to bring up and I kept forgetting and I knew there was something I knew there was something


Okay, that's it for now if you can't tell I'm really excited about that if you're not excited


Then you haven't worked with other libraries in that is much more challenging


Concept Learning with Energy-Based Models (Paper Explained)

Hi there, what you're seeing here is an energy-based model that learns the


concept of a shape from a demonstration on the left. So on the left you can see


demonstration of data points sampled from a shape, in these cases circles or


squares, and then the corresponding energy function that the model infers


from that and then it can replicate that shape on the right using that energy


function. So the paper we're going to analyze today is called Concept Learning


with Energy-based Models by Igor Mordach of OpenAI and this is a very cool paper


or at least I think it's a very cool paper but it is also a very hard paper.


So therefore first I want to kind of make a bit of an introduction into the


concepts that we are facing in this paper. So the first thing you need to


know are energy functions or energy-based models. What is an energy


function? An energy function, sometimes called E, is simply a function with one


or multiple inputs, let's call them X, and you can make the... if the energy function


is happy with X it will be the value 0 and if the energy function is not happy


with X it will be a high value like larger than 0. So this is happy, this is


not happy. So let's give some examples of this. We can formulate almost any


machine learning problem in terms of an energy function. Let's say we have a


classifier. The classifier is... takes as an input an image here, maybe of a cat.


And a label. So if the label is cat then the energy will be 0 if the energy


function is of course working correctly. And if we give the energy function


the same image but we give it a wrong label, dog, then it is very high. In the


case of the classifier of course we can simply take the loss function as the


energy function and we automatically get an energy based model. So the loss


function here would be something like the negative log probability of the...


sorry, of the correct class. But in any case it is just going to be a high number,


let's call it 10 to the 9. So the energy function says, ha, this is very... this is


very bad. This thing here is very bad, the entire thing you input. It won't tell you


yet what's bad about it. So that also means you can change any of the two


things to make the classifier happy. Now usually we're concerned with changing


the label, right? It's like, tell me which other label do I need to input to make


you happy? And if we make the labels differentiable, of course we never input


the true label, we actually input like a distribution, softmax distribution over


labels and that's differentiable. We can use gradient descent to update the dog


label. We can use gradient descent to find a label that would make the energy


function more happy. So we could use gradient descent to get the cat level if


we had a good classifier. But we can also, we can also optimize the image to make


it compatible with the dog label, right? That's things that if you ever saw Deep


Dream or something like this, those models do exactly that. They optimize the


input image for a particular label and there you can view the entire neural


network including the loss function as the energy function. So what's another


example? Another example is let's say you have a k-means model and the energy


function is simply input a data point and for the data point what you're going to


do is you're going to find the min cluster index, the min k over, you know,


you have your multiple clusters here and your data point might be here. So you're


going to find the cluster that's closest and then the distance here, this distance


d, will be the energy of that. So the model is very happy when your data point


comes from one of the clusters but your model is not happy when the data point


is far away and that would be the cost function of the k-means function. So


that's an energy-based model too. Now currently energy-based models have come


into fashion through things like GANs or any sort of noise contrastive


estimation. So in a GAN what you have is you have a discriminator and the


discriminator will basically learn a function to differentiate data from


non-data. So that by itself is an energy function. The discriminator will learn a


function and that function will be low wherever the discriminator thinks there


is data. So it will usually do this around the data points. So the data


points form the valleys right here and then the generator will basically take


that discriminator function and will try to infer points that are also in these


valleys to produce points that are also in the valleys. And then you basically


have an energy learning competition. The discriminator now tries to push down on


the energy where the true data is and push up on the energy where the


generated data is and that will give you basically a steeper energy-based


function in the future. I hope so in this case the discriminator neural network is


the energy function and the degenerator just tries to produce data that is


compatible with that energy function. So I hope that concept of what an energy


function is a bit clear. Again any machine learning problem can be


formulated in terms of an energy function. Now what is not done so far is


what we alluded to a little bit before in the classifier example and also here.


So right now when we want to train again we simply take the generator to produce


data. Now what's the generator's goal? The generator's goal is to hit those valleys


in the energy function. And we produce a generator in one shot produce this


data. But what we could also do is of course we could just start somewhere. Let's say


here we pick a random data point and then we use gradient descent because the


energy function in this case is smooth. We use gradient descent to just drop down


this valley and then find ourselves in this valley. So without ever training a


generator we can use this methods to produce points that are in the valley of


the energy function. And this I don't know if people I guess people have


trained GANs like this. The reason why it doesn't work let's say in the real world


is because that procedure will just produce adversarial examples for the


discriminator and those usually look like nothing like data because if you


keep the discriminator just stable and gradient descent against it what you'll


get isn't really qualitatively good. But in principle if the discriminator was a


good energy function for the data to describe the data we could use gradient


descent. The same up here. In order to find a good label for an image given


that we have a good energy function. So this is that. We could simply gradient


descent on the label in order to find a better in order to find a better label.


So in this paper we're going to have a situation where we say we're given an


energy function and we're given a bunch of inputs. They are then called X, A and


W. And if I have my energy function already if I have given my energy


function and I have given two of those three things any two right I can infer


the last thing simply by gradient descent on my energy function because I


know the energy function is zero when all of these when the energy function is


happy with the input. So when all of these things agree basically the energy


function is happy it will output zero otherwise it will output a high value.


Therefore if I'm given any of those two any two of those three things I can find


a compatible third thing by descending. And then of course over here in these


machine learning problems the task was always actually to learn an energy


function. So usually in the training data set we are given images and labels and we


want to learn this energy function which would be parameterized so we want to


learn the parameters. And the same here in our general case if we are now given


three things but we are not given the parameters of the energy function we


don't know what those are. As long as we're given all of the inputs in our


training data set and our training data set guarantees these are actually you


know these are inputs that are compatible with each other the energy function


should be low we can simply gradient descent on the parameters of the energy


function. So in a sense there are four things right there are these three


inputs and then there are the parameters of the energy function. If we're given


any three of those four we can gradient descent on the rest and that's going to


be the basis. So the X here is going to be the so-called state and the state in


this paper is going to be images of entities. The entities sorry it's not


going to be images but the entities are these little circles that you're going


to see and each of those entities can have an X position a Y position and I


believe a color so R G and B. So each of those can have that and then the


concatenation of all of those attributes is one big vector and that is your X


that's your state. So state is number of entities and their attributes. A is going


to be an attention mask over the state. So A is going to be here you have four


entities so A will have four entries telling you which of these entities you


should pay attention to right now and W is going to be a concept vector so


called. So W is going to be the embedding of a concept. Now what a concept is in


this case is very general. I can give you an example one a concept is do any of do


the entities that the A pays attention to are they close to each other. So in


this case you see we have two entities that A has a high value on and this is


this ball up here and this ball down here. Now if the concept vector is the


embedding for the concept of being close to each other then the energy function


would be very happy if those two things are close to each other and it would be


very unhappy if those two things aren't close to each other but in the very same


situations of the same X the same attention mask but a different concept


so a different W vector right here then the the energy function would be maybe


very happy if the two things are far apart and maybe unhappy if the two


things are close. So the question is always how are the three things that you


put into the energy function compatible with each other and given all but one of


these things you can infer the other. So let's say you have a perfect energy


function for this all of these for the situation you're just given the


energy function you can trust it and you are given let's make an example you are


given the X so you're given the state I'm going to draw the state down here


right okay this is the state and you are given the W and the W is the embedding


it's a vector but the in embedding space but the embedding is for a line right so


the geometric the geometric unit of a line now your task is to find a the


attention mask that will make the energy function happy and as you can see right


here what you would do is you would put a lot of weight on this this this and


this ball and no weight on that ball because those make a line and since


everything here is differentiable so the state is differentiable the attention is


differentiable and the concept or vectors they're differentiable you can


use gradient descent to find that another example if you're given again


the same W so line and you are given this following thing and you are given


now you're given the attention on these three and you say please find the X


please find the X the state that makes this energy function happy now this here


you would call the starting state the X zero your your task is going to be find


the X one find the state how do you have to change this state such that the


energy function is happy and of course the answer is going to be is to push


this ball here inward until it is in the middle of the two others so the three


form a line right these three form a line you you don't you don't have to do


anything to this ball up here because there is no attention on it and the


attention it's only is the concept for the things that you put attention on and


the state are those three in agreement then the energy function is happy okay


we have covered the basics now let's dive into the paper I think this is the


longest introduction ever but I think it will pay off once you see so they they


specifically or this this author I think it's a single author identifies two


different things that you can do with an energy function here of course you can


do more as we saw but they identified to so here is where you have given the


initial state and an attention mask and you want to find the X one the state


that satisfies the concept and attention the most this the author calls generation


as you can see here these four things that you have the attention on are


pushed around until they make a square because the concept right now is square


and in the other case where you are given this X zero index one just call


this X right here just call this thing X if you're given those two and you are


given the concept square and your task with finding a the attention mask of


course you're going to put the attention on these right here and that is going to


happen through gradient descent again we're not learning a model to give you


that attention like in again we're learning a generator to just one shot


give it to you right now what we're going to do is we're going to gradient


descent optimize on our smooth energy function to give us that perfect


attention mask that satisfies the energy function alright so this is the


difference right here gradient descent is part of the output procedure of the


model usually we just use it to learn and we learn a one-shot model but here


gradient descent is part of the model so they introduce energy functions here and


they say okay we can have a policy on X so if we're given a concept W and if


we're given an a we can have a policy over X which basically means we can find


X's that are compatible with that by running gradient descent here you see


there is an X K minus one and we are running gradient descent on the energy


function with respect to X to find a better X that satisfies the energy


function given those inputs and the same if we want to find an attention mask we


are running gradient descent on the attention mask again in order to satisfy


the same energy function so you see the inputs are both times the same the


concept here we can input square here we can input square but the difference is


what we're running gradient descent on and what we keep constant and I would


get I would add a third line here actually because we can also if we're


given an X and an a we can also infer a W and that's going to be an integral


part so if I have this right here and this situation and I have say I have


attention on these four now I can ask the model so I'm given X and I'm given a


I can ask the model to infer W and the model should ideally output ha this is


square now the model isn't going to output square the model is going to


output a vector representation of square right so the model is going to output


square but as a vector of numbers because that's how we've trained it W


isn't it is the embedding but what we can then do later is we can say okay I'm


not going to tell you it's a square you just come up with a vector W to


describes this situation and now I'm going to take that vector W that you


came up with miss mister or mrs. model and I'm going to take tell you a new


situation this situation right here and I'm going to now give you X and I'm


going to give you the W that you yourself have output and now please tell


me what's the a and then the model is of course supposed to tell you oh these


four here are the a so without without ever telling that it should be a square


what you can do is you can let the model infer a W from one example situation


and then transfer that W to a new situation so it can identify you can


just say whatever concept I have up here please apply that same concept which is


the W down here and this is the entire paper now this is the concept learning


through energy based models okay so that is kind of a third line I would add down


here you can infer a concept vector if you're given the X and the a so in order


to do all this their energy function is going to be a so-called relational neural


network so what you'll have is you'll have a simple neural network a


multi-layer perceptron that always connects two entities to each other with


the concept vector and then this is a belief a sigmoid that connects the


attention masks of the two and you simply sum over all pairs of two entries


in your model and then you send that through an MLP sorry through an MLP


again this I believe is not so important it's just important that they can feed


this entire situation the X the A and the W they can basically feed into a


neural network and the neural network comes up with a number of how well those


three things fit together and then you can transfer these concepts that's


pretty cool now the only question is of course we've always said we're given an


energy function we're just we just have it but of course this is a neural


network and the neural network has parameters and the parameters we don't


know what good parameters are at the beginning so we need to train this thing


and again the reason why these are toy problems right here is I mean we'll get


to why it's computational but this is kind of a new field I believe in machine


learning at least I come from classical machine learning and we only ever have


used like SGD to train and we only ever have produced models that one shot


produce something and here we this is a I believe this a new concept where you


use gradient descent as part of the output and that makes a lot of trouble


so that's why we work in toy problems so what this this here is the situation I


described you have a demo event where you're given the X and the A and you're


supposed to infer the W so the question here is what's the W and the model will


come up with a W and you're not gonna do anything you know right now you're


simply going to take that W and tell it oh well here is a so-called test event


so please apply the W you came up with in this test event and please find me


the the a in this case that satisfies the W and the X I give you here and of


course the a right here is as you can see even you don't know that it's a


square and the actual concept here is move the gray ball to the middle of the


square right that that is it here but no one has told me this I just looked at


the picture so the the correct answer here would be to place attention on


those four things and then to take this thing and move it to the middle right


here in in the in this over here so that would be the correct answer now the


question is how do you train something like this and they they show that they


so this is the loss function right here the loss function is they give you a a


concept and an initial situation and you're supposed to infer the X 1 and the


a and the loss function is simply the negative log likelihood of that but what


does that mean so we'll we'll make it easier if if you have this this procedure


right here where you have a demo event this up here this is demo and this is a


test event how are you going this entire procedure how are you going to learn the


energy function well in this case this entire procedure this entire thing is


one training sample sample but usually we have input and label and now here


it's much more complicated because so we have input okay that's this X and this a


cool but then we have SGD as integral part of the procedure to determine the


W and now what we could do is just apply a loss to the W but we don't because we


don't know what the embedding space for the concepts is we could maybe train a


classifier but in this case we want to train the ability to transfer these


concepts so our training sample needs to be one time transferring a concept so SGD


for one is part of our process here and not only that but then this this X here


of course is also part of our training sample right this up here is X 0 and this


here is X 1 and now we need to find this a this attention mask and that is an SGD


again remember inferring anything through the energy function is a gradient


descent process so ultimately our one training example consists of X 0 a at


the beginning so let's call that a 0 it consists of the SGD procedure to find W


it consists of X 1 and it consists of the SGD procedure to find a the a 1 the


output a and then that will give us the output a the a 1 so this here is our


input in the classical machinery this would be our X and this here would be


our label Y and that's what we train on we train so such that the output right


here the a this is of course sorry this is of course the Y hat this is what we


predict and in the training sample we just write a little generator that will


you know make this situation that knows what the concept is ready will say ok


I'm gonna make an example for a square then it will make this will make the


attention mask for a square and then it will make the new situation again with a


square but not tell us the attention mask there and it will make the


attention mask into the true why so at the end we can compare what our model


output the attention mask we output here without ever knowing that this should be


a square and we have the true label which comes out of the generator that at


the beginning decided that it should be a square and then the loss the distance


between those two that's our loss this is an this is an enormous procedure to


get a loss and most crucially you have to back propagate through optimization


procedures and this is something that we just can't do yet in our models if you


take an image at resnet 50 right right now we do one forward propagation to get


a label in this procedure if you had to back propagate through the optimization


procedure for each sample you would need to basically back propagate through 50


forward passes of the resnet if you if your optimization procedure is 50 steps


long and that is just not feasible right now so that's why we don't do it but I


believe maybe once we find a smart way of back propping through optimization


procedures a whole lot of these things will become the new a new wave in


machine learning I'm excited by this I'm pretty sure it doesn't work yet and this


is very figly fiddly work but I'm excited by the prospect that we can do this so


this is the training procedure right you were given x0 x1 and a and you optimize


in order to infer the concept behind it right the generator that your level


generator of your training data it knows the concept it has a concept in mind


when it generated this but you're not telling your model what the concept is


it needs to infer that and then using the thing that the model inferred you


can either give it x0 and x1 and infer a or you can give it the X and the a and


infer X you can do either of those right these are called identification or


generation respectively and then you compare the output here to what the


generator at the beginning thought again it's not telling you it's that's because


that's the label and you compare this to that and that will be your loss to train


your energy function parameters so your training samples if you think of this


entire thing as one forward pass of the model then it's just classic machine


learning right you have a training sample which is one forward pass and you


have a corresponding label that you infer so let's jump to the experiments


right here the experiments are actually pretty cool so what they've done is for


example have taken the concept of being far apart from something now being far


apart so that the little X needs to be as far away as possible from the ball


that has the attention on it right so if you do generation and you start the


little X right here and you ask the model where please infer the next state


of the world it will push that little X away right here and in color you can see


the energy function valleys of the position of the X so it pushes it away


from this thing but if you take the same concept embedding the concept embedding


of being far away but you don't do generation you do identification which


means you infer the a then it will simply tell you that this ball right


here is the furthest away from the X right so you can do all sorts of things


like this and transferring concepts I find this here pretty interesting so


they had have two different concepts one concept is red as an identification you


need to identify the red ball but the other concept is you need to turn


something red right you need to take a ball that is maybe not blue and of


course the color you can gradient descent on the colors you need to make


it red and since the energy function it just takes three input X A and W it


doesn't you you're not going to tell it right now in which situation you are it


has to create create this W embedding space through learning and if you do it


with those two concepts then it will put the make something red concept and the


is something red concepts in the same places so this is a PCA and in blue I


think these blue is the attention codes for identify the red things and in red


are the generation code for make something red and they will be put in


the same place which is pretty cool it means that the energy function really


learns the feature of something being red I find this pretty pretty neat and


then here they have some experiments where they basically show we need that


gradient descent optimization procedure because only after many steps will will


the energy function basically be aligned with the concept that you want so


if you have a zero shot model like just one forward pass as we do here you'll


see that the energy function that is supposed to make a circle from samples


right this is the example concept right here it if you just have a one-shot


model it will it cannot but in this case at least it doesn't learn to one-shot


produce only if you optimize for a few steps will it get this so you optimize


that inference time and that seems to be very important you can see again here


demonstrations of this so the example is this and then the model as you can see


after 20 steps learn optimizes the points to go to these locations whereas


after only one step it didn't do that yet so there are complex things at work


here and this column here is where you don't have a relational neural network


so you can't basically capture dependencies between things so you you


have no chance of making a square because you don't know where the things


are in relation to each other but that's more of an engineering question their


point is basically that if you have models that do optimization at


inference time they are much more powerful than models that just do a


one-shot forward pass it's sort of like an autoregressive model in NLP versus a


non-order aggressive model that produces all words at once if you produce all


words of a sentence at once no word can depend on any other word and you can


just come produce independent or you can just you produce independent things


which will make the sentence often not make any sense they also have this KL


objective which is a regularizer which I believe that's just a trial and error


they built it in because but it is a regularizer I don't want to really go


into that and then they they do demonstration in and they reenact it on


a robot the demonstration here is that there is a situation where two things


have attention on and you're supposed to move something into the middle of the


two things so that's the content you don't tell the robot the concept it needs


to learn that from data and then infer that this is the concept that you want


and then transfer that to the other environment now you know this it look


you know there's this robot environment but ultimately they still encode the


positions of these things and the position of that and really all you have


to do different here is that instead of moving this actuator directly you need


to like calculate what you need to do to the individual joints in the robot so


I think this is maybe because it's open AI and it needs to you know look robot


and stuff but the problem here is not really different it's it's not even it's


not real world transfer or anything so yeah let's go through some of the


things they can learn with this so you can see here they can learn these


regional geometric shapes and so on the left is the example event that the model


needs to take the concept from now this is this is I believe very much


identification so what they did is they trained with a data set where all of


these appear right so there are squares there are lines there are circles so


this is maybe my criticism here that it is not so much to generally infer a


concept it is more like identify the concept so the model basically just


needs to decide is this line is this circle or is this square because that's


was those things were in the training data set it would be nice to see how


this generalizes to general concepts or if we can even make that if we can have


a zero shot concept inference and then transfer those concepts to other things


maybe that's already happening I don't I don't know so here the spatial


arrangement is to either be close to something or to be between two things so


if the attention is on two things you want in between so you see the top ones


are the demonstrations it needs to recognize the concept and it needs to


basically optimize to fulfill that concept shapes so to make shapes is hmm


oh yeah there's a triangle right again this this this this very much I believe


relies on recognition and not actual understanding of what a triangle is


here you have proximity being closer being far apart what else is cool oh


yeah do you have the recognition for the same task right you need to identify the


ball that is closer for and here you really also see the optimization


procedure in action where for example at the beginning of each flicker you kind


of see the attention being everywhere and then stabilizing to one or two


points so if two points are equally close or far apart you'll see the


attention being on multiple points which is pretty cool right so that means the


model really learns this this this concept here's the count quantity so you


can either have one two or larger than three or something yeah that seems like


they tried three and four and didn't work so they just said oh we'll just do


larger than three and here is this robot thing where it also always needs to move


in between now this this is the part that I'm not really impressed with but


you know whatever whatever you want okay I hope this was a good introduction to


energy functions what you can do with them what I think of them and of this


paper it is a pretty cool paper yes it only works on toy problems so far but I


believe this is one interesting direction of future machine learning


and something yet to be very much explored if you like this content please


subscribe tell all of your friends about it share and I'll see you next time bye


Object-Centric Learning with Slot Attention (Paper Explained)

Hi there, today we'll look at Object-Centric Learning with Slot Attention by Francesco


Locatelli, Thomas Kipf and others of Google Brain, ETH Zurich and MPI.


On a high level, this paper recognizes scenes of objects from single pixels and it's best


I show you a picture of what's going on.


So you have scenes like this, where there is some sort of an arrangement of objects


and there are multiple tasks you can do here, specifically they consider the task of unsupervised


recognition of objects which they call object discovery and supervised classification of


objects.


The difficulty being that these are sets of objects so there is no ordering to the sets.


They do this via a thing they call slot attention that basically is a permutation invariant


attention mechanism over these objects in both the supervised and unsupervised domain


and they do this in a fashion where they iteratively route the attention in order to make the different


slots compete for attention over these objects.


So that's the sort of high level.


If you are in this field, you probably know right now what's going on.


If you're not, we'll dive into it together.


So stay tuned.


If you like content like this, consider sharing it out, leaving a like or tell me what you


think about it in the comment.


I appreciate any suggestion for making these videos better and so people can learn more


from it.


Alright, so the problem I've already described the problem a little bit but let's go a bit


deeper here.


You have images like this and the images we're considering are going to be images that have


some sort of arrangement of objects or what we humans would call objects.


In this case, you can see there is this gray square, not sorry, this gray cube right here.


There is a smaller green cube and then there is a yellow cylinder.


Now in the task of object discovery, what you're supposed to do is you're simply supposed


to say that there is an object right here, there is an object about here, and there is


an object here, right?


So basically, you're supposed to point to the pixels where there are objects and you're


supposed to segment the objects from each other.


You can see right here that this model, we don't know how it works yet, but it separates


the left cube here, the bottom cube here, and the top right cylinder right here.


In the task of set prediction, you're supposed to say what objects there are.


So you're supposed to say there is a gray cube right here, a green cube right here,


and there is a yellow cylinder right there.


Actually, you don't have to say where they are, I guess.


There are many different variants of this task.


But mainly, you're supposed to classify them, meaning you have to say there is a gray cube


there.


I believe in this case, it's with coordinates, but you can do it without the difficulty here,


of course, being that these are sets.


So there is no natural order in it.


So if you say there is a green cube and a yellow cylinder, it's going to be the same


as there is a yellow cylinder and a green cube.


So you have to build an architecture that is somehow invariant with respect to the labels.


And we've seen a lot of the concepts in this video in this paper before.


This video is sort of a a kind of a mash together of different concepts of other places.


So what you'll see is, for example, this property of the fact that here you see are the labels


for these objects.


This could be there is a green cube.


There is a gray cube.


And you'll have to come up with an architecture that if here you predict that green cube,


you consider it correct, even though the corresponding label isn't the one for the green cube.


And we saw this, for example, in this DETR architecture by Facebook, where they use a


matching loss, but we'll get into that.


Okay, so these are the tasks, the tasks are object discovery and set prediction.


So how does this paper deal with this, they use this thing called a slot attention module.


Now the slot attention module is in essence, it's pretty simple.


What it does is it has these different slots right here, as you can see, and it divides


the input into features.


So you can see there is a CNN encoder because we're working with pixels, it's natural that


we want to encode these into a CNN.


The CNN will probably downsample the image a bit and divide subdivided into this grid


right here.


So you have a fairly coarse grid, the grid is actually not a bit finer than you see here.


This is just for example.


But you'll have ultimately a number of features.


So each pixel right here is going to be a feature, each feature will have not only this


one channel as you see here, but many, many channels of information down here.


So the CNN will encode each of these regions in the picture into a feature vector.


And then you have these slots.


So what you'll want to do, we maybe look at this, so you'll have the features right here.


These are your features.


And you'll have the slots and the slots, let's say there are fewer slots than features to


three, three slots, four slots, as in this case, four slots.


What you'll want to do is you'll want to assign the features to the slots.


So you maybe say, okay, this feature right here, and this feature right here, they go


to this slot, and then these two features go to this slot, and this these two go to


this, and that feature goes to that.


And that's equivalent to basically subdividing the picture into these slots.


Ultimately your goal is going to be to say that these features right here, these pixels


right here, are going maybe into that slot.


And then these ones right here are going into that slot.


And these ones here going into that slot, and the rest, so all of their background is


going into that one.


You can see that if you have a system like this, if you can train it correctly, then


it becomes pretty easy to so it becomes super easy to classify it right here, because you


can just take each slot and independently classify it, right?


Because you already know, you already have assigned all the pixels where the object appears