NeurIPS 2019

I don't know any of these things I'll just try to get back later going to work


NeurIPS 2019

a bit now get back later get my ticket and then I hope I can report a bit from


NeurIPS 2019

the conference over the next few days I can get some good nuggets out of there


NeurIPS 2019


Learning World Graphs to Accelerate Hierarchical Reinforcement Learning

Hi there. Today we're looking at learning world graphs to accelerate hierarchical reinforcement learning by Wen-Ling Shung et al. from Salesforce Research.


This work is based in the world of reinforcement learning and especially hierarchical reinforcement learning.


So in hierarchical reinforcement learning, the idea is that in order to perform a task like in this case, they perform all of their experiments on mazes like this.


So imagine you have this maze and this red thing here is the agent and the goal is the green square.


And the gray things obviously are walls and the black things are everywhere the agent can move.


The agent can always move one step in any direction that it wants and that isn't blocked by a wall.


So in order to fulfill such a task, the agent needs to take many, many steps like go here, here, here, here, here, here.


Each one of those is a step. In addition, this specific maze has an additional property, namely that there is a locked door here.


And first you need to pick up the key to basically to open the locked door.


So in order to reach the goal, the agent needs first to pick up the key, then open the door, then go to the goal.


And each one of these, it has to traverse many, many steps. So the idea in hierarchical reinforcement learning is that you have two parts to it, to the agent.


So your agent, which is this entire box here, is divided into what's called a manager and a worker.


And this is a divide. So what the manager sees, the manager sees basically, I do an example here, they do it differently.


But the manager could see large, could see the world basically only in these large chunks.


Right. And it doesn't really care what is in or it cares what is in the chunks, but it doesn't distinguish points within the chunks.


It just knows about these chunks basically. And what the manager will say, oh, first I need to go to this chunk here, then because there's the key in this chunk.


And then I need to go to this chunk here because there is the door. And then I need to go to this chunk here because there's the goal.


So the in the view of the manager, which has a very high level view of the world, is the the action sequence is down here, over here, then over here.


These are like three actions. That's a pretty simple. And then the manager would pass this information to the worker and it would say, hey, worker, please go to this state here.


Please go to the first state. And then the worker would be tasked with basically moving the individual steps to go not to the final goal, but only to go to that chunk.


And then in that chunk, the worker would go to the key. And then once it has the key, the manager would say, good job.


Now, please perform the second action, which is go to to this chunk here.


So the second action that the worker would so you basically get the idea. Whoops, I am doing something here.


You get the idea that the I'm creating text boxes that the worker and the manager work together and that the manager has a high level view of the world.


And then the worker can basically execute the actual actions that the manager has decided on in a fine grained way.


So this is gives you several advantages, namely the manager can plan high level and far away things.


And then the worker really only has to care about its close neighborhood because each step the manager proposes is a fairly short range.


So the worker can implement it. They do this in a kind of different way.


So let's actually start from the back from of this paper, which is I find is a bit more explanatory and it makes a bit more sense to look at it.


What they propose is to learn a world graph. So in a world graph, what is a world graph?


A world graph consists of two things. First, a set of states, which is the or the blue states here.


So all these blue states, which are so-called pivot states or important states.


So these are states in the world that are very important, determined by some measure.


Right. So these are basically states that look at look at where they are.


They're often like narrow passes. You see here, here, they're at these narrow passes.


So basically, if you if you reach those states as an intermediary goal, then you can basically go a lot of places from here.


So these are very, let's say, powerful states. And these states are connected by a neighborhood graph.


So basically, which states of these are close to each other?


And for example, here you would connect, of course, those two because they're neighbors.


Those you would probably connect those. So I'm attempting to to kind of draw the world graph.


You might connect those. It doesn't need to be like a tree. It can be like such.


So you see that the graph kind of takes shape. These are fairly reachable.


So whenever a node in the graph, whenever one of these important states is fairly easily reachable by some other state, it's designated as a neighbor.


So with that, with this world graph here, this is what you get.


You get an abstraction. Basically, you get a set of states with connections between them that says,


how easy or hard is it to reach from one state to the other?


If you have these things, then you can really easily imagine a hierarchical reinforcement learning algorithm that now incorporates this information.


Namely, the manager will only use the important states to plan.


So, for example, if the goal, the goal isn't drawn in here, but let's say the goal is here and then the door.


The door is here. It's a locked door here. And then the key. Let's draw in the key. Come on.


OK, this doesn't want to. All right. The key is somewhere. Let's say here. There's the key. Here's this.


All right. Then the. No, let's put the key further away.


Come on. Door here. I'm off with the colors and key here. All right. So what would the manager do?


The manager would then say, ah, OK, the key is here. So this would be a good state to reach of my importance.


The manager is only allowed to go important states. Right. So the manager says because it has the graph.


Right. It says, ah, this state is easily reachable from, let's say, this state.


And this state is easily reachable from this state. So it plans go here and go here and go here.


Then get the key. Right. This is a kind of a micro action that is not in the important state.


Then I need to, you know, go here. This is reachable from this state.


That's reachable from this state and from this state. And that's reachable from my origin.


So from the key, then next go here, go here, go here, go here and then open the door.


And then, of course, go here and solve the the task.


The worker then would only ever need to implement the following.


It starts here and it says, ah, I need to go here. What do I need to do?


I need to go, for example, down and over. And now once I've done this, I need to go here.


So I need to go right down. Right. So you see the worker only ever has to care about going from one hop to the next hop,


making it really easy for the worker, while the manager only has these blue states available,


which makes its search space much more, much more condensed and much more,


much more over viewable, especially with the nodes in between the world graph.


So that's if you have the world graph. Right. If you have this set of states and how important are,


how easily they reachable, reachable they are between each other.


You can very easily do a reinforcement learning approach that that is hierarchical,


has the manager plan on the world graph, has and then has the worker implement the fine grained actions.


And there is already a method that does this. This paper here uses feudal networks.


So we won't go into that later. Just saying it's pretty easy if you have those things.


So the real question is, how do they learn the world graph?


And what they do is the following. And they describe it here in kind of this.


Sorry. This way. What they want to finally learn is a prior that tells them for a given state,


how important is it? And that's a beta prior.


A beta distribution is a continuous approximation on a kind of a binary zero one variable.


So how do they do it? They use an LSTM to encode trajectories.


So these are trajectories from kind of rollouts of policy.


And then the LSTM encodes it. And for each step, it outputs this posterior over the what's called these latent variables here.


They say how important is a state. So these are the posteriors, whereas this over here is the prior.


And the posterior, of course, only makes sense in context of a trajectory.


That's why the ultimate decision happens for the prior, because the state needs to be important or not important to any trajectory.


So what they do is they roll out policies and they have certain methods of doing this.


So they have random exploration of curiosity goals, but they also train this continuously.


So they update it continuously via this what's called a goal condition policy.


And what a goal condition policy is basically is you put the agent somewhere in the maze.


Actually, let's use this maze over here. You put the agent somewhere in the maze.


Let's say here you, for example, make a bunch of run, make a random exploration.


Let's say here. So you know these two things are reachable. And then you train the agent, say, go from here to here.


Right. This is your goal. Now the agent tries to kind of reconstruct this random walk to there.


And you can. So this is how you train an agent to basically go from any two well reachable states to each other right from here to here and so on.


Now you won't train it to go directly from here to over here because a random walk would be very hard for a random walk to find its way over there.


But what you end up with is is somehow an agent that is able to reach close by states.


And that's exactly what the worker is supposed to do. Right here.


And so off of these trajectories, you can then unroll them and decide on the kind of on these on these pivotal states.


So how do you do that? And this is where this top part here comes in.


So down here, you input the trajectory and you output how important is each state.


All right. And now you see in this example here, the light color means that the LSTM decides this state isn't important.


And the darker orange color means the LSTM decides this state is important.


So what you do next is the states where it decides it is important.


And notice the beginning at the end are always important. It feeds to a second LSTM as an input.


You see here, here, here. So in this case of these of these six states in the trajectory, three are important, namely the start, the end and this one here where the LSTM decides, hey, that's important.


That goes into a second LSTM, which is generator.


So this here is an encoder and this here is a decoder. And what it does is it decodes the sequence of actions.


Right here, given nothing, just given this, it decodes a sequence of actions.


And at the end, what you want is that the actions output here reconstruct the actions input.


This might sound a little confusing, but the core value of this is what you want is to reconstruct the actions of the trajectory taken given only the important states.


What does this mean in our example? In our example here, this means if I have to go from here to here.


Right. And for example, I took the following path. This is this. So right, right, down, down, right.


This is these were my sequence. Now, if I only have the start, the end and one state in between, let's say this one.


Right. Then can I reconstruct what actions were taken?


And if I erase the blue thing and I tell you I went from here via here to here, then you could very much reconstruct the actions here.


So this state here is a good candidate for being an important state.


Whereas if it were a different state, if it were, for example, if I told you I went from over here to here and then to here, you'd say, well, this could be either something like this.


Or it could be a path like this. Right. It could be many, many paths or like this could be many paths leading from here.


So this state here is not probably not very important.


So that's kind of how they how they learn which one are the important state via this encoding trajectories in an LSTM and trying to reconstruct the state,


the actions taken in the trajectory, given only the states that were deemed important by the LSTM.


So that's how you train the LSTM to recognize important states.


And once you've recognized the important states in a trajectory, you can then use those to learn prior.


So basically, you ask over all possible trajectories, which of the states are generally important.


And that's how you end up with these blue states. Right.


And then the last part is to connect the blue states. And that is fairly easily done in their approach.


What they say is, all right, we have blue states, we should pick one.


And we do a random walk from it, right? Random walk, random walk, random walk.


If we hit another blue state, like this one here, in the random walk, we simply say, well, they're probably neighbors.


So we do this a bunch of times. If you hit the blue states, of course, without hitting another blue state first, then you connect the two in a graph.


So these would be connected, these would probably be connected.


What we ended up at the beginning, right, you have this graph, maybe these two are connected, and so on.


So this gives you this world graph. And now you end up with a set of important states and connections between them that tell you which ones are easily reachable from each other.


So you can train the manager on that.


You can train the worker, as we said before, to simply select two close by states, train it to go from one to the other.


That way the worker will learn that. So in essence, that's how they do it.


You can look at the experiments themselves. They show that this basically transfers.


So if you train like this, pre-trained, then you can give more specific and more complicated tasks. And this will rapidly accelerate the learning of this.


Datasets for Data-Driven Reinforcement Learning

Hi there, today we're looking at data sets for data driven reinforcement learning by Justin Fu,


Aviral Kumar, Ophir Nachum, George Tucker and Sergei Levine. So this is a what you would call


a data set paper or a benchmark paper. And the main point or the main area of the paper is was


called offline reinforcement learning. So offline reinforcement learning, usually in reinforcement


learning, you have this task, right, you have the agent, and you have the environment. And


the agent gets some sort of observation, and has to come up with an action in response to that


observation. And then it gets back a reward, and another observation. And again, has to come up


with an action. And the goal is to maximize the rewards over time that the agent gets while


interacting with the environment. So usually, this is organized in what are called episodes,


which basically means if you have some sort of environment, right, and here is the agent,


and here is the goal, right, the goal is a inverted triangle. And there are a bunch of walls


right here, right. So it's kind of a maze that the agent has to navigate. Then one episode is could


be the agent moving around until it either finds the target, or hits a wall, or just kind of goes


around and around. And then at some point, you say, all right, that's enough. Over game over. And


usually, in reinforcement learning, you perform many of these episodes, and then you learn from


them. So you perform episodes, and each episode gets into usually some sort of replay buffer,


right. Let's call this replay buffer. And you do this many times. And at the same time that you're


doing this, you're using the things that you stored here in order to learn, right to the agent learns


from these things, right. So it acts with the environment in this loop, in this fashion,


then once it has done an episode, it puts it into the replay buffer. And then it learns from the


actions it has performed. This is what is usually called online reinforcement learning. Right, so


this loop is online. Online means because the agent learns from its own actions, right. Now,


in contrast to this, there is offline reinforcement learning. So in offline reinforcement learning,


the agent has to learn from someone else's actions, right. So this connection here is severed.


Instead, you have other agents, let's call these agent one, agent two, multiple agents,


agent three, they all have their own interaction with the environment, right, environment,


environment interactions. And they feed their experience into they perform these, these episodes,


they feed their experience into the replay buffer. And then the agent just has to learn from that.


So whatever happened here, this was previous, right. And now the agent has to learn how to


maximize its reward, just from the experience that is in the replay buffer. From these other agents,


this is what's called offline reinforcement learning means the agent learns from someone


else's actions. Usually the power of reinforcement learning, of course, comes from the fact that you


learn from your own actions. It means that, for example, if you already have some successful


trajectories here, right, you found the target, you can try to replicate that because you know


which actions you performed. And if you don't, you know, change anything, you're probably going


to find the target again, just by randomness, all right, because you've done it already once,


and so on. So you kind of know all the intrinsics of your own algorithm that led you to reach the


target. Now, this is an entirely different case with all of these other agents, you have no clue


how they were acting, why they were acting, right, you just know, okay, they did a series of actions,


and that gave them some kind of reward. And you have no idea what their reasoning was or anything.


All you really can learn from is their sequence of actions. Now, why is that problematic, right? So


if all of the agents, for example, if this is, if this is an actual platform, and this really,


right, steep here, or this is all of here is really steep cliffs, right? And you can actually


fall off. But the agents, they're, they're humans, right? So they don't want to fall off. So what


they're gonna do is they're just gonna take steps that are maybe like this, or maybe like this, but


they're, they're humans, they're smart, they're never, they're never gonna, they're never gonna


fall off here, right? Why is this a problem, if you're not trying to learn from this experience,


and your policy, by some chance, because you might have some entropy in there or something,


you do know what happens if you make a move like this. And you also know what happens if you make


a move like this, right? Already, two humans have done these moves. But what happens if you make a


move like this? You just don't know, right? In classic reinforcement learning, you would get a


negative reward, and you could learn from that to not do this action anymore. But in in this case,


in this case, you simply don't have any data to, to tell you what happens when you go off there.


So you see that there's a there's a problem if you are not able to learn from your own experience,


but you have to learn from some thing or someone else's experience, the distribution of experience


that you have available to you might be on, like not, not fully specific of the of the environment,


it might be very different from what you would do. And it might be very not conducive to what you


want to do with it. So the task of offline reinforcement learning is harder than online


reinforcement learning. But it also has many, many applications. Sometimes it's just not possible to


do online reinforcement learning. When, for example, in medical field, right, think of,


think of the medical field, where you want a robot to perform a surgery, you can just do


reinforcement learning in with our online techniques, because they're just gonna try a bunch


of things and see what works. And you don't, you don't, like, maybe you want that, I don't want


that. So necessarily, you're going to be left with, let's have this robot learn from human experts,


right. So that's a task for offline reinforcement learning. There are many more tasks. For example,


if you think of search engine, you will have many, many, many logs from human searching things. And


you simply store them, you simply have them in a buffer. Now you want to maybe train a reinforcement


learning agent that I don't know, serves the best possible ads or something like this. You want to


do this in a way that you can use all of that data, even though that data wasn't collected by


that particular agent, right. So there's the crucial difference to supervised learning, again,


is that you have this, this interactive structure, right, this multi step interactive structure,


because in a supervised learning, you also have, you also have this buffer here, right, in supervised


learning, you simply have your labeled data set. But the difference is in supervised learning,


you always know what the right action is currently, right, because you have the labels.


In offline reinforcement learning, you don't know, you right, you might be here. And there are three


actions available, right. And if you all you know is that the, the, the demonstrator a these actors


here, one of them has done this, and then this, and then this, and then got a two, you have you


have no clue what happens if you do this, and then this, and then this, right, all you know is that


this action here might eventually lead to a two, right. And you also can't try it out. Because you


can't try out this path, because you don't get a reward here, you have to find and this is the


this is the task here, you'll have to find some other example or stitch together, they make a good


example here. So this paper, basically proposes a benchmark for offline RL algorithms. So what they


do is they have a bunch of data sets, right? They have a bunch of these replay buffers around for


different tasks, the collection of this, that they collected with various techniques. So there's human


demonstration, there is other agents and so on. They have that. And you're supposed to take one of


them, learn something, learn an agent, and then evaluate it on an environment. Right. And they


propose which ones are suitable for this, they give you the data. And they give you the environment


to evaluate it on. In the end, you'll get a score and you can compare your offline RL algorithm


with others. They also provide some benchmark implementations for algorithms that already do


this, and they show that they don't really work well. So one of the one of the tasks is this maze


here. In this maze, you will, the task is you are somewhere, let's say here, and you need to go


somewhere, let's say here, and you need to find your way. Right. And the demonstrations you have,


the data in your replay buffer, is such that this is the same task, but never the same start and


end points like you are tasked to. So you might have one in your replay buffer, you might have


one trajectory, one episode that went like this from one to two, right. And you'll be able to see


the reward of that. And you might have one trajectory that was from two to three, like this,


right. So both of these things actually give you really high reward. So if you were an agent,


right, and you had to learn, and now the task is please go from one to three. What you could do


is you could simply say, Ah, I know, I know the green thing gave a pretty high reward. And the


yellow thing gave a pretty high reward. I know the green stink thing started at one and I know


the yellow thing ended at three. And I know they both have this common location. So what I might


do just is I might go to that common location, and then go on on the different path, right. So


you have to somehow stitch together experience from other agents in order to make your task work.


Now, this is a very explicit example. Of course, what we want to do is we want to do this in a more


implicit deep learning way, ideally, and not manually stitch together other trajectories,


though I'm pretty sure that would not that would not be so, so dumb, right? I'm pretty sure there's


a lot of data augmentation you could do during training simply by stitching together other


trajectories, right. So from this trajectory, you could actually, not only could you make other


gold conditioned ways, for example, from here to here, or from here to here, you could make from


here to here, anywhere where you have shared points, you could go you could train a policy


that goes there and then goes further or something like this. I'm pretty sure there's already an


algorithm that does things like this, but I'm just thinking aloud here. All right, so this is one of


the tasks and you see that the the that that you will have to learn a policy to go as fast as


possible from any point to any other point. And you're all you're given is a database of experience


that already exists from some other agent, but never will probably never the exact route that you


need to learn right now. All right. So the goal is how fast or how efficiently can you do this,


this is one task in this data set. The next task is very similar is this grid world here,


where there is this red square, a red triangle, that's your agent. And then there is the green


square, that's your goal, or vice versa. And so you're basically tasked to not hit the walls here,


and and to go about your, your way finding the target. There are more elaborate things like this


mojo co environment here, or the ant maze where you have this little ant with you know, the spider


legs. So this is no longer you can just move in either direction, you have to actually control the


legs. And there's also this arm, this robotic arm. So you see, there is a wide diversity of tasks.


And also, there is a wide wide diversity of how the replay buffer was constructed. So in some cases,


the replay buffer is actually constructed by a human performing in this environment. So in


this hand in this hand manipulation task, you'll have demonstrations from humans, you see, it's not


particularly many samples here, it's 5000 samples, which I guess are are is a chopped up version of


I'm not really sure how the human things are constructed. But you can clearly guess that


the degrees of freedom that you have in a robotic hand is much, much higher than you could learn


just from these 5000 samples, if you were to, you know, an online URL algorithm that just does random


exploration will need much more than these 5000 samples. And the 5000 samples won't be


IID distributed with all the degrees of freedom, it will just be here's what a human does, right.


And so you can think of algorithms like inverse reinforcement learning or something like this.


But here in inverse reinforcement learning, usually you assume that the expert, the expert


is kind of trying to achieve the same reward as you do. But this is not necessarily the case here,


you have a given reward structure, but you are tasked to simply learn from these demonstrations.


You can see it's also possible that there is this is constructed by a policy. And that usually means


that they so either it's, it's constructed by, let's say a reinforcement learning algorithm


that was trained in an online fashion, but maybe not as well. But also, I think they have a behavior


cloning policy that they got from human demonstration, I think so that there are many ways.


Also, sometimes you have a planner, which is, can you imagine it's an algorithm that wasn't


machine learned. So I know almost unthinkable, but in these in these kind of mazes,


you can actually do planning algorithms that can can sort of, so I know this is crazy and


crazy talk, the niche topic, but there exists things like a star search, where, where you can


construct the kind of shortest path through these mazes and things like this. So yeah, that's, I know,


I know that is that is very niche. But you can construct policies like this, and then you can


use those as your replay buffer filling. And you can already see that this also will be a massively


different distribution of data than you would get with an online RL algorithm. Right. So in conclusion,


they do test other, they do test other algorithms on this. In conclusion, they say that most offline


RL algorithms nowadays, they don't work well on these on these data sets. The only data sets where


they do work well, is where the replay buffer was generated by some sort of,


like here, by some sort of policy by some sort of reinforcement learning policy. So what they would


do is they would train an online policy, and the experience generated by that online policy while


it learns will make up the replay buffer. And if you use that replay buffer for offline learning,


then they say it tends to work okay. But if you have other methods of course, you can use that


if you have other methods of collecting the data that are very different from this offline, sorry,


from a reinforcement learning collection approach, then it tends not to work as well.


All right. So if you are interested in offline RL, please check out this paper, all their code is


available right here. Note that the link in the paper doesn't seem to work. The true link is here.


Learning Rate Grafting: Transferability of Optimizer Tuning (Machine Learning Research Paper Review)

Alright, so I just got done making a video about this paper and I was trying to upload


it, so I looked at the open review page and I read the first review and I just thought


I had to show you this.


Now before you see the review of the paper, but just look at this review.


So the paper is about optimizer grafting, it's about transferring the learning rate


of one optimizer to another optimizer, it has some experiments in it and proposes this


algorithm to investigate sort of learning rate schedules.


Okay, main review, S1, which I guess is strength one.


A large amount of experiments is conducted, plenty of results shown in the appendix.


As to a novel optimizing mode of grafting two different optimizers is proposed.


So you know a little bit about what's in the paper.


This one, the paper structure is strange.


I recommend to read some published proceedings to try to make this paper more clearly.


What just to say these these are accomplished researchers, right?


That are the authors of this paper actually show who the authors are.


The structure is strange.


I recommend reading, you know, read a bit, maybe maybe a book, maybe, you know, you'll


learn something.


Weakness two, some form it may not be legal.


Okay.


Weakness three, the theory is not reasonable.


Not really.


By the way, the paper proposes no theory.


The theory is not reasonable.


In other words, you just tell me you do it like this, but not why it's reasonable.


Okay, I mean, that is a that is a even though the paper explains clearly why they do everything


that might be a criticism like you haven't really given a theoretical foundation for


your reasons.


But then, actually, I don't think Adam grafted onto SGD.


So this is the new method they propose.


It's SGD with the learning rate of Adam.


Actually, I don't think Adam grafted onto SGD will be better than Adam.


Notice this is what they show in the paper that they make experiments to show that this


is the case.


And it's not like this person has tried it out and has said it doesn't work for me or


it doesn't work in this other paper.


No, no, no, no.


The entire thing that this person says is I don't think this will happen.


No reason.


Why?


What is this?


This is this is a type of reviewers that people have to fight with.


And then there's like some hubbity, hubbity, hubbity, hubbity.


I'm sorry, if they show in the paper that this is the case, then either you claim they


are lying and or you have conflicting evidence or anything like this, but simply sitting


here saying, I don't think so.


What?


I mean, ah, then we can.


This is this.


This is why I'm confused.


In my view, this is method is more like an SGD with multiplying a large constant to its


gradient.


I mean, at the end, that's what it is.


But like, has this person actually read the paper?


Weakness for I have a question.


That's a weakness.


A weakness is I have a question.


How to compute the norms?


How to compute these norms?


It's norms like the paper clearly like they don't.


OK, they don't say it's L2 norms, but they clearly you know, how do you compute the norm


of a vector?


Is this calculated with this is answered in the paper.


This is clearly answered throughout the paper.


If not, figure one is a wrong example.


Well, it's it is.


So it's like, how is it a weakness if you have a question that is answered in the paper


and then weakness five, the results shown in tables are not strong enough.


Right?


A large amount of experiment is conducted and plenty of results are shown in the appendix.


The result shown is not strong enough.


Well, what what do you mean not strong enough?


Like not highly performant enough because that's not what the paper is about.


Not strong enough.


You mean not enough because, well, the other reviews.


It's not like the other reviews are necessarily good reviews of the paper, but at least they


have some criticism like, hey, you know, you're not theoretically motivated or something like


this.


And they are a bit extensive.


But like, this is what this is.


You know, it's it's it's I guess, you know, if you're some company researcher and so on,


you know, your bonus might depend on a submission being accepted or not, which, you know, if


you're at Google or so, I mean, you're doing well.


Right.


But if you're like a PhD student and you need to get papers accepted in within a certain


amount of years and then I don't think that what you clearly show in the paper is the


way it is because I just pull it like somewhere out of here.


OK, enough of me ranting.


Let's go into the paper.


By the way, I make one mistake.


I make one mistake in the paper, which is kind of similar to what the person is here.


So I there is a diagram and I'm going to just going to describe it right here where I where


I say there's an arrow like this and an arrow like this.


And I say, well, the combined update step would be something like in the in between,


which is is not the case.


It would be actually one of the arrows just rescaled.


Errata OK.


Bye.


Last thing.


This is the best.


I forgot.


Confidence.


You are absolutely certain about your assessment.


This is the highest score.


This is the review rating themselves.


You are very familiar with the related work and checked the math and other details.


Really?


Because here it says I'm confused and I have a question.


The following is a community inspired paper review, which means that we have talked about


this paper in our discord paper discussions.


We do this regularly and I can take a lot of good opinions from there and bring them


into my videos.


If you're interested in joining these paper discussions, join our discord and watch the


events channel.


Hi there.


Today we're going to look at a paper by Naman Agarwal, Rohan Anil, Ilad Hassan, Tomer Koren


and Cyril Chung.


But it is not the paper that you see right here.


See this paper is called Disentangling Adaptive Gradient Methods from Learning Rates and it's


on archive with the authors.


Now allow me to present this paper right here under review at iClear with anonymous authors.


That's called Learning Rate Grafting Transferability of Optimizer Tuning.


Now suspiciously the two papers have pretty much exactly the same content.


So you know, safe to assume that we might make an educated guess about who these authors


might be.


I'm going to review the obviously newer version because newer is always better.


So what is this paper about?


This paper is about a technique called learning rate grafting and grafting means that we transfer


the learning rate from one optimizer to another optimizer.


They have a bit of a graphic right here.


So what we would do is we would take two different optimizers and think of things like SGD or


Adam or something like this.


So these are fairly popular optimizers in deep learning.


We would take one of them and that one would give us the information of what the direction


of updates of our weight is.


So let's actually say SGD here is this purple one in this direction.


You can see that will follow in general the direction that SGD tells us to go.


However, we don't go exactly what SGD we don't do what SGD tells us to do.


Instead of we take the learning step size or the learning rate from Adam and we go that


far.


So one algorithm dictates where we go the other algorithm dictates how far we go.


And what this does is it implicitly transfers transfers the learning rate schedule from


one optimizer to another optimizer.


And as a result of this many many things happen.


So one simple thing that results from this is we're able to investigate some of the differences


between the optimizers.


Surprisingly one of the things that this paper finds is that maybe the different optimizers


it's a bit over let's say over described overhyped what the differences really are between them.


A lot of times it simply comes down to the learning rate schedule that the optimizers


induce and as soon as you transfer that to another optimizer the other optimizer will


perform just as well.


So the differences between a lot of these optimizers might just come down to the learning


rate schedule.


Another thing that they can do is they can for example transfer these learning rate adapt


adapt sorry adaptations that one does to the other and that makes it in practice that gives


you benefits in practice.


For example Adam let's look at Adam Adam maintains three buffers for every single parameter.


So for let's or let's go SGD SGD for every parameter W it has one it essentially just


updates that parameter.


If you have SGD with momentum then you also have the momentum parameter that it maintains.


So for every parameter there is a momentum parameter and then as a gradient comes in


it updates the momentum parameter and that it uses that to update the weights.


So one buffer essentially per parameter that we want to treat.


Adam on the other hand maintains like three buffers.


I don't exactly remember what they all are but they are like the squared sums of gradients


and then they are somehow the current gradient squared or some exponential moving average


across that.


In any case it maintains like three different buffers per parameter and that also means


that it has like double at least double or three times the memory requirements of SGD


right.


SGD even with momentum needs a lot less memory than Adam and that's a big deal because memory


is one of the things that especially on GPUs is a limited commodity.


So if you're able to reduce the amount of memory that your optimizers need then that


means that you can train bigger models because now you have a bunch of free space.


So what this grafting method allows you to do is it allows you to essentially run SGD


adjusted for the learning rate schedule of Adam but without having to run Adam.


You can simply transfer the learning rate schedule or the adjustments to the learning


rate from Adam to SGD and you know that's a that's a pretty cool thing.


So we're going to look going to go look into how this paper does it what it suggests and


it's pretty straightforward paper.


I think it's pretty pretty short pretty cool to read and yeah.


So what is what exactly is grafting.


They first do a little bit of an excursion into preliminaries and that essentially presents


these adaptive optimizer these adaptive methods.


So if you look at SGD what it does is it pure plain SGD its update rule which they characterize


as an algorithm a right here that takes in the current weights of the neural network


or whatever system you optimize and the current gradient right.


So W are the weights G is the gradient both at time step t it will output for the next


weight so a always gives you W t plus one it will output the current weight minus a


step size times the gradient.


This is classic gradient descent.


Now this right here is a learning rate schedule.


So even in gradient descent people do learning rate schedules.


Sometimes there is a bit of a warm up and then you might reduce it over time maybe after


some epochs I go down and so on or you might not write.


But these are usually handcrafted learning rate schedules.


Now when you go to other things such as Adam or Ada Grad or anything like this of all of


these Ada Grad is probably the most simple.


So the reasoning behind Ada Grad is the following.


If if you have a loss landscape which we are going to draw here as some sort of a topological


plot so every line is in sort of a same loss height and this is the global optimum right


here.


So you start out somewhere here you calculate the gradient the gradient maybe goes in this


direction so that's the local the local tangent to these these ISO lines.


That's pretty simple right.


You see you go straight here even if you have some sort of a bit of a mistake at the beginning


because it's stochastic you can see in general you go downhill.


However what if the landscape doesn't look like this but it actually looks you know like


really skewed in one of the dimensions.


So it's really steep in one of the dimensions and it's really flat in the other dimension.


Now what happens here is that if you start off the same thing maybe you have a little


bit of noise you tend to make if you do this step.


So if you look at this what you're going to do is probably you're going to make a big


step into this and then it's really steep right.


Now it's really steep into this direction so you're going to bounce over here like really


far and then it's really steep in that direction so you're going to bounce over here really


far.


So because it's so steep in that direction you want to bounce around way too way too


with way too big of a step size just because one direction this direction is way steeper


than this direction.


So what do methods like AdaGrad do.


AdaGrad flattens out this landscape by observing.


I mean the algorithm doesn't see the landscape it only sees these points where you're at


and the corresponding gradients.


So what AdaGrad does is it simply says I'm going to look at one of these gradient steps


right let's say I'm here.


This is my gradient here.


I'm going to look at what's the change in this direction what's the change in this direction


and then I'm going to normalize by it.


So the update rule for AdaGrad is something like Wt plus one equals Wt minus some step


size times the gradient.


But now the gradient gets scaled by the sum of square gradients and the square root of


that.


So what this means is that I'll take all of the gradients that I've seen so far I square


them and then I sum them all up and in essence this is element wise by the way so these are


vectors and we are talking about diagonal AdaGrad.


So in essence what this says is that if I have my gradient vector here I'll put a matrix


in front of it and every entry in this matrix is one divided by the square of the gradients


I've seen so far.


So it's a bit of a normalization if my gradients in this particular direction were really large


I'll divide by a lot.


If my gradients were really small I'll divide by just a little bit.


So you can see that it transforms a landscape like this to implicitly look much much more


well conditioned and you can even see because we have a total sum right here that goes on


with time that there is a little bit of even a decreasing learning rate built in because


the square is always positive.


So we're simply going to add on to these buffers and that means that we are going to decrease


our learning rate implicitly over time.


So here you can see you can see two things you can see that you know these these preconditioners


they have their reasons for existing first of all but much more important.


They introduce an implicit learning rate schedule right this thing right here is an implicit


learning rate schedule and all of these algorithms like AdaGrad, Adam and so on.


They introduce exactly that.


So this part right here that's the implicit learning rate schedule and we're now we're


now wondering so how much of the success of these optimizers comes from the fact that


they do something like this right here where they you know look at each of the coordinates


and they you know they they adapt with respect to how steep they are and so on and how much


or how much simply comes from the fact that they say well now you need to go far now you


need to go not so far now you need to make a big step now you need to make a small step.


So that's what we're wondering and grafting allows us to answer these questions.


So in grafting what we do is we leave the optimizers as they are.


So here we would leave SGD to do SGD right.


So again we're at the start here I'm running out of colors to draw over top of one another.


Let's go with green.


We're at the start right here and we want to and let's say we've made the step now we


want to go into this direction right SGD would make a big jump right here and Ada, AdaGrad


or Adam maybe would do two things.


It would say well since this one direction is very steep I'm not going to make a that


big of a step into that direction and maybe make a smaller step and I also adjust my direction.


What grafting does is it says okay we're going to take your suggestion of how far we should


go but we're still going to go into the same direction that we originally went.


So we're taking the step size that the one optimizer suggests and we'll transfer it onto


the direction of another optimizer.


So this allows us to answer the question what's really important here the step size schedule


or the direction the particular direction that these optimizers produce and the answer


is going to be the step size.


So the grafting algorithm is detailed here.


This is the simple version which is I believe called global grafting.


So you can see we're going to note we're going to take this right here this notation so M


is stands for magnitude algorithm I guess I don't know I've invented it.


D stands for direction algorithm and M hash D is the combined grafted algorithm.


So what we're going to do is we're going to feed the same input the current weight and


the current gradient to both of the algorithms.


They will manage their states internal states independently but yet they will not yet update


the weights they will simply suggest each an update.


What we'll then do is we'll look at two quantities this right here and this right here.


So this is the step that this here is WT plus one according to algorithm M and this is WT


plus one according to algorithm D.


And we're going to look at both of the steps that they would suggest right if we subtract


this here this is what step do you suggest.


And then what we do is we compute the norms of these steps and we'll simply normalize


the quantity of D right here by the ratio of these norms.


If we rewrite this a little bit you can see much more clearly what's going on.


This is WT plus and then I'll write the norm the first norm here WM minus WT and then I'll


write the second thing WD minus WT divided by the norm of WD minus WT.


So there you can see that we'll take the direction we'll take the direction of the D optimizer


and we take the direction because by dividing by its norm we normalize it.


So this always has length one right.


So this is simply the direction of the step that the D optimizer would do and we multiply


it by the norm of the step that the M optimizer would do.


Notice M only comes in here through this norm so M has no influence on the direction that


we go while D has no influence on the magnitude of the step because we always divide by its


own magnitude.


So that's the grafting algorithm and they have some properties right here.


It's you can graft an algorithm onto itself it won't do anything.


You can graft multiple algorithms and so on it's not commutative yada yada yada it's not


necessarily a descent method which is interesting but I guess irrelevant because that I consider


that an edge case and now they have one more trick up their sleeve how they make it more


interesting namely this is what they call global grafting where it's just one global


learning rate right this whole norms these whole norms here they are they are just one


number at the end.


They can also do this for example for each layer individually so they divide up the parameters


into layers and then do it for each layer individually.


If they were to do it for each parameter individually right then it would be it would not have any


effect so if they do it for each parameter individually I think it would just revert


to being the old sorry it would just revert to being the M algorithm right this that's


what they say right here if they do it for each parameter individually they might as


well just run M because the magnitude of each parameter is dictated by fully by M and we


don't so well we don't calculate the direction of D because each of the entries is separately


divided by itself so D will just output a bunch of ones so yeah that's that's the reason


and because the norms are just of size one in any case that's a bit of that's a bit of


pushing it to the limit we can either do this globally or we can do it for each layer individually


that's this partition parameter right here so what does this where does this go what


they try is notice that we're still in the case where we need to run both algorithms


simultaneously right so for each step we're here for each step we have to consult STD


what would you do and then Adam what would you do and then we do the grafting between


the two things and then we maybe get this direction right here we go on we again ask


both optimizers we go on in the experiments they do a good job of controlling for the


actual compute that they give to these experiments and and therefore you can make some assumptions


but one worrying thing about me just as a side note is that Adam has this for example


this internal state right so it has these it accumulates the gradient into buffers and


so on yet we make an update step that is not into the direction that these buffers would


suggest so technically these buffers are wrong for the path that we're taking the buffers


expected that we're going to take this path right here and I'm not sure how much how much


you know we how much we actually miss due to that I also don't know how we easily would


correct it but I would just wanted to say that the internal state is updated as if we


were to actually take the step that the algorithm suggests however we're not going to take that


step at the end so this is a bit of a shady practice in this grafting algorithm in any


case as we do run both at the same time you can see right here so there's an experiment


where experiments for implicit hyperparameter transfer comparing hyperparameter search for


SGD with momentum versus grafting with and then M is SGD sorry so it's Adam grafted onto


SGD is that is that true M because it seems like D is SGD right it's always M hash D and


then SGD is at the end huh well maybe that's wrong I don't know as the way I understand


it is that you have the trials with SGD you have trial with Adam which is in blue right


here and then if you take this grafting approach and you do Adam along with SGD so you do the


direction of SGD but the step size that Adam would do you see that you almost get the same


performance in fact in this particular case SGD with the Adam step size even outperforms


Adam like a tiny little bit if you go to a higher batch size that's no longer the case


but also here you see that it seems to be that as soon as you get this step size right


not only can you not match it with any humanly chosen let's say step size of SGD which would


be all the gray stuff but also immediately most of the or all of the benefits of the


Adam optimizer versus SGD vanish so it really seems to be a thing of the step size and as


far as I understand it that's the global grafting yeah they they do make some like they mentioned


a bunch of times that this number right here no it's layer wise sorry it's layer wise grafting


they mentioned a bunch of times that this is higher than just using Adam but I'm not


sure how exactly robust this is especially as you see here if you go to the higher batch


sizes it is a different different story they also do some experiments with with Resnets


which aren't as cool like they're not as performant so here you see a lot of the times that they


take SGD which is a good algorithm for these types of problems by the way SGD was a bad


algorithm for BERT that's why they used it as the direction and grafted the learning


right onto it in these particular cases SGD is actually pretty good and so is Adam as


you can see right here and the other algorithms AdaGrad seems to be kind of bad if they now


graft SGD or Adam on to AdaGrad which you can see here with the layer wise or the global


grafting it helps a little bit right compared to just AdaGrad but it's not like it's not


like that it really gets into a highly performant region so I guess the conclusions of this


is that sometimes or is that the the step size schedule is an important parameter it


does it is part of why some of the optimization algorithms outperform others it might not


be all of the reason I guess that's that's a a cautious a cautious thing you can say


right here they go into a little bit of analysis for example about this giving you sort of


new bit of new insights so for example people have come up with this yellow learning rate


schedule for SGD there's a bit of a warm up and then there is just a decay after every


few epochs and so on and if you transfer that to AdaGrad so if you graft that on AdaGrad


right the trick is we don't transfer it we don't simply say well these are the steps


we always we ask both optimizers and then the resulting learning rate schedule might


be a different one from either of the two and the cool thing is that here the algorithm


seems to really decide kind of on a on this polynomial warm up for AdaGrad before then


using this decay that comes from SGD so it's pretty neat that it allows you to kind of


gain an insight into what these algorithms are doing they do a last thing right here


where they say can we get away with not running both algorithms at the same time and that's


what they they do right here so what is this they take AdaGrad and they know they take


Adam sorry they take Adam and they take SGD and they run it for just 2000 steps this is


very small number of steps let's say in training of Bert so these is just the first few iterations


they run both and what they do is they observe the norm ratio during grafting so they do


this grafting where they run both and they observe the ratio of norms between what one


and what the other one would suggest okay so essentially they do this grafting and they


observe the how the step sizes between the two relate and then they say okay we'll just


take the median over these 2000 steps and that is going to be our learning rate correction


to SGD essentially we're saying we're going for 2000 steps how does the learning rate


of the the implicit step size of Adam compare to SGD over these steps maybe it's always


10 times higher for some layers maybe it's 50 times higher for other layers you can see


they split this up into different different layer types like embeddings or self-attention


and so on and then they say well okay so from here on out let's just run SGD only SGD but


always correct the step size by this ratio and that actually works apparently so I don't


think there's a plot necessarily right here but you can see this is one of the results


so with Adam you again get this 69.5 SGD is way worse because this is BERT but then the


combination as far as I understand it that is this this discovered per layer learning


rate correction so that's one number per layer at even then SGD is better if you have this


learning rate correction given by Adam then just Adam itself a little bit but still it


is or is it not no this is grafted sorry I think this is the one this here is the one


where they keep it constant and that is not better but it is at least it is the same right


like I hope the rounding that rounding was in their favor right here otherwise they'd


have like added added like one one digit and had to could claim that they're better but


in any case it's pretty cool to see that the performance here jumps by quite a bit and


it's not that much worse as if you had executed Adam alongside right that's the 70.1 on the


bottom here they have different different kind of even more quantizations which make


the result worse most often but it seems like if you get them exactly correct then it can


improve by a little bit not too big of a fan of these kinds of things it shows that you


can go simpler but yeah you have to kind of hit it exactly right with this hyper parameter


and that defeats the purpose a little bit in any case I think this is a two powerful


things from this paper first of all this can be used for investigating these optimizers


right because you can now see aha here is you know here is the exact effect that the


step size schedule is having on one or the other optimizer you can sort of mix the step


size from one with the directional update rule of another one the second one is that


something like this where you simply quickly observe how to optimizers stack up against


each other match each other in the step sizes they would suggest maybe you need a little


bit more memory at the beginning because you up you execute both of them however you only


need to do this for a few number of steps before you can then go ahead and simply take


what you learned and save a whole bunch of memory because as they do right here they


are they only from here on out they only execute SGD no more Adam the ratios are fixed and


they are per layer so that's that's pretty cool and pretty powerful especially I'm wondering


how these things generalize so can I take sort of these can I take the ratios of one


network and transfer them to another one with a slightly different architecture maybe a


bigger network or a different problem a different data set so this seems to be a pretty exciting


future direction because it makes everything a lot more efficient if we simply know that


aha embedding layer okay you know let's just multiply that by 50 or something like this


and then lastly this is a bit of my worry is that I don't know where we go if we if


what I said right here the internal state of the optimizer assumes we're taking a certain


step yet we take a different step I don't know how that influences the entire grafting


algorithm they have a lengthy appendix if you want to go into that of a lot of a lot


of different results right here and but I don't want to go into that right here in the


conclusion they say we've introduced grafting a binary operation which blends the behavior


of two optimization algorithms towards investigating the entanglements between widely used adaptive


preconditioning rules and learning rate schedules yada yada yada furthermore we have shown that


grafting can be used to extract standalone learning rate corrections enabling us to train


a transformer using SGD with momentum for the first time well I guess people have been


able to train them before just not to satisfactory to satisfactory accuracies we hope that this


finding will simulate further empirical research power of simple per layer learning rate schedules


okie dokie the empirical phenomena examined in this work seem to be unexplained by current


theory that is also an interesting point we hope that the experiments enabled by grafting


will aid in developing more robust beliefs both adaptive methods and learning rate schedules


and guide future theoretical inquiry all right theory people here's something for you to


explain all right I hope you have enjoyed this overview of learning rate grafting sorry


for de-anonymizing the paper right away but yeah that's a bit silly anyway in any case


if you like this hit subscribe smash like get enough sleep and I'll see you next time


[Live Machine Learning Research] Plain Self-Ensembles (I actually DISCOVER SOMETHING) - Part 1

Hey, what's up? So I've had this relatively dumb research idea and people


have been asking me for more coding videos and so on, so I thought why not do


a video where I take a research idea and implement it from scratch just to show


how one would go or how I would go about implementing something like this. Now


this was simply meant as sort of a demonstration but then at the end it


actually worked and so yeah that was unexpected and my initial reaction was


just to be like oh crap just hold everything you know stop video making


develop the idea write a paper about it okay and I was about doing that when I


realized that you know I'm always the one complaining that research is not


transparent enough and people aren't open enough and so on so I sort of


thought I might do a different thing right here in that I will actually share


the process of this non finished research project so currently I am in


the middle of this I have no idea whether it's going to work out or not


and that's it and I think we can do open source software development you know


completely in the open whereas with research we're all like super scared


that people are gonna scoop us and we people just keep it keep their work


hidden until they're done and then boom they put it on archive and all I want to


go to a world where we collaborate much more in research and it's much more like


open source software development so here is my way here's my process of


implementing this idea and it's fairly long so if you just want to get to the


results you can just skip at the end I'll put timestamps in there's this new


YouTube chapter video so that'll be very helpful I guess yeah and with that being


said I hope you enjoy this let me know what you think of videos like this and


I'll see you next time hey what's going on today we're going to take a research


idea and implement it as fast as we can so this is not really to show you the


best research idea because it's not and it's probably been done before so I have


no high hopes here but this is just to show that if you had like some research


idea and you've actually done the literature research and figured no one


has done that yet which I haven't because probably someone has done that


how you could take this and like get started up initially pretty quickly and


this is just the process that I would go through and I'm going to go through with


you today and we're going to try to get this up and running as quickly as


possible so I had this idea that looking at SimClear v2 there's a lot of things


to be done still in the space of let's say self teaching self distillation and


so on you know there's mean teacher and then there's what not and this is all


usually done in the semi supervised very few label regime and so on but we know


that these self supervised techniques can help you and supervised learning


and then in SimClear v2 you do semi supervised in that you do self


supervised and then fully supervised and then distillation like self


distillation there's there's all these kinds of interleaving stuff and I


thought okay what if I just take a pre-trained network that performs really


well on something and I self distill it into a bunch of student models like a


number like ten or so and then I like that's my ensemble model will that


perform better than the original model like this is a terrible idea and it's


probably not going to work like there's 99% chance it's not going to work but


let's try to test this today so I got my drink I got my carbs since it's


weekend and we're going to give this a shot all right so first thing we need


some sort of base to go from in research it's good to build your own stuff but a


lot of times if you want to be as fast as possible you want to go as quickly as


you can so here I found this repo thankfully with an MIT license so shout


out to who even fun I guess for training for putting up a repo training


these c410 models or training these Pytorch vision models on c410 c410 is a


small enough data set so that we can kind of work with it and these models


are already pre trained so I've cloned this repo and we're going to adjust that


so there is a first of all there is a download as you can see here which in


this report says it downloads these I have not done this before I have no clue


how this is gonna work out in this download script here downloads the


weights from box and I hope you can see this and then I guess you can load the


the pre trained weights with pre trained equals true and yeah we'll get into all


that later so the first thing we got to do is get this to run let's say so let's


look at this downloads thing first so the download thing is going to have a


URL it's going to use requests to get that URL and then save this into this


state dicts thing now what I usually want to do is I don't I want my folder


of code to only have code and not to be intermixed with data and code because


this is the thing that I'm gonna ship around to various servers and so on so


I'd rather have the code in one folder and then the data and like a central


folder so I'm not really fine with this sort of downloading the this right here


into into the folder that we have so what I'm going to do is I'm going to


change that such that it downloads it into a central folder so first we


already have OS so what we're going to do is we're going to get some like data


path going which is going to be our home folder OS path and I guess I also I


already have a c410 folder right here so we'll use this and then so half dot join


will join that and that's going to download it's not really data is it it's


more like models okay let's do this cool so data path is this is the models that


is going to download all right and then it unzips the file again so here it


unzips the file to the current working directory I don't want this so I'm going


to change that again to the models path all right


nope directory to path to zip file directory to extract to I think we're


fine right now so this download script is going to download the path the all


the weights there now I want this to happen sort of automatically while this


is in a server or while this is on a server so what I'm going to do is


probably just to so if this script runs you can see it runs the main but in the


other script I might just want to do this automatically so let's go to the


test script right here or let's say we go to the train script this is probably


the main script right here the train script so we have to somehow call this


other script here probably in the main function all right so let's import this


other so import c410 what was it called c410 download okay and here we're going


to call that and does this does this not download it if it already exists we have


to check that so a lot of this is just going to be you know beating the stuff


into beating stuff into into existence so if this zip file already exists we're


not going to we're not going to do anything right which leaves us open if


like if the unzipping fails then we're going to be in a kind of dumb path but


you know we'll risk it zip path would be that so let's if OS path exists zip path


then return okay so we're good in the download script what else do we need the


data set I probably already have the data set from torch vision so that's not


going to be an issue okay so here we're gonna call c410 download dot main all


right and that should do we can't really call that yet let's actually just run


this download script no no such file a directory probably need to make that


probably need to make that directory right okay if OS make there's models


path exist okay true yeah that should be something all right and we're downloading


so this is 2.4 gigabytes which can you know be put by itself let's put that


over there and while that's downloading let's check out the test script actually


let's check out the test script so this simply takes in this c410 module and


instantiates a trainer and as you can see it calls test on it so this should


not be too hard I'm going to guess this c410 module is a lightning module as you


can see right here it is we know how tensor sorry pi torch lightning works if


you don't know how patrick's lightning works pretty easy you configure this


module right here you configure a bunch of stuff like the data sets the training


step and so on and you're good to go so I guess what we're going to do is we're


going to change this train script and change it to our needs okay so let's


copy that let's go with train ensemble bang so this is what we're going to


change all right so first if the GPUs is a string then yada yada yada if it's to


then wow that's that's that's kind of a weird engineering quirk right here okay


what I want to do is make the GPU use transparent so we'll only ever use one


GPU so let's call that CUDA and put that to true and then we'll say da da da da


oh come on there is like a lot of stuff going on here let's so and then torch is


called torch I'd hate that can I do this can I import it like twice with


different names probably it's probably not very good but I'll do it um okay so


if CUDA is not available we'll just set the CUDA to false if th dot CUDA dot is


available cannot if it's not available then age params dot CUDA equals false


and then we'll set the GPUs to zero comma I guess that's what it expects if


else none and that should do it for the GPUs okay so second thing that we need


we're going to need we're going fit here and there is this logs directory where


the checkpoints are going to be saved I'm fine with that I just want to kind


of remove the logs directory at the beginning so I'll do that and whenever


we start this I'm going to remove the logs directory this is a controversial


move but you know um remove tree recursively delete delete the directory


yes logs good okay our download is done so what do we do next we might want to


do just try to test test something and here in the test thing we might want to


set the GPUs I don't have a GPU right here so none and the data directory is


going to be yeah I'll put it so


nope nope nope okay it doesn't find the it doesn't find the the state dicts and


so on now we're going to have to fix this we're going to have to fix the fact


that it doesn't load okay okay and that's probably going to be here in


these models so if I look in the dense net for example which we can learn and


there's this pre trained argument and what's that going to be it's oh that's


bad okay it like has a hard code at the fact as hard coded the fact that there


are there is this state dicts directory okay um yeah that's terrible terrible


terrible terrible so I guess this is going to be in every single one of these


models and that's not good so what we're going to do is probably always late


loaded without the pre trained and then kind of loaded ourselves from the from


the correct directory so what's the correct directory again we're going to


set the model deer we probably can just take that from the download script like


that state dicts okay and then we want the architecture I guess that's a thing


we can actually put the classifier here right here that's something we can so


it's going to be the classifier if you look in the state dicts directory I'm


going to guess you can models c4 10 state dicts we haven't unpacked it where


have we not where have we unpacked it to help help oh no have we unpacked it to


here we have not we have not so what is in here ah it's a c4 10 models something


and then state dicts okay so it's always going to be the architecture plus PT so


we can you know we can deal with that so it's going to be c4 10 models state


dicts that's fine and then it's always going to be the architecture plus a PT


so let's look at one of these models to see how this is loaded we've saw we've


seen this here so we simply want to load this state dict in and here it


constructs the thing this is let's do proper string interpolation shall we oh


device where this device come from we should check that out device is given


device device device CPU where is device given okay dense net device CPU oh I


guess device is always CPU and then then we map it to wherever I'm not entirely


sure so here we say set device I guess we can just get the device from


somewhere let's try it out okay so we're going to need this right here so we're


going to OS path join models path and something that's dot PT so and here we're


going to get the architecture which is the classifier cool so that's how we


load something and then the device maybe we can just go torch CUDA dot get device


is that possible let's try


nope okay nope no get device device maybe nope


map location was given okay so we have to figure out where this device comes


from honestly here no module there's this get classifier right here but


that's just says pre trained this device always CPU I just can't believe that I


guess I'll believe it will always load to the CPU okay cool we can do that I


guess pytorch lightning will then put it on the GPU for us cool so this is about


how far I got when I try to do this by myself and now the problem starts missing


keys in state dick a lot of missing stuff we can't we can't possibly load


that yeah no not going to so we can't load stuff what does it do


load file name equals and then let's paste this and let's put some kind of


breakpoint here so we can check it out okay that exists no she feels like that


should exist yeah that exists what's the what's the deal what's the matter here


so we got model which is I guess a resonant 18 and we got this thing that


we might want to load so why doesn't it work torch load load file name see that


works so that's the state dict is that let's look at its keys we got a you know


a bunch of stuff okay so why can't we load that model load state dict state


dict and now unexpected keys in state dick missing keys so this is always


prepended with model dot and here it's not okay what do we do about that I


guess this is because we loaded ourselves okay cool so our model is not


yes so our model has the sub path model so we need model dot model dot load


state dict right look at us we made it so this is testing I guess this is this


resonant 18 or whatnot so we can leave that to run for itself so we figured out


how to load this stuff took us a while now let's go ahead and we know how to


load the models we know how to load the weights so this is our teacher model


right our teacher model is supposed to load up the weights and then and then


teach the student models so here what does this training thing do we download


the thing we make our GPUs to be really good okay and then we instantiate this


module right here as you can see so now we're going to check out this module by


the way the testing is done and as you can see there's an accuracy of 93.33


which I'm pretty happy with this is congruent with what we saw right here


the resonant 18 do okay and we can I guess we can take a resonant 18 or a


resonant 50 they're both fairly small right here so a lot of them are going to


fit on our GPUs once we use the GPUs so let's change this module around right


here to actually do the to actually do the let's say the the proper thing that


we wanted to do so here we have self dot model as you can see and it's get


classifier and the question is does it load it pre-trained so what we want to


do is this is going to be our teacher model and this in this get classifier we


want pre-trained to be false always right here we don't want any sort of we


don't want to load the pre-trained instead what we want to do is we


actually want to have the we want to load it ourselves right so here pre-trained


false and now we're going from our test script we're going to take over the path


they think the code that we used to load this okay all right so but a beam but a


boom OS we don't have OS that coming along just fine yep yep yep so now here


we're going to have our self teacher model to load that state dict all right


so this is it for initialization now we also need our student models of course


so our student models are going to be a bunch of models models are going to be a


bunch of models where what do we say so this is going to be a torch or like a


module list there's this module list I don't know them


torch dot nn dot module list right so I initialize that with a list and the list


is going to be get me the classifier and we're just gonna go for the same kind of


classifiers right now to really boil it down to have the same architecture for


the student and for the teachers for bar in range in range and here we probably


need a flag so H params dot num students okay so these are going to be our


student model so let's quickly create this num students thing right here I'll


probably have to have an integer and we'll go with five students for now okay


so we're creating five students all of them are not pre-trained so we're going


to are we going to train them from scratch or do we want actually to take


over the weights we probably don't want to take over the weights let's just


train them from scratch in a distillation mode I have no clue about


this stuff by the way okay I guess this concludes this already concludes what we


what we wanted to do so I guess this module list what can we do with it does


anyone know I don't know by the way I'm sorry for the switching between the dark


and the bright background I don't know how to fix that so pi torch and an


module list it would be nice if we could give them some names right so I guess


that's just an iterable right here so probably there's nothing that we can do


to give them proper names or we'd have to hack around and I don't want to do


that so I guess we can just check if that actually computes until here so


let's check it out let's try the ensemble it doesn't data set not found


or corrupted okay so what we'll have to do is we'll have to implement have to


change this data directory right here so the data deer is going to be OS this


whatever my c410 directory is no such file directory logs okay so logs doesn't


exist so let's actually make it still no such file a directory logs why why


doesn't it make it no such file a directory logs ah okay we need to ignore


errors here and we're good okay so it computes until the point you probably


you probably can't see that right I guess now you can see it let's check


yeah now you can see it all right so where are we we are at the point right


here in our module after we've created the teacher and the students so if we


look at self technically we should be able to see right here a whole bunch of


resnet 18s whole bunch so here you can see the teacher model right and I'm


going to guess you can see layer 4 and here you can see the student models so


the student models are going to be in a whole list of models and now we're going


to train them so since they're initialized differently our hope is


going to be that they're sort of going to end up at different places we're


going to train them with the same like we're going to be really really stupid


about this okay all right so let's be really stupid about it so what what are


we gonna have to change here is our training step and our training step is


actually fine we'll simply forward we'll get a loss from that and then we are


going to return that and that's going to be back propped so in our optimizer


wherever we initialize our optimizer we should probably give it the parameters


that are not only the student model parameters right not the teacher model


parameters so it should only train the student models okay and even even like


that we should probably always set the teacher model in Eve in eval mode but


we'll do that in the forward step right here so in the forward step we get


images and labels and here it runs it just forward through the model we want


to change that we actually want to have teacher predictions which we're going to


have the teacher model we're going to forward this through the teacher models


now the criterion I'm gonna guess is a cross entropy so the predictions here


are actually going to be logits right and this is this is good except that


what we want to do is have a distribution of over labels so after the


teacher here runs through and let's put a break point right here and actually


look at it I find it's always easy if you go and just run until the point


where you're are at the code and then you can just look at stuff so here


there's oh there's a validation sanity check okay probably don't want that and


now we have the break right here and now we can look at teacher predictions dot


shape so that's a batch size times ten and if we look at it I'm going to guess


there's some negative numbers in there so that's not going to be that those are


going to be logits now we want them that to be a soft max over the last dimension


and that's going to be of the same shape but of course now we're going to have a


proper distribution so if we sum over the last dimension you should see a


bunch of ones all right so the teacher predictions are going to be soft max


over the last dimension and since we since we don't want to back prop through


the teacher we can do this in an environment of no grad right here so we


have that with not being stupid and we also set the teacher model into eval


mode so I guess that does it set train no


that should do it I have no idea yeah let's let's run it again we could have


done that there okay so so far so good so we have the teacher predictions now


what we need to do is run them through the student and use them as labels so


we'll go for student in student models we'll go student forward or we simply


run the images through that and that give us the logits and then we use our


loss function on the logits and that not the labels but the teacher predictions


right so we never actually use the labels here as you can see and that's


going to be the student loss and now we have a bunch of losses and we're going


to append that


nope like this and our loss is simply going to be the sum of all the student


losses not even the average I guess we could losses I guess we could make it


the average just so if we change the number of students will will get some


kind of some sort of a better sense of the actual numbers what what what okay I


think over here we're good yeah so our teacher model is not in training mode


but our student models hopefully are in training mode no is this the eval pass I


guess this is the eval pass this is the validation sanity check pass okay so


this is going to be our loss and our accuracy now right so okay what's going


to be our accuracy our accuracy is going to be we have these student losses all


of them and what we are going to do is we're simply going to take the maximum


prediction across the students pretty easy pretty easy but we need to collect


the logits so come on so we'll also have the logits append the student logits


okay so we have a whole bunch of logits right here and we'll get some


predictions out of that now the question is do we want to simply take the mode or


do we actually want to run a softmax over each and then take the average


prediction I'm not super super sure but we can try to do it in different


different ways so right now we might just want to take the maybe the average


logit and then run a softmax on top of that because I'm gonna guess the logits


are outputs of a linear layer so they might behave more in a linear fashion


than if we were to average the actual probabilities that come out right maybe


let's let's do this okay so we'll go we'll take these logits they're all and


we need to somehow concatenate those or stack them so how we're gonna stack them


so they're 256 their batch size by number of classes so we'll just stack


them at dimension zero I guess that's fine and then we are going to mean also


across dimension zero so those are going to be our logits or final logits and


then our predictions are going to be the argmax of the logits in the last


dimension yep that should be pretty straightforward I guess that's it


easy as that yes the rest here should just do by itself and I'm going to go


ahead and run give this another run and see where we run into problems can't


really see how this could ever go wrong


we'll just take everything over okay we actually got a problem 1d target tensor


expected multi-target not supported so the cross entropy loss in pytorch does


not support that let's let's give it a shot make this a little bigger for you


and let's go for the cross P loss I can't type today so here we have the


cross entropy loss and the cross entropy loss is useful when training


classification problem with the classes yada yada yada wait should be one okay


criterion expects a class index as the target okay so what we need is like a


soft loss right we don't need this cross entropy loss we actually want we want to


have soft targets so what do we do we want to do I think the cross entropy


loss is a combination of the here of the log softmax and the NLL loss can we take


the NLL loss maybe so the NLL loss right here is going to be the target that this


loss expect should be a class index no okay that's not good so next let's go do


we have we somehow need a soft cross entropy loss let's search for that


pytorch soft cross entropy soft classes I guess people do that kind of stuff so


the problem with these kind of losses is that what you do what you have to do is


kind of protect yourself against against numerical instabilities right so what we


want to do is find a function that does this for us I guess if we do the lot the


the log softmax that should take care of it for us okay this is tensor flow okay


following thread cross entropy loss I guess people just do really the log


softmax and then do that and we should be fine with this okay thanks


K Frank yeah maybe maybe this has advanced since then so we can give like


a last look at this and this is a bit too big I'm sorry your eyes are gonna


have to suffer and we're going to look at loss functions and we're going to


just look through them multi-label soft margin loss mmm multi-label we don't


really want multi-label right we want this but not with the targets okay I


guess we're just gonna have to write this ourselves so ultimately what is the


cross entropy that cross entropy is simply the probability of the true label


times the log probability of the wrong or of the predicted label if you as you


see right here so we are going to simply multiply target times the log


probability of the predicted label and then some some don't take that mean


across the batch I guess yeah that should do we can implement this let's do


it so this criterion right here is going to be our loss function and that's only


used once so what we can do is going to be a function so we're going to take


student logits and we're going to take teacher probabilities okay so how's that


gonna work out we're going to do the log soft max from the student logits so


and then dot does that exist log softmax functional okay we need functional and


student logits of that dimension so now we have properly normalized student log


it so that's going to be student log probes and then what we want to do is


simply multiply the teacher probes times the student log probes and the negative


of that is going to be our loss the question is do we want to sum that I


guess across this dimension or mean it I guess the sum sum should do all right


this is it easy as that why have we searched for so long so the criterion we


can simply replace that now by our loss function cool so let's run it again


yada yada yada okay sanity check maybe we should have taken like a smaller


model it sometimes pays off to you know start with a really small model small


model just so you can you can do these kind of things fast so here we have


dimension out of range today okay which is where is that in forward in line 78


let's go there line 78 okay here the max is not going to be as much fun let's


like let's go there I think this is some of these things change over time in


Pytorch so this code might be written when you know so what we have is we have


a max over the predictions and predictions oh it's already an arg max


so I guess we can remove this all whether or not that agrees with labels


dot data we don't need any of that dot float and we also don't need that we so


the accuracy is simply going to be the mean of this no I guess so we're here


can't we just do predictions equals labels yes and we want the sum of that


actually we want the float first and then we want the mean yeah that seems


reasonable so let's do it like this yeah that it doesn't float mean perfect how


could this be any more easy but this right here is all of it so validation


step accuracy corrects we'll just look at it once we done it and what I do is


usually just run it until it doesn't give me any mistakes anymore and then I


know I sort of have succeeded


okay we're pretty close I feel so it says grad can implicitly created only


for scalar outputs which probably means our loss function is not a scalar so


when we return the loss right here here we have the sum of the losses divided by


the length of the losses let's go here and check out what's up with that so


what we'll do I see this loss function here will output basically one loss for


each data point so what we need to do I guess is call mean on this or sum when


they created the criterion in the original one now we've thrown it away


look at the git diff right here so I guess this reduces how does this reduce


the cross entropy loss when we don't do anything cross entropy loss reduces


reduction mean okay so let's reduce with the mean so if we call loss then after


that we should call mean and then here I'm not so sure anymore we should divide


here because the learning rate is kind of tuned to the original loss size so I


guess we'll be content for now with summing up these things and over here I


guess we've we've solved that right no losses yeah see our losses is going to


be an entire tensor and now we just fix that right now okay so let's try it


again in the meantime what can we do we can so we already take care of our GPU


we take care of the logs one thing to do when with respect to this download stuff


right here is you know if you have a server or something and you let a lot of


things run in parallel what you want to do is make sure they don't all download


the same stuff at the same time that's pretty bad so what you want to do is


ideally have some sort of lock such that they coordinate and I usually use a file


lock for this so I'm gonna create that right here from import the file lock and


then I simply create the lock let's go file lock and here you have to input a


file so sometimes like yeah data lock I don't know you just pick some file and


that's the file that these these processes are going to sync on and then


once you do this you simply wrap all of it in a with lock so only one at a time


can go in in this function all right so that's that that should make us safe and


right here we're now training this is excellent we are training the students


now we need to do that on an actual GPU so I have multiple tools to ship this to


a GPU so first of all I can try to ship this to and try to ship this to a let's


say to to one GPU so the way you do that is first of all I want some sort of


unbuffered version of Python and then I have this do I even have this tool I do


have the tool okay so I'm gonna call one of our servers


and we don't know what's going on okay so cannot import name seed everything


from PyTorch lightning seed everything is that some kind of new thing in PyTorch


lightning I guess I have it apparently so here seed everything with zero why I


don't need that we are running without any seed here we are being really cool


okay next mistake cannot okay still same mistake of course since we don't yep


next mistake we'll just go through the mistake learning rate late learning


rate loggers so I guess we need to update PyTorch lightning on the servers


and I'll do that quickly okay I've updated PyTorch lightning let's check


out whether or not we can actually run something yeah we can run something so


this again is now downloading this on the server while this is happening


there's another thing we can do namely I have sort of sort of made a system to


run stuff on servers which I like a lot honestly so I guess we can try this out


how do I hidden with I okay so I want to first of all delete this delete this


yes I guess delete this cool and this git folder is a bit annoying let's


restructure because otherwise it will always ship the git folder with


everything up here does it do that yeah I don't like to have the code in the top


level so quickly make a sources directory move everything in there so


move the c410 models into the source directory move all the Python files into


the source directory no clear the logs and we're much better much better okay


much better so what we what what we will do is my system requires like a file and


I'm just gonna copy one from another project quickly okay so we're back and I


copied that over as you can see you basically give hyperparameters and it


blasts ever the hyperparameters through in a kind of a random search fashion it's


not too sophisticated but we can work with it so 10 that's the file right for


10 train ensemble yes that's the file cool and here we're just going to put


all of our hyperparameters and that will remove the logs file I'm okay with that


but on this


bang cool so what do we want we want basically we just want to try it like a


bunch of times and then see like average across it right that's all maybe we want


the the architecture to change so let's say the classifier is a resnet 18 or a


resnet 34 or a resnet 50 just so we have a bunch of stuff to to do okay and this


downloaded and is training on GPU hopefully if this works then we can


ship this off and we'll make this other hyperparameter that I like to use called


rep which is just basically a dummy parameter and so I can just repeat the


experiment a bunch of times and let's put that in here so this is really that


this has this has no effect except for randomizing it a bit I guess we can try


to seed stuff so whenever it says seed everything we'll just seed it with this


we'll call it seed no is it here this seed everything yeah so H params dot rep


sorry seed cool what this this is doing something nice you can see it so this is


unbuffered Python output thanks yeah so what other classifiers do we have we can


again we can try a bunch of them we can try all of them why don't we try all of


them like this then let's go into this rat file I don't know why I called it


rat I just want it like some three letter thing so yep like this and then


we can just take all of that crap and delete it and delete this and those are


going to be all our models so our classifier is going to consist of all of


this stuff let's I know I know I know I suck at them don't tell me actually tell


me I want VIM tips trying to learn something new like each week in VIM but


it is hard and tend to make myself actually do it so let's go let's go with


just one repetition so far if we if we are not sure we can still up the number


of repetitions we don't even have the rep right now this is called seed all


right so we have different classifiers and what we're going to need we also have


this num students right let's go with one with five and with twenty so here we


got one epoch done and we get a validation loss do we get a validation


accuracy validating validating I have no idea we'll cancel this right now and


we'll go ahead and just blast this on to our servers and hopefully that that's


gonna work I have no idea is everything fine everything's fine go no what what


cool and let me get back to you once this is finished all right we're back so


I've just written some code here to extract the results of that run and


something you know it's pretty interesting what came out so in these


plots you'll see on the x-axis of the number of students in the ensemble


remember these students are all trained from the same teacher the teacher you


can see an orange that's just a single teacher for reference you can see that


if you have one student model it sometimes underperforms or in sometimes


outperforms the single teacher model but then if you have more student models you


can see that there is a pretty monotonic relationship so here it's the reason


this fit doesn't finish here is because there is not enough space on the GPU for


that many student models but you can see that the relationship here is fairly


monotonic here it's a bit of a kink so the first idea like this this is really


astounding because these students have all been trained from that single


teacher and they have been trained for as long as the teacher has been trained


so they don't have more compute than a teacher they've been trained from


scratch not from some checkpoint or from the teacher weights it's simple