git for research basics: fundamentals, commits, branches, merging

Hi there. Today we're taking a look at Git and especially Git as it is used maybe in research collaborations.


So Git is like a tool to collaborate but when you research, like when you work on a paper together with other people,


you won't use a lot of the features that Git offers and that are usually described by Git.


So in this series I want to talk about how, what's kind of the most simple way to collaborate with people on like a research project using Git.


And today we're going to go over just the fundamentals, which makes everything else a lot easier.


So what you need to understand about Git is that fundamentally Git is a graph and it's a graph of commits.


What I mean by this. So let's say you have your paper, you write some things and then this is kind of version one.


And then you have another paper or the same paper and you kind of change this line here. That's version two and so on.


You have kind of this chain of versions that you would like to keep in store.


This is the classic example of version control where you would like to save these versions and do it in a way that you can at any point in time go back to any version previously.


And this is exactly what Git does without you having to kind of rename like people usually copy the file and then rename like this version two, version three, final version, really final version, really final version corrected, blah, blah, blah.


All right. So Git fundamentally is a graph and a graph of an object we call a commit.


So commit, which I'm going to represent as a bubble here, is simply a kind of an image of your hard drive or one folder of your hard drive at a particular point in time.


So this will contain all kind of files. Let's call this file a file b.


Oops, well, I meant to make a square here, but all the files that are in your folder, which is called the Git repository or it's not correct, but bear with me.


You have this folder and all the files in this folder.


When you make a commit, all these files are kind of saved as they are into one of these bubbles and they're saved forever basically in this in this status that they are.


So what you can do now is you can go ahead and make a second commit.


So you change a bunch of files. Let's say the file b is still the same. And but the file a has changed is now a prime, you make a second commit.


And the second commit references the first commit. So part of a commit, except the very first commit, part of a commit is always a pointer to its parent commit.


And especially if you look at the commits, they all have names. And the name of a commit is always its hash.


And the hash includes basically the hash of all the files that are in there. So a hash could be something like F5C259 and so on.


And for the next commit, the hash also includes the reference to the parent. That's why the integral part of a commit is to which parent it belongs.


This ultimately is what makes the graph kind of the graph. Every commit and references its parent.


So you can address every commit by its name, as I said, which is the hash of the commit. And you can also so the hash is like really long.


But you can also simply reference it by kind of the first couple of letters, as long as that's unique.


Git will let you do this whenever you need to reference some commit.


Alright, so we've discussed that basically a commit is a bunch of files as they are, and it saved in this state.


So Git's of course smart, it will only kind of save the diff from one to the other commit.


But you can just imagine that a commit is simply one, the status of a folder at a particular point in time.


So let me just take away these files here. There are a bunch of other things in Git.


So the kind of one concept that Git has is called a tag. A tag is a name for a commit that you give yourself.


And the tag is like a little flag that sticks in a commit. And you may say this v1, version one.


This is simply a tag. And as you make new commits and so on, the tag simply stays there.


And at any time, if you don't want to remember this big long hash, you can simply refer to this commit as v1.


Because that's the tag. It's kind of simple.


The next form of a little flag that you can append to a commit is called a branch.


And a branch, the difference between a tag and a branch. So a branch is also this flag, and we'll call it, I don't know, blah.


The difference is that when you are on this commit here, right here, and you make a commit on top of this commit.


What's called, you've checked out the blah branch. So right now you're looking at blah, which is this commit.


And you make a commit on top of this commit. What Git will do automatically for you is it will erase this flag and move it to this next commit.


Blah. So this is kind of, so you might know branch is kind of from Subversion or other version control technologies.


It's very similar, but in Git, a branch is simply like a tag. It's simply a name for a commit.


But with the additional property that when you make a commit on top of that commit, so when it has the commit as its parent,


then Git will move the branch, the little flag, to the new commit.


So basically, you always have that one branch, which is called master. It creates this automatically for you if you just have this little flag master.


Right. And you make a commit on top of master, which would cause master to go here. Master.


And so on. So people usually say they work on the master branch, which means they're simply making commits on top of the commit that currently has the master flag.


Git also allows you to move around both tags and these little branches basically to any commit.


So I could forcefully go erase this here and simply stick the master flag here.


And sometimes if we kind of decide these two commits are no good, we would simply do this.


We would simply take the master flag, put it here, and then when we make a new commit on top of the master now,


what we would make is we make a new commit point here. Then Git would move the master flag because it's a branch.


Master. And then we simply continue working here, working here and Git will happily move along this master.


So in Git, there is no need to actually delete commits or something like this.


What we can simply do is kind of move the branch that we're working on to the commit we like and garbage collection ultimately will at some point go and delete these two commits.


This is a bit more difficult once you collaborate with other people because they might actually have made commits that reference the commits that you just kind of deleted or so.


So it's a bit tricky, but ultimately this is something you can do.


So the next thing we're going to talk about is multiple branches.


Having multiple branches basically boils down to you have a few commits, you have your graph, and let's say this is your master branch.


So here we have master, but also, or let's make the one before, otherwise I don't have space.


Master. So what someone else would like to do, like, is say, hey, I want to like try out this new feature in code, it will probably change the code base and so on.


But I want to try it out. Maybe it'll introduce some bugs and so on.


And then what you can do is you can make a new branch f1.


It's called f1 for feature one. So and then you can I can make a commit on top of feature one.


Which would then move the feature one flag to here and so on.


I can make second and third commit and so on. Meanwhile, the other people working on the project or maybe even you yourself can work on top of this commit of top of the master branch.


So in kind of software engineering, this is typically used when one part of the team wants to implement a new feature.


But the other part of the team kind of continues to do bug fixes or things like this development on the version of the software that doesn't yet have the new feature.


But they kind of need to fix bugs. And since the new feature is not complete yet, they can both work on the same code base.


So. Each work on on their own branch, so to say.


And at the end, when feature one kind of is ready, the people say like, OK, we've implemented it. It's all good.


There's no bugs. We would like to integrate the feature one into the the main software, basically.


What you have to do is you would have to do a so-called merge.


A merge is a process that generates a merge commit and a merge commit is this thing here.


As you notice, it has more than one parent. It has in this case two parents where it kind of combines.


So from this commit here, both branches are based off of this commit and then changes were made, individual changes in this branch and in this branch.


So there's the possibility that, you know, people change different things.


And what the merge commit needs to do somehow is to bring together these changes.


So actually both branches might have changed the same file, but in a different way.


And now the question is, how do we merge these different files?


And that's kind of the last topic we're going to today.


How does Git do a merge?


So when we talk about merging, Git has a bunch of built-in algorithms that helps you with most of the time merging is automatic.


So if you have like files here, A and B and one in one branch, A is changed some here and in one branch B is changed.


Git simply assumes, well, this one branch has changed, A the other one hasn't.


So the changes, you know, they mean something. I'll take them.


And as well in B. So Git basically, whenever something has changed in one branch and not changed in the other,


it will assume that the changes are the thing that needs to continue to live basically.


It assumes that the changes were made for a reason and that reason should continue.


So one might be a bug fix. The other one might be the new feature.


The same goes in the same file. So when you have the same file and one branch in one branch,


something on top is changed and the other branch, something kind of on the bottom has changed.


Git simply assumes both changes are wanted and takes both.


The only kind of time when Git doesn't know what to do is when both branches change the same line.


So I'm going to represent this with, I don't know, but when both branches change the same line in the same file,


kind of or close by. So there are these algorithms that Git determines when there's a so-called merge conflict.


That's the only time where Git doesn't know what to do. And so as preliminary, it's a good idea to structure files in a line based fashion,


especially if you write kind of a lot.


A good practice is to put every sentence on a new line and not have like giant lines of multiple sentences,


because if you put every sentence on a new line, then you immediately kind of see where something was changed.


Whereas if you have this big paragraph and Git will simply tell you this line has changed,


which is an entire paragraph, and you don't see what's happening.


So when you have a merge conflict, Git asks you what to do. And it does this in a very simple way.


We're just going to kind of take a look here as a final demonstration.


So I have a Git repository here. Let me clear that. So as you can see, there's simply this test file in here.


And I've just made one commit, the initial commit. And let's look at this test file. It simply says, hello.


So what I can do is, for example, I can say hi. When I want to make a new commit,


first of all, Git status will always tell you kind of what you can do, what's happening in your Git repository.


Here it says change is not staged for commit modified test.txt. And it also tells you what you can do.


So it tells me, for example, use Git checkout dash dash with the file name to discard changes.


Or use Git add to update what will be committed.


I'll use Git add with this. So it tells me changes to be committed. Now it's green, as you can see.


So when I now type Git commit, it should commit these changes.


And this is a common occurrence in Git. Whenever you see a text editor opening,


Git expects you to type a text message, like a commit message in this case, like a log message basically.


The hashtags are comments, which will not go in here. This is all described right here actually in these comments.


The thing about these things is when you type an empty message, then Git will abort the commit.


So if you notice you've done something wrong, you can simply save this file with being empty, being nothing but comments basically.


Git will abort. So it's super useful. I'll just say added hi and then save this file.


So this is not a special thing. All you need to do, this is an editor, a text editor that edits a text file.


You simply need to save the file and close the editor and Git will be like, OK, cool, I'll continue.


So with Git log, now you can see we have two commits. We have my initial commit and we have the commit called added hi.


If you look at the test file, you see hi. So what we'll do now is finally we'll make two branches as we discussed before.


So this is my initial commit. I've made one more commit and we're on branch master right now, which Git status will tell you.


See, on branch master. So this is now master. What we'll do is we'll make a new branch called F1.


We'll make a commit on F1, meaning we'll move this F1.


Then we'll make a commit on top of master like this, which meaning we'll move this master.


And then we will merge F1 back into master such that kind of this master is here and at the end we can even kind of remove the F1 branch.


And we'll do this while we're having a merge conflict so that you see the whole process.


So, OK, so what I want to do is first I want to make a branch F1. For this we can use checkout minus B for making a new branch F1.


If the branch already exists, you simply need to check out, which means I simply go to where this branch is to the commit that the branch references to.


We also say we put head where to this commit. Head is always the thing you're looking at basically, the thing you've currently checked out.


So make a new branch F1 and we'll immediately switch to F1. If I type status, it says on branch F1.


It's still the same commit, but we're just in a different branch.


So we'll make kind of a change to this file here. I'm going to say hello. Cool.


Save the file. Status, it says it's modified. I want to add and commit it.


And there's a shortcut commit minus A minus M.


So the A simply says all the files that have changed, add them. So I don't need to add git add all the changed files separately.


Though this only counts for kind of changed files. If you have completely new files that git isn't tracking yet, you need to add them yourself.


But so here with a minus A, I skip the need to first add the files and with the minus M I can give directly the commit message. More O.


Cool. So now what we've done is we have made this commit here and moved the F1 flag to this commit.


What we'll do now is we'll go back to this commit, which is currently master branch, and we'll make this commit.


So first what we need to do is we'll go back to some commit, which is a checkout.


Checkout master since master is still referring to that commit.


As you can see, when I open the test file, there's no hello. It's the status from before. Hello.


I can now say I can now change the file in some other manner. In this case, I say hello because I want many E's.


And I can say I can commit this because I'm now on the branch master. It will make this new commit here and move the master branch to that.


More E. If you look at git log, you see all these commits on this kind of branch.


You don't see the commit on the F1 branch. For that, I would have to go back to the F1 branch.


I log and you see here it's a different story. After the added high commit, there's the more O commit.


Whereas up here, after the added high commit, there's the more E commit.


So merging. And by the way, merging also is not only happens when you have different branches,


but also because it's the same thing. When you collaborate with other people and these people make commits and you make commits independent of each other


and you kind of try to synchronize your work, often you need to do a merge. And then merge conflicts can also happen.


So what we do now is we come back to master because we get checkout master. There are shortcuts for all of these.


So we're on this branch right here. And what we want to do is we want to make the merge commit.


And so we want to merge F1 into master. So while I am on master, I can say git merge F1.


And it will try to merge, but it will tell me conflict, automatic merge failed, fixed conflicts and then commit the result.


I can say git status and it will tell me you're currently merging. You have unmerged paths.


And this test.txt file is both branches modified it. So I go into the test and this is very strange if you see it for the first time,


but it's actually very intuitive. So what it will do is wherever the line is that both branches have changed or wherever the block of lines is that both branches have changed.


It will basically indicate this with by writing directly into the file. So it will make these smaller, smaller, smaller, smaller, smaller than sign.


And then it says head, which means this is the thing you're currently looking at, which we know its master has changed this first line to this.


Hello, then it will be like equal equal equal equal. And then it will say, down here, we'll say the F1 branch has changed this line the same line to hello.


And it will it will denote this with like the end of this with like, larger, larger, larger, larger, greater than signs.


So what you need to do to in order to merge is simply you need to make this file as you wish it is in the merged state.


First of all, you can always start by kind of removing actually good practice maybe to remove these equal lines.


And then within this delimiters kind of change what you how you want the file to look. In essence, I simply want to have these O's here at the end.


There's one too many like this.


Or like this. I like that. So I'm going to call that the merge. And then I delete these lines. So this is the file that I would like the to the merged commit to have.


So what I can do, I save this file. Again, I say get status, it still tells me it's unmerged, but it tells me what to do.


It says use git add to mark resolution. Right? I've resolved it. Git add test txt.


Git status, and it says all conflicts fixed, but you are still merging use git commit to conclude merge. All right. Git commit.


Bam. And I'm still I'm still have to enter a commit message, which is already predefined here.


Saying I merged the branch f1. And there's conflict, there were conflicts. But that's fine.


I like this message. So I'm simply going to save the file right here.


And when I look into git log, it now gives me the full story. First, I have this added high commit, then I have the more o commit and the more e commit, which were in parallel to each other.


And then I merged both branches into one. So we're now right here.


What I can do now is I can even delete the f1 flag because I don't need it anymore, basically,


which I do by git branch minus the f1. And it says deleted branch f1. So no commits are actually deleted when you delete the branch.


It's simply the little flag is deleted. The only danger is when you delete the little flag and the name and you're unable to reach the commit from any other name.


But here, of course, we have this master. And by following this edge here, we can reach this commit just fine.


So git won't delete it or garbage collected. But it will also tell you when you're about to do something dangerous.


So don't worry. All right. So with this, I think you should already have many tools or many, many insights into git.


And another video, we're going to look at how to collaborate online with people, which isn't much harder than this.


It's simply like two more steps to do to push and pull your work from a server together with other people.


Reinforcement Learning with Augmented Data (Paper Explained)

Hi there. Today we're going to take a short look at reinforcement learning with augmented data.


This paper is by Michael Laskin, Kimmin Lee, and others from UC Berkeley and NYU.


So the reason why this is a short look is because I believe the statements made in the paper are quite short and small, but they're quite grandiose.


So we'll dive into it. The paper basically combines two things, reinforcement learning and data augmentation.


Now, reinforcement learning, we've talked about a number of times. It's basically where an agent is in a world and has to learn to solve an optimization problem by repeatedly interacting with the world.


You can see here, for example, this is the walker task, where this walker thing, it has two feet and basically needs to stand upright and walk for a number of steps.


The further you go, the better. So by repeatedly trying this and getting better and better at it, that is reinforcement learning.


The second part is the data augmentation. Now, data augmentation is a pretty standard practice in supervised learning.


What does it mean? So if you have a supervised learning task, for example, an image classification task, here is a picture of a cat and the label is cat.


Then you can feed this through your neural network, right, to arrive at a loss.


And but you only have so many pictures, right? You have a database and maybe you have, I don't know, one million images.


OK, usually what people do is they go, let's say a number of times, like 20 or 50 times through that database to basically have the model learn each image multiple times.


But what turns out to be more successful is if you do data augmentation, that means you have an in-between layer right here that takes this image and some modifies it in some small way.


This could be, for example, it blocks out part of the image. So it simply blocks out the square here and then you feed that through the model.


And then the next time the image comes up, it does something different. For example, it randomly crops the image to only the top right part here.


And then the next time it does a bit of a color jitter and then the next time it goes to grayscale and so on.


So supervised learning has found data augmentation to be quite beneficial because not only do you make the model learn what this picture is,


but you also make the model kind of learn some small variations of that picture where you can be pretty sure they would not change the label.


So you would not feed the model false information that generally makes it more robust to test time discrepancies.


So this paper has basically claims. If you want to do reinforcement learning, if you do simply do data augmentation with the input data to that reinforcement learning, it works much, much, much better.


Now, of course, we can expect since in supervised learning, this is a general trick that it would do something for reinforcement learning as well.


But this paper basically claims that this one plug in like here. So this is basically you plug this into your pipeline in the reinforcement learning.


This is basically as much of a gain as pretty much the last five years of research on reinforcement learning on these things.


So let's dive into it. This paper proposes just what I said, just plug in the data augmentation and then do reinforcement learning on the augmented data.


They use these data augmentations. So crop, we've already discussed it's a random crop.


Grayscale means that the picture goes to gray, black and white with a certain probability.


Cutout means that there is a little patch missing, like I said, cutout color the same but in a random color.


Flip means you flip the image horizontally or vertically according to a random probability.


Rotate is the same but you instead of flip you rotate it.


Random conv means you randomly convolve it with a filter. In this case, some red or blue or yellow filters.


And color jitter means that you kind of jitter around the colors in a sort of way that doesn't mess up the image too much.


So you basically just kind of change the colors on the image but the overall image still looks the same.


The only thing you have to pay attention to is that in your reinforcement learning pipeline, usually if you have a walker like this,


what you want to do is you have your network here and then you have your policy and your value function.


If you don't know what these are, I've treated them many times in reinforcement learning videos.


What you want to do is you simply don't want to take this one current observation in here but sometimes you want to take kind of the stacked of the last few frames


so that the model kind of gets an idea what happened during let's say the last one second.


So it can determine in this walker for example, it's not only important where the legs are which are up here right now.


It is also important their momentum, how they're moving right and you can determine that from the last few frames.


So sometimes it's beneficial to feed the last few frames and they say the important thing here is that these augmentations are applied consistently across the stacked frames.


So basically you select on an augmentation and on the scale of that augmentation and then you apply it to these stacked frames all the same.


And then in the next forward pass you have a different set of stacked frames then you can decide on a on a different augmentation.


So that's basically the only difference between the supervised setting and this setting is that you have to consistently apply the augmentation.


You have to consistently apply this here and during training.


So they formulate the classic proximal policy optimization here which is an actor critic method and the only time you have to really pay attention is when you plug the observation into these models here right here.


Then it needs to be the same augmentation sorry the same observation so that means the observation augmented with the same data with the same augmentation procedures.


All right. Getting it together.


Cool. So when you do this when you do that let's say when applying RAD which is the random random data augmentation to SAC which is soft actor critic right.


Our data augmentations are applied to the observation pass to Q and pi.


So sorry. This is the thing up here.


This is soft actor critic which is a state of the art of policy algorithm for continuous control problems and also you have to pay attention that when you feed the observations there the same observations like here and here.


And then proximal policy optimization is the one is a state of the art on policy algorithm for learning a continuous or discrete control policy.


OK. So as I said they simply drop this in there and then it turns out they outperform or match performance of many many baselines.


Here you can see curl.


I've made a video on curl which is another way of augmenting or pre training a for reinforcement learning then state of the art things like planet or dreamer.


I've made a video on dreamer as well and then pixel SAC and state SAC is sort of a cheating algorithm because it has access to the state whereas all the other methods only have access to the to the pixels.


And you can see that the data augmentation method which is basically just plain RL plain pure SAC plus the plus the data augmentation outperforms in many times all of these other baselines.


Now here is a criticism of me.


In order they never investigate they simply say wow this reaches the same performance or outperforms these other methods.


Now so it's the state of the art algorithm.


It's important to note here that this is on the DM control 100 K and 500 K benchmarks which means that there is a limit on the number of I believe frames from these control tasks that you get.


So you either get 100 K or you get 500 K frames.


So the difficulty is learning from limited data.


It's not state of the art reinforcement learning method overall.


It is the state of the art on this particular task on learning from limited data.


Now while I can believe that the augmentation would help here I it is completely unclear whether or not the augmentation gives the same benefits as like something like dreamer or whether the benefits from dreamer and the benefits from data augmentation are completely orthogonal.


So in this paper given that the claim is so simple that they make I would I would expect like an investigation what happens if I do dreamer plus the data augmentation.


Maybe they've done it somewhere and I just haven't seen it.


But it just seems like they they put this on the base basic RL algorithm and then they claim well look here it works well but they never show that.


So it could be that dreamer all this architecture what it simply does is basically recover these gains that you could get by data augmentation or it could be that it actually does something different.


But just reaches the same amount of gain right it just reaches the same amount and improvement and by combining them you could improve it further.


So not not just to get like a better number but combining the two would actually give a lot of hints as to whether or not this augmentation works in line with the other methods or whether the other methods are really doing something meaningfully different or not.


But this is just not done here and so they go into the they go into a a question of which data augmentations contribute the most and they get to the point where they say random crop is extremely effective.


So they have this table here where they just basically combine two augmentations and do you see. So for example this thing here means that you apply grayscale and then the rotate augmentation and that gets you to whatever 300 points in this walker.


If you apply crop and then crop it gets you to 920 points and beats everything else. So they say OK crop is the most the most effective.


And I have I have the sneaking suspicion that these augmentations are so effective simply because of how we set up these tasks right these reinforcement learning tasks they don't tend to be a real world.


They tend to be somewhat simulated. And as you can see here the the image is pretty clear so you can pretty clearly see that here is a thing there's no natural background or whatnot it's procedurally generated right there are these stars that could confuse the model a bit.


But still it is so easy visually this task that I'm going to guess the whole reason why these image augmentations help is simply because of the way these reinforcement learning tasks right now are set up.


And I would guess that if we had reinforcement learning in something like the real world that the image augmentation methods would help in about the way they help unsupervised tasks in in the same data for example ImageNet.


So that is my sneaking suspicion and this paper I want to say it sort of over claims it's how how absolutely great this works.


Of course it works great on these things but I think there needs to be an investigation of where why.


So here they have some attention maps on where the algorithm focuses and you can see when there is no data augmentation it sort of focuses on good points but when you do crop it focuses on this ridge here which makes sense right because that's the thing that needs to be kind of vertical in order for the walker to be stable.


And if you do other things then you can see it it doesn't really focus it focuses on different things so the crop method seems to make the model focus on the most important part of the image.


And as the same with the cheetah task here so if you don't do augmentation and some of the augmentation you can see that actually focuses on some of these background stars whereas in the cropped version it focuses on not on the stars but actually on the cheetah as a whole which probably makes sense.


Now again I have a bit of a I have a bit of a worry with these kinds of experiments because we already know that crop will give you a much better score right.


So who's to say that if we could train this thing here to the same score it wouldn't be paying attention to the same part.


What they're trying to make clear here is that it is dependent on the particular type of data augmentation that the model gets a better grip on the input but it is not really a valid comparison if we know that the crop agent performs a better score.


And it could simply be that that's the reason why the attention is better right that that it is actually solving the problem better.


So I mean of course this the fact that it's working better is due to the fact that you have crop augmented the data but the fact that is focusing on the correct parts is not a property of the crop augmentation but the property of the fact that it reaches a higher score.


That was a long winded complaint but I hope you get what I mean here.


The last thing they do is they investigate generalization performance.


So improving generalization on this open AI proc gen.


Now as I understand it this is a reinforcement learning task or a suite of tasks where you have procedurally generated levels.


So you can sort of train on a bunch of levels and then test the generalization to new levels that you haven't seen before.


So there's a jumper here and star pilot so they seem like this like a jump and run game or big fish.


I don't even know what you have to do in big fish but you can see that the levels seen here.


This is one example and on scene.


So in this example the background is very different and I'm going to guess in the jumper thing not only is the background different but also the kind of generated level how you have to jump is quite different.


So they investigate whether or not a agent trained on only the scene ones can generalize to the unseen ones and this table presents the results.


And as you can see the RAD with the crop or with other things outperform the pixel paste based PPOs.


Now there is some nuance to this table here.


First of all you can see that these this crop thing is now only the winner in one of these three tasks right in the in the big fish thing.


In there is another augmentation technique here that wins over at star pilot but you can see the difference is not that high and in the in the jumper with 200 levels.


So this is 100 or 200 levels.


The the original method is even the best.


So here again I believe this is evidence that that it is very much an interaction of these augmentations with the way the task is set up and not the augmentations themselves or the fact that you're augmenting.


For example if we look at this big fish we've seen okay the what seems to change here is mainly the background where as in the jumper example the entire level structure seems to change.


So then the the augmentation all of a sudden is not super effective anymore actually hurts.


So I'm just not I'm just not super convinced by the claims are making here and one of the claims I find is in particular rad with random crop achieves.


Oh wait there's that point down here.


Oh yeah achieves 55.8 percent gain over pixel based PPO.


Okay trained with 100 training levels outperforms the pixel based PPO with 200 training levels on both big fish and star pilot environment.


This shows that data augmentation can be more effective in learning generalizable representations compared to simply increasing the number of training environments.


I this statement so again how like why do you compare two different things if you don't like if you don't show that maybe they're orthogonal in fact they are probably orthogonal because even on the 200 levels you you gain over the pixel based PPO right.


So why the comparison.


And then second of all so here we see on the 100 levels this method is better than the pixel based PPO and then they claim that okay they are even better on 100 levels than the pixel based PPO on 200 levels and why that is true.


If you know if if if a is bigger than B then probably A is going to be bigger than B plus some epsilon and right.


And that doesn't I just think that doesn't really warrant their statement where they say oh look this is even better so as if the 100 levels of additional training were the standard measure of more data.


Like if there is going to be if you're better at the beginning there's going to be a certain amount of data where you're still better than the other method with more data and I don't find this super duper surprising but they make a big claim here out of this.


All right, so in conclusion, I hope I'm not too harsh on this paper it is a cool paper and of course it is cool findings but I have a big suspicion that the augmentation here works so well simply because of how we set up these RL tasks because they're visually quite.


Let's say easy and therefore these augmentations that are also our sort of easy abstractions of when an image is visually similar because all of these things right to us as humans, we say probably doesn't change anything if we just rotate the image.


And we, this is our prejudice, and we built this prejudice into these simulators for the RL tasks, so they will match up extremely well with these augmentations, and that's the reason I believe these things work, and maybe not as much the, the fact that you're augmenting.


Neural Networks from Scratch - P.8 Implementing Loss

What is going on everybody?


Welcome to part 8 of the neural networks from scratch videos.


In the previous video we learned how to calculate the categorical cross entropy as our metric


for loss.


And in this video what we want to talk about is how to incorporate that into our sort of


overall neural network framework that we've been building in this series.


So when we go to apply it there's a few changes that we need to make.


We need to make it a little bit more dynamic as well as address one of the hurdles that


you're going to find when you go to actually run this.


So let's get to it.


Up to this point with calculating loss we've only considered calculating this categorical


cross entropy from the softmax output of a single sample at a time.


But in reality we'll be feeding in multiple samples at a time or a batch of data which


will return to us a batch of softmax outputs.


From here we'd also have a batch of targets.


So let's imagine our three classes are dog, cat, or human.


Let's first consider a scenario where our targets are sparse as opposed to where our


targets are one hot and coated.


So imagine that our batch of targets are that the first sample is a dog, the second sample


is a cat, and the third sample is a cat.


That would mean then that the dog class represented as a single sparse value would just be a zero.


The cat target would be represented as a one and the final cat target for that third sample


would also be a one.


So this would be our sparse target vector for the softmax outputs.


So to visualize how they line up that first value corresponds to the first sample's softmax


outputs and the zero means that the target class was a zero.


So to get the confidences for this class we just grab the value at the index of zero of


the first of the softmax outputs which is a 0.7.


For the second value in the class target vector we have a class of one.


So we would be interested in the confidence of the element at the index of one for the


second softmax output vector.


This is a 0.5 in this case.


And then finally for the third sample in that batch the target was a one so we would again


match that to the softmax output confidence of 0.9 in that third sequence.


Now that we know how these targets line up to the confidence values that we're interested


in how might we program this in pure python?


So one way might just be to zip these two lists together then iterate the target index


and the softmax distribution where the desired confidence value would be the value at the


index of the target index in that softmax distribution.


That said we're going to incorporate numpy now which is faster and more concisely implemented.


So we can actually get rid of the zipping and iteration and instead we can do all of


this exact same thing in one line since the softmax outputs is now a numpy array with


numpy array properties and the output will now be a vector.


So what's happening in this last line might be a little confusing or hazy for some of


you.


So let's talk about it for a moment.


So because softmax outputs is now a numpy array we can actually index this numpy array


in a variety of ways thanks to numpy.


So the way that we're doing it here is we can actually pass lists of indices that we


are interested in and then it's by dimension.


So this will be at the first dimension what are the indices that we're interested in and


then the second dimension what are the indices that we're interested in.


So the 0, 1, 2 this is your 0, this is your 1, this is your 2 and then class targets is


right here 0, 1, 1.


So again that's going to be 0 here and then 1, 1.


So if that's kind of confusing to you I would suggest maybe just kind of tinker around with


this.


So for example if we replace class targets with just 0, 0, 0 it's going to take the 0th


element from each of these so 0.71 and then 0.02.


But in our case we're going to use class targets because these are the values that we're actually


interested in.


So it becomes just a really nice concise way to get at the exact values that we're interested


in in one clean line.


Now rather than hard coding the range of however long our values are we can actually just use


the Python range of the length of the softmax outputs.


And as we learned in the previous video categorical cross entropy in the case of classification


with neural networks simplifies to being the negative log of the target classes confidence.


So we just wrap this in a negative np.log the resulting values being our losses.


From here we calculate the loss for this batch as just the mean of the losses.


And then with that you might be thinking hey we're done right?


And you'd be wrong because we're eventually going to hit a problem with this and that


problem is 0.


The negative log of 0 is infinite so we have a slight issue.


Now if the confidence for the correct class is a 0 I suppose you could argue well it is


infinitely wrong but it's kind of a problem for us.


So if for example if we wrap this in negative np.log and we check our results right everything


is looking pretty good.


You get a higher loss if the further away you are.


So for example if we decrease this to 0.2 we can see the loss definitely increased to


that 1.6.


But if we make it a 0.0 we do get a warning it does still run but we see that the loss


is actually infinite.


And this wouldn't necessarily be the end of the world because that's still it's not not


necessarily wrong but the problem is when we go to wrap this in a numpy.mean so as soon


as you throw in one infinity everything just dies out because now the average the batch


loss is also infinite so just one time you get a 0 and it's game over for you.


One option that we have to handle for this is to clip the values in the range by some


fairly insignificant amount so for example 1e-7 and then we're no longer dealing with


infinity and we want to clip because we don't just simply want to shave off from the low


end of a confidence and cause some sort of bias so we're actually going to clip 1e-7


from 0 so we'll start the range at 1e-7 all the way up to 1-1e-7.


Thus we will just simply clip all of these predicted values with numpy.clip to ensure


that we don't wind up with this infinity problem.


So we'll start by defining a common loss class so we'll call this class loss and in this


loss class we're going to have a define calculate and the calculate is going to take self it


will take output and y so output is the output from the model y is the intended target values.


Inside the calculate what we want to do is we want to get those sample losses so we're


going to say sample losses equals self.forward and the forward method here will pass output


and y so the forward method is going to vary depending on which type of loss you are actually


using so that will vary but all of the losses are going to need to calculate loss so they


will share this kind of calculate method here.


So that's our sample losses then we want to again that will be like a vector of values


so what we want to do now is data loss or batch loss is going to be the numpy mean of


the sample losses and then finally we will just simply return the data loss.


So in this case we want to define the forward method for at least one loss function that


we intend to use which is going to be categorical cross entropy.


So just below this class we'll define another class and we will turn the page of our book


and this is going to be called class loss underscore categorical cross entropy.


This is going to inherit from the base loss class.


Now we just need to define our forward method that will take self and then y prediction


and y true.


So again ypred these are going to be the values from the neural network y true these are the


target training values.


So first thing we're going to do is we need to know what is the total length so we're


going to say samples equals len of really arguably whoops arguably either of those but


we will go with ypred.


Now what we want to do is clip these values so ypred clipped is equal to numpy dot clip


and we want to clip the y prediction to be between 1e negative 7 so a number close to


zero but not quite zero for infinity purposes and then to be not biased we're going to do


1 minus 1e negative 7.


So once we've got the clip done the next thing that we need to consider is sometimes people


will pass targets that are either in the form of scalar values so class one, class zero,


class one, class one or it will be one hot encoded so it might be zero one well let's


make it the same as it was above so no I had it right anyway and then one zero and just


to save myself from typing out the rest I'm going to say like this so in this case you've


got two samples but they were just scalar values in this case you've got two samples


but they're one hot encoded values so we want to be able to handle for both of these because


it's pretty common practice that people will use one or either it's not really any one


that is more common than the other people are kind of comfy using using both so you


want to be able to handle for whether someone has passed one or the other so the way that


we can find out which one has been passed is we can just do if len of y true y true


dot shape is equal to one this means they have passed scalar values so not one hot encoded


values just true scalar class values so in this case we're going to say correct confidences


equals y pre-clipped and then we're going to say range of samples and then y true so


let's solidify what's happening here in the correct confidences in the case where the


targets are a scalar form so the targets array is just a one dimensional array so first you


have the y pre-clipped values and we're indexing those by the range of however many samples


we have in the batch which really just means we want to reference all of the samples in


the batch along the second dimension of the batches clipped outputs we want to grab the


index of the y true value so this means from the zero is sample we grab the zero with value


from the first of the sample we grab the first if value and then from that final sample index


of two we grab that first if element again then we'll come down and then handle for the


other possibility here which would be L or well we're in the book we're saying L if but


I guess we'll handle it with an if L if len y true dot shape double equals two so in this


case it would be a one hot encoded vectors that are being passed here if that's true


what we want to say is correct underscore confidences is equal to NP dot sum of y pre-clipped


times y true and that sum operation is happening on axes equals one spot check make sure I


didn't screw that up I think so cool so in the second case the targets y true is a two


dimensional array because it's an array of one hot encoded vectors so the length of y


true dot shape is two in this case we're going to do a sum of the clipped values multiplied


by the y true array which is that array of the one hot encoded vectors and we're doing


this sum along axes one so what's in what ends up happening is we're multiplying each


of those vectors and so when we do that we're multiplying each individual element at the


same index in one vector with the other vector and so what's happening here is everything


winds up getting multiplied by zero except for the target class's confidence because


in the one hot encoded vector everything's a zero except for the target class so summing


all these together along axes one is giving us just those values that we want so these


are the two methods that we have for dynamically handling either one hot encoded vectors or


scalar class values it doesn't matter which ends up being passed here either one will


work so that's how we can handle for the correct confidences and then all we want to do from


this point is to do negative log likelihoods is equal to the negative np dot log of correct


confidences and then we return the negative log likelihoods so once we've gotten to this


point these negative log likelihoods it's going to be again the vector of values that


comes from to here to sample losses we take the mean we return that data loss that batch


loss so now what we'll do is using what we've just coded we'll define our loss function


as the loss categorical cross entropy object we will then calculate our actual loss by


saying loss function dot calculate and in here again we pass the y prediction versus


the target so the y prediction will be whatever your final output layer was that used that


softmax activation in our case that's activation 2 so basically the output is going to be activation


2 dot output and then our target was y because that just comes from all the way up here so


this gives us our actual loss value and we can even print that out print loss colon and


then loss we'll save that run that and sure enough we get a loss of 1.0986104 which does


indeed match the book so briefly we'd like to show a way to calculate accuracy since


you'd likely be calculating accuracy in your code around this step but we'll probably revisit


accuracy later so given your targets and your softmax outputs you can actually calculate


accuracy from here pretty easily you can run the outputs through a numpy dot argmax calculation


which will return the index value for whichever value is the greatest you can run this np dot


argmax on the entirety of the softmax outputs by utilizing the axes perimeter which notifies


numpy of which axes you actually want this calculated on and that would be one in this


case since it's a batch of these outputs so you can then say predictions is equal to the


argmax of axes one of the softmax outputs then we're just going to compare these two


and wherever the values are equal we have a true or a one a match if not then it's a


zero we got it wrong and then we would just sum up the average of this so this means that


the accuracy is going to be the mean of the equivalence check of these predictions which


were just argmax outputs from each sample output and the class targets so while accuracy


is practical and useful the metric that we are most interested in when training a neural


network is actually the loss metric since it is a metric of how wrong something is and


our goal is to decrease that loss now how do we decrease that loss well we know that


we can adjust the weights and biases of a neural network but how do we do that so the


optimization of weights and biases is the subject of the next and likely next few videos


so stay tuned for those again if you are interested in this and want it in physical form you can


get the neural networks from scratch book you can get that at nnfs.io otherwise I will


see you guys in the next video


Do ImageNet Classifiers Generalize to ImageNet? (Paper Explained)

Hi there, today we're looking at do ImageNet classifiers generalize to ImageNet by Benjamin


Reckt, Rebecca Roloffs, Ludwig Schmidt and Vaishal Shankar. So the premise of this paper is pretty


simple. We've been training models on ImageNet now for a while, almost 10 years to be exact. ImageNet


is this data set with a lot of images, millions of images, categorized into many thousands of


categories. Now the classic part of ImageNet that people know is that it has about 1.5 million


images in 1000 different classes. And this ImageNet was one of the main data sets now,


or has been for the last few years. And as you can see on the right here, the error rate year


after year was pretty much, I think, cut in half every year since 2012, when the first net, AlexNet,


was using deep learning instead of the classical visual computer vision approaches. So we've been


training on ImageNet for a while. And the idea or the question this paper asks if we collect a


second test set, right, so for ImageNet, we have a train and a test set. If we now collect a second


test set here, test v2, right? If we have a model that was trained on training here, and evaluated


on test, does it also perform well on this second test set, right? The idea of this being that maybe


over the years, we've tuned our hyper parameters and all such that the models perform well on that


particular test set, let's call this v1, right? And it might not, it might not be as successful


on a new test set. So this paper goes about collecting a test set to ImageNet in exactly


the way that the v1 test set was collected, right? So they, they try to match exactly the process of


how v1 was collected to create another test set, and then they evaluate models on that new test set.


They do this not only for ImageNet, but also for CIFAR 10, which is a much smaller data set, but


also a lot of computer vision algorithms are evaluated on CIFAR 10. So let's just put up a


hypothesis here. The hypothesis is that we have pretty much overfitted to ImageNet by now, this is


a very prestigious number to get if you have state of the art on ImageNet, and therefore tuning your


hyper parameters and your learning rate and everything such that it performs well on the test


set v1 is very likely. So this paper has the most important plots are like this, it has two axes,


and on the bottom axis is the performance on v1, right? And so performance, and that means accuracy,


basically, so accuracy, and on line two is v2 accuracy. Now, here is one, here is zero. This


line here means that if a model is performing 50% accuracy on v1, it is also performing 50% accuracy


on v2. So being on this line would basically mean we have not overfitted, and the model is


performing equally well on both both sets. So what what now, if we assume that we have overfitted,


if we have overfitted, we would assume that you know, the models that perform really poorly might


also, you know, they perform kind of, they're not really overfitted, but over the years, as we've


gotten better on v1, we stray away from this. So we get better on v1, but we don't really get


better on v2, right? And that means we've overfitted to v1. And this might even go down,


right? The more the more kind of we might overfit to v1, the worse we're actually going to get on v2.


So this, this is kind of a meta overfitting. So this is what we would expect if we overfit,


right, or overfit to v1. And I think this was the initial hypothesis behind the people that ran this


experiment to check, can we see an effect like this? Or is the effect more a continuous one,


where we don't overfit? And what they found was neither. And these are basically these interesting


plots here. So again, the dashed line here would be the not overfitting line. So what they find,


if you, for example, look at ImageNet, every dot here is a model, right? So this model right here


is performing with like a 67% accuracy on v1. And it is performing with something like a 53%


accuracy on v2. If you look at this line here, what that means is every model kind of drops by


about this much, right? So not only not, we don't see this, and we don't see this, but we see this


line is shifted down. And if you look closely, especially on c410, you can see the line rather


than being tilted like this is actually tilted a bit slanted upwards, right? So the angle is not,


is higher, is greater, the slope is greater than the one to one slope. This is extremely interesting.


If you think about what does that mean? It means that if you take a model, right, right here,


if you look at its order, it's, it's, it's a this, let's, let's look at this model here. This model


is number one, best model in the world, right? It will still be number one on v2. This model here


is number number three, rank three on v1. It will also be rank three on v2, right? So the order of


models is pretty much constant. So if a model is doing well on v1, it is also doing well on v2 in


relation to other models. But every model experiences this drop in in accuracy. And the


most interesting part is that and again, you can see this more here. The, the better you're doing,


the smaller this drop gets, right? This drop here is smaller than this drop here. This is exactly


counter to the notion of overfitting where it seems the more accurate you get on v1, the more


you're able to close this gap between v1 and v2. And if you extrapolate here, you might as well


think that once we are at at, or actually sorry, 100% is already here, if you could go high, maybe,


or maybe you can see here that in the end, the these will actually converge. But nevertheless,


if the models that are doing better on v1 aren't only not overfit, but they are actually experienced


less of a drop with regards to a new test set. So they generalize better to the new test set than


the the worst models, right. And that is crazy. And it is not only neural networks, right. So up


here, for up here, you have the deep neural networks and whatnot. But you can also go with,


I believe some of these or even further down here are k nearest neighbor, sorry, k nearest


neighbor classifiers, and things like this. So it doesn't seem to be a property of neural networks,


it really seems to be a property of the data set. And this paper, first of all, goes over how they


collected these. And second of all, their hypotheses and investigations into why this phenomenon exists,


why we are all of a sudden, worse on the new test set, but completely worse in a different way than


we expected, compared to the original test set. Alright, so they first say potential causes of


accuracy drops. And they propose a model, they say, here are two, here is the entire difference


between two data sets with regards to a classifier, it can be decomposed into three different gaps.


And you see the first and the last part here are the ones from the left side. So this is an expanding


sum in the middle. So there is the generalization gap. The generalization gap refers to the gap that


you have between different data sets of the same distribution. So this is like you know from the


train versus test set, you train on the training set, and then you have a generalization gap to the


test set. In this case, the generalization gap simply refers to the difference between the


generalization to the first and to the second set. They argue that this isn't really an issue here,


because the, because the, they say the the confidence, they can put up confidence intervals.


So if those were identically distributed, how much would the generalization gap be at maximum,


given some kind of confidence interval, 95% confidence interval would only give you plus


minus a 1% difference in generalization gap. So they rule out that this is the reason for the


big discrepancy, then have two others, they have the adaptivity gap and the distribution gap.


So the adaptivity gap is what we hypothesized at the beginning, it is the overfitting to the first


data set or to one of the two data sets. So if you have a big adaptivity gap, then you have


outfitted much more to one than to the other data set. Now, because of the shape of the curve,


being the way it is, they also rule out the adaptivity gap. And we we went over why, right,


because it would look completely different than it does. Now, the only thing remaining here is


this distribution gap. So they explain that this difference here most likely comes from the fact,


fact that the old and the new test set have a different distribution. And they go into why that


is. And I'm going to compress the, I'm going to compress their sort of hypothesis of why that is


into a, into a short summary, let's say we won't go over the entire paper. They basically say that


the mechanical Turk has a, the mechanical Turk part of the processing pipeline has a very big


influence. So what happens when you collect an image net test that you start with Flickr? This


is a big image database. And the images as far as I can understand, they they are tagged and you can


search for them and so on. So they start by going to Flickr and searching, searching for images. And


their ground truth class labels come you may know this from a system called WordNet. And WordNet is


sort of a linguistic classification of words into groups. So it would have hierarchically would have


animals, animal being a word and then below animal, it would have dog and then it would have


terrier. And it would have these hierarchical groupings of these words and the search on Flickr


for images. And then they put the images to a human rater. So the human rater in on a system


called mechanical Turk, you may know it's that you can just sign up there and do these kind of


tasks. They present the human with a grid of images and a class terrier. And they say, please


select all the images where a terrier appears. So the human might select this, this, this, and this


one, right. And that will give you what they call a selection frequency. So selection frequency. So


how often was a particular image selected, given that class? And of course, the higher so you do


this over many, sorry, the selection frequency is across many of these mechanical Turk workers. So


if for a given image, the selection frequency is high, let's say going towards 1.0. That means


every single human selected that image to be in the in this class. So you can be pretty sure it's


in the class. If it goes towards zero, then you can be pretty sure it's not in the class. And


this selection frequency, criteria, the paper thinks that this is the main criteria why the


data sets are of different of different difficulties, let's say, because even even though they try to


match the process exactly even what the questions they post to the Turk workers, they even restricted


their flicker date range to the date range where the original image net was collected. They still


think that there is a difference in how the mechanical Turk workers basically rated the


images or then after that, how they were selected using the selection frequency. So what they do is


they do different, they select different images depending on criteria, so they can test these


hypotheses. Their original v2 test set is called matched frequency. Now what you do in matched


frequency is you kind of play a little game, what you do is every now and then, you will implant an


image here of the v1 test set. So thereby, and of the v1 test set, either of this class or of another


class, so you can kind of do a quality control. And from this, you can now find out what is the


selection frequency of images in the v1 class for Terrier. And then you can simply select the same


one in the v2, right? So if if you know the selection frequency for v1 was point eight,


you can just put the threshold here at 0.8. And you know, you can be reasonably sure that you


have selected a similar difficulty. Or so you would think, right? So if they do it like this,


then they get this drop that you saw at the beginning. They also do this threshold point


seven, which I guess is arbitrary ish. And then they also say top images, where they say, for each


class, we chose the 10 images with the highest selection frequency. So these would be the sort


of easiest, easiest ones. And if you look at the graphs, and this is ImageNet for these different


datasets. So if you do the threshold point seven that they selected, now the old line was somewhere


here. Now the new line is much closer, right? You see that here. And if you do these top images,


so you just select the easy ones, the new line is actually above, right? This is now note, note that


the red line here is above the black line, while here it's below. It is still extremely interesting


that still there is this almost linear relationship between the V1 and V2 accuracies. And even here on


this easier data set there is. So they basically hypothesize by thresholding differently for the


new data set, you can, you have a very good grip on the difficulty of the new, of the new data set.


And so this process of matching the selection frequency, so this matched frequency data set,


it might actually not result in the same difficulty in data set. They do have some more


experiments where they experiment with different, different difficulties. So let's actually jump


down there. It's a bit of a jump because there are over 70 pages in this paper. The appendix has its


own content directory. That's how crazy that is. But I want to show you these plots. So here, what


they do is they do different bins of the, of this set. So they have bins of easy samples, less easy


samples, and so on. And you can see that you have a pretty good grip on this, where this line is. So


if you only take the easy samples, you're up here. And this is what we saw before. If this is the old


line, it's the red line, right? If you take the entire new test set, if you just take bin, the


second hardest bin, you're somewhere here, or here, or here. So you have a good, you have a good hold


on where this line is. But that still doesn't explain that if you try to follow the protocol


exactly, why does the accuracy drop? That, that is still a mystery, even though they say, here is a


variable that influences this a lot. If they try to set the variable as it was set in v1, it is not


equal. So the still mystery remains, I would say. So the last thing they do is they try to, to just


come up with a model for this. So their hypothesis now is that the new test set just is harder. And


they have a analytical kind of a formal model of why, if you assume certain things, this results in


this line, right? The, the, the really interesting thing about the paper is that the, the accuracies,


they all fall exactly on this line, there's this linear relationship, especially they say, if you


do like probit scaling of accuracies, there's this line. And so they put up a model where you say,


what if we assume that each example i has a difficulty, right? It's just a number how


difficult it is. And each model j has a probability of correctly classifying an image with difficulty


tau here, given by this function, right? So this here is the probability that the model will


classify an image correctly, giving that it's tau hard, right? And so this is an increasing function,


and they put up this following parameterization. This, this is the CDF of the, so they've put up


a model for this function now, right? For this, they say, if we assume that it is like this, that


each model has a sort of a skill number, so each model has a skill, and each image has a difficulty


that is tau. If we, if the skill is higher than the tau, probably it will classify correctly. If


the skill is lower, probably then this number is negative, it will classify it incorrectly. So this


is the CDF of a normal distribution, it goes something like this, right? And if these, the


zero point is here, so if this number here is zero, it's like 50-50, whether it will classify it correctly.


So if you assume this, and you assume a bunch of other Gaussian error distributions, then the, the


difficult, sorry, the performance of a model on the test set v2 is exactly the performance of the


model under test set v1 times this scalar here, plus this scalar here, which is a linear relationship.


So they put up a model for for this, of course, this doesn't explain anything, right? This doesn't


explain the phenomena, but they, it still gives a clue of why the linear relationship here might


result from the test sets having a different difficulty setting, or a different difficulty


properties. So they go on, after discussing related work, they go on to say what can one do, and


suggestions for future research, I especially like the super holdout. So if you ever make a data set,


if you ever make a data set, then make a super holdout set. And once you're like, almost out of


your career, just come up with it and say, Oh, I have this last data set here, that I made way back,


it will be fantastic. Alright, so I think this paper is very interesting. And I think everyone


that sees and reads this comes up with their own hypotheses of why this is and what's going on here.


They have investigated a lot of this, especially I want to highlight an experiment where they take


in part of v two here. So they split this one into a train and a test, sorry, train and a test. And


they put this and this training together into like a super train, right? So, so you train on both


things together, and they see whether it improves at this test set, right? You would think that if


you put this training in there, that it would improve, and it does improve, but it improves by


like a minuscule amount. So they've done a whole bunch of experiments like this to investigate


what's going on. This is all in this 70 page appendix that you can go over. Right, that was


what I had to say for this paper. If you like this video, consider subscribing and comment what you


Imagination-Augmented Agents for Deep Reinforcement Learning

Hi, today we're taking a look at Imagination Augmented Agents for Deep Reinforcement Learning.


This is a paper by DeepMind and has been in the news a bit recently, so we're going to


have a look at what it's all about.


Basically they claim that agents who have a model of the world perform better usually


than agents who don't.


But of course usually we don't have a model of the world, so they make the agent learn


a model of the world which it can then use to plan.


Now this learning of the model can of course be imperfect because it's learned, and so


they provide a way to work with imperfect environment models and combine them with a


model-free approach.


So what do we mean by models and model-free?


Basically what you can say is if you have a model of the world, you have kind of a machine,


say a box, and in this box you have a state s, and you feed the state to the machine and


you feed an action, and the model of the world will tell you what the s' the new state is


going to be.


So this is in the case where you exactly know how your environment works.


Now in a model-free approach what you would do is you would plan basically, you would


have a state and you would put that through some kind of a layered neural network and


out would come what action should I take right now.


So in the model-based approach you're trying to try out all these actions and tell you


look which one gives me kind of a desired final state, and in the model-free approach


you simply use the rewards to go directly and say here's my state, what should my action


be?


So this paper is a combination of both.


The basic architecture is here, so let's start from the very right.


We have two paths divided along this line.


The final policy, so which actions you're going to take and what kind of values you


can expect is going to be a result of two different models that are combined.


There's a model-free path, which means this is what we talked about.


Simply here is the state and you simply feed it through this neural network thing, blah


blah blah blah blah blah, out comes a policy or an action you should take.


But then there's also this other path, and this is the imagination path.


Basically consists a bunch of these rollout encoders, and these rollout encoders is just


the agent imagining the future.


So the agent doing some actions and looking at how they will perform.


So the agent has this done, there's this imagination core thingy.


What this consists of is a policy network and an environment model.


This environment model is really the core of the entire thing.


So this environment model, you basically learn from what you've seen so far.


So far you've taken certain actions, you're in certain states.


You've used this to learn the environment model that gives you from one state the next


state and the next reward.


So that's what you learn, of course also using neural networks and whatnot.


And you use that environment model to imagine the future.


So here in this imagination core, basically you put in your state, you get out some new


state and some reward, you feed the new state and you imagine another action.


Of course the actions aren't random, the actions you also take via this thing.


And this is where it loops all back.


This is now a model-free policy network that works with the environment model.


So basically in your imagination, you only use, if you look at the very right here, you


only use this right path because your imagination doesn't need to be super exact or super well


planned.


You can use the model-free approach that we kind of know kind of works for some problems.


You use this to generate your actions that you imagine and you use an environment model


in order to look how these actions will play out and that's how you imagine one step of


the future.


And you simply repeat this a couple of steps and then you have an entire what's called


a rollout which consists of these pairs of states and rewards.


And what you do then is you encode this rollout via this encoder which is in this case an


LSTM or something like this I think.


You encode all these states into one vector, into one embedding basically for this rollout.


And this embedding describes kind of this future imagined path.


Of course what you're going to hope is that somehow this encoding captures how you will


do in the future and how good this will be, so these states and rewards.


Once you have a couple of these rollouts, so once you've imagined a couple of different


futures you then aggregate them in this aggregator.


I think in their case they just concatenate these rollout encodings.


And then you feed this too to the big aggregator on top.


So the big aggregator on top can now combine the model-free path and the imagined futures.


So if the big aggregator thinks that the imagination isn't correct it can resort to the model-free


path, but it can also think that maybe it's correct or it can be kind of, if it's sure


it's correct it can fully trust these rollouts and perform actions according to that.


All of this is of course trained end-to-end.


There's a tiny piece we haven't looked at yet, namely how this policy network on the


left is learned.


And this is simply learned by, and I have to pay attention that I'm doing the right


thing here, so you take this big thing here, your final policy network, and you perform,


you kind of learn to copy its actions simply from the input.


So from this model-free input over here you take this input and you take the output of


your big policy network and you try to simply make a neural network that copies the outputs


given these inputs.


And that's kind of your small policy network in here that's simply model-free.


So the loop closes in a way that you use your learned model to then again imagine the future,


but of course for imagining the future, within imagining the future you can't have another


instance of this network because it would be infinite recursion, so you can only have


a model-free network.


Alright, that's it for the model.


Of course, yeah, there's a couple of tricks and how to encode these things.


Maybe they perform experiments, and this is maybe what you've seen in the media so far


of this game.


And this game is a game where you have to push around the brown boxes onto the red squares


using the green avatar that you have.


So this game is difficult because first of all the levels are generated randomly, so


there's no way you can hard code anything.


And second of all, if you push a box, say this box here, if you were to push it to the


right into the corner, you would have no way of getting it out again.


That's why I have to plan ahead and avoid such mistakes because they're not fixable,


so once you make the mistakes you can't go back and that's where planning comes in so


handy.


If you imagine this future and if your model is correct, or approximately correct, then


you can avoid such mistakes.


Of course that's the difficulty in this game and that's where the planning helps.


Note that they don't code in how the game works, so all these models get this pixel


input of the game and they have to kind of imagine the pixel output they're going to


get.


So that's increased difficulty.


So technically the method is model-free in the sense that there's really no coded model


of the world, just the pixels.


So they have performance comparisons where if you... and I find this on the right here


interesting you can see according to the unrolled depth, so how much steps into the future you


imagine, you can see it kind of flattens out after only about five steps.


Whereas the game usually lasts for about 50 steps, they say, so only imagining five steps


is already really helpful.


What I don't like here is that they compare to what they say this copy model... because


this here is a standard model-free comparison, so it's just a model-free agent and of course


it... or not of course, but it performs worse right here.


Because it has no imagination, but it also has less parameters, so they're trying to


compare it to something with the same amount of parameters and say, oh, we have this copy


model agent here and what the copy model agent is doing is simply it's for the environment


model.


It's the same architecture, but for the environment model it simply predicts the output as the


input.


So it simply says, oh, you do this action, the environment's going to be exactly the


same as it is now.


And I don't like it because basically this entire branch here becomes rather useless.


And so even though you have parameters in here, they're not useful.


So to say that this is a comparison with the model of the same amount of parameters, I


don't know, technically true.


Another thing that they do is they pre-train the environment model with a model-free agent.


So first they code a model-free agent, then they pre-train the environment model to then


use with this agent.


So it's not fully learned intent.


I can imagine they tried and it didn't work and this is how you get it to work.


So they also experiment with imperfect models.


And so they train the environment model only imperfectly.


And as you can see here, this is kind of the output you can get.


Say duplicates, you have kind of errors, twice your character here, you have like boxes within


the wall or all kinds of things.


And they basically show that if you try to classically plan using these models, these


bad models, you get nowhere.


This is a Monte Carlo sampler planner using a poor model and it performance degrades significantly


from when you use the good model, which is right here.


And the imagination agent is not affected by kind of the bad model, except that it takes


kind of longer to reach its high inaccuracy.


So there's a couple of other experiments and a couple of Pac-Man experiments where they


show you can learn one model to transfer kind of to play different games in this Pac-Man


world.


And that just works the more if you have very sparse rewards, which you can imagine, yes,


if you need to plan, then that's what you get.


You get the ability to earn more sparse rewards because you can kind of look ahead.


All right, so I think I'll conclude here with the discussion of this paper.


Stochastic RNNs without Teacher-Forcing

Hi everybody, my name is Florian and Janik was nice enough to host me here as a guest to talk about


Stochastic RNNs without Teacher-Forcing

Stochastic RNNs without teacher forcing. This is based on recent work,


Stochastic RNNs without Teacher-Forcing

Deep State Space Models for Unconditional Word Generation, which we presented at this year's New RIPs.


Stochastic RNNs without Teacher-Forcing

And if you feel like any more details, please check out the paper.


Stochastic RNNs without Teacher-Forcing

We focus on a de facto standard training hack for any RNNs that generate text.


Stochastic RNNs without Teacher-Forcing

It's called teacher forcing and it's used in any model, whether unconditional or conditional,


Stochastic RNNs without Teacher-Forcing

such as in a sentence autoencoder or in a translation model.


Stochastic RNNs without Teacher-Forcing

To understand where teacher forcing comes from, we first need to understand where text generation comes from.


Stochastic RNNs without Teacher-Forcing

For the good or the bad, and here we will focus on the bad, text generation has its roots in language modeling.


Stochastic RNNs without Teacher-Forcing

So language modeling is the problem of predicting the next word given all the previous words.


Stochastic RNNs without Teacher-Forcing

People used to use n-gram models for this, but today people use recurrent neural networks to do that.


Stochastic RNNs without Teacher-Forcing

Such recurrent neural networks, or RNNs, factorize the joint observation probability of a sequence,


Stochastic RNNs without Teacher-Forcing

that I here depict as w, into independent softmax distributions over individual tokens.


Stochastic RNNs without Teacher-Forcing

So for every time step, there's a softmax function, and the softmax is conditioned on a hidden state.


Stochastic RNNs without Teacher-Forcing

And all the magic of the RNN goes into the function that gives you the new state given the old hidden state.


Stochastic RNNs without Teacher-Forcing

Usually this is called a transition function, f, and as an input it gets the last state and the last word.


Stochastic RNNs without Teacher-Forcing

So f could be a GOU function or an LSTM function.


Stochastic RNNs without Teacher-Forcing

Just like any other language model, you can turn this into a generative model of text.


Stochastic RNNs without Teacher-Forcing

Let's look at the dependencies that you would have at test time.


Stochastic RNNs without Teacher-Forcing

There's an initial hidden state h1, we sample a new word, we use our transition function f, and it gives us the new state h2.


Stochastic RNNs without Teacher-Forcing

Then we can sample a new word w2, feed it back, get a new state, sample a new word, feed it back.


Stochastic RNNs without Teacher-Forcing

It's important to note that all the stochasticity in the output is solely due to the stochasticity in the sampling process,


Stochastic RNNs without Teacher-Forcing

because the transition function is deterministic.


Stochastic RNNs without Teacher-Forcing

So far there's nothing to complain about.


Stochastic RNNs without Teacher-Forcing

But so far I've only talked about test time.


Stochastic RNNs without Teacher-Forcing

At training time there is a catch.


Stochastic RNNs without Teacher-Forcing

This is where teacher forcing kicks in.


Stochastic RNNs without Teacher-Forcing

It turns out that you can't learn this model by basing the evolution of the hidden states on your own predictions.


Stochastic RNNs without Teacher-Forcing

You have to use teacher forcing and that means you substitute your own prediction by the ground truth.


Stochastic RNNs without Teacher-Forcing

So at training time there's no sampling loop, you just take the ground truth token and feed it into your state transition function.


Stochastic RNNs without Teacher-Forcing

So that feels unintuitive because at test time we do something else than we do at training time.


Stochastic RNNs without Teacher-Forcing

It's also known in the literature for a few years to cause biases.


Stochastic RNNs without Teacher-Forcing

So why is that problematic?


Stochastic RNNs without Teacher-Forcing

Remember we come from language modeling.


Stochastic RNNs without Teacher-Forcing

In language modeling we could argue that if our only goal is to predict one word given the previous words,


Stochastic RNNs without Teacher-Forcing

then of course we can use the ground truth context to ground truth previous words.


Stochastic RNNs without Teacher-Forcing

But if we're interested in generating like longer sequences, then we need to learn what to memorize.


Stochastic RNNs without Teacher-Forcing

And in particular we need to become robust against our own predictions,


Stochastic RNNs without Teacher-Forcing

because we might make mistakes at test time and there's no ground truth at test time.


Stochastic RNNs without Teacher-Forcing

Just to get this confirmed by somebody who has worked in the field for years,


Stochastic RNNs without Teacher-Forcing

at the NeurIPS representation learning workshop Alex Graef mentioned teacher forcing as one of the big three problems for autoregressive models.


Stochastic RNNs without Teacher-Forcing

And in his own words, teacher forcing might lead to predict one step ahead, not many, and potentially brittle generation and myopic representations.


Stochastic RNNs without Teacher-Forcing

How have people addressed teacher forcing so far?


Stochastic RNNs without Teacher-Forcing

There are approaches to try to mitigate the problem, for example by blending together these two views, training time and test time,


Stochastic RNNs without Teacher-Forcing

so that sometimes you use your own prediction during training, but sometimes you use the ground truth.


Stochastic RNNs without Teacher-Forcing

We believe for rigorous model of text generation we need a rigorous model of uncertainty.


Stochastic RNNs without Teacher-Forcing

This should be an integral part of any generative model,


Stochastic RNNs without Teacher-Forcing

and therefore it should be the same model both at training time and test time without any hacks.


Stochastic RNNs without Teacher-Forcing

We propose a fundamentally different approach by proposing a new transition function.


Stochastic RNNs without Teacher-Forcing

The new transition function is non-autoregressive.


Stochastic RNNs without Teacher-Forcing

That means it depends on the last stage, ht-1, but it doesn't depend on the last word.


Stochastic RNNs without Teacher-Forcing

That means teacher forcing is not an option anymore, but it also means teacher forcing is not a problem anymore.


Stochastic RNNs without Teacher-Forcing

Instead, the transition function accepts a white noise vector as the second input.


Stochastic RNNs without Teacher-Forcing

Now you might wonder why do we need noise at all as an input to the transition function?


Stochastic RNNs without Teacher-Forcing

Well, for a given prefix there might be different continuations,


Stochastic RNNs without Teacher-Forcing

so we need some source of entropy to model the entropy in different continuations.


Stochastic RNNs without Teacher-Forcing

The rest of the paper pretty much focuses on the following two questions.


Stochastic RNNs without Teacher-Forcing

A. Which function f is powerful enough to turn the most simple noise source, just the standard Gaussian vector,


Stochastic RNNs without Teacher-Forcing

into something that is powerful enough to replace the autoregressive feedback mechanism of a standard RNN?


Stochastic RNNs without Teacher-Forcing

And the second question is, of course, how do we train this? What framework do we train this in?


Stochastic RNNs without Teacher-Forcing

And it will turn out that variational flows are suitable functions f,


Stochastic RNNs without Teacher-Forcing

and variational inference is the right framework to train them.


Stochastic RNNs without Teacher-Forcing

So here's the roadmap to complete the model.


Stochastic RNNs without Teacher-Forcing

First, we need to cast the generative model as a probabilistic method, because so far I've only sketched a procedure


Stochastic RNNs without Teacher-Forcing

that involves sampling some noise and then applying some function and then predicting observations.


Stochastic RNNs without Teacher-Forcing

Then we need to propose a variational inference model so that we can do maximum likelihood training.


Stochastic RNNs without Teacher-Forcing

We will derive an elbow, which is our objective.


Stochastic RNNs without Teacher-Forcing

Then in the paper we also describe how the tightness of the elbow can be improved.


Stochastic RNNs without Teacher-Forcing

And here I will finish by talking a bit about evaluation and what we do to inspect the model.


Stochastic RNNs without Teacher-Forcing

Since this work is based a lot on variational flows, let me give you a quick summary of variational flows.


Stochastic RNNs without Teacher-Forcing

A variational flow is a diffimomorphism f, which maps from what I will call a simple noise space, x i, to a complex noise space, h.


Stochastic RNNs without Teacher-Forcing

And here I'm already using the notation for our sequence model.


Stochastic RNNs without Teacher-Forcing

Simply by the change of variable formula, we know that the probability of an event h in the complex space


Stochastic RNNs without Teacher-Forcing

is simply the probability of the event in the simple space, x i, as given by the inverse of f,


Stochastic RNNs without Teacher-Forcing

times a Jacobian term with respect to f, evaluated at x i.


Stochastic RNNs without Teacher-Forcing

How can we use this in our sequential setting?


Stochastic RNNs without Teacher-Forcing

First, let me fix some notation, because sequential models are pretty prone to overloaded notation.


Stochastic RNNs without Teacher-Forcing

I'll write time as t, running from 1 to capital T.


Stochastic RNNs without Teacher-Forcing

And whenever I talk about a sequence of variables, like w, I don't index them, I just write w without an index.


Stochastic RNNs without Teacher-Forcing

And only when I need a specific element, I'll write it as w t.


Stochastic RNNs without Teacher-Forcing

Let's formalize the generative model.


Stochastic RNNs without Teacher-Forcing

We start out with the probability of observing a sequence w.


Stochastic RNNs without Teacher-Forcing

And since we use the latent variable model, we marginalize out the latent variables h.


Stochastic RNNs without Teacher-Forcing

And then we will assume that the overall dependencies between hidden states h and observations w


Stochastic RNNs without Teacher-Forcing

follow like an HMM type of dependency.


Stochastic RNNs without Teacher-Forcing

That means the new state only depends on the last state, and the current observation only depends on the current state.


Stochastic RNNs without Teacher-Forcing

Now the question is, how do we model these transitions?


Stochastic RNNs without Teacher-Forcing

I've so far pitched the ideas of sampling noise and then using some transition function f.


Stochastic RNNs without Teacher-Forcing

And we've seen flows already. Now we are ready to combine the two.


Stochastic RNNs without Teacher-Forcing

We propose a transition function f g, which has the signature as I mentioned before.


Stochastic RNNs without Teacher-Forcing

It gets a hidden state and a noise vector as an input, and it gives you a new state as an output.


Stochastic RNNs without Teacher-Forcing

This can be seen as a conditional flow, because any h t minus 1, any last state, inserted as the first argument into f g,


Stochastic RNNs without Teacher-Forcing

induces a flow which maps from the simple noise distribution to the space of new hidden states.


Stochastic RNNs without Teacher-Forcing

And as I've said before, for the prior distribution in the simple noise space, we simply assume it's a standard Gaussian.


Stochastic RNNs without Teacher-Forcing

Let's look at this graphically, because in the end this is a graphical model.


Stochastic RNNs without Teacher-Forcing

I copied over the formulas from the last slide, and at the bottom you see the graphical model.


Stochastic RNNs without Teacher-Forcing

First we have a sequence of stochastic variables psi.


Stochastic RNNs without Teacher-Forcing

Those deterministically induce via the transition function f, via the flow, a sequence of hidden states.


Stochastic RNNs without Teacher-Forcing

And those independently predict the observations.


Stochastic RNNs without Teacher-Forcing

All the magic is in the transition. So let me sketch this process here in the big circle.


Stochastic RNNs without Teacher-Forcing

How do we get from the last state h 2 to the new state h 3?


Stochastic RNNs without Teacher-Forcing

Let's say h 2 encodes a prefix and there are two possible configurations.


Stochastic RNNs without Teacher-Forcing

They're equally likely in the corpus, so there are two potential new states, the blue state h 3 and the yellow state h 3.


Stochastic RNNs without Teacher-Forcing

I've sketched the standard Gaussian noise distribution at the top.


Stochastic RNNs without Teacher-Forcing

There are yellow samples and there are blue samples.


Stochastic RNNs without Teacher-Forcing

The flow realizes a mapping that takes any yellow sample and maps it to the yellow hidden state,


Stochastic RNNs without Teacher-Forcing

and it maps any blue sample to the blue hidden state.


Stochastic RNNs without Teacher-Forcing

So with probability 1.5 in this situation, we either get a blue or a yellow sample from the simple noise distribution,


Stochastic RNNs without Teacher-Forcing

and it will induce new states, blue h 3 or the yellow h 3.


Stochastic RNNs without Teacher-Forcing

So far we have proposed the generative model.


Stochastic RNNs without Teacher-Forcing

Now the question is how do we train it if we don't know the hidden states?


Stochastic RNNs without Teacher-Forcing

The answer is variational inference and in particular amortized variational inference.


Stochastic RNNs without Teacher-Forcing

The key idea of variational inference is to introduce a parameterized approximate inference model.


Stochastic RNNs without Teacher-Forcing

How do we propose such a model?


Stochastic RNNs without Teacher-Forcing

Well, a good recipe is to first look at a true posterior, the probability of a state sequence given an observation sequence.


Stochastic RNNs without Teacher-Forcing

The true posterior turns out to factorize into individual components,


Stochastic RNNs without Teacher-Forcing

which give us the probability of a state given the last state and the future observations.


Stochastic RNNs without Teacher-Forcing

It turns out that we can formulate this inference model using two ingredients that should be familiar.


Stochastic RNNs without Teacher-Forcing

First, we use a transition function, Fq, which induces a flow.


Stochastic RNNs without Teacher-Forcing

It has the same signature as Fg for the generative model, and we use a noise source, Q.


Stochastic RNNs without Teacher-Forcing

But now the noise source isn't uninformative anymore.


Stochastic RNNs without Teacher-Forcing

In variational inference, the inference network is informed about the data.


Stochastic RNNs without Teacher-Forcing

So there's a base distribution Q of xi t, which is allowed to look at the data Wt.


Stochastic RNNs without Teacher-Forcing

Now compare this to teacher forcing.


Stochastic RNNs without Teacher-Forcing

In teacher forcing, we substitute our own predictions by inserting ground truth information into the generative model.


Stochastic RNNs without Teacher-Forcing

In variational inference, it's very clear how to use the data.


Stochastic RNNs without Teacher-Forcing

The data enters through the inference model and it enters in the form of future observation


Stochastic RNNs without Teacher-Forcing

because the past observation we want to store in the hidden state.


Stochastic RNNs without Teacher-Forcing

It remains to derive an ELBO, which is the usual evidence lower bound objective used for variational inference.


Stochastic RNNs without Teacher-Forcing

Any ELBO, whether it's in a sequential setting or not, factorizes into two parts,


Stochastic RNNs without Teacher-Forcing

a reconstruction loss and a model mismatch term.


Stochastic RNNs without Teacher-Forcing

Here reconstruction loss means probability of an observation given a state.


Stochastic RNNs without Teacher-Forcing

And model mismatch is between the generative model P and the inference model Q.


Stochastic RNNs without Teacher-Forcing

This is what is usually written as a KL divergence.


Stochastic RNNs without Teacher-Forcing

To derive our ELBO, we follow the literature on flows.


Stochastic RNNs without Teacher-Forcing

In the first step, we introduced the flow on the inference model FQ.


Stochastic RNNs without Teacher-Forcing

We turn the expectation with respect to the complex state space H into an expectation with respect to the simple noise distribution.


Stochastic RNNs without Teacher-Forcing

And then of course, at the same time, the flow appears inside the expectation.


Stochastic RNNs without Teacher-Forcing

And we got the log determinant terms that I've mentioned before.


Stochastic RNNs without Teacher-Forcing

In the second step, we introduced the generative flow FG using the same change of variable technique.


Stochastic RNNs without Teacher-Forcing

It's possible to write out the ELBO in a way so that there's only one Jacobian term for both flows


Stochastic RNNs without Teacher-Forcing

and so that the generative model always appears as the inverse concatenated with the inference flow.


Stochastic RNNs without Teacher-Forcing

In a second, I'll show you what the interpretation of that is.


Stochastic RNNs without Teacher-Forcing

Let's quickly recap what we've seen so far.