Stochastic RNNs without Teacher-Forcing

There's a generative model. It consists of a generative flow FG and an uninformed noise source.


Stochastic RNNs without Teacher-Forcing

There's an inference model which contains an inference flow FQ and a simple base distribution across the noise variables Q of xi.


Stochastic RNNs without Teacher-Forcing

In the ELBO, the two flows appear concatenated and we can interpret this in the following way.


Stochastic RNNs without Teacher-Forcing

The inference model Q proposes a noise vector xi t that is informed about the future.


Stochastic RNNs without Teacher-Forcing

The inference flow maps this to a hidden state.


Stochastic RNNs without Teacher-Forcing

At the hidden state, the reconstruction loss lives.


Stochastic RNNs without Teacher-Forcing

This is where we pay a price for making a bad prediction.


Stochastic RNNs without Teacher-Forcing

However, the inference model cannot encode all the possible information about the future into the hidden state ht


Stochastic RNNs without Teacher-Forcing

because the mapping continues to the simple noise base of the generative model.


Stochastic RNNs without Teacher-Forcing

And the inference model must make sure that the proposal also covers significant probability mass under the uninformed prior.


Stochastic RNNs without Teacher-Forcing

This trade-off between reconstruction and model mismatch is common to all ELBOs.


Stochastic RNNs without Teacher-Forcing

But here we highlight the special situation where we have two flows, one for the inference model and one for the generative model.


Stochastic RNNs without Teacher-Forcing

In our paper, we also show how we can use the recently proposed important weighted autoencoder to improve the tightness of our bound, but I'll skip those steps here.


Stochastic RNNs without Teacher-Forcing

Instead, let's quickly talk about evaluation.


Stochastic RNNs without Teacher-Forcing

We apply our model to unconditional generation.


Stochastic RNNs without Teacher-Forcing

So why in hell would somebody look into unconditional generation?


Stochastic RNNs without Teacher-Forcing

Well, actually it turns out it's harder than conditional generation.


Stochastic RNNs without Teacher-Forcing

If you know what a French sentence looks like, it's much easier to continue a partial English translation.


Stochastic RNNs without Teacher-Forcing

But it's not only harder, it's also more interesting to inspect which information does a sequence model need to store and which information can it forget.


Stochastic RNNs without Teacher-Forcing

We use two metrics to evaluate our model.


Stochastic RNNs without Teacher-Forcing

First, we look at sequence cross entropy.


Stochastic RNNs without Teacher-Forcing

So we compare the model's sequence distribution to the data sequence distribution.


Stochastic RNNs without Teacher-Forcing

Usually estimating the data distribution is impossible.


Stochastic RNNs without Teacher-Forcing

You don't want to say that the probability of a sentence is how many times the sentence has appeared in the training data.


Stochastic RNNs without Teacher-Forcing

However, for words, we can use unigram frequencies of words in a corpus as a pretty reliable estimate.


Stochastic RNNs without Teacher-Forcing

Also, we can get an estimate of our model's probability assigned to a sequence by using MC sampling.


Stochastic RNNs without Teacher-Forcing

We take the marginal likelihood, sample k trajectories, and assess the probability that the trajectories assigned to the given sequence.


Stochastic RNNs without Teacher-Forcing

Since our model is not autoregressive, a sequence isn't tied to an observation.


Stochastic RNNs without Teacher-Forcing

So we can actually use the same sequences of hidden states to evaluate probabilities for all the words in the vocabulary.


Stochastic RNNs without Teacher-Forcing

Since we've pitched our noise model as the key to contribution to our generative model, we want to empirically verify that the model is being used.


Stochastic RNNs without Teacher-Forcing

Working with a clean probabilistic model allows us to use tools from probability theory to assess that.


Stochastic RNNs without Teacher-Forcing

We use the mutual information between a noise vector at time t and the observation at time t.


Stochastic RNNs without Teacher-Forcing

So this measures how much information in the output is actually due to the noise model.


Stochastic RNNs without Teacher-Forcing

Before showing you the numbers, let's quickly go across the parameterization of our model.


Stochastic RNNs without Teacher-Forcing

For the flows, we look at shift scaling transformations.


Stochastic RNNs without Teacher-Forcing

And if the scaling g is lower triangular, we can compute efficiently the Jacobian determinant.


Stochastic RNNs without Teacher-Forcing

We also look at real MVP and we compose flows by concatenation.


Stochastic RNNs without Teacher-Forcing

The base distribution of our inference model depends on the future observations, which we summarize using a GRU RNN.


Stochastic RNNs without Teacher-Forcing

The base distribution itself is a diagonal Gaussian.


Stochastic RNNs without Teacher-Forcing

We use a state size of 8 and also run some experiments for 16 and 32.


Stochastic RNNs without Teacher-Forcing

All the numbers are in the paper, so here are just the take-home messages.


Stochastic RNNs without Teacher-Forcing

We on par or better than a deterministic RNN with teacher forcing trainer at the same state size.


Stochastic RNNs without Teacher-Forcing

Also we observed that a powerful generative flow is essential to achieve good performance.


Stochastic RNNs without Teacher-Forcing

Furthermore, we can confirm that important weightless elbow improved the results.


Stochastic RNNs without Teacher-Forcing

This is the first model applying generative flows to sequence modeling.


Stochastic RNNs without Teacher-Forcing

So naturally, we are interested in comparing the expressiveness of fg and fq.


Stochastic RNNs without Teacher-Forcing

Our paper has a table that compares four choices for both flows.


Stochastic RNNs without Teacher-Forcing

Our findings are that the generative flow should be powerful and the inference flow should be slightly less powerful.


Stochastic RNNs without Teacher-Forcing

To understand our noise model, we look at the mutual information at every time step and show a box plot for all of them.


Stochastic RNNs without Teacher-Forcing

Initially, the mutual information is highest, which means the initial character is most important to remember.


Stochastic RNNs without Teacher-Forcing

The noise model is never being ignored and we see increased variance in the remaining time steps because we are averaging here across different sequences.


Stochastic RNNs without Teacher-Forcing

A non-autoregressive model needs to have lower entropy in the observation model, because any under entropy under the observation model is being forgotten because there is no feedback.


Stochastic RNNs without Teacher-Forcing

The purple line shows you the observation model entropy during training.


Stochastic RNNs without Teacher-Forcing

The dashed red line shows you the entropy on the observation model of a baseline.


Stochastic RNNs without Teacher-Forcing

So indeed, we have lower entropy in the observation model and at the same time in green you see the mutual information increasing.


Stochastic RNNs without Teacher-Forcing

Let's summarize our findings.


Stochastic RNNs without Teacher-Forcing

Using variational flows, non-autoregressive modeling of sequences is possible and teacher forcing is not necessary.


Stochastic RNNs without Teacher-Forcing

At the same time, we get a noise model that is the driving factor of the sequence model and is easy to interpret.


Stochastic RNNs without Teacher-Forcing


Matplotlib Tutorial 7 - loading data from files

Hello everybody and welcome to part 7 of our matplotlib tutorial series. In this video


we're going to be talking about loading data from files. So it's a really popular request


for people like how do I actually get data from a CSV file, let's say, is usually the


most common example, but I mean there's other files. How do I get that data from a CSV to


matplotlib? Now, depending on the file type and how that file is structured and all kinds


of stuff, I mean this process can vary, but for the most part there's two major ways to


do it. One, you can use the built-in CSV, but two, you can use NumPy. If you don't have


NumPy, same holds true for every time, it's basically what I told you in the beginning


too. Pip install and then it says 6 NumPy. That's it. If you're on a 64-bit version of


Python, then you might not have the ability to just easily download NumPy, so what you


would want to do, I'm pulling it up at the moment, is you want to head to this website


here. It's got the unofficial binaries website for all kinds of Windows packages and usually


what I do when I get here, control F, NumPy, hit NumPy, bam, here you go, these are the


wheels. If you don't know how to install a wheels file, the easiest way to do that is


to head on over to pythonprogramming.net, start learning and let's say here, pip, just


search for pip, how to download and install packages with pip, click on that bad boy,


wouldn't you know it, there's a tutorial for that and we'll move that ad for now. And then


so you can come down here and learn how you can install modules either with pip or even


wheels with pip, so if you're not familiar, go for it. So anyways, once you're ready and


you've got NumPy and all that, we can begin this tutorial. So, let's reinstate our PLT


legend and our labels here. We no longer want to have a pi at all, so just get rid of the


rest of the data basically. And so first we'll consider using the CSV, the built-in CSV,


so import CSV. So let's say you had a file and that file was called example.txt. So what


we can do is we can add that file, so here we go. This is just where I'm building, this


is where I'm writing the script. So this is the script that we're working on right now.


So wherever you've got it, you can just right click, new and let's make a text document


and we'll just call this example. The way that you can use CSV to open a TXT file is


you can use NumPy to open text files that are actually.csv, it does not matter. So


what we can do is edit with notepad or we can even just open it. This one's pretty simple


actually, we can just use a regular notepad here. So let's say you've got a file and it's


got separated by new line as the variable. So you might have a 1, 2, 3, 4, 5, 6, 7, 8,


9 and a 10 and you've got comma, comma, comma, comma, comma, comma, comma, comma, comma,


and then you've got random variables here. Let's just, you can put whatever you want,


you don't have to worry about copying me, just put some data in there. Okay, cool. Got


our sample file, save that, we'll close that and we're pretty much all set making the


file. Now how do we use CSV to read that file? Well first, let's make some empty lists. So


x equals an empty list, y equals an empty list, good to go. Now what we're going to


say is with open and because I'm on Windows I can reference this file locally. If you're


on Linux you may or may not be able to do this or if you're on Mac, I'm not sure if


Mac allows that or not, but you may or may not need to give the full path of the file.


On Windows you can just give the local path and that would be example.txt. I want to open


that with the intention to read with R and we're opening it as.csv file. Then what we're


going to say is the plots will equal csv.reader and we're opening or reading the CSV file


and then delimiter will be equal to a comma. Delimiter is, you'll hear what is it delimited


by or something. This is what they're separated by, so csv comma separated, you know, variable


or values I guess. The delimiter is a comma. In our case the delimiter is a comma, but


in some cases you might have a colon as the delimiter or who knows, it could be a lot


of stuff. So that will be plots. Then we're going to say for row in plots x.append and


right now when you read files generally it's loaded in strings so we need to convert them


to either a float or an int. Since we know they're all integers we'll convert them to


int. So int row 0 and then we'll say y.append int row 1. So plots was each line, right?


So it was basically reading it line by line or actually plots would be like the whole


list separated line by line. Then we're saying for row in plots we're going to append and


then each row is separated by these commas. So it's almost like it's already built and


we might even, we could have possibly gotten away doing this another way, but that's okay.


I don't really tend to use csv but I thought it would be worth showing csv since it's built


in. Anyway, so now we've got the data anyways, whatever. We can do plt.plot x comma y because


we just populated them and then we could say the label here is loaded from file. That should


be it. We can save and run that now and there it is. There's our data. It loaded this interesting


graph for you to check it out from our file that we wrote. Awesome. Now we can close this


and now let me show you another way and this is probably the more popular way and the reason


why is you can use this to work with dates data as well pretty easily. Also numpy is just


faster. It's not going to require us to do any conversions. This is pretty messy. So


I'll leave this here. I'll just, maybe what I'll do is I'll call this part one and then


I'm just going to comment it out with triple quotes and now we're going to do part two.


So part two will be import numpy as np. Again that's just, that's standards basically. Then


we'll make some space here and then this is where we're going to write our numpy code.


So what we can do with numpy is first of all we can unpack into x and y and then we're


going to say np.load txt so we can use numpy to load a text file into it and again this


doesn't have to be a.txt it just needs to be, it could be a.csv, it just needs to be


a file with text in it. So that's it. Anyway the file that we want to load same as before


example.txt we already made that. You do need to specify a delimiter that is a comma and


then you have to say unpack equals true. So unpacking like with most, many functions you


don't have to say like yeah unpack equals true you just do x comma y or whatever two


variables and it unpacks but basically what unpacking does is numpy.load txt basically


returns this list that each list has two elements in it so what we can do is like here we were


a little unable to unpack it we would have had to basically map the list or basically


to do that but we could have done this all in one line but it would have been complex.


But we're going to use numpy because numpy is complex but we don't have to worry about


the complexity we're just loading it we say what the delimiter is and we say hey we want


to pack or unpack the variables separated by that delimiter into x and then y. So that's


good but this will only work if you have two variables like if you added a z here you'll


get an error it will say too many values or not enough values to unpack and if you just


had an x it would say too many values to unpack. But anyway x y that should work and then again


we basically use this exact line so I'm just going to take this line copy paste and that's


that let me clean this up a little bit and now let's save and run that and again same


graph just used numpy to load it instead of csv. So anyways that's just a really quick


tutorial on loading information from a file another popular source of information is online


so we will actually be covering how to load data from the internet as well so don't worry


about that. So that's it for this tutorial if you got any questions or comments up to


this point please do feel free to leave them below otherwise as always thanks for watching


[ML News] DeepMind fails to get independence from Google

Hello, everyone. Today we're going to look at some news in the machine learning world.


The Wall Street Journal here writes Google unit DeepMind tried and failed to win AI autonomy


from parent. So apparently DeepMind has sought to become more independent of Google in the


past. And here they write that it's been founded in 2010 and bought by Google in 2014. And


starting in 2015, there were already talks as far as we want to be more independent.


Now apparently DeepMind told staff late last month that Google has called off those talks.


Here it says DeepMind founders had sought among other ideas, a legal structure used


by nonprofit groups reasoning that the powerful artificial intelligence they were researching


shouldn't be controlled by a single corporate entity. On the other hand, from Google's point


of view, the proposed structure didn't make financial sense for Alphabet given its total


investment in the unit and its willingness to bankroll DeepMind. So DeepMind sold itself


to Google because of money needs. Their research consumes ginormous quantities of energy and


of researchers and that costs a lot of money. So they cashed in 500 billion as a price.


Said it bought the startup for 500 million and the losses of the company were about $660


million. This company makes giant losses, because what they do is essentially PR. So


the position of Google here is that they want to bring the teams closer together and have


a stronger impact rather than separating the teams. This is an asset to Google a tech asset.


So for DeepMind it's pretty easy to push for a nonprofit structure given that you know,


they will never make profit ever and their claims to wanting to be open and not in the


hands of a single thing. I could take it more seriously if they were ever to publish in


open access journals, which they don't they publish in nature. Oh, you got to pay 20 bucks


for that article. Thanks DeepMind. Surely you don't want the technology to fall into


the hands of a select few. If they were to actually open source their code and not just


some crappy pseudocode that has lots of mistakes in it. I'm sure you want to just distribute


that stuff out of there. Because if it's just in the hand of a single minority, that would


be terrible. Right? Right? No, I think what they want is they recognize they got something


good going there. They got someone paying for their bills and they don't want someone


from top down telling them, hey, make it more into a product. Hey, give it to us. We need


it to make money. What are you talking about? Google wants this technology in their products


as fast as possible as best as possible. And DeepMind researchers are just really, really


smart people that output these things. Lastly, I want to show you this rendering of the proposed


new DeepMind offices in here. Like if that is not the most dystopian future picture I've


ever seen. I mean, it does look cool, but it is a bit on the elitist side, I would feel


it's a cool office, like, sure, I take it. Absolutely great. What I'm saying is you want


this on one hand, but then also you want giant loss making and independence on the other


hand, maybe that's not possible at the same time. I'm just not really sure that that is


the reason DeepMind seeks independence. All right, that was it for me. This is already


too long. Tell me what you think in the comments. What should DeepMind do? What should Google


do? Who's the good guy? Who's the bad guy? How should AI benefit all of humanity? Or


Python Pickle Module for saving objects (serialization)

What is going on everybody welcome to another Basics Python tutorial video.


This video is going to be concerning another standard library module called Pickle.


So what is Pickle? What are we even using it for and all that?


So Pickle is a serialization and deserialization module.


Basically it converts Python objects to byte stream and then back again.


Cool. So what does that mean?


Okay so basically what it's doing is, and in fact if you're coming from other languages,


you might know this as like flattening, serialization, marshalling, stuff like that.


But in Python everyone's going to refer to this as Pickling.


So try not to use those other words, some people might not even know what they mean.


So Pickling, again, is used to store Python objects.


So this could be like a whole lot of things.


This could be maybe an object you've created from a class.


This could be a variable. This could be obviously a variable.


But this could be like a list. This could be all kinds of things that you can Pickle.


So why might you want to Pickle or when are people using Pickling?


An example from our own tutorials is for example when we've Pickled a machine learning classifier.


So with machine learning when you create a classifier you have to, you create it by training a classifier.


The training process takes a while but what your end result is an object and that object is your classifier.


So you have one of two choices.


You can either somehow save that classifier or every time you want it you actually have to go back and retrain it every single time.


That's inefficient.


So we use Pickle so we can Pickle or serialize into ByteStream and then de-Pickle or de-serialize back to a Python object.


So let's go ahead and just show a quick example.


Let's go ahead and import Pickle.


Another example why you might use it too is say you have a large data set that you're using very frequently.


Every time you read in a data set either way you're connecting to like a MySQL database.


Maybe you've got a CSV.


Maybe you're just using a flat file or something.


The way you want to read in processes takes a while and if you Pickle it it's going to be a whole lot faster.


So keep that in mind.


It's going to be, I mean sometimes your loading time let's say compared to JSON or pulling in from a SQL database,


we're looking at 50 to 100 plus, sometimes up to a thousand or more times faster.


So it's useful just for anything that requires a lot of processing and ends in an object.


So anyways, importing Pickle.


Next thing, let's just create an example dictionary.


So we'll say example dict equals something and let's just say it's one colon something,


two colon something, and then three colon something.


And then let's just put some arbitrary things in there.


Good.


So now we have our example dictionary.


Now how will we output this or how do we Pickle this?


Well, we can do something like this.


Pickle out is going to equal open and we want to open dict.pickle.


And we're going to open it with the intention not only to write but we're writing bytes.


So WB for bytes.


If you're coming from Python 2.7 and you're watching this and you refused to make the switch to three,


just know you don't really need this.


This is a requirement for three.


So now you've got Pickle out.


Now that we have this Pickle out file, what we're going to do is we're going to say Pickle.dump.


So here we're referencing the Pickle module.


So Pickle.dump and then we pass some parameters.


First of all, what do we want to dump?


Well, we want to dump the example dictionary.


And then the second parameter, where do we want to dump it?


We want to dump it to Pickle out.


So we save that and then don't forget, obviously, after you write anything to files, go ahead and close it.


So let's say Pickle underscore out.close before we make those mistakes.


And that will save something to this dict.pickle.


It will go in the same directory as your script.


Feel free to go check there if you want.


Now how do we load the Pickle back in?


So how do we depickle something?


Well, we basically go through a pretty similar path.


In fact, let me pull up, let's pull up the Pickle.


This is the Pickle here.


Let's open it with Notepad, take a peek.


Okay, so that doesn't really quite look like anything we could read.


But that is, you know, our information.


It's just we don't recognize it ourselves.


It makes a whole lot of sense to the computer though.


So anyways, that aside, let's close this out.


How do we read that dictionary back into memory?


Well, let me go ahead and just comment those out.


If you don't know how to comment bulk, it's highlight everything, Alt 3.


And that will bulk comment something.


So now, let's pickle in.


So we're going to say Pickle, whoops, Pickle underscore in equals open.


And then we're going to open that same file, dict.pickle.


And we want to open it with the intention to read, but read what?


Bytes.


So now we have the Pickle.


And then we're going to, we just, now that we have that,


all we have to do is we're just going to say Pickle load.


So we would say something like this, example dict.


So we're just calling it the same thing.


But again, this is all commented out, so it's doing nothing.


Example dict equals pickle dot load.


What do we want to load?


We want to load this pickle in file.


And that's that.


So now what we can do is we can say let's just print the example dict.


And then to prove that it is indeed acting as the object that it truly is,


because when you save an object, it also saves the data type and the object type and all that.


So to prove that, let's just do print example dict and then to.


So if you recall, this would be the key of two.


So we should return a value of two.


I have no idea what just happened.


There we go.


I have never used this separate point clear break point.


Anyway, example dict to print that out so we can see that it's acting like a dictionary.


Sure enough, here is our dictionary and here is our value.


We can also do like three just so we can get a different number or letter.


Actually, I used F.


So that's pickling.


That's why you would pickle.


And it's pretty useful.


The only caveat you should take note is you might use pickles to transfer data between maybe some servers


or you might pick up a pickle or you might pickle some data from another source.


It's unlikely that you're just going to parse a website that has pickle or anything,


but you are going to you might have it being transferred via network.


Pickle has no security in place.


So you load that object, you run that object.


You better be certain about the security of that object.


Otherwise, you're going to run some you could run some malicious code.


So anyways, that's pickle.


In the next tutorial, we're going to be talking about eval, which is sort of like pickle.


If you wanted to save something into a string form or something like that,


but it's more commonly used if you want to read someone else's string version of something.


Anyways, that's what's up next.


If you have any questions or comments on this video, please feel free to leave them below.


Otherwise, as always, thanks for watching.


Thanks for all the support and subscriptions.


AlphaStar: Grandmaster level in StarCraft II using multi-agent reinforcement learning

All right, let's talk about AlphaStar Grandmaster level in Starcraft 2 using multi agent reinforcement


learning.


The corresponding paper looks like this and is by Oriol Vinyals et al from DeepMind and


has been published in the Journal of Nature recently.


Now let me say this first, stop publishing in Nature.


This is a journal is not open access, it makes its readers pay for getting the article.


So actually, you can access this article or a public version of it for free, but you can't


print it, you can't download it, unless you pay for it.


And this to me, it seems it seems ridiculous, because none of this money goes to the authors


of the article, none of this money goes to the reviewers, the review quality isn't notably


better at least in the field of computer science.


All of this is, is a publicity stunt by DeepMind because Nature has been kind of impactful


in the last decade.


It's like, Oh, look at me, I got a big dick I publish in Nature.


Not not nothing more than that.


It's like OpenAI saying their model is too dangerous to release to the world.


I guess DeepMind might make the same claim about AlphaStar.


It's like too dangerous of a Starcraft player.


Yeah, so stop, stop this.


Push your research in open access.


Nature or journals like these for computer science, it's a remnant of the last century.


So go on and join everyone else in in distributing knowledge.


Alright, rant over.


Let's jump in into this article so that the article describes how to train a reinforcement


learning agent to play the game of Starcraft 2.


So Starcraft 2 is this game for everyone who doesn't know, just very quickly explain the


game.


Starcraft 2 is a real time strategy game.


And you're kind of in this top third person view, and you control your units.


And the goal is kind of to move your units around.


And first of all, build up buildings.


And using those buildings, you can then produce more and more diverse units.


And ultimately, you want to kind of produce some sort of army that can go to the opponent


and destroy the opponent's base.


So you control all of this on a computer using a mouse and a keyboard.


And Starcraft is notable for being very balanced.


So there are three different races you can play.


So first are the Terran, which are kind of human human ish.


They have Marines and tanks and helicopters, I believe, and things like this.


Then the Protoss are some sort of alien race that are super advanced.


So they can teleport and have energy shields and things like that.


And then last are the Zerg.


And the Zerg are kind of icky ground dwelling creatures that infect things and spread like,


you know, like a disease.


So the interesting thing here is compared to other real time strategy games is that


the three races, they play very different.


So they have a the game is almost a different game if you play as a different race.


But they're so well balanced that almost any matchup is kind of a fair game between equally


skilled players.


So that's that's makes Starcraft pretty unique.


Also pretty unique is the very, very high action per minute rates that pro players get


to like they're in play this insanely fast.


So game lasts about 10 to 15 minutes.


And as I said, the goal is to destroy the enemy base.


All right.


So to train an RL agent to play this is very hard because like the action space is very


high.


You have to target with your mouse, part of the screen, right, you have to look what is


on the screen, what can I do?


Right, there's this mini map down here, there are things you could do, there are opponents


you can target and so on.


So all of this is very, very, very difficult for an RL agent.


And at the end, right, after whatever 10 minutes, right, you play, play, play, play, play, and


after 10 minutes, you either win or you lose.


And can the RL agent has to figure out which of the actions that I did during those 10


minutes, right?


Is it was this one was this one, which led to me winning or losing.


These are very hard problems for reinforcement learning.


And the DeepMind has combined almost every trick in the book known so far to RL to achieve


this.


Now the main, the main contribution, I'd say, here that is novel is what is called leak


training.


And we'll get to that.


So first of all, if you don't know what reinforcement learning is, reinforcement learning is basically


what I just described, you have an input, right, which is could be this thing here.


And you have a set of actions, you have a set of actions that you can do, which the


set of actions here is anywhere you can click, right, you can click anywhere on the screen.


And you have to do this over and over and over and over again, until you either win


or you lose.


And from that, you will see you will at the end receive Yeah, you win or you lose.


And then you have to kind of learn to play the game.


So it's machine learning hardcore, because you get minimal information and have to achieve


a lot of things from it.


So the first thing that DeepMind actually does is it does supervised learning.


And we'll get into how exactly the the model works later.


But first thing DeepMind does is it trains an agent to simply imitate humans, right?


So you have human data.


And from the human data, you so these are games played by humans, good humans, right?


Not not people like me.


So these these are games played with humans from from a significantly high elo.


And the first thing you extract is this Z here.


Now Z is is called a statistics vector.


And as I understand it, it's mainly the build order, which means in which order do you build


your buildings and units.


And this is very important in StarCraft.


This is a strategic decision where you say, Okay, first, I'm going to build three worker


units, this is like three workers, worker, worker, worker, and then I'm going to build


a house and then I'm going to and so on.


So these are major strategic decisions that that you kind of have to make with minutes,


minutes ahead of time to plan in advance.


And this this is kind of stays constant for the game.


So this is extracted and provided to the model as an input.


So what is the current strategy, basically, the current overall strategy?


The second thing that is extracted is this is at every time step, the observation that


the humans had, so the screen that humans see, and also the actions that the human did,


right?


So the human takes its mouse and click somewhere, right?


This is supposed to be a mouse pointer, and clicks here, right?


And then the model, this part here, this is the model.


And this is the policy function of the model.


So the policy decides what to do, right?


is trained to match the action that the human did.


So in essence, first, you train an agent to simply imitate humans.


And this you can do by supervised learning, right?


This is classic machine learning, each, each step you have this input, which is an image,


and you have the strategy you're trying to follow.


And from these two, you're simply trying to match the action that the human did, assuming


the human made a good decision.


So this is how you initialize, right?


You don't start from scratch.


Now I have to say that even though this name is alpha star, it has surprisingly little


to do with alpha go or alpha zero, that DeepMind has done before.


Mainly this is entirely model free reinforcement learning.


And and goes more into the direction of Yeah, classic, classic deep RL.


And you can see with the human data, you can already get pretty far.


So these down here are the leagues of StarCraft.


And this this here are percentiles of players.


And you see with the supervised training, you can get almost, you can get better than


80 85% of human players already.


Right?


So pretty, pretty impressive already simply by imitating humans.


Now so the the way to, to further improve this, and let's actually go first into how the model


looks like.


So down here, they describe this model.


That's it.


So the model is supposed to map from input to output.


So from the screen that the agent sees, right and some other things to what the agent is


going to do to an action a.


If you simply do this at every time step, then you have a game playing agent.


So first, the question is, of course, how does this happen?


Now the input isn't only the thing that the agency sees, which is this the mini map and


the mini map.


I believe that's the mini map or the entire map.


Well, it's in essence, it is a picture.


It is also a list of entities.


So the the game engine extracts a list of entities and these can be inside the screen


here and outside the screen.


For friendly, so the assumption is the agent knows about all of its units and where they


are and what their statistics are.


So in this entity thing, for each entity, you have a list of what is its health?


What is its type?


What is its position?


Does it carry any items and so on all the things you need to know about this entity?


This is in this list of entities.


And along with that also opponent entities, but only the ones that are on screen.


Right?


So all of this goes into this list of entities.


And then the next features are scalar features.


And as I understand it, scalar features are things like what race are you playing currently?


What time is it in the game and so on.


So these are additional features.


And also, baseline features.


And this, this is mainly used to train the value network.


And if you, this is not going to make sense if you know nothing about reinforcement learning.


But one main contribution of this paper is or not contribution or but kind of thing that


they claim is that for computing the value network, they also use the observations.


So all of this for of the opponent player, because you know this during training because


you're doing self play, and you don't need this value network during inference.


You can actually do this and this improves performance significantly.


Alright, so that's just for people who know RL very well.


Everyone else don't, don't worry too much about these things.


Alright, so these are the inputs, the scalar features, the entity and the minimap.


They go each one goes through separate encoders.


So the minimap goes through a ResNet, which is a convolutional network.


And the entities go through a transformer, which is kind of a thing to it's appropriate


to encode a set of entities, right?


scalar features go through a classic feed forward network, MLP.


All of these get combined here into a deep LSTM that goes over time.


Now the deep LSTM is what really makes the strategy because each time step, each time


step a screen like this is input into the into the thing that but the agent also needs


to remember what did it do last steps two steps ago, right?


This is important because you don't have full observability, you need to know what did I


do in the in the past.


And that's where the so if the last step you saw this screen and the step before you saw


this screen, right, then all of this would go through this encoding step into the LSTM,


right?


So the LSTM will encode now, over time, all of these different steps.


And so you can kind of say, Alright, if I have just started building a building, I should


probably not build the same building again, even though I can't see it on the screen,


Because I know that three steps ago, I did start building a build build a building.


Yeah, so this is kind of the LSTM is basically where you integrate your strategy over time.


So from the LSTM, you have to make two predictions, you have to make a prediction of what to do.


This is the action.


And how valuable is your current state?


This is called the value network.


Usually, this is a core component of deep reinforcement learning.


These two components, one is called the policy, which would be everything over here.


And what is called the value network, which is called everything over here.


These are the things you need to do actor critical learning.


And actor critical learning is the current state of the art in deep RL.


So deep mind does nothing else here.


Except, as I said, they use these baseline features for the value network.


But if you don't know what a value network is, don't worry about it.


The important part for playing the game is actually the part over here, the called the


policy.


So first, you need to do to decide what action you do.


And there are many action types in StarCraft.


As I already said, you can build a building, you can move a unit, you can actually move


the camera.


But it's not an action type, right, because you want to maybe see what's over here or


over here or over here.


So that's an action you can do.


And if you have decided on what action you want to do, you have to decide when do I do


it.


So you see the action type once you figured it out, it goes into the next neural network.


And that decides, okay, when do I do it?


When do I do this action?


So it specifies a delay.


Then once you've decided what to do and when to do it, it goes into the next neural network.


And that decides, should I put this into the queue of actions, because the agent here is


limited to a certain number of actions per second.


And I think it's 22 actions per five seconds or something like this.


So it in order to mimic, you know, human limitations.


So the there's a queue of actions to be executed.


And the agent needs to decide do I really want is this action so important to put it


into the queue?


Alright, if you have decided what to do, when to do it, whether you would like to do it


at all, right, then you have to, you have to say, it goes into the next neural network,


and you have to say, alright, which units do I want to do it with?


Right?


If you want to build a building, you can have to choose one or many workers to do it.


I don't actually know how StarCraft works in this.


I'm bit of a noob.


But you have to you have to select units with which to do the action for most of the thing.


And there I like the use of a pointer network here.


So what a pointer network is, is a network that can point to its own input.


It's sort of like an attention network, but not really in a pointer network, if you have


a set of inputs like we have here, so entity, entity, entity, entity, entity, right, all


these entities.


And you can see the entity embedding, the entity encoder actually has skip connections


that go here, right.


So this network directly gets these these entities as input, it can then write you then


you have a neural network on top of that neural network that the neural network takes all


of these things as an input.


And what the neural network will output is a pointer to one of these things, right?


You can say, look, I point to this thing right here.


This is a called a pointer network.


And yeah, as I said, it's different from an attention network, which might so an attention


network is where you get a distribution, actually, you get a distribution in both cases, there


is a difference, but we don't we don't have to really time to go into it here.


But in essence, with a pointer network, you can select which of these entities you want


to do something with.


All right.


Now you've decided on which action when whether to queue it with which unit to do it.


Now you have to decide for some actions, for example, if the action is attack, or heal


or something this target unit, which unit do you want to target?


Or which, which location on the map you want to target?


This is the target point here.


And you can see again, here are skip connections from the entity encoder, and from the spatial


encoder to these things.


And while the target unit is an attention network, as this like a much like a pointer


network, you will kind of point to places in lists.


The target point is a deconvolutional resnet.


What that means is so you have the spatial encoder here will embed the mini map.


So there will be a neural network right here.


Actually, let's draw the neural network in this color, right here, it will give you a


an embedding of that right and that's what you what you feed into.


That's what you feed, for example, into the LSTM.


But then what you do is you have a deconvolutional network, which again produces a mini map.


But on this mini map, there is not it's not the original mini map, but it's kind of a


distribution of locations.


So it said, here, here do I want to point?


All right.


So the that this neural network is responsible for producing this dot on the mini map, basically


saying, okay, I know what to do when to do it with which units to do it, and so on.


I want to do it right here on the mini map.


Okay, and now you have it right, you go from the input, which are these things, the mini


map, the entities, and so on, to what do I want to do where when with which units and


so on.


Right?


This is called a policy.


And it's extremely complicated.


Every one of these boxes here is a neural network.


And you can see it's it's very, it's a lot to train.


And they of course, they have a lot of resources since they are deep mind.


But that's the the main thing.


They have a few tricks to train this, and we won't go too much into this.


But one of the tricks is v trace from the impala paper, one of another trick is up go


up going policy update.


And a third trick is TD lambda learning here.


And all of these are kind of improvements onto classic actor critic reinforcement learning


style like a to see your a three c.


If you are interested, then you can, you know, look into these things.


So that's how they train it.


And the question now is, what's the protocol for training it we saw, okay, there is supervised


learning, cool.


Then there is reinforcement learning.


All right, but you can't just apply and this is in reinforcement learning.


This is what we said you get kind of a reward.


And the reward goes into this TD lambda, and v trace and and up going policy update to


train the value function and the policy.


But the special thing that this paper introduces is what's called leak training.


Now in in papers like alpha go, or alpha zero, what had been done is called self play.


And self play basically means you have an agent.


You have an agent, right, you have this agent.


That's this is supposed to be an artificial intelligence, right?


How do I make it artificial?


Okay, it has a little hat, right?


A funky hat.


It's a robot, and the robot will play a copy of itself, right?


And the copy might be slightly different.


But the it basically, these two, these two play each other, and thereby become better


and better and better.


And you can see this like over time, as as the purple one gets better, the blue one gets


better as well, because they kind of play against each other.


And when one falls behind, right, when one falls behind, then they simply copy over from


the other one, they basically copy the other one, and then they catch up again, right,


they catch up, right, and they continue competing.


So by competing against each other, they get better.


And this is called self play.


Now, people have noticed this kind of leads to instabilities, because you can get kind


of trapped, get trapped in cycles, like rock, paper, scissors cycles.


So what they do is they will actually, as they get better, so this is the first version,


right?


And the second version, they are a bit better now.


So they have bigger hats, right?


And here, bigger, bigger, larger hats, right?


And down here, they are even better, so they have like ginormous hats, but they might have


some weaknesses because they only play against each other, right?


So this is the same players, but over time, what they will do is they will actually play,


occasionally play old versions of the other player or of themselves.


Right?


Occasionally, the new versions will fall back and play old versions, or not only the current


versions of the agent or old versions of themselves.


So this, this is called fictitious self play in that you always play the, you not only


play the your current kind of opponent or your current self.


I mean, it's the same anyway, because you keep copying the weights, you also play the


old ones.


And this paper goes a step further and says, actually, we, we do this, but we want to prioritize


the good ones.


So for example, we know that we know that the current ones are good, right?


But we know that this particular one was also pretty good so far.


So we are we keep making, we keep making these these new ones play against this one more


often.


And this has led to kind of an improvement in these kind of self play algorithms.


The real new part of this alpha star paper is the fact that they do this league training.


And in the league training, they this this is what it looks like.


But I find this graphic rather confusing.


I'd rather explain it like something like this.


All right, so there is your current, your current strategy.


And you have a hat, right?


And you do all of the you do all of the all of the I play against myself with the smaller


hat thing, right?


I play against past versions of myself.


Fine.


But then you also do you have what's called exploiters and exploiters.


And exploiter is a let's call it a triangle hat because it's very evil.


What it does is it specifically targets only the current good agent, right?


So this this agent right here is tasked with playing old versions of itself and playing


the exploiter both at the same time.


But the exploiter is only tasked with playing this thing.


So what it can do is it can specialize in exploiting whatever weaknesses this player


has.


And of course, the hope is that the this player will become better in response because there's


a player trying to exploit it.


Right?


So every and as this as this player becomes better than this player here is reinitialized


and tries to find new weaknesses, right?


So as this as this one continues to learn, so the exploiters, they are initialized, you


can see this here.


So these are called the main agents, and you can see they play against each other, right?


One of them they play against each other, they play against past versions of themselves.


So these are past versions of themselves.


But then there are these main exploiters and the main exploiters, they're constantly reinitialized


from human data, right?


You can see this here, they're reinitialized.


And they only play against these main players, right?


They don't have to deal with any of the past players or playing against themselves stuff.


They only try to exploit the main players, and thereby the main players get better.


Once they get better than an exploiter, they are reinitialized.


So the exploiters are reinitialized to find new exploits of the main agents.


The third component is what's called a league exploiter.


And a league exploiter is the following.


So the league, let's the league exploiter here, and its hat is a wavy hat.


And what the league exploiter does is it plays against past versions of itself and others.


So it does play against the league exploiter, sorry, with smaller wavy hat.


It also plays against this thing.


By the way, this here also plays against past versions of this and of everything else.


You can see here the past version arrows, it goes against all past players.


So this represents all the past players that ever existed.


And so does the, so here, but also against past versions of this main exploiter here.


And the important thing is the current main exploiter doesn't play past versions of itself.


So this also plays this, and this plays this, and this plays this, and this also plays this.


So the league exploiter, they do take part in this whole league business, like playing


against past versions of all the players.


But it only plays against the main against the main exploiters.


And this is a thing that I find missing here, honestly, I don't know if I don't understand


this, but I'm pretty sure I do like these also play these.


And that's an arrow missing in the in the drawing.


The league exploiters play the main agents.


But the main difference between the league exploiters and the main agents is the league


exploiters, they don't play themselves, right?


There's no there's no playing themselves on the league exploiters.


So the league exploiters, what they can do is they can find weaknesses of the entire


league and kind of train train the by playing against the main opponents using those found


weaknesses.


You bet that the main the main agents will get better against those major weaknesses


of the entire league, right?


So the main agents for first of all, they get better by playing the main exploiters,


because the main exploiters are mainly trying to exploit the main agents.


The main agents also get better by playing the league exploiters because the league exploiters


find weaknesses of the entire league.


So and the main agents, they also get better by playing each other.


So that makes these these main agents kind of you can say they're trained against everything


under the sun, right against any possible exploit that can be found either in selves


or generally, and thereby they get really good at Starcraft, because they can counter


pretty much everything.


So this is how league training works.


And this is what I feel is the main contribution of this paper to the reinforcement learning


world.


Now they do an ablation study here, you can see where, where where this ends up.


So these final agents here, they end up in grandmaster level Starcraft and beat 99 point


some percent of human players.


So really, really good.


They do an ablation study of all of the tricks they use.


So this is pretty much all tricks they use.


And you can see here, this includes this league composition, right?


What happens if we only have main agents, then main exploiters, league exploiters, and


you can see the elo going up.


Then you can see, okay, multi agent learning, how much does this fictitious self play?


The fact that we prioritize to strong players and so on, how much does this help?


And you again, see the elo going up.


How much does it help that we use human data?


How much does it use that would help that we use these different networks, and so on.


They have very, very good kind of ablation studies of how much each of the each of the


things help.


Here, they investigate what if we didn't have a camera interface?


So what if we could see the entire game at once, and not only kind of the opponents that


are within the camera?


And what if we didn't need to move the camera?


They investigate the off policy learning corrections that we mentioned, and so on.


So I find this very cool that they do these huge ablation studies to show really how much


each of these tricks that they used helps in generating their superior performance.


And here you can see how these agents develop.


So over training, and they have a massive infrastructure, and they train for days, you


can see this here.


But you can see that the the main agents just get better and better and better and better.


While the main exploiters, of course, they stay the same, but they're they kind of keep


getting re initialized, right?


So this main agent is trained to exploit these.


These sorry, this main exploiters trained to exploit these main agents, this one is


trying to exploit these ones, they're not by themselves really good agents, but they're


simply trained to, to find and exploit weaknesses of the main agents.


Likewise, these league exploiters, they do get better with the league, but they are only


concerned with exploiting current and past versions of the league, right, also to make


the main agents better.


So everything is geared towards making these main agents better.


And you can see it actually works.


They have some analysis of which builds which which units these agents build, I'm not too


versed in StarCraft to comment on this.


But all in all, I find this to be a very cool paper.


And I, I find it to be described fairly clear what they do, though they do not release the


source code.


They release some kind of pseudo code.


But the analysis and the ablations are very good.


The results are, let's say questionable, because, of course, the of course, the the you can't


compare machines to humans, especially in a game where you have to make quick actions,


even if you limit the actions, right, they did, they do this here, right.


So they have kind of a this monitoring layer, which limits the actions and introduces delay


and so on, but still like if if it is not the same as a human who might not always be


able to do these 22 actions per five seconds, who, if something quick happens, may need


to have some kind of relaxation phase and so on.


But they try with these kind of delays and action limits, they try to model this, this


these kind of limitations.


So that's, yeah, that's, I find this as fair as possible.


This is what I find kind of problematic.


So they own units, as I said, the agent can also see the ones that are outside the camera.


And that seems kind of shady, because, of course, you can you can claim humans can do


whatever command groups to also control units outside the camera, but it's not really the


case.


So that's that's sort of a distinct advantage that the machine has.


But yeah, in any case, I find it to be very well done.


And I hope this made it a bit clearer what the exact contributions are.


And with that, have fun time playing against AlphaStar.


CornerNet: Detecting Objects as Paired Keypoints (Paper Explained)

Hello there, today we're looking at cornernet detecting objects as paired key points by


Heilaw and Jia Ding.


So on a high level, this paper detects objects in images.


Let's say this is an image and here's a chair, you know, you have your chair.


And the way you detect the chair for this paper is going to be you detect the bottom


right and the top left corners of the bounding box of the image.


So rather than detecting the middle and then specifying height and width, like we saw in


the Facebook DTR paper, you detect the two corners.


And this paper goes through what they have to do to get this to work, including a new


pooling method called corner pooling.


So that's the gist of the paper.


As always, if you like content like this, consider subscribing and sharing it out to


other people.


That would be very helpful.


So a commenter actually recommended this paper to me after I made a video on Facebook's DETR


object detection pipeline.


I said something like, okay, since that paper always would detect the middle of the object


and the height and width, couldn't you make something that also that detects the corners


here and the corner here and then that would define a bounding box just as well.


And in the comments and thank you very much for that.


Someone made that pointed me to this paper.


It's a bit older, as you can see, but it's I still think it's, it's pretty cool.


So we've already seen the problem.


Like the problem isn't hard.


And it's detecting bounding boxes and images.


And in these data set, the problems, the difficult parts are that you sometimes have multiple


objects like here, if two humans, they can be overlapping, they can be of different sizes,


that could be like a third human, like small back here, there can be other objects, you


don't know how many there are, and so on.


So it is it is a fairly complicated problem.


But as I already said, the way that cornernet here does this is by predicting the locations


of the top left and bottom right corner, thereby defining a bounding box.


And it does this independently.


So there's one network basically, that does the top left and one that does the bottom


right and they are then combined and at the end, they're sort of refined, I think.


So the architecture is pretty simple.


First you put the image through a ConvNet, which is like a feature extractor.


So this is the basic part.


It was even the basic part of Facebook's DETR pipeline.


First you have some sort of ConvNet.


Now they in this case use in this hourglass architecture that described down, down here


somewhere.


And this basically compresses the image into a smaller resolutions, I would take that image


and compress it down to very small resolution, but many, many channels.


So it's sort of forced to learn a global semantic representation, and then it up samples the


image again, and down samples it again, and it up samples it again through.


So at each of these steps, there are many convolutional layers right here.


And because that would lose you too much space, like local information, there are skip connections


built in between pairs of layers, where information can travel without computation, basically.


So this is a this is fairly standard architecture right here.


But then after these hourglass CNN, you get to these prediction modules.


Now let me switch back to the top drawing.


Basically what you want as an output of these prediction modules is two things.


So first of all, you want these heat maps, sorry about that.


And these heat maps will simply tell you where are the corners.


Okay.


Now the heat maps, their dimensions are the height of the image.


Sorry, the height here, H, come on.


And the width of the image.


And this here would be the number of classes C. Okay, so you have one channel for each


of the classes that you predict.


And the heat map will basically be very high at the location and channel where there is


a corner of that.


So you see have one heat map for the top left corners, and one heat map for the bottom right


corners.


And then also what you want to predict are these embeddings.


Now, simply because you have, you know, I said there can be multiple instances of the


same class in the in the same image.


So now you have, in this case, particular case, you're going to even if you predict


absolutely correctly, you predict two top left corners and two bottom right corners.


Now this isn't particularly hard because there's only one configuration that can possibly be


but there could be situations where there are multiple.


And that's why you need to somehow match these corners, you have to match, you have to know


which ones of those are the same objects.


And they do this by a second output in their heads called this embeddings.


Now these embeddings, they're simply vectors.


And the only thing that they're asked to do is they're asked to have a large inner product


whenever they belong to the same object.


And they are asked to have a small inner product, sorry, when they're when they belong to the


different, to different objects.


So this orange thing here would have a large inner product with this green bottom right


corner embedding.


Okay, so you train these embeddings, they don't need to mean anything, you simply train


them to predict the same thing for the same objects and different things for different objects.


So after that, when you match the corners, you can simply go over, you can say, ah, this


which one of these two right here has the larger inner product, or you can do like some


Hungarian matching, and maximize the total inner product or something like this.


This was quite surprising to me that it works, but it's based on a line of research that


is already has already established that this can work.


Because ultimately, these things, these two pipelines do not really communicate, right.


So I'm going to guess what they learn is sort of a sort of a descriptor of the actual object


that's there, because if both describe the objects that that's there, with their embeddings,


their embeddings are going to have a large inner product.


And if they describe different objects, then their embeddings are not going to match, right.


So even though you train the this objective, I still think that these embeddings would


pick up something about the object, something about the visual characteristics of the objects


will be very interesting to see whether someone could actually parse out what they what they


do, because it's almost impossible otherwise for these things to be learnable.


All right, so that's the that's the goal right here, you want to get these heat maps and


these embeddings.


And the way you do it is fairly easy architecturally, you have these two prediction modules, one


for top left and one for bottom right, and each of them have three outputs, the heat


maps, the embeddings, and here the offsets are simply a way for you to deal with the


fact that you downsample and by downsampling, you have to round certain pixels to certain


locations, and then the offsets, they they compensate for this, but I don't want to focus


on these right now.


So you simply have these two outputs right here.


Now we'll look at corner pooling in a second.


But how do you train this?


So you can now say, okay, if I have a picture like this, there, there is exactly two locations


in the class human, where the the top left corner is correct.


And that's right here.


Okay, so two locations.


So I fill I make my matrix, my target matrix with a one here and the one here, and zeros


everywhere else, right?


And I train my network to give me this particular thing as an output for these heat for this


heat map in the channel human.


This this might work, but it is more profitable, let's say, if you allow for some slack.


So what they say is, you know, since if I'm anywhere within this orange circle right here


with my prediction, my resulting bounding box is still going to overlap fairly well


with the ground truth bounding box.


And the accuracy measures for these things, I think are based on how much you overlap


with the ground truth bounding boxes.


So what they do basically is they, they give, they put a one in the spot where the actual


corner is, and then they put like a 0.9 around it 0.9 0.9, and so on, and they kind of flatten


out.


So this is sort of a Gaussian right here in multiple dimensions, if that drawing makes


any sense.


And they say, well, you the closer you are, basically, the more reward you get.


So you train it to predict in this general location.


Now of course, the size exact size of this Gaussian has to be dependent on the actual


size of the box itself.


And they have, they, they regard that and say exactly how they calculate these Gaussians.


But for the understanding, it's just important that they do give some slack here in how they


compute the loss with respect to the heat map.


Now the loss with respect to the, to the embeddings is pretty simple, pretty straightforward.


So remember these embeddings, you have two embeddings per, you have the top left embedding,


that's the E tk, the top embed, the top corner embedding, and you have the bottom right embedding.


And what you want is for them to be close together when they describe the same object,


right.


So this is this push and pull losses.


So in the pull loss, what you want to do is you want to minimize the distances of these


two things to this thing right here.


And this thing is simply so E k is simply the the mean, so it's E tk plus E bk divided


by two, that's simply if your top left corner is here, and your bottom right corner is here,


and they have embeddings, this one has this embedding, and this one has that embedding,


then the mean of the two embeddings, which I guess is whatever this right here.


Yeah, that's about the mean.


So the location is not important, actually.


So it's about the embedding vectors, it's not about where the corners are.


The two embedding vectors must be close together.


And you model that not directly by making them close to each other, but by making both


close to their mean.


And that probably saves you some back propagation troubles, where you, if you if you have two


moving parts in a loss function, and you optimize both, then you tend to, so you have two things,


you want to bring them closer together, they might tend to overshoot or something like


this.


Okay, so this brings those two closer together.


And in the push loss, what you want to do is you want to simply make the mean between


the two, remember, this is this is the mean, this the mean embedding of this object far


away from the mean embedding of any other object in the picture.


Okay, so this here is a is a margin loss, which means that you cap it at some point.


So the if they're close together, you if two different if the embeddings of two different


objects are close together, and see here, this quantity will be small, and therefore,


it will lead to this delta, you give a loss of one, the delta here is one in this case,


but as they get further apart, you're more and more happy.


And you reduce your loss until you don't give you so you don't give any any bonus for them


being super far apart, you don't simply don't want them to be closer together than one.


All right, in their case, I think they have a the dimension of these vectors is actually


one, which basically means they just put a single number, which I find astonishing that


that works.


Yes, they use embeddings of one dimension.


So they just use numbers.


Astonishing that it works, but okay.


So that's how you train the the embedding output embeddings close together of the same


objects of the two corners and embeddings far apart for different objects.


All right, so we can now predict where the corners are, and we can match them.


Now, one center part of this is the corner pooling.


And why is the corner pooling necessary?


So what's the problem with this sort of approach?


The problem and they have a an example right here.


The problem when you want to predict a corner of an object is that in a CNN, what the CNN


can is good at is like local neighborhood information, right?


So if you have to predict, let's go for the moon, actually, here, let's predict the location


of the moon.


If I have to predict the location of the moon, and I'm a CNN, and I have this receptive field


I'm like, Oh, yes, it's like in here.


And then I have this receptive field.


And I'm like, yes, it's in here.


And then I zoom in on the corner, not on the moon itself, but on the corner where I need


to predict, right?


At some point, I like I'm sort of, I'm like, wait, wait, where is it?


Because in this particular field receptive field of this resolution, I have I have no


clue if the moon is close, right?


So at the location where the actual bounding box is, I have no local information of the


object, because usually objects are not squares, they're sort of round like the moon, or like


here the plane, these corners, they have no local information about where the plane is.


And corner pooling is a method to propagate that information along the axis.


So what corner in corner pooling, what you would allow the location here in the CNN to


do is to not only look at its itself, so its own location, but actually to extend its field


of view over to the right, and down to the bottom, it's asked to predict a top left corner.


So what you do is you max pool everything from here to this corner detector.


So the corner detector will basically be able to detect whenever in either this band right


here.


So whenever in this band right here, there is the top, like the top of an object like


the top of the moon here, this corner detector can say, ah, that's probably the right height,


right here for a corner.


And it combines this with the information of this, this side here, where it also says,


oh, there is the side of the moon, that's probably the correct, you know, up down.


So there's probably a corner right here.


Okay, whereas a, like a location right here, which would get the same signal from the right,


or like almost the same signal plus this signal right here.


But in essence, it would also detect the top of the moon, but it would not get the same


signal from down here.


And therefore it says, ah, even though to the right, I see some the top of an object,


I don't see the left of an object to my bottom.


So I'm not going to predict a corner right here.


Alright, so this corner pooling goes for the top left, and of course, equivalently goes


for the bottom right, that can always max pools to up and to the left of itself.


And that's exactly what you see here.


So in this corner pooling, what you can do is you can propagate signal to the left, and


to up, and then you add the two informations.


And that will give you your output feature.


And you can calculate this actually fairly efficiently by doing like, like you do a cumulative


sum, you do like a cumulative maximum across the different axes.


And then you simply add two arrays.


And that's it.


So you simply put the corner pooling before you predict the the these different outputs


right here, the heat maps and the embeddings, which means that this hourglass network is


not affected by this.


Just the predictors of heat maps and embeddings, they then get the information from this hourglass


network into these into these directions.


Okay, I think that's a pretty, pretty neat method of solving this.


And here they show how you can calculate this.


And then the corner pooling is right here, they do add a skip connection here.


Because sometimes, if you just aggregate this information, you might, you might actually


get confused because so the trouble, of course, comes when there are multiple, like, different


objects that have, you know, the same top, and then there's also a person right here,


that so it gets like a signal, it gets another signal that there is the left side of a person


right here, or maybe, you know, like this, so it will, it will predict like a corner,


maybe here, where there is none, so it's, it's, sometimes it is important to have local


information still.


And that's exactly what this skip connection is supposed to address, I guess the situation


up here would be resolved by the different embeddings, but still, so you have that you


add and you put another bunch of convolutional layers on top of that, and then you'll get


your predictions.


And that's it.


You mix all the losses, so there is a detection loss from the embeddings, there's a, sorry,


from the heat maps, there is the pull and push losses for the embeddings.


And there's this offset loss that you train to compensate for the down, the downsampling


errors.


And they ablate the various things here, basically, they show that they're better than other one


shot, or one stage predictors is so apparently, there's one stage predictors where you have


a single pass through a neural network.


And there's two stage predictors where you have multiple or two passes through different


neural networks.


And they compete in the in the one, one stage neural network category, if so to say, and


they show that they get significant improvements with and due to this corner pooling, which


is pretty cool to see, because it makes sense.


It sort of makes sense how you would like to think about it like this.


And to see that it helps is pretty neat.


Yeah, they also investigate how large they have to make these these Gaussians and so


on.


And these are some qualitative examples, you can see that without the corner pooling, what


you'll get is that so the top here and the left and the right are correct are detected


correctly.


But you can see that probably the network thinks that there is an extension of the object


right here, and therefore doesn't do doesn't do a good job.


Because this, this position right here, it has no access to sort of it has to use like


a long range access.


It can't it can't really look in detail at the features here or here.


So when it scans up and down the side, where the bottom corner where the bottom break is,


it can it can only look at very coarse features, because it has to basically transmit information


in the CNN of a higher layer.


And the higher layer has a higher receptive field, which means it has a lower resolution.


So it can't really go and look very in very detailed fashion at this border right here.


So it misses it.


Okay.


Same right here, as you can see, so there are a number of failure cases that they can


now solve using this method compared to if they didn't use the corner pooling.


They show some also some times where their method fails.


For example, here, it matches the top bot, top left and bottom right corners of two different


objects, because they their embeddings were close enough.


And yeah, that's what I'm saying.


I'm I'm wondering what these embeddings actually learn, because they are generated independently.


So not entirely sure.


It's also not exactly what I had in mind when I formulated this idea in the last video.


But I'm actually not sure what I had in mind myself, to be honest.


But in my mind, it seemed to be like you should be able to train a network.


If there is an object right here, you could train a network to predict for any given location,


let's say how many pixels to its bottom right, or maybe you want to normalize by the area.


That's there are part of a particular object.


And then you could use you could predict each pixel and use like the differences between


the differences between the points as as scores for bounding boxes.


I don't know if you see what I mean.


You could basically tell the you have you'd have one network predict everything to the


bottom right, and then you'd use the differences.


And the transformers would be very good at that because they can sort of have this attention


between each pairs of points and so on.


I'm not entirely sure, but this might just be crap.


Yeah, here's some more examples.


This appears to work really nicely.


But of course, in the qualitative qualitative examples, it always works nicely, but they


also demonstrated.


All right, I found this paper all in all pretty cool, pretty neat.


It's a simple idea.


It's executed.


Well, I don't have the feeling that there are like too many tricks in here.


And they show really that the improvement seems to be due to their their corner pooling


method.


And that's pretty neat.


So if you like this paper, make sure to check it out.


And I'll see you next time.


Turing-NLG, DeepSpeed and the ZeRO optimizer

Hi everyone, today we're going to look at Turing NLGA 17 billion parameter language model by Microsoft.


The latest and greatest of language modeling by Microsoft.


What is this? It is a language model. A language model is basically a model that learns to produce language, given language.


So if you start a sentence, it's supposed to finish a sentence. If you start a paragraph, it's supposed to finish the paragraph.


That's a language model. Ultimately, you can make it do different things like answer questions, have a conversation with you, anything to do with understanding language.


The special thing about this one is that it's ginormous. So if you look at the scale of kind of language models.


So BERT was quite a large thing in its back in its day. Ye olde BERT. You can see here it has about 340 million parameters.


Now, I have to say all of these language models are transformers. This is kind of the state of the art today.


So all of these are kind of our transformer based models.


And then GPT-2 here, you can see that was the model that was so large, it was too dangerous to be released into the world.


That stands at 1.5 billion parameters. Megatron LM by Nvidia 8.3 billion.


And now we are at 17 billion parameters for this language model.


And it is a bit better. So like people just throw more and more and more resources at this language problem.


And it really, yeah. So what you can do with it, you can, of course, do language modeling.


So what happens is you take a bunch of text like all of Wikipedia and all of the Internet and all of Reddit and so on.


And you let the model train on it to understand, to basically produce that sort of language.


And then you can measure it. For example, it's a perplexity on a validation set.


And the Neuroturing NLG is currently state of the art on that.


It can also do, for example, question answering. So you can ask a question and give it a passage about that question.


And it will then tell you the answer that it deduced from that passage given the question.


As you can see here, what is more interesting is that a usual QA system will point to the passage.


So it will point to the words Tristan Prediman.


Whereas with a generative model like this one, what you can do is you can make it actually output an answer as a sentence.


So it will generate the text, Jason Braz was engaged to Tristan Prediman. Sorry, Prediman.


If you ask a question without giving it a context and just ask it to generate an answer, it will do so as well.


I don't know if these answers are cherry picked, but they call this zero shot question answering.


So if you ask when did World War Two end and it can output World War Two ended in 1945 simply out of regularities, it detected in the training data.


So, I mean, that's what I'm kind of wondering. At what point are these models, do they have so many parameters that they simply reproduce the training data?


Right. I mean, this clearly some article from the training data is about World War Two or many are.


And it simply learned that following a question when did World War Two end, it needs to answer with the appropriate passage.


I'm not sure that is a proper measure of language understanding if you simply can bake more and more of the training data into these many, many parameters.


But I'm not the judge of that here. It can do it very well. So, yeah, what I'm actually more interested in is this thing is called the zero optimizer that they use to train the model.


So the model is just a transformer. It's just a big, big transformer model.


There is nothing really special about the model, except that it is larger than the last model, and therefore a bit better.


What is interesting is that this would have been pretty impossible to train if it weren't for this zero optimizer of these deep speed library and Microsoft has released this deep speed library.


It's compatible for now with PyTorch. You can check this out. I'll put a link into the description.


And I want to dive into this a bit. There's a paper by Samyam Rajbhandari and all and by all by Microsoft.


The paper describes in detail the optimizer, but it's not very visual. That's why we're going to the blog post.


You can see it gives many speed ups over the previous Megatron LM model that NVIDIA just trained using kind of just what NVIDIA has.


NVIDIA has machines that are interconnected within the machine with very fast buses between GPUs.


But this zero optimizer can now also go over the network and make it pretty fast. So let's explore that a bit.


I have the copied this here. So we'll look how the zero optimizer works. So usually what you do is if you have multiple GPUs, you can do something like this.


And this is called data parallelism. What you have is you have a model and the model in this case fits on your GPU. It fits on a single GPU.


So the blue thing here is the model. I'll actually draw this. So the model, let's say, is a somehow a neural network.


So it has a bunch of layers, layer, layer, layer, layer, right? And what you want to do is you pass data forward.


Here is some loss and then right into the loss function, and then backward again. That's basically what you need to do.


You need to pass it forward and backward in order to do back propagation training, right?


If this all fits into one box, that's completely fine, right? If this fits into one machine, cool.


We can just put many batches of data through batch one, batch two, batch three, and so on. Train the model.


If you want to do a speed up using this, you can do so. If you have lots of data, you can do what's called, and I'm always confused.


I think this is called data parallelism, or is it called model parallelism? Because in any case, what you can do is you can take a second machine,


or many of those, replicate the model. Now, these models, these two models here are exactly the same, right?


But what you do is you take your data and you split it up. Basically, you take double the amount of data,


and you put one batch of data through the top part, and you put the other through the bottom part.


And you do your forward passes on the machines, and you do your backward passes.


And then what you want to do is you want to sync between the machines what they learned from the data.


So each machine has a different set of data points. Each machine calculates its own parameter updates, learns from the data it has,


and then they communicate to keep because this here and this here should be the same, right? It's the same model.


So they have to keep in sync. Now, this can be usually can be done fairly efficiently, especially if these aren't actually two machines,


but just two GPUs inside of one large machine. So if this is a large machine, this is GPU zero, and this is GPU one.


This is pretty standard because especially on video machines, they have these whatever I think they call them infinity band or so or is that so


in video has these connectors that can connect the GPUs together really fast. So you can keep these in sync.


But now the problem becomes what if you want to train a model that is larger than this.


So let's forget about the data parallelism for now, if that is what it's called. And just consider a model that is too large.


So a model that is too large will not fit into a machine. So this is a model as a large model.


So what you want to do is you want to pack some of the model onto your first machine, and then take the other part of the model.


Sorry, and pack it onto another machine, right, you separate the model and put it on different machines.


Now, if you have a batch of data, what you have to do is you pass it, pass it, pass it forward propagate as you regularly would,


but then you have an intermediate result, you send that to the next machine. And you forward propagate that at the end here, you have a loss, right?


You want to back propagate regularly through this machine, you have an intermediate result of back propagation,


send it over the network, and back prop all the way through the model. So that's how you can train a model that is too large for one machine.


If you have multiple machines, the problem here, of course, is this part. Just as you had to keep in sync the model before,


now your problem, your communication problem becomes one of you have to send the intermediate stages to that model.


And you have to send the intermediate stage of the back propagation back to to that part of the model. And while this part is working,


this part is kind of idling away. And the network overhead, sorry, is just very costly.


Especially if your model is so large, you can't even fit into one of these single boxes, right? So this, this is very, this is very problematic here,


is still doable. But what the zero optimizer does is it does both data and model parallelism. So it can train models that are too large, right?


Too large for a single machine. And it can do data parallelism at the same time such that basically everything is working all the time,


there is no wasted, or not much wasted computation, the communication is efficient, and so on. So it's really a technical achievement.


It's not so much in a scientific advance, it's really a technical achievement, this optimizer. And we'll shortly go through,


there is a kind of an animation, but it's on the website, but it's super slow. And I think this might be the first time that I will be faster at explaining something


than a video. Alright, let's see here. Alright, cool. So what do you do is let's just consider these three GPUs. Before that, it would all fit on one machine.


But now let's say you don't actually have that much, that much memory, you don't have these, these giant, empty blocks here, you just have a bit of that.


So you have to split your model. The blue parts here are your model, right? These are, these are model parameters. The orange parts here is memory you need to store gradients, right?


You need as many gradients as you have model parameters, as because you do gradient descent. The green stuff here are what's called optimizer parameters.


Now, if you just have SGD, these would be non existent. But if you have something like AdaGrad or Adam, they have additional parameters for each model parameter that they need to keep track of.


So these are stored here, and there can be significant overhead. There's also like a floating point 3216 conversion going on here, don't want to go into that.


Alright, so you split your model onto these three machines, let's say that's your entire model, your model is six blocks wide, right?


And you need to forward propagate now through everything. Alright, so here is what zero does. And I think it's pretty cool.


So what we need to do is we have these three different batches of data. And we want to forward propagate them all through the model through the same model at the same time, as if the model were actually stored on all these machines, like if all of these machines had the entire model.


And we can do a bit of communication. So what we do first is this one's easy, right? Data zero through the first two layers here is easy, right? Because we have them, right?


So bang, you go through the first, you get an intermediate result here and here. Right? Okay, how do we propagate data one through the first layer?


We can't send data one here, that would be too expensive, right? And that's the whole point would be lost, right?


We want to actually compute data one on this GPU at the same time. What we do is before we start, we actually communicate these two blocks here to GPU one.


Alright, we send these parameters around and fill them in here. I should actually make them blue, right?


We send them here. And we also send them here, right? We send the parameters to all the machines. Right? Then we can actually forward prop data one through this, and data three through this.


Right? So we can do forward prop. After we've communicated all the GPUs can be working. Same with layer two, right? So layer two, simply can send it's, it's these, these two here, you can see that these two here to the other machines.


Right? Now, while it's doing that, we've already propagated to through the first layer. So we've already propagated here, and here through the first layer.


So we can actually delete these again, right? We can, we can delete these first layer parameters that we sent around again. Right?


So here's how you see how we can save memory, we don't keep all the model in sync and all the machines, we send whatever we need on the other machines.


And then once the computation is done, they can delete it again, right? Because there's always one machine, right, this one here for the middle parameters, that keeps track of the parameters and that can at any point if they're needed, send them again.


So that's the big kind of catch. So you can forward prop now through these two, right? They here, they're already present. And then you can delete those again on the machines where they're not natively stored.


And you can send from here, you can send those two also up here, you can send those two and forward prop your model through to the end, right? Oops. Yeah, that was a mistake. They should be here.


Then each machine calculates its own loss, right? loss, loss, loss, and the backward propagation happens in much the same way. So as you can, if you follow so far, you can already imagine, right?


So what I can do is, so now this is a loss, the loss is different, because there's a different batch of data going through each machine, right? So there's a different batch of data going through each machine, but each machine has computed with the same model due to the communication of the zero optimizer.


So that's pretty cool. So you get the benefits of data parallelism, right? Lots of data on the different machines. And you also split up the model across the machines. But the you don't actually store the model on any of these machines, you don't store the model, you only send right from here, you send as you need, and then you delete again, right?


And for the backward propagation, same thing, right? Do backward propagation, you calculate gradients. Now you calculate gradients here, you send the gradients as needed to the other machines, right?


You calculate gradients here and here, and you send them to the machine where they're actually needed. This is a weird pen, sorry, you send them to that machine, that machine will aggregate all the gradients of all the machines. What is up with this pen?


Okay, it will aggregate them, and then locally, it can compute using these optimizer parameters, and so on, it can do all kinds of optimization locally, because it has gathered gradients from all the other data.


So what you end up with, for example, GPU two here, for these two layers, it has effectively broadcast the layers such that much, much more data than it just had itself could run through the layers.


It has aggregated gradients from all of that data. And now it can use all of these gradients together to make a good update, using the optimizer parameters to make a good update to these model parameters.


And then in the next iteration, it can go ahead and broadcast the model parameters, the new model parameters again, so it is able to compute with much more data than it can just fit by itself.


And it is just doing its part. So zero and deep speed, zero is the I guess the protocol and deep speed is the actual library, they will do all of this communication and splitting and so on for you over the network in a way that is efficient in a way that everything runs at the same time.


And the communication overhead is minimal. And you can actually choose which stage you want. So what your trade off of communication and memory saving will be.


So this is extremely cool. They say this goes up to whatever 100 billion parameter models. If you use item. This isn't something for you know, your average collab user, this is really something for the for big players.


But that being said, I don't think language is solved by simply throwing more parameters at it. I think we're still a bit of a breakthrough.


Author Interview - Memory-assisted prompt editing to improve GPT-3 after deployment

Hello, this is an interview with the authors of the paper on memory assisted prompt editing


to improve GPT-3 after deployment.


If you haven't seen it, I've made a comprehensive paper review on this paper.


And I released that yesterday.


So the authors that I'm having on today as guests have seen that paper, and we're able


to dive right in.


So if you haven't seen it, might be a good place to check it out.


I wish that you have a lot of fun following this interview, or that you learn something


or that you're entertained, ideally, all three together.


And yeah, have fun.


Bye bye.


Hi, everyone.


Today I'm here with Aman Madan and Niket Tandon of the paper memory assisted prompt editing


Aman and Niket, thank you very much for being here.


Welcome.


Thank you for inviting me.


So you've set out to write this paper.


And I guess the viewers have probably seen the review.


And this is really cool, because these large language models, sure, we now have a fine


tuning endpoint at GPT-3.


So it is a little bit possible to adjust it to your use case.


But I think what you're doing right here comes the closest to what people imagine when they


hear AI, like when someone when when I go to someone and sell them an artificially like


an AI system, they imagine a computer program that learns immediately, right that they can


they can like tell things to and it adapts, you know, it gets smarter as they interact


with it.


And largely the AI community has not delivered on that promise.


We train things on static data sets, and then we deploy them and they're frozen.


And yet your system, I think, yeah, it comes the closest to really to live up to that promise.


So I think that's, that's really cool.


How did you?


How did you go?


Like, how did this come to be?


How did you figure, you know, let's, let's build something, let's build a plugin for


Our original motivation was, can we personalize very large models such as GPT-3 rather than


having many copies of a giant GPT-3 model trained in one place on one static data along


the way with the user, the models can improve, personalize over time.


This was the original motivation why we started with this part.


And GPT-3 was a great example to start with because it is such a large model that at the


time of writing, it was not possible to fine tune these models.


Yeah.


So I think similar to that, one of the reasons why we specifically thought of having a plugin


for GPT-3 is, so I was using Copilot for some time and Copilot makes the same mistake every


time I write a print statement.


So I'm using something like Python 3.7, which has S strings, which is a way of displaying


the output, which you can nicely splice strings with variables.


But the Copilot will always use the older version of print statements.


And I would have to go back, edit it and, you know, make it the S string that I want.


So it was natural to, you know, kind of, there was this urge, you know, I wish there was


something that could personalize this ID to me, but this instance of codecs to me, and


you know, something like a hash map would work in that case.


So whenever GPT-3 completes it with an older print statement, I can just have a regex that


replaces it with an S string.


And that kind of motivated this whole idea of having a small plugin outside of GPT-3


that stores these error cases and can correct them on the fly.


And in the first version, we had some sort of proof of concept with kind of mixed output


kind of thing.


But the idea is to kind of not have to train the model and having something super light


that can adjust to these things that are not equal repeatedly.


Yeah, it's cool.


And you don't even need to be open AI to do this, right?


Because most research sort of assumes you're in control of the model.


But this is really something you can just hang in front of whatever model that you're


consuming, which is pretty cool.


So I think, you know, it is important to say that I was quite critical of the paper in


some places, and is good to inform the viewers that there is actually a v2 out that addresses,


I think, almost all of these criticisms in one batch.


So I just quickly want to show that.


And you told me that it got done, like just in time last night or so.


So there is a new version of the paper, which is on GitHub right now.


I guess that's also coming on archive in the near future.


And that does have a lot more experiments, because I think one of the issues I had is


that you said, well, we just want to present the framework of things.


And you did some experiments.


But can you maybe, you know, just talk about what new experiments you've added and how


those turned out in this in this new version?


Because if you know, with new experiments, and being state of the art, it is it sort


of invalidates my point of well, you just present only a framework.