then trying to understand every everything. But the things I do read, I try to understand


How I Read a Paper: Facebook's DETR (Video Tutorial)

thoroughly. Okay. Then there's the architecture, okay. And that again, I read closely and get


backbone, okay, transformer encoder, okay. And now I understand much more closely decoder, okay. And


here I get now finally, I get what this is about decodes and objects in parallel, yada, yada, yada.


These input embeddings are learned positional encodings that we refer to as object queries. And


similarly, to the encoder, we add them to the input at each attention layer. So now they name,


I've already seen these object queries here. And the only word I actually need from this sentence


are learned. The fact that they're positional encodings, I just kind of ignore. As soon as they


say learned, I know, aha, these things here are learned, they have actually, they're always the


same for each of the images. They're just overall learned. Okay, so now I feel I understand the


entire model. And yeah, so they then they say auxiliary decoding losses. And this sometimes


you have to pay attention to like, auxiliary auxiliary things, like,


like, auxiliary auxiliary things, because those are the the things that here they say explicitly,


we found helpful to use auxiliary losses. Sometimes, they won't say why they did it,


they'll just say, our loss consists of three things. And, you know, if you look at the three


things, only one of the things is really a part of their story so far. And that you should


immediately conclude that they've put in the other things, because they tried it and it didn't work,


right. So you can also kind of get an estimate of the brittleness and so on of the system,


in that you see how many unnecessary things are there or how many things are not straightforward,


how many things are the easiest thing that you would do when you would go about and do what they


did. Okay. So then you this concludes this model or method. Usually this section is called like


method or model or something like this. And you go to experiments. Now, the main question I have


so far, or I have, maybe I have some more questions about the model itself, that I haven't


been able to pick up from this section, which is not the case here. But I simply keep those


questions in mind, and see whether they are resolved later, right. So I keep an awareness


of what I don't understand. But from here on, my main issue is, are they demonstrating that their


story works, right? So they're here, they're, they're proposing a loss and a model. And in my


mind, they now need to convince me that that works. And that's, that's, it's not as easy as


simply to show me some numbers that they are good at some benchmark, they need to show me


that they get those numbers, because of what they claim. So here, they claim, well, okay,


they propose a new, they propose a new architecture. So what they need to convince me


of is that the architecture itself makes sense, right. But in other papers, when, when you


propose, like, and when you say like, oh, we, for example, in an LSTM, when you build in an


attention mechanism, and you claim, oh, we, you know, the attention mechanism can look back at


the source sequence in one step, then you need to convince me that that actually happens, right. So


you need to not only do you need to perform well, you need to convince me that you perform well,


because of what you claim your model does, right. So and that's often difficult. And I specifically


look out in the experiments for usually the question is like, where are they trying to


bullshit me? Right? Where are they trying to our or are they trying to bullshit me? Are they trying


to cover up the fact that something doesn't work? Now, all the experiments are always in the best


light possible, of course, and you have to keep that in mind. But a lot of times you can also


already see from the experiments that, okay, are they doing something weird? Are they not showing


me some obvious experiment? Or? And that's a lot of times, because is there an easier explanation


for why they get the results that they get, other than their explanation, right? And it is it is


their job to convince you that their explanation is the correct one for these numbers. And


especially if there is an easier one that they haven't excluded, and then I don't believe the


experiments if that's the case, right? If there is an easier explanation for the effect, I'm,


I'm very skeptical. But some papers have an easier job here than other papers. So in this paper,


they basically show results on a on a on a task. And since their paper is about, hey, our pipeline


is just easier than other pipelines, what they first of all need to do is they just need to like


match the numbers of other pipelines. And here I see that, okay, in these results, often you have


maybe a table or something. Here you see like this, their model, other models, and their model


is the best model in a lot of cases. Now, if the best thing is, of course, if their model


throughout is the best, the worst thing is if it's like scattered, like this, even if their model is


the best, but in every single benchmark, a different configuration of their model is the best.


That's, that's sort of a bad sign unless they can explicitly explain why that is. And it's also not


that good of a sign if these things are spread out like this, like sometimes the baseline is good,


sometimes their model is better, and so on. So pay attention to that. Now in this paper, it doesn't


matter so much, that's actually fine, because what they're trying to show is that their model is on


par and way easier. And they've already made the case in what way it is easier, it's easier in


terms of architecture. If they were to say it's much faster, then after that, I would expect,


you know, an experiment in speed while these numbers are matched. But since they say it's


easier, I've already seen the architecture, I'm convinced of that. Now that they show, okay, our


numbers match, or actually I'm surprised they even outperform a lot of times, then I'm quite happy


with these experiments. So also, look for differences between numbers and the spread of


numbers. Now, it's not easy to say what if like point one is a big or a small difference that


depends on the task. But if you know, pay attention to these things, pay attention to the fact that


these results are noisy. And oftentimes, there is a lot more hyper parameter tuning going into the


model of the paper, then into the baseline model. So I do want to make your look your stuff look


as good as possible. And here is a little bit where the institutional credibility of someone


like Facebook comes in, in that I tend to believe their results a bit more than other results, not


mega, but a bit more. Yeah, also look at patterns that they don't point out in the text. So if


there is like a pattern, if you see like an interaction between the number of parameters and


the score or something like this, just try to be on the lookout of that and see if you can spot


something that you think or think about whether that makes sense or not in what your hypothesis


would be. So here we go on and okay, then they go into ablations and a lot of a lot of these


papers do ablations and I generally appreciate that. So here they visualize that the attention


mechanism in their model actually refers to different instances, right encoder self attentions


for a set of reference points, the encoder is able to separate individual instances and you can see


that pretty clearly right here where and even here with the overlapping cows. And this is the sort of


experiment that I would expect that actually convinces me that their architecture does what it


says that it does right. And something like this where you see like totally overlapping things with


the attention of the individual things visualized. So telling me like especially this one right here,


the the foot of the back elephant actually being focused by the attention of the bounding box of


the back elephant. That's the sort of experiment that convinces me that their claims like that


their numbers really come from what they claim it comes from. Okay, so at the end of the


experimental section, you should always ask yourself, have they really convinced me that


their story is true, right? That the improvement or when whenever they get an improvement or


whatever they get, whether is, is due to the story that they want to sell me? Or could there be like


an easier explanation? Or does something not fit is like, are there are the experiments different


than from what you would expect here? Okay, so these are these are my main questions. Are they


are they convincing me of their story? It's not? Do they have state of the art numbers? I don't


care. I don't care. Even though, like sometimes. So there is a bit of a catch. I, I don't care about


state of the art numbers. Now, let's say you have a table like this. And you have a computer vision


model. And one of the models is like on the CIFAR 10 data set. Now, if your baseline model has like


a 91 92% accuracy on CIFAR 10, when I know the state of the art is 96, I don't care, right? I know


like I've done CIFAR 10, I know with like, I don't know, five, six layers, CNN, you can reach these


91 92 93% accuracy. And to get to the 96 97, you'd actually be like in the region of a wide res net


and whatnot. So I, I know that even though you're a few points behind state of the art, I know,


you know, this, this is valid still, so I don't care. But if you were to be like at 80% accuracy


on CIFAR 10, then I then I get a bit like, hmm, I like it's pretty easy to get to 90% plus with


like a standard CNN. So there, I immediately start to wonder why is there an explanation? Now, this


could be like a theoretical paper that says, Oh, we investigate MLPs. And that's why we only get


that number. So that's that would be fine. But if something is out of the ordinary like this,


then I pay attention but never because something isn't like the latest and greatest state of the


art. That's just dumb. Okay. And also, if only evaluate what the paper claims it does, right?


If the paper says, we want to show that we are on par with current models, then don't be mad if the


paper doesn't outperform these models. They didn't claim that right. So yeah. So after these


ablations, I'm actually pretty happy right here with the results. And this right here, when I saw


this, I didn't, I didn't expect that. But I read the experiment description that these are these


different learned object queries and what they do. And that gave me an increased understanding of


how these object queries actually work. Right. So at that point, I still had like a vague, I knew


that these are learned. But reading this and sort of looking at it, studying it a bit, I was like,


oh, okay, then I understood even better what they are. So again, when I say understand everything


in the method section, you can still have questions. And but you just have to keep it in


mind for later. And then here I go on. And there's this detr for panoptic segmentation. And


they here they propose like a new model. So I first look at it and I'm like, okay, they propose


a new model, they can do stuff like this. Now this is not object detection. And again, I'm not sure


is this like a is this like a an add on to the method? Or is was was this up here just an


intermediate step to this? And honestly, after reading that I still wasn't sure. It seems like


something in between, of course, the paper is also a bit longer than other papers. It just,


it seems it's too long for just being a side note. But it's too short for being its own thing. So


that was just a bit weird. And I treated it as as just like a, oh, we can also do this with our


model. But I didn't pay like too much attention to that. Okay. So at the end, I you know, look at


conclusions. Now the conclusions of a paper are much, much often, they are not nearly as informative


as the introduction, the conclusions, they all often tend to be very generic and kind of hedging


a bit against criticisms, saying what would be up for future work, which is again, hedging against


criticism. Because you simply say, well, we didn't do this. That's future work. Yeah, so again, I


read it, but I don't really pay attention to it. And then I gloss over the abstract, I just would


kind of scroll through the abstract, if there's something that catches my eye, I would look at it.


And if not, then not. And then I basically go to the start. And whenever I didn't understand


something, I go back, I look at it again. And I try to think are all my questions answered? And


have they sufficiently convinced me that their story is the thing that really has the effect


right here. And then, if I now were to make a video of this, I've often found it useful to


just put the paper away for a while. And it's, I usually get the best results when I read the


paper the day before and then make a video the day after. Or if not, I'll just, you know, put it away,


do something else, do some email responding, programming, going outside, eating lunch, just


some kind of a break between first read or between first couple of reads. And just, I don't even


think about the paper. I just kind of, it's just in the subconscious, it kind of brews, right. And


I happen to think about the paper every now and then. But I don't make a conscious effort to be


like, Oh, how am I going to explain this and so on. But I just found the worst videos are the ones


where I immediately make the video after reading a paper. And I've just discovered that if I kind


of take a break, and then I look at it again, right, I look, I don't read it fully again. But I,


if I have, if I have the feeling I've understood it, I don't read it fully again. But I just kind


of look at it and go again through the story. And I think that's even if you, you know, want to,


if you want to talk about a paper in a reading group or tell, you know, explain it to your


friends or whatnot. This is often very useful, just put it away for a while, let it mellow.


And I find that helps a lot. Okay, that was my process of reading this particular paper.


Now we again, this this is a high quality paper. So it's, I find it's a pretty easy read in that I


simply need to understand what they did. And I'm pretty happy with their experiments. I maybe next


time I can find an experiment or a paper where I'm initially more skeptical about what they did.


Where I'm initially more skeptical and not as happy with what I find. But yeah, let me know if


you enjoyed this or if you would like to see any other explanation. I don't exactly know if this is


what you expected from a video like this. So let me know. Maybe I've misunderstood you completely,


or it's way too long, way too detailed or way too undetailed. Yeah, leave me a comment and I'll see


Did I crash the NFT market?

Has anyone noticed this odd coincidence where I upload a video where you can make your own


Did I crash the NFT market?

NFT using machine learning and then like two days later the entire NFT market crashes


Did I crash the NFT market?

I don't want to imply causation, but that seems pretty odd


Did I crash the NFT market?

So still if you want to create your own completely fungible completely free


Did I crash the NFT market?

Bored ape NFT you can go to whykilture.com


Did I crash the NFT market?


Blockwise Parallel Decoding for Deep Autoregressive Models

Hi there, today we'll look at blockwise parallel decoding for deep autoregressive models by


Mitchell Stern, Noam Shazir and Jakob Ushgharaj of UC Berkeley and Google Brain.


So this is a bit more of an engineering paper than usual, which I find cool.


It's basically an engineering trick to get these autoregressive models to decode faster,


while you can either preserve fully their performance or suffer a bit of a drop in performance,


while even speeding them up more.


All right, so let's dive in.


Actually, the the paper starts out with a description of what autoregressive models


are and what decoding is in them.


So let me try to quickly, quickly explain this.


So what is an autoregressive model?


So basically, we're talking about, let's say language models.


So language models are the classic examples of these models, where you have a language


model is a model that simply predicts the next word in a sequence.


So you could have something like a cat sits on the and then here is blank.


So the language model is asked to predict which word is the word that follows.


The language model basically does this by predicting the probability distribution over


the next word.


So w t plus one, if this is this is t here, this is t minus one, and so on, w t is given


all the w's smaller or equal than t.


So all the words that come before should lead to the next word being predicted.


So the language model is tasked to ask, what is the next word in the sequence?


Or what's the probability distribution over the next word?


And then you can simply, you know, pick the maximum probability word or something like


this.


So that's, that's pretty standard so far.


So what is the autoregressive part in here?


So basically, the autoregressive part means that in order for me to find this word here,


this next word, I will look at all of these words here.


And what does it mean then when I want to use this language model for generating generating


a sentence, let's say, so I'm now I've trained the language model, it's really good at predicting


the next word, I want to actually use it to do something more interesting.


So I, I want it to generate a full sentence, what I do, let's say I pick the first word,


the right, I picked the first word.


And I simply asked the language model, why, what's the next word?


Right?


And the language model can do this, it can assess what's the probability distribution


here over words, and it will, for example, give me some some distribution over words,


and I pick the maximum one, I say, Okay, the maximum one here is house.


Okay, the house, the house.


And then I go back and ask the language model, what's the next word then?


See, clearly, you're a language model.


So you can give me based on the two previous words, you can give me the next word, what's


the next word, and the language model was maybe say the house is, and so on.


So you can see how you can generate a sentence by simply basically feeding the answer that


the language model gives feeding it into the next step of predicting.


So all of these now go into the next step.


And once you've predicted the next step, the house is on.


Once you've predicted that, then you can take that and in conjunction with everything you've


predicted so far, to predict the next step.


So you can use the language model that is trying to predict the next word to predict


an entire sentence.


And the autoregressive part basically means that its own predictions will serve as the


basis for the next predictions.


And this introduces a fundamental problem, namely, that I have to basically wait for


one prediction.


So I have to wait here for is before I can predict on.


And this means if I have a, I basically can't help but so if this is my language model,


it's a box, I can't help but go to the language model, wait for a response, okay, then go


to the language model again, wait for a response.


Again, this is inherently sequential nature here where I have to do like m steps if m


is the length of the of the sentence that I want.


And we can't make use of batching.


So usually what you do during training during training, you have a whole bunch of data,


right?


You have the cat sits on the mat.


You have the house.


The house is blue.


So I can generate just from these two sentences, I can generate a bunch of training example


I can ask, I can this is a training example where the input is the cat and it's meaning


to predict sits, then this is a training example where the input is the cat sits and the language


model has to predict on this here is a training example, this this is a training example.


So I can chunk this up into a whole bunch of training examples.


And all of those I can write I can feed in parallel into a big, big matrix, I can all


put them here, and then run this thing through my language model in training mode, because


each of them is already like is in the corpus.


And I can, I can, I can batch the training, but I can't batch the prediction because of


what we've seen before.


Because inherently, the next predicting the next word depends on the last word that the


model itself has output.


So there is no no training corpus around since we're, we're not training.


Yeah, so this is the fundamental problem.


And these authors tackle this problem, say, how can we make this faster, this decoding?


So they introduced greedy decoding here where they say, Okay, this will just what we've


just seen, the probability of the next word is like the the maximum, the maximum log probability


here in that case, if the model predicts a log probability over the words that we've


input so far, right.


And this x here is so this is, for example, a translation task, a machine translation


task.


So the x would be the source language sentence.


So maybe like a French sentence, and the y smaller equal to j would be the so far decoded


English sentence if we're trying to translate to English, and the y j plus one would be


the next word that we're trying to predict in the English sentence given the English


sentence so far and the French sentence, the total French sentence.


So greedy decoding just does this one step after another.


And we will we try to go to what they call blockwise parallel decoding.


So we can just jump to the graphics straight away, because what they do is pretty straightforward


and is best illustrated in this graphic actually.


So they go from the from this situation where they already have this here, they have a saw


a dog ride.


This is the the sentence that has been decoded so far.


And we'll, we have to try to complete it.


And naturally, we'll ask what's the next word.


But they say, Okay, what if we could predict not only the next word from this, but the


word two positions away or three positions away, we could do this all at the same time.


Right?


I mean, I can certainly build a model, a language model that doesn't only predict the next word


predicts the word after that as well.


So of course, if then the this word, the predictor for this word still only gets this as an input.


So this is the important thing here.


So the the part of the model that predicts the is two words away, doesn't isn't being


informed that this word is being produced here.


So naturally, you would expect the quality to be worse because the the word one position


away two positions away and three positions away are each predicted basically independently


of each other, just from the from the source context.


So there is no, there is no, you can't expect like a coherency between the words or not,


not a lot.


So this is the fundamental trade off with such a model, you can predict farther into


the future at the same time, but then these predictions can basically depend on each other.


And this degrades your performance quite a bit.


So what these authors do is to remedy that they say, well, these these things here, we


can I mean, we can produce a bunch of them, right?


Since all that's required as an input is this, we can actually produce, like we can produce


a batch of them at the same time.


So we can produce one, two and three words into the future.


And we can do this like 100 times in parallel, no problem.


Alright, and we can, we can sample this, we don't have to always take the most likely


word, we can actually sample a bunch into the future.


And this now gets smarter, because now I have a list of 100, basically, suggestions of what


the continuation here could be right, I have, I take this not as a given, but I take these


outputs as suggestions.


And then I can have another model that this is called verify here, and have another model


that scores all of these different all of these different decodings in parallel.


Both of these can be done by the same model, we saw the language model can be either used


to predict or to score something since it inherently predicts the probability of sequences


or of following words, we can, we can let it output this probability, all in parallel.


So this, this also can count as a score.


And what I'm trying to say is you can you can, since the language model is constructed


as a as outputting probabilities anyway, like such, we can use it both to predict the next


word, and also, if we have a suggestion, we can use it to score that, and to say, Okay,


how likely is that?


And then what we can make sure is that the suggestion we are looking for the suggestion,


basically, that has the highest score.


And if you want to be really true to your original model, you say, I want to look for


the suggestion that has the the maximum that would have had the maximum score had ID coded


one by one.


So then basically, you retain the original performance, and you gain a speed up as long


as the what the greedy decoding would have produced is in your suggestion in your box


of suggestions that you produce, as long as that's in there, you gain a speed up if that's


not in there, then you can always you always have the the one word ahead model because


that's you have that anyway, you predict the next word anyway.


So in case none of these suggestions work out, you still have this one word prediction,


basically, which is the the class, the model you started with.


So at worst case, you're as fast as the the greedy model.


And in best case, you always your your suggestions are so good, that they're always the one that


would have been decoded anyway.


So you can basically in this case, do three steps at once.


Alright, so this verify step here is is shown here.


And you see it will decode.


Now this is just one suggestion, keep in mind, they can produce many suggestions at the same


time if if there's memory, or, and they can actually they can score each of this.


So they can score this, they can score this, and they can score this also independently


as a batch.


So they can do this in parallel.


And here you see, yeah, here is the executed in parallel.


So the model will go and will score this word in and says, Oh, this would have been this


is the argmax of the greedy decoding anyway.


And it can also score this step and say, Aha, given that there is an end that this the is


the argmax anyway, right?


And you can score this step and say a given that there's in the the argmax would have


been car.


So that's not bus.


So we reject the suggestion, but we keep that part of the suggestion and say, Okay, the


in the is basically what would have been decoded anyway, according to the greedy decoding.


So we can basically accept this here and continue from there.


This is the accept step here.


So this basically, so you can see in this one step, which we'll call one decoding step,


we have basically done two of the greedy decoding steps in one go.


So by predicting into the future, and then selecting the one that agrees with the original


model, because we can, the fundamental thing is we can score in parallel, but we can greedily


produce not in parallel.


Alright, so they actually push this further.


Um, by also eliminating one of the one of the evaluations here, by combining basically,


the next predict step with the previous verify step.


And it's, it's pretty cool to look at that.


So we're in the same situation, you have this and you suggest this continuation.


And then the scoring model, again, will go will go here.


But while you verify, you also do the next predict at the same time, since you've built


your model, since it's the same model.


And this model, every time you execute it, it outputs a distribution over the next set


of positions, you might as well take the outputs of it, right.


So when you then decide to accept this here, you will already have the outputs computed


for the next three positions.


So this you can feed directly into this next predict step, you basically don't have to


execute it, you simply go to the one you've accepted.


And then you look at the outputs that you get anyway, from this model, and use them.


So you might ask, okay, which which, how does a model look that like scores and predicts


into the future?


And this the answer is here, it's a bit out of order, I would have maybe liked this more


previously.


But in any case, this is what they do.


So they, they use a transformer architecture.


And you have to imagine it starts down here.


Actually, there is a huge network down here, right?


This is just the output layer.


So there's a giant transformer network down below, and it produces this output representation.


Now normally, from this representation, you would go to this what's called p layer here.


This is a output vocabulary projection.


So this has one entry for each of the words in your vocabulary, so the a cat and so on.


And you would then for each one predict a probability.


So with this representation, you basically project it onto this vocabulary and predict


the probability distribution over the next word.


But what they do is they say, No, no, we not only need the next word, we need the next


three words.


So let's actually split this output signal into three output signals.


And they do this by introducing this, this hidden feed forward layer here, or a hidden


transformer layer.


It's a hidden layer.


Yeah, we insert a single feed forward layer with hidden size, okay.


So they insert a hidden layer.


And then they also add these skip connections here.


Right at the skip connections, which basically just means they feed through this output directly


to to here, and add it to that.


So basically, the feed forward layer needs to transform these this output here into the


vocabulary input one step ahead, two steps ahead, and three steps ahead.


And you can see here, the those are independent, right?


They don't depend on each other.


There's nothing feeding back p1 here into the decision of p2.


So they can be executed in parallel, but they lose the dependence on each other.


Alright, so that's, that's the architecture.


And you can clearly see here, it's able to predict three steps into the future at the


same time.


So yeah.


Alright, so they also do different adjustments where they say, now, yeah, we can also do


also kind of sacrifice a bit of a bit of the fidelity to the original model by not requiring


that the basically, we don't, we don't only accept when the suggestion is the perfect


best suggestion that would have been decoded by the greedy model.


But what we could do is we could just, if it's in the top k, we could accept it if it's


in the if it's good enough.


Basically, one of the suggestions that we have is good enough, then we'll accept it.


Or when you have like some sort of distance metric, they say here, so the distance between


our suggestion, and the maximum, so the what would have been best by the greedy should


be smaller than some constant epsilon.


And that way, you can sacrifice a bit of performance, but your suggestions will be accepted much


more often, and thereby your speed up will be much higher.


And they also experiment with whether or not they should fine tune the original model along


with their model.


And also the experiment with knowledge distillation, where they basically have like some, some


teacher model, and you train the your model on the output of the teacher model, don't


want to go too far into this since these these are mostly kind of things to make it work


even better.


And you can see here that this is for example, a machine translation task.


So this is the WMT 2014 English German translation.


And there's a regularly they get a blow score of 26 in here, higher is better.


And if you can see, they get a fairly sizable speed ups by keeping the blow scores fairly


constant.


So they they almost speed up by two x.


But if they allow the blow scores to go down a bit, they get a much higher speed up of


like three.


And then if they do like distillation and fine tuning, they actually manage to keep


up the performance, even though they get very, very high speed up.


So they get speed ups until like five x, not dropping the blow scores very much.


So that's, that's pretty impressive.


Another experiment they do is image super resolution, where you can see here with regular,


they try to really keep exactly the original model output.


And it doesn't, it doesn't speed it up too much, but when they allow for a bit of a mistake


to be made.


So here, this is image super resolution, so values are between zero and 255.


And they allow epsilon equals to two of that.


So that's this kind of less than 1% error on the individual pixel, then they get a speed


ups of seven x or something like this.


And you can see in this region here, that when the K is four, and K is the number of


steps that you decode ahead.


So and the mean block size is 3.75.


That means on average, 3.75 steps ahead are accepted, which means basically their their


suggestions are almost always good enough to be accepted.


So they get this massive speed up by basically being able to jump these decoding steps.


Yeah, so they have a bunch of other results here.


They're show their wall clock time speed up since iteration speed up as well.


But if you have to pay in huge computational costs, it's not so good.


But they also show that they have big kind of wall clock speed up up to up to four x


here in super resolution and over three x in translation.


So it's a pretty cool paper.


They give some examples here on two more tables, some examples of their super resolution.


And yeah, if this might be something for you, then use it.


I think it's a pretty neat trick and especially for production systems.


All right, that was it.


A. I. Learns to Play Starcraft 2 (Reinforcement Learning)

StarCraft II is a multiplayer game where the objective is simply to eliminate the other players.


You do so with an army of attack units. We like this. We do not like this. Those attack units cost


resources to make, so you probably need to set up a base, build various buildings, collect resources,


and then you can build that army. All of this involves seemingly endless strategizing and


knowledge of the game, along with years of tuning muscle memory to achieve high actions per minute.


But we're programmers and we can just use libraries that let us use Python code to play the game for us.


Even this, however, takes a while to master all of the strategies. But we're also machine


learning practitioners. Could we just simply import some reinforcement learning from Stable


Baselines 3 and play the game? I guess it's going to be a little harder than that. But let's say


that we really do want to use deep reinforcement learning to play some StarCraft II. How might we


actually do that? Like most deep learning problems, at the macro level we have to consider what our


inputs and what our outputs are going to be, as these are the things that we actually have control


of and we have to determine the characteristics of these ahead of time. For the input, what


information do we think the neural network will actually need? In this case, it could quite


literally be the game visuals, like the frames from the game. It could be some other graphical


representation or it could just be a vector of values, like number of units and minerals and


locations of things and so on. Then for the output of this model, we need to consider what this neural


network can actually decide. Essentially, with deep reinforcement learning in this case, this is


going to be the actions that the model can actually take, like whether to expand and build out our base,


to attack the enemy, and so on. For the input into the model, I think something like the mini-map is


actually a really good candidate for model input. Showing the actual game would mean that the agent


would have to come up with some logic to either move the camera around or it would probably wind


up mostly paying attention to the mini-map anyways. So I think building our own version of the mini-map


with the data that we think is important, I think that's the best place to start from. For the input


map, we're going to start with a blank canvas that is just simply the size of the game map. Then first,


we're going to draw the minerals. Not only are we going to draw them, but we'll also make them more


bright the more minerals there are left. We'll use the same concept for minerals as well as the gas,


but also a similar idea for building and unit health. Here's the result of this so far. We can


only gather information on minerals that we can see with units, which is why some of them are just


very dim and then some are very bright. The dim ones we just we haven't seen so we have no idea


how many minerals are there. Next, we'll mark the enemy starting location with a red dot. Then we're


going to use a similar strategy for enemy units in another red and then enemy structures as well


in a red color. Then we're going to mark our own nexus buildings in a light blue color and then


all the other structures a greenish color. We're going to mark Vespene as a pink. We're doing this


after our structures because if we just slap a structure on top of the Vespene, we wouldn't


actually know what the amount of Vespene was, right? You can only have one thing on a square.


So I just went with draw the buildings first because we end up building a building on top of


the gas to collect the gas. So draw the buildings first then draw the Vespene. Then finally, we'll


draw all our units which will be green by default, but we're going to color the void ray unit which


is our attack unit specifically a blue. I think this gives us a pretty good final result where


it's fairly easy to see what's going on in the game. I did most of my R&D here in Linux which


didn't allow me to run the graphical representation of the game and just by watching games with this


representation, I think it's really easy to see what's going on, how we're doing, and all that. I


think you could definitely make good decisions with this information. So I think this will be


a pretty good starting point for input to our model. Will it work? Maybe. Remember the best


thing that we can do is keep things simple to start. One issue I can already imagine here is


that this is just a ton of mostly noise per input. The image is mostly empty so it's going to be an


area that we could probably improve upon later. But now that we have network input, we need to


handle for the output logic which will be the macro actions that we're going to allow the


reinforcement learning agent to take. For action zero, I'm referring to it as the expand action.


This is the code for it. Essentially for expands, we want to make sure we have enough supply to grow


and build as many void rays as desired. If we have that, then we'll check to make sure we have


enough probes. These are essentially the worker units. If not, build more of those. If we don't


need those, then let's build assimilators to collect the vespine gas. Finally, if we have all


of this stuff, then let's make another base in another resource area. For action one, it's all


about building stargates and to build stargates, we need gateways and cybernetics cores to also


exist. If those don't exist, build them. Otherwise, build a stargate. Right now, the code is limiting


one stargate per base, but this could be changed to allow the AI to make endless stargates,


theoretically speeding up the ability for you to build void rays faster. For action number two,


if we can afford and we can build a void ray, build one. Action number three, this is to scout.


Just not too often. Essentially, scouting is sending a poor probe to a certain death


and we're just doing this to see what the enemy is up to. Initially, we're going to start out


completely random. So theoretically, one out of every six frames, we could be sending a probe


and this would be faster than we could possibly build probes. So just to limit this, we're adding


a once every 200 frames. That's when you can send another scout. Action four is to attack. First,


we're going to look for any close units, then any close buildings, then any known unit, then any


known building per void ray. If all else fails, then we're going to send the void rays to the


enemy start location. We could probably get a little more fancy here since it's totally possible


that we eradicate the enemy from its starting location and we have no idea where it is from


this point, but it's somewhere, building its base and collecting resources and all that. So we might


want to have some sort of ability to hunt or seek out the enemy in other locations. Finally, action


five is mostly tuck tail and flee. The void rays should also be capable of just simply returning


to the base after a fight. So it's not just to flee, it's also just to bring our void rays back


home. The void ray units are really our only base defense here. So if they're not busy elsewhere,


we should bring them back sometimes. At this point, the last thing that we need to do is


calculate a reward for the agent to learn from. But first, is this even a worthy experiment at all? I


can already hear some people complaining that the agent isn't controlling every single action in the


game and we must ask ourselves, how will we know if this agent is good or not? We need a benchmark


to compare to. So before we waste our time on a reward mechanism and all the R&D required there,


which honestly is often the hardest part of any reinforcement learning development,


we really do need to test random, which is really just a question of how good does an agent do if it


is just randomly picking actions. We have a lot of helper code here, some fail safes. Isn't it really


just possible that randomly picking actions will also win games or at least be remotely comparable


to any reinforcement learning agent? Well, we can test this really easily. After running 200 games


running random actions, we have not won a single game. Okay, I think we have something we can


certainly work for. All right, now let's talk reward. In order for the agent to learn anything


at all, we need to reward it for doing quote unquote, good. Rewarding agents in reinforcement


learning is often the hardest part of all. Take our problem for example, the actual goal here


is to win games. It's pretty simple. If we win, we could give the agent a reward and if we lose,


a big negative reward. The issue is, as we've seen already, the agent starts off basically random and


the number of observations and actions per game can be 5,000 to 10,000 or even more sometimes.


If we go through 10,000 actions and lose the game, as we pretty much know we initially will


every single time, how will the agent know which actions were the ones that lost the game and more


importantly, which actions would be good to take? With this many actions to sift through before


getting an actual reward, it's far too challenging for a model to discern which one was the good


action and which ones were the bad actions. So instead, we're going to need some sort of


intermediary rewards to help boost the activity that we think will be useful for victory. To do


this, you can just give a very small reward per step for things that you think might be helpful


and then you can still give one large reward at the end for actually winning the game.


So what might we reward to help win? We could reward resource gathering but this would probably


just result in an agent prioritizing lengthening the game so it can collect as many resources as


possible and it probably wouldn't want to build many attack units since these cost resources.


We could go off of total units and buildings as well as a sort of reward for the overall size of


our military but again this would disincentivize the agent from actually winning and completing


the game. Instead, it would just want to build lots and stay alive for as long as possible.


So for me, I personally settled on something extremely simple. A reward for attacking. For


every void ray, which is our attacking unit that we have in our army, if that unit is actively in


combat then they earn the agent a small reward per step. This incentivizes actually eliminating or


at least attacking the enemy. It will still possibly be the case that keeping the enemy


just barely alive to make more units and buildings for us to destroy will happen but my hope is that


a large reward for actually finishing the game will overcome this. Okay, so we've got some great


ideas here but we still have to actually connect Stable Baselines 3 in this Python StarCraft 2


environment and to do that we also need to convert the Python SE2 environment to an OpenAI Gym


environment in order to work with Stable Baselines 3. Due to the way everything runs and can or


cannot communicate while it runs, this presented a far more difficult challenge than I expected.


If you're looking to learn more about Stable Baselines 3 and custom environments, I'll put


a link in the description for a tutorial series I did. It's a lot more simple and easier to understand


if you're new to it compared to what I'm going to be doing here. So for everything to communicate


here I ended up going with a state file. This file contains the state which is going to be our mini


map, the reward, action, and whether or not the game is done. And this shared file is how we'll


communicate between the systems here. Is there a better way? Absolutely. Will this work though?


I hope so. So this is definitely going to be very confusing with everything that has to connect and


wait on each other and all this but let's just go in logical order. So to actually run everything


we're just going to have a training script train.py and this is what's going to actually run and


start up basically everything. So for train.py there's really nothing fancy here. It's a typical


Stable Baselines 3 training script. The script runs the sc2env file which I haven't shown yet


for our custom environment. So now let's peek at this sc2env file. So what this is is a custom


OpenAI gym environment. Essentially we have an init step and a reset method. Again to learn more


about this check out the SB3 tutorial in the description. It'll be much more easy to understand


than this but basically all this does is initialize the action and observation space for the agent


immediately. Then before the environment actually runs it is reset and here is where the magic or


hacky code happens. In the reset method we empty the map and prepare our state pickle file. Note


that the action in this pickled dictionary is none to start. After creating and saving that pickle


we open up another process to actually run the StarCraft 2 game. So this is actually running the


game with the logic that we've kind of described up into this point as well as through the Blizzard


API and actually playing StarCraft 2. Finally the reset method actually is just returning an


observation which is that map that blank map that we've created. So at this point theoretically the


StarCraft 2 game is just kind of hanging out waiting for us to issue commands to it. So we've


returned this observation from the reset method which Stable Baselines 3 via this train.py file


passes through the reinforcement learning model. In this case that's a ppo model and that model


returns an action. Stable Baselines 3 then passes that action back to our environment's step method.


In the sc2env step method the first thing we wait for is an action to be picked by the model. Once


we have that action we add it to the state dictionary. Keep in mind again up to this point


the action has been none. So what's actually waiting for an action? Our actual StarCraft 2 game


and the script that communicates with it which is Incredibot.py. Once that script detects a new


action it performs the action actually in the game. It redraws that map that we've created.


It calculates any reward then replaces the previous state dictionary with a new one where yet again


the action is none. But we're returning the reward and a new observation. We'll also handle at the


end for if the game actually ends. From here which script is waiting on an action? That's the sc2env


script and the cycle goes on until the game ends which will issue a done and then the environment


resets and we continue on indefinitely. So yeah this is definitely the most spaghettified code


for reinforcement learning that I've written but does it work? Of course it works! Around this time


I spent quite a while tinkering with various rewards and logic but the one that worked best


was just a simple reward for attacking and of course winning or losing. While the models were


training I was obviously tracking things like overall reward. The reward mechanism itself


changed quite a few times so each variant isn't necessarily directly comparable to the other one


as well as I have personally found that it's best to try multiple times with the same exact


reinforcement learning algorithm as it seems the random initialization starting point of all models


can actually play quite a significant role with reinforcement learning and that actual model


learning over time. As you can probably tell I made many attempts before settling on that very simple


reward mechanism of just simply rewarding for combat and then at the end a reward or punishment


for a winner loss. The best model landed on a reward of about 200 at its peak. That said what


does that actually translate to? As models trained I also kept a log of wins and losses since that's


actually what we're truly after so I wanted to measure the winning rate. In the case of this


model at the same time as its 200 reward peak we can see game victories at around 70% against the


hard computer bot which isn't too bad especially considering random was literally 0% we were never


winning. Despite the amount of effort put in so far I would still consider this just the beginning


of any reinforcement learning with StarCraft 2 and I'll have to think more on how I might expand


this to do better at both macro tasks like we've done here as well as some sort of micro reinforcement


learning algorithm probably in the form of two separate algorithms running at the same time


where one algorithm is focusing on macro like what we have here and making large overarching decisions


like build buildings and expand or attack and so on and then another one that focuses solely on


micro tasks for example like what coordinates to go to or build on and so on but overall I'd say


this is a pretty good start. I did also attempt to build some sort of hunting capability for the


void rays to find enemies at more locations than just their base or if they just happen to show up


since it seemed like many games were lost because the enemy just hid when we had the advantage but


later came with a bigger army and defeated us. I ended up allowing void rays to look at random


coordinates if no other enemies were present but my results here were actually worse than


when we weren't looking at all. I still think there's something here to be fixed logically


probably fairly simply but again I wonder about maybe instead making a reinforcement learning


algorithm that chooses what coordinates to fly to rather than me hand coding even more logic.


I think making a micro strategy reinforcement learning algorithm to pick coordinates for travel


is very likely the next step to try. That however is a project for another day. I've hosted all the


code as well as a decent model that you can play with if you'd like you can find that in the


description of this video. If you're interested in learning more about neural networks and how


they work then you might want to check out Neural Networks from Scratch a book by myself and Daniel


Kukiewa inside of which you'll learn how to code neurons activation functions how to calculate


loss do optimization and back propagation and of course apply everything that you've just learned


to a problem. The book itself is in full color for graphs and code syntax highlighting everything's


on high quality pages with full font and a comfortable line spacing to make reading easy.


Finally every concept including the math and coding is broken down into the truly atomic


parts and then put back together step by step to give a full understanding of neural networks and


how they work including all of the math behind them. By the end of the book you will have created


your own neural network framework from scratch. This is the book that I wish existed when I was


learning deep learning and I'm extremely happy to be able to offer it to you. If this sounds interesting


to you then you can head over to nnfs.io to learn more and buy yourself a copy.


Running locally and getting surrounding data - Halite III coding competition p.2

What's going on everybody and welcome to part two of our highlight 3 and slash highlight 2018


tutorial series. In this video what we're going to do is start adding some logic to our bot to


hopefully get it to not move so randomly maybe move with a purpose. So I'm going to start doing


this locally so if you want to follow along locally just go to the learn tab here and then


downloads and then download the python3 version for your operating system. When you do that you


will get something that looks like this and this is the file that we're going to be modifying them


my bot file. So go ahead and do that and let's go ahead and get started. So the reason why I


really want to do this is because the logging doesn't seem to work as well on the web editor


for some reason like I don't actually get all my log output on the web editor whereas locally when


I do logging.info I get it and you can't print with the you know built-in web editor because


of the way Halite is actually processed you're gonna it's gonna it's gonna see those things


that you print and think that's an input to the game engine and it's not. So anyways we're gonna


start working locally so and then this file is exactly the same file you get if you download it.


I just removed basically all the comments because we went through everything I only left this one


because it's important to remember that this is where the two second timer begins. Okay so let's


go ahead and get started and basically our main issue here is that our ships are just moving


randomly to collect Halite and now what I want to do is start moving towards Halite instead. So


after a quick you know kind of poking around the documentation I found that every ship you know a


ship has a position and from that position you can so from any position so it's not it's not


a ship attribute it is a method of any position you could need. So from any position you can do


a.get surrounding cardinals so that means get the north, south, west, and east so you can get


that the information not only the position of those positions but also the you can use that to


start figuring out okay well how much Halite is at each of those positions so you could start at


your ship's position and be like okay what are the surrounding cardinals here where is the the best


Halite in the next closest move I could possibly make and that's what we're going to do but you


could later you know kind of expand that out as as many iterations as you could possibly handle and


then choose the direction that is the most advantageous to you probably taking into some


consideration the other ships that you've got but yeah so anyways we're gonna we're gonna make use


of this this method here so coming back over to our bot for ship in me.getships the first thing


that we want to do is our choices to move because each ship can only move like for each turn each


ship can only move once like one position and it can only go up down left right so we have choices


so choices will be ship.position so ship.position is that position object and then we can do the


.getsurrounding was it get surrounding cardinals yeah so we can get those surrounding cardinals


now what we can do is and in fact I think I might sit down less than I might I might have us actually


keep this logic this is probably pretty good starting logic here but we don't necessarily


want to move randomly but the next thing I'd like to do is let's just do a if game.turn number


equals 15 we're gonna logging.info choices so at turn 15 we're gonna save a bunch of choices now


let's just run this game really quick so to run it you can look at this run game batch file and


in fact this should work right out of the box because we've kept our name is my bot.py so you


can either run it right from here or you could you know like you could modify it and then just


you just double click this or you can just open a command you know.exe and run it but I'm just


gonna double click it that was pretty quick and I also don't see any real errors let me do that


again okay so I think we're hitting some sort of error here so let me go ahead and just copy this


I want to actually see the error so I'm gonna open up command prompt let's run this real quick


oh okay some sort of Python issue about tabs and so I just put it I just installed sublime


text let's see so I'm pretty sure we're the ones yeah ooh I don't know what the option is in shoot


okay in then using spaces maybe whoa okay now I got it okay now we just got to find the rest of


them so it looks like this one might be the last one okay now let's try again run game okay it


looks like it works so whenever a game runs you get your logging information comes here so if we


click on one of these we'll see indeed at turn 15 we got the positions the cardinal positions at the


you know surrounding cardinals per ship and we should have gotten the same thing for bot one


yeah there it is also whenever you run you also get replays in here and some of these are error


logs from before so in fact let me just delete those and then let's just run this one more time


so we get the right log so now going into replays we can see we got a replay and then we get error


logs for each of the bots so I'll click on this and this just currently just contains warnings


about about our ships so for example each time your ship moves it needs to exhaust halite so in


this case it was directed to use for halite so it could move south but only had zero halite so the


ship could no longer move for example so anyway closing out of that we've got a replay and if you


want to watch your replay you can just go in here click watch and then just click and drag the replay


anywhere on that page and boom here's our replay the other thing we can do is change our theme


which I'm going to do now to oh there's a new theme so there's colorblind halite 2 which is what


I had been using and then roguelike I'm not sure what that does besides apparently not let us change


it immediately interesting it's like I've never heard of roguelike before but it looks like ASCII


or something stuff okay anyway I'm gonna go with halite 2 though drag that in okay so yeah because


with roguelike roguelike how could you you can't see the actual halite so I'm not really sure what


the deal is there anyway cool so now that's basically kind of the workflow that I tend to


use at least when I'm starting eventually if I'm just tweaking little things I'll just go to play


copy and paste my code in and look at it really quick there make quick tweaks but as long as I


want to log info I'm gonna do it locally so okay so now that we've got the get surrounding cardinals


the next thing that we want to do is get the halite for those surrounding cardinal positions


so that's what I'm gonna be doing in the next tutorial if you've got questions comments concerns


SynFlow: Pruning neural networks without any data by iteratively conserving synaptic flow

Hi there, today we're looking at pruning neural networks without any data by iteratively conserving


synaptic flow by Hidenori Tanaka, Daniel Kunin, Daniel L.K. Yamins and Surya Ganguly.


So this paper on a high level does what the lottery ticket hypothesis does, but does so


without any data, it prunes a neural network at the beginning.


And it does so, it's able to do that, because it claims that its algorithm avoids this problem


called layer collapse, and then is based on conserving a quantity they call the, the synaptic


flow.


And we're going to look at this.


And it's pretty cool algorithm, it seems to work pretty well.


As always, if you want to help out, you can share this video.


And let me know in the comments what you think of it.


I do read the comments.


And I would love to hear from you.


Alright, let's dive in.


So they're saying pruning the parameters of deep neural networks has generated intense


interest due to potential savings in time, memory and energy, both during training and


at test time.


Recent works have identified through an expensive sequence of training and pruning cycles, the


existence of winning lottery tickets or sparse trainable sub networks at initialization.


So what is this paper talking about?


If you if you don't know much about pruning, here is kind of a basic overview.


So if you have a neural network that consists of many, many layers of neurons, what you


can do that one way of pruning that what what the goal is, is to end up with a small neural


network that performs well.


But for now, we have a big neural network that doesn't perform well, it hasn't been


trained yet, right.


So what you can do is you can first train the neural network.


And then you have a big neural network that performs well, and then you can prune it.


Now a lot of times, a lot of the time this this has been seen as sort of the pruning


way, you would train the big neural network, and then you would prune it, because the other


way was not feasible.


First pruning, and then training was not feasible.


You might you might ask, okay, we might just want to start with a small one.


And yeah, that's correct.


So what does this first way by you, this first way is buys you mainly two things.


So imagine this network right here is much smaller than the original network.


So it is less, it uses less storage.


So you can potentially if you want to ship it to like a customer over the internet, you


maybe instead of a gigabyte, you only have to transfer a few megabytes.


And that's pretty cool.


The second thing if you prune in the correct way, you can also make it faster because now


there's less weights to multiply with, you can actually make it go faster.


So pruning is a now this this combines with techniques called distillation and so on is


our ways to make the networks smaller and faster.


So if your customers are for example, on on mobile phones, then you can ship you can train


a big network to a good performance on your big GPU server, and then ship it out to a


mobile phone once it's small, and it will perform fairly well on that mobile phone without


GPU.


So what about this other way?


Now, in order to do the other way, we would sort of have to have an idea which one of


these big networks which sub parts of these layers are the good ones right in order for


us to do this first prune and then train.


The interesting thing is that the paper the lottery ticket hypothesis, I've done a video


on this and we've also interviewed the author on our ML street talk podcast.


This paper has shown that this is in fact possible.


A long time people have thought we need the big network in order to train right we sort


of the bigness of the network, the full connectedness of the network is required for the training


dynamics.


But this paper has shown this is not possible, you can prune at the very beginning.


Now what does it do?


It first trains in neural network, like in the in the olden days, then it prunes the


neural network, and then it remembers which connections of the train neural network it


has pruned.


And then it simply goes back to the beginning of training right here, up here, and says,


I now know which connections are important.


And I'm simply going to prune all the other connections other than these ones.


And then interestingly, if you prune first, and then train, that works just as well and


can actually work even better.


The interesting thing here is that I mean, this is a big, big cycle.


But the interesting thing that the paper demonstrates is that this is even possible, right?


People thought it wasn't possible.


And this paper demonstrates if you only knew if you only knew which ones you must retain,


you can prune at the beginning of training.


The lottery ticket hypothesis paper, though still requires to actually train the full


network and then do the pruning, like in a classic way in order to find out which ones


you need to prune and which ones you don't.


This paper right here takes that idea and says, can we find?


Can we find a pruning algorithm that prunes at the beginning of training, yet does not


have to train the full network, in fact, doesn't look at any of the data?


Okay, and this is going to be our starting point.


So their story is going to be, it's quite an involved story.


And I think the overview is important as we go through the paper.


So first, they named this problem called layer collapse.


Now layer collapse is going to be whenever a pruning algorithm removes the entirety of


a neural network layer, which means that no information can flow anymore, and therefore


the network can't train.


And they claim that this is the main problem why these current pruning algorithms cannot


achieve very high pruning ratios.


So can like very high compression ratios is because they do premature layer collapse.


They then formulate this maximal critical compression axiom that has sort of a guiding


principle to build pruning algorithms.


Second, they show that this quantity called synaptic saliency, a general class of gradient


based scores for pruning is conserved at every hidden unit layer of a neural network.


So they show that these are conserved.


And they show this because this is their their argument is going to be first the argument


is layer collapse is a problem.


The second argument is these things are conserved and these the conservation of the synaptic


saliency leads to the layer collapse.


And we're going to see how that happens.


And then third, they say the solution to that is iterative pruning.


So they show this at the at the example of iterative magnitude pruning, which we know


avoids layer collapse.


So iterative magnitude pruning is something that happens in this lottery ticket way of


of doing it.


This lottery ticket way, you can actually do it not in one step, but it tends to work


better when you want to go from 100% of your weights to just 5% of your weights, it tends


to work better if you do it in stages.


So first you go to 90% to 80 to 70 and so on, down to your desired thing.


And this iterative procedure, they claim is what is what circumvents this problem of layer


collapse.


And then at last, they say we proved that a pruning algorithm avoids layer collapse


entirely and satisfies blah, blah, blah, if it uses iterative positive synaptic saliency


scores.


So they bring it all together and say, if an algorithm satisfies our axiom, and if the


algorithm is an algorithm that uses these saliency scores, like this one here, and if


the algorithm is iterative, then it is not going to be subject to layer collapse, and


therefore, it is going to be able to compress to a very high compression ratio.


And then they actually do suggest an algorithm, this synaptic, sorry, iterative synaptic flow


pruning SynFlow, that does all of this and never looks at any data.


Alright, this is quite a story.


But remember what we're doing first, layer collapses a problem.


Second, why is layer collapse a problem?


It's because of this synaptic saliency conservation.


Third, we can avoid it by doing iterative pruning and lastly, this algorithm does it


without looking at data.


Okay, so layer, layer collapse, layer collapse is a pretty simple phenomenon.


I've already said it, if you have a neural network, and it has a bunch of layers, and


let's draw a couple of neurons here, and the neurons are connected to each other via connections,


connections, connections, connections.


And you have a pruning algorithm.


Now the prunings algorithms they consider here are so called single shot pruning algorithms.


What they do is they look at the neural network and this can be before training or after training.


But they at some point, they look at the neural network, and they each they assign a score


to each of these weights, like they'll say, you're one, you're a five, you're a nine,


and so on.


And then they simply prune away the lowest scores.


Okay.


And you tell the network what compression ratio you want to you tell the network, for


example, please prune away 90% of the connections.


So these algorithms would look would assign the scores once and then remove the bottom


90% of weights.


Okay, like this.


So those are the single shot pruning algorithms.


Now what is layer collapse?


Layer collapse is whenever an algorithm removes all of one layer, because maybe so here was


a nine, maybe you have like 11, 12, 13 here, okay, so and then you're in you're in this


situation right here.


And the algorithm is pretty, it's pretty dumb.


It's simply removing the bottom 90% of the connections.


And here it figures I need to remove one more to meet that goal.


I remove the one with the lowest score, I'm going to remove this one.


And it's pretty obvious that now no more information can flow from the beginning to the end of


the network, because, well, what's, where is it going to float to?


It's a bit more complex than that, like, you can just retain a layer like a connection.


For example, if this were a connection, there would also be no information flow because


you'd have no outgoing connection here.


But ultimately, layer collapse is whenever an entire layer is removed.


Okay.


And they they do say somewhere that that's the case.


I think layer collapse here, layer collapse occurs when an algorithm prunes all parameters


in a single weight layer, even when prunable parameters remain elsewhere in the network.


So I'm not as such, I'm not sure that this is like a giant problem.


It gets to be a problem, but it could be circumvented fairly easily, right by simply saying, if


you're about to prune a connection, that's integral to the information flow from the


start to the end, don't prune that connection, prune some other connection, right.


And then you could simply avoid that.


And I'd be interested in how that works out.


So but in this case, for purposes of this paper, they simply consider algorithms that


assign a score and then prune the bottom couple of percent, okay, for so we don't want any


like handcrafted rules in here or something.


So they look at this quantity called the max compression.


The max compression is a quantity that's basically the maximum achievable compression whilst


still avoiding layer collapse.


And they say, for example, for a network with l layers and n parameters, the max compression


is n over l, which is basically means every layer only has one parameter remaining and


therefore, if it's the correct one, therefore, information can flow from start to the end.


All right, so this is the maximum achievable compression, anything beyond that would automatically


induce layer collapse.


Now, anything before that could induce layer collapse, but there is a way to compress the


network to the same level without inducing layer collapse.


And their point is basically that these other compression algorithms that they compare with,


they always they always induce layer collapse before they actually have to because they


cut off a connection that leads to layer collapse before they like there would be another connection


that they could cut off that would not lead to layer collapse.


And of course, if you are layer, if you have done a layer collapse, then you accuracy immediately


drops to zero or two random, because no more information flow.


So they look at these things here, random pruning is where you simply assign a random


score to each connection.


Magnitude pruning is what the lottery ticket hypothesis does, but just they look here at


a single shot.


So you simply look at the magnitude of the weights, and this can be before after training,


I think they do it after training here, which is classically done, you look at the magnitude


of the weights and you prune the bottom 90% away.


There are also two more advanced methods, these SNIP and the GRASP, so SNIP and GRASP,


which look at the gradient of the training loss in the network, and they decide according


to that gradient, which which things to cut and which things not to cut away.


The GRASP even involves the Hessian right here.


So they're fairly, you know, complex method that have some thoughts behind them about


why they do what they do, yet they all induce layer collapse before they actually have to.


So they define this thing here called the critical compression.


The critic the critical compression is the maximal compression ratio a given algorithm


can achieve without inducing layer collapse.


So the critical compression here is basically whenever that algorithm goes to zero, that's


the critical compression, that's kind of the farthest you can push the algorithm without


him without sorry, German speaker, without it induced without it inducing layer collapse.


Okay, so you can see that for these baseline algorithms, the layer collapse occurs way


below the theoretically possible max compression.


And we're going to see that in their algorithm, this sin flow, that this max compression is


achieved.


And it's actually achieved without any of these handcrafted rules that I mentioned,


it is the algorithm by design already achieves this maximum compression ratio.


So they formulate this here as a guiding principle.


They formulate is an axiom, I would, I would rather say it's like this kind of a, it's


kind of a guiding principle of building these algorithms that any algorithm you build should


have.


So the critical compression ratio of a pruning algorithm applied to a network should always


equal the max compression of that network.


It basically means when you build a pruning algorithm, if you push that pruning algorithm


to its limits, it should not do layer collapse unless it absolutely needs to.


Okay.


Again, the extent of this problem, I don't, I don't know, but they do demonstrate that,


that they can push their algorithm a fair bit further.


Now without inducing layer collapse, you already see that these other algorithms like in this


regime, apparently layer collapse hasn't happened yet, because they still have sizable accuracy.


But there's still a, you know, there is a reasonable difference here between those and


the sin flow algorithm.


So I'm not too convinced yet that layer collapse as such is the problem, because they have


a difference before their layer collapsing, as you can see, right here.


And I have a feeling that this difference here is due to this iterative procedure and


not actually due to the phenomenon of layer collapse.


But yeah, so if it were only layer collapse, what you'll see is that they do the same,


the same, the same, and then at some point is like, boom, now I have layer collapse.


Yeah, so the layer collapse story, I'm not sure, but it's part of the story.


So let's, let's go with that.


The second part, which is kind of disconnected.


So they established two things, they established a layer collapse problem.


And now they establish the synaptic saliency, which then later they're going to connect


to the layer collapse.


So the synaptic saliency, they say, is a score, is any score metric that can be expressed


as the Hadamard product of this thing with the parameters.


Okay.


So each parameter is going to be multiplied by the gradient of some function with respect


to that parameter.


They say where R is a scalar loss function of the output of a feed forward neural network


parameterized by theta.


Okay, so many of these pruning algorithms can be formulated in this framework right


here.


And their, their algorithm can also be formulated in this framework.


So you can see the score that the algorithm assigns to a weight can be defined as such.


And as I said, many fall into this category or are similar to this, especially, for example,


they say when R is the training loss L, so this is the simplest case, you take you put


data through the network, and then you take the training loss of that data and you sort


of back propagate it.


And now you're going to prune these connections according to how big the gradient is.


If you say the gradient is very big, that must mean the connection is very important.


Because there's lots of information flowing through it.


So if it's the training loss L, the resulting synaptic saliency metric is equivalent to


the score metric used in skeletonization, one of the first network pruning algorithms.


The resulting metrics metric is also closely related to this right here.


Now this you can see it's not exactly the same, but it's closely related to the one


used in this snip baseline, and also closely related to this thing right here used in grasp,


where it's not just the gradient, it's actually the gradient multiplied by the hessian to


account for curvature.


Okay, so they're going to investigate this synaptic saliency in neural networks.


They formulate two theorems right here about the conservation of synaptic saliency.


Remember synaptic saliency is any score that respects this, that is built like this, any


score s.


The conservation of synaptic saliency, all synaptic saliency metrics respect two surprising


conservation laws that hold at any initialization and step in training.


So these are not usually like in distribution or something like this with high probability.


These things hold at any point in the neural network.


First is the neuron wise conservation of synaptic saliency.


For a feed forward neural network with homogeneous activation functions, and a homogeneous activation


function is an activation function that is can be expressed like this, for example, ReLUs


fall into that category, the sum of the synaptic saliency for the incoming parameters is to


a hidden neuron is equal to the sum of the synaptic saliency for the outgoing parameters


from the hidden neuron.


So what does it mean?


It's actually pretty simple.


If you have a hidden neuron, and you look at all the incoming weights, and you look


at their synaptic saliency, which is this s score of each of these weights, like what


would the pruning algorithm assign to that, and you look at the outgoing ones, then the


sum of all the incoming ones is going to be equal to the sum of all the outgoing ones.


So that's pretty interesting.


And they extend that to layer to the entire network.


So an extension of that network wise conservation of synaptic saliency, the sum of the synaptic


saliency across any set of parameters that exactly separates the input neurons from the


output neurons of a feed forward neural network with homogeneous activation functions equals


that.


So it basically says it remains equal.


It doesn't mean to exactly separate the input from the output.


That's basically the definition of a layer in a neural network.


So what they're saying is that you have a bunch of layers.


And if you look at a particular layer, like this one here, and you look at the incoming


connections, and you sum up all of their synaptic saliencies, that's going to be equal to the


sum of all the synaptic saliencies of the outgoing connections of that layer.


And it can also apply to like a group of layers and so on.


But the synaptic saliency is conserved in that way.


Now why is that important?


And here is where we make the connection with the layer was it later drop layer, whatever,


okay, the fact that the fact that these algorithms tend to drop entire layers before they have


to, if you have in your network layers that are of different sizes, so you have large


layers, and then smaller layers and smaller layers, what will happen is that since the


synaptic saliency is conserved, the sum is conserved, if you have more connections in


one layer, so lots of connections, lots of connections, and in the small layers, you


don't have as many connections, the sum is equal.


So that means each individual one here is much, much smaller.


So the S is very small for each individual one here, and the S is very large in there.


That means the pruning algorithm is going to really, really kill off this these connections


in the big layers, right.


And it's actually going to kill them off to a point where it probably is going to eliminate


that layer before it even prunes many of the connections of the small layer, just because


of that conservation fact.


And they do experiments like this.


I think there's an experiment up here, where I like this one down here better, where they


basically show that you have inverse layer size on the bottom, and you have the average


score that the pruning algorithm assigns to any connection.


Now these, as we've seen, they're not exactly assigning the scores of this saliency, but


they're very close to it.


The Synflow algorithm does exactly assign the synaptic saliency as the score for the


pruning.


Now, we've basically seen that this leads to a bad result, but the synaptic flow is


going to compensate for that.


But in essence, as you can see, as the layers get, so inverse layer size grows, which means


that layer size shrinks, as the layer size gets smaller, the average score of the connections


in the layer is higher and higher, which basically means that the pruning algorithm, if you just


let it go by itself, it's going to kill off the smaller, sorry, the larger layers first,


because they have the smaller scores.


And you can see that even though the other algorithms don't conform exactly to that,


they conform to this approximately.


So these here, because their score is closely related to what the Synflow does, and the


magnitude pruning, because mostly because now I'm not sure if that's at the end of the


training, at the beginning of the training, if you just initialize, then the score is


going to be proportional to, it's going to be proportional to their magnitude and their


magnitude is determined by the initialization scheme.


And the initialization scheme is most of the time like modern initialization schemes compensate


for the fact that you have different number of incoming and outgoing connections, and


therefore, they would automatically assign higher number, sorry, a higher initialization


constant to layers that have the lower number of parameters.


So even the magnitude pruning will conform to this.


Now it might be absolutely reasonable to say that that's also the case at the end of training,


because most parameters aren't going to move super much during training.


So this still approximately holds as you can see here.


Of course, the random one doesn't do that.


Yet, because you prune randomly, you're still absolutely subject to this layer collapse.


In fact, in the random one, the smallest layers would be the ones to go away first, because


it's just more probable.


Okay.


So we've discovered that if you do something like saliency scoring, or something that's


correlated to it, then you're going to remove the biggest layers first.


And that's a problem.


And that's what they say, this con, this fact of this conservation laws, and the single


shot nature of these algorithms that they only assign scores once, and then they prune


away whatever the bottom such and such percent are, leads to layer collapse.


Right?


That's, I think we've established this now that the combination of the two things leads


to layer collapse.


Now they make a little bit of an excursion.


And they say, there is actually something that doesn't run into layer collapse.


And that's iterative pruning algorithms.


So specifically, they look at magnitude pruning.


They say magnitude pruning, which remember is also, if you do it single shot, it also


runs into layer collapse, magnitude pruning avoids layer collapse with conservation and


iteration.


So because it iterates, it avoids that.


And that's what these lottery ticket hypothesis paper does.


It does it iteratively removes a couple of connections, then it retrain the network basically


recomputes the magnitudes and therefore recomputes the scores, and then it prunes again, and


then it recomputes and compend prunes again.


And by recomputing, you can basically these some of the connections that weren't important


before, but just survive the pruning, they can now be like, wait, I have now way more


responsibility as a connection, and they will shoot up in importance to avoid being pruned.


So you can see if you push your network to a sorry to a high compression ratio, then


if you just do this single shot pruning, you run into this layer collapse at some compression


ratio, you simply crash to random performance or zero performance.


Yet, if you do multiple iterations, you can see here already two iterations, then it's


much longer before you run right here into layer collapse.


And if you do three iterations, you do much more.


Now this, the three iterations doesn't mean you prune more like at this, at this point


right here to tend to the one.


All of these things prune nine out of every 10 connections.


It's just the thing that has three iterations prunes, maybe first three, and then again,


three out of the 10.


Whereas the one iteration would prune all of the nine at in one go.


And they give a reason for this they give they say that it's the fact that gradient


descent encourages conservation.


So they give a little toy example here they say to better understand the dynamics of the


IMP algorithm during training, a little smaller, we will consider the a differentiable score,


this one.


So this is not exactly magnitude pruning, but it is very close, right?


The squared, it's just the square of the parameter instead of the absolute value of the parameter.


They say it's algorithmically equivalent to magnitude score.


Consider these scores throughout training with gradient descent on a loss function using


an infinitesimal step.


In this setting, the temporal derivative of the parameters is equivalent to that and thus


the temporal derivative of the score is this.


So now they're going to look at how does the score evolve when they train the network.


And the score evolves exactly as the negative to the saliency.


Surprisingly, this is a form of synaptic saliency, and thus the neuron wise and layer wise conservation


laws from section four apply.


In particular, this implies that for any two layers of a simple fully connected network,


then this quantity holds.


So that this is not new.


But what it basically says is that through training, these connections equalize the saliency


again.


So if you have a very big layer, and here a very, very small layer, and because it's


a big layer, these scores are very much lower, right?


It's just little s and here it's big S per layer.


But then if you prune away, and you run gradient descent on this, these scores will tend to


become bigger.


And in this case, these weights will tend to grow in magnitude, because you've pruned


away the others, they now have more signal, probably flowing to them and more gradient


flowing to them.


And therefore, they're going to grow in size.


And therefore, their score is going to be bigger.


So this gradient descent of this iterative procedure makes the scores better for that.


So basically counteracts the layer collapse.


So they put all of this together and say, theorem three, iterative positive conservative


scoring achieves maximal critical compression.


If a pruning algorithm with global masking, and global masking means that you rank all


of the connections and then prune from all of the connections, it's a difference to layerwise


masking where you say I want to remove 90% of each layer, which sounds like it would


avoid layer collapse, but also it works a lot worse than the global one, the global


strategy, assigns positive scores that respect layerwise conservation.


And if the algorithm so respecting layerwise conservation, it basically means you your


score should be or if your score is a saliency score, then that's the case.


And if the algorithm reevaluates the scores every time a parameter is pruned, then the


algorithm satisfies the maximal critical compression axiom.


So that's basically saying that if you have any algorithm that prunes with a saliency


score, like theirs is going to do, is going to be able to be pushed to the limit until


the maximal capacity is reached, if you reevaluate the scores every time a parameter is pruned.


So this is basically saying that whatever the lottery ticket hypothesis paper did with


magnitude pruning, if you do it with saliency based pruning, you're guaranteed to achieve


the maximum possible compression if you if you push it.


But of course, we know that whatever the whatever the lottery ticket hypothesis paper did is


impractical because it needs to retrain the network every single time it wants to prune,


right?


If we're going to do this after every parameter, that's going to be a long time, it's going


to be impractical.


We ideally want to prune the network before we even look at any data.


And they're going to do exactly that with the Synflow algorithm.


They say theorem three directly motivates the design of our novel pruning algorithm


Synflow that provably reaches maximal critical scores.


Oh, no.


Okay, this was bad.


Malarial critical compression.


First the necessity for iterative sorry, first the necessity for iterative score evaluation


discourages algorithms that involve backpropagation on batches of data and instead motivates the


development of an efficient data independent scoring procedure.


Second, positivity and conservation motive motives probably motivates the construction


of a loss function that yields positive synaptic saliency scores.


We combine these insights and introduce a new loss function where the one is the all


one vectors.


Okay, so this is the loss function of their saliency scores.


And this might seem like so what do we have?


We have the parameters of layer L, the absolute product, sorry, the absolute value of those


parameters.


And then you simply multiply all of the layers together.


And you have this product here with the ones on the side.


So this is a quadratic form sort of.


Okay, this might seem a bit weird.


But but in practice, and this is also what happens in their code, you can do something


pretty easy.


So first, you have to transform all your weights to their absolute values.


Now in their code, you can look at it, they they do remember the signs for later.


So but first, you convert all of them to their absolute values, then second, you simply take


a data point that is filled with ones that literally the number one.


So if your if your input is an image, you just put a one at each pixel, you feed it


through the network with all of these positive weights, and you get out some output, you


get some output vector, okay, then you simply you need to do this.


This inner product with the one vector, which is simply a sum, right?


I don't, I don't get why they, it's a bit of a funky way of writing a sum, right?


You simply sum that up to get a to get a single number.


And this single number now is your is your pseudo loss function.


It's simply the loss function that an all one data point gets when the when the loss


function is just the sum of the outputs.


That's that's that's it.


And then you back propagate that loss to you back propagate that loss to the layers.


Right?


So this is our remember, this is not the score itself, but our score is going to be the derivative


of our with respect to a weight times that weight.


Okay, so you want to back propagate, and then you multiply each of these weights by the


back propagated signal.


And that's going to be your score for each parameter.


Now this doesn't seem too hard, right?


You just need you don't even need a batch, you need a single data point, one back propagation,


and then you get your scores.


Okay, you don't need expensive training, or anything like this.


This seems pretty cool.


And they give an example here.


For example, for a simple, come on for a simple, fully connected network, ie, this, so they


consider here a linear network, right, just so we can look at exactly what happens for


linear networks, you can often compute quantities exactly.


So if we look at just a linear network, without non linearities, we can factor the synaptic


flow score for for any parameter as such.


So the score, this is now not the R, this is going to be the score is going to be this


thing right here.


So you can see that the parameter is multiplied by this thing, and by this thing.


And other than for example, magnitude pruning, this actually takes into account all the input


flow because it goes from this one, sorry, goes from this, goes from this one, it goes


through all the network, right, every path that arrives at this particular weight is


going to be considered.


And every path that goes out from this particular weight is going to be considered.


And the saliency score is going to depend on all of these paths, all of these, all of


the information flow from input to output that goes through that weight.


And if you do this, then you get a really good pruning algorithm.


So yeah, the algorithm is is I've already described it.


And in their experiments, as you can see, right now, they have a bunch of networks,


these VGG networks, or like wide ResNet, they have a bunch of data sets like tiny image


net or c410, where they experiment with these different baselines.


And you can see that the baselines often run into this layer collapse problem.


Sorry, often run into this where all of a sudden, let's actually look at let's look


at this ResNet 18, right here.


Maybe you can find a connection between maybe there's differently sized layers in ResNet


And that's why the collapse happens even earlier.


But you can see right here, there's a collapse if you do magnitude pruning, even also if


you do random pruning, it falls down pretty hard after a while, the baselines, they hold


up better, but you can see in different models and different data sets, that the baselines


crash at some point as well.


Now I've already said the comparison here, it seems a little bit unfair, I might, I might


have over read something, but I'm pretty sure that the baselines remain single shot.


While the sin flow algorithm here is now of course, no longer single shot, it's actually


multi shot, and they've made the exact argument that the single shot is the problem.


And therefore their algorithm is multi, multi shot.


And it seems like they should give the other algorithms the opportunity to also do multi


shot, just to compare them fairly.


As I said, maybe they're doing that, but I'm, I haven't read any anything.


So it, you know, it just seems like the comparison is a bit unfair, if you identify the problem


and then just leave the other algorithms with the problem.


Sin flow is still different from these other algorithms, even if they had the multiple


steps.


Now the counter argument to this, of course, is that these other algorithms all require


the training data, they require actually passing the data or training the network in the case


of magnitude pruning and so on.


So that's pretty expensive, whereas sin flow, you simply pass forward one data point, and


that's it.


That's a good argument.


But it seems like the effect of the synaptic saliency scores, and the effect of the multiple


steps aren't really disentangled in these experiments right here.


It simply shows that it consistently outperforms other pruning methods.


And what I'd like to see is really where that outperforming comes from.


Okay, so that's what I think of this.


And that was the paper, basically.


I'm even even if I'm not convinced quite yet, this is pretty cool, right?


And I think this will, if not be, if it's not used itself, it will inspire kind of a


line of work into pruning at the beginning of training without looking at data.


And maybe, you know, maybe we can even think of building networks, like, instead of just


pruning them, we can think of constructively building networks that observe these properties.


And therefore, we can just construct initialized networks already with good properties such


that we don't even have to go to a bigger network and then prune it down.


It seems wasteful.


It seems like we should just be able to derive principles of what we want in the how the


weights are structured, and then construct networks that are according to that.


And I guess that's what's going to happen in a few papers that are coming.


Alright, again, if you liked this video, consider subscribing, giving it a like, commenting,


and let me know what you think.


Apple or iPod??? Easy Fix for Adversarial Textual Attacks on OpenAI's CLIP Model! #Shorts

OpenAI has a new network called Clip and it is easily confused.


You might remember the experiment from the paper where it confuses an apple with a label saying iPod as an iPod.


Now I've managed to actually reproduce that experiment.


So with my own apple on the left Clip will confidently predict an apple,


but on the right Clip will confidently predict an iPod.


So it turns out if you just give it the opportunity for a third label saying,


wait a second this is just an apple with a label saying iPod,


it will confidently predict that for the picture on the right.