to be zero. However, if the mask is one, what does it mean? If the mask is one, that means that this


Addendum for Supermasks in Superposition: A Closer Look (Paper Explained)

this entire feature right here, let's call it f is positively impacting is positively contributing


to this particular neuron right here. So if the mask is one, this is this, it means the addition


of that feature more of that feature makes that log it go up. Okay, so if the mask is one during


training, it means that the feature positively contributes to the task. So if we look at the


gradient with respect to this function with respect to the the log it and the function


basically means it's just measured measures how high these superfluous log it's are.


Then, what why do we find a negative interaction there? Because if you look at the neural network,


and you forward pass, and this particular feature is important. And you look at the loss g and you


backward pass through the log, it's if it is smaller than zero, that means that there is a


negative interaction right here. So that basically means that if we make this feature higher, then,


in this case, we make this g function go lower. Okay. And that is the case for the correct task,


because if this is the correct task, and the mask is learned adequately, that means it should


assign a low weight to the superfluous neuron whenever the input features, you know, are of


that task. And so it makes sense that this here would be a negative number, because what we want,


if the mask deems the feature important in a positive sense, we want that if the feature goes


up, g goes down. And that's exactly why we have the negative interaction right here, right. So


the negative comes from this being negative. I hope this sort of makes sense. So if the mask is


one, the mask says, basically, if that feature goes up, the loss goes down. Now g is a measure


of the superfluous neurons, the superfluous neurons should be small if the loss is small.


So if this is really from the task, and this feature is really useful, that means if we


increase the feature, the g function should go down. And therefore, this product here is going


to be most likely negative. Okay. And the contrary is, you know, analogous right here. If this is


not of this task, the mask can either be zero or one, right? If it's zero, then this quantity is


zero. However, if it's one, it's more likely that the that there, the feature here, because it's,


I is not the correct task, which basically means that this feature, it is for a different task,


it is good for a different task. So the mask of that different task says it's good right here.


And we have no reason to believe that this would decrease the loss of the loss of this particular


data point in this task. So it's kind of the inverse reasoning. If you look at the actual


derivation here, it's fairly long. And it goes over the cases of the interactions between actually


this initialization and the mask. So the initialization can be positive or negative,


as you can see right here. And I think, I just think that the, the intuition here is that the


superfluous neurons react differently to a data point of the trained task, because they have been


kind of made to decrease for that task. And for that particular mask, as they do for when the data


point doesn't match the mask, when the data point doesn't match the mask, there is no reason for the


logits of the superfluous neurons to be low. And if the data point task does match the mask, there


is ample reasons for those to be low. I hope that sort of makes sense. It is sort of it's a bit more


of an intuition. But if you really want to dig into it, look at the derivation right here. Okay,


second point is the fact that the masks and the superpositions don't really have to do anything


with each other. And that's, you know, I've said throughout the video, like, remember, these tasks


are super easy, yada, yada, yada. So let me make it clear. In this in this diagram right here,


the super masks, these are simply a way to train a neural network in a crude way, right? I don't


think there is, you know, this distinction between mask and network. I don't really like that much,


because ultimately, what you're doing is simply you're training a neural network in a kind of


weird way. Okay, the fact that you always use the same underlying, you know, gray neural network,


doesn't really matter right here. It's still what you do in the supermass training is you provide a


severely over parameterized network, and then the mask simply gets to choose which weights to mix


rather than you get to adjust the weights. If you adjust the weights, you usually get more accurate


than with the mask, but it's sort of like a quantized neural network that you train right


here. So that's the supermass thing. Again, I don't think it's important that the underlying


network is always the same. The only advantage you have is it saves space, because these masks


are very small. The supermass on the other hand, this idea that you overlay all of the


masks together, and then you look at where this at the gradient of the entropy, and you look at


which of the of the mixing factors the gradient pulls the most. That's a different idea. And the


question here is, wouldn't that isn't that independent does really depend on the masks,


or doesn't it? And the you know, the hypothesis would be that if I simply train, you know, three


different neural networks for three different tasks, could I not do the same superposition


trick? Like, could I not just add all of them with a respective alpha, look at the entropy,


calculate the gradient with respect to each of the alphas of the entropy and then decide which task


it is, you know, don't need masks, simply mix neural networks in superposition. So I did it.


And I actually tried their code is available. So big props for their code being available.


I tried their code. It's actually very few changes. And I'm going to append my live coding of


this at the end of this video. So if you want to, if you are interested in watching that you can do


so. But you know, the outcome is if I train neural networks, and I have, you know, done super quick


and initialize them wrongly, probably in all, but if I train these neural net, if I train the masks,


you get to like 92% accuracy in their tasks in each of the tasks. And then also in the average,


if I train the actual neural networks, I get to a higher accuracy, like 93 something, it doesn't


matter, it's just higher. Okay, so that's hypothesis one is the training masks is just a way of training


neural networks, the fact that the masks and the network training itself are that close, I think,


is a testament to how easy these tasks are, like how easy eminence is, I'm going to also hypothesize


that if the task gets harder and harder, and I don't mean 10 class image net, I mean, a thousand


class image net, then these masks are going to degrade severely versus training the actual neural


network, I might be wrong, I mean, you can over parameterize really heavily, and they will still


work. Okay, but in any case, I trade the train these neural networks, and they reached higher


accuracy. And then I did the exact same thing, I laid them in superposition to determine what task


it is, and I could achieve the exact same result. So here in their example, they have 100% accuracy,


100% task classification accuracy, and I reached the exact same thing code worked. I'm not going to


try to scale this up to 250 or 2500 tasks, but I'm going to assume that with, you know, tuning and


stuff, that it's going to work about equally well, you could make an argument that the masks being


sparser, they might be differentiated from each other more accurately. But I'm not sure, maybe,


but it's not a cool, it's not a qualitative difference, right? So these two things are really


two separate ideas that find their way together in this paper, but ultimately have not much to do


with each other. Okay. At least that's from what I can tell, I might, I might be wrong here. And I


might be wrong with respect to their G objective and whatnot. And, you know, but I think that that


these are two cool ideas, but they can be applied independently. So the last thing I want to look at


is their broader impact statement right here. Now, there is a reason. So usually, I kind of track


these broader impact statement, because I think this is this here is sort of fundamental research,


right? This is fundamental machine learning research, we do architecture, we do multitask


learning task isn't really important, as long as we have kind of the same tasks right here on


correlated and so on the same hardness. And I've also made the point that it's really important for


these tasks to be the same hard for this to work in this place a role right here. So um, they do,


they do describe some of this in this conclusion with, you know, limitation that we observed has to


do with task identity inference when models are not well calibrated models that are overly confident


for the wrong task. Okay, so in order for them to infer the correct task, they sort of, so if you


look at your entropy of the models for the tasks, that means you're going to select the model that


is the most sure about the task. This only works if the tasks are equally hard, okay, if one task


is much, much harder than the other task, this other task is always going to say, well, I'm


really confident about this one, because the task is just easier, it's going to be, it's going to


train a neural network is generally more confident, and you're going to misclassify a lot of the tasks.


So, so here, what does this have to do with the broader impact statement? If you look at the


broader impact statement, what they say right here, so they say, a goal of continue learning


self-many tasks with a single model, however, it is not exactly clear what qualifies as a single


model. Therefore, a concrete objective has become to learn many tasks as efficiently as possible.


We believe that sub-sub is a useful step in this direction. However, there are consequences to more


efficient models, both positive and negative. So this is sort of what the community does. So


there are three things that I've seen so far in broader impact statement. First,


some people say this is not applicable to us, which I agree for most fundamental research,


broader, like the broader impact statement is supposed to be what does this particular method,


how will this influence broader society? So not applicable, completely valid for most of these


research papers, because guess what, you can use any method to do good or to do bad. And that's,


that's the second, second part. Second method is basically, you just say generic statements,


how you can do good and bad. And usually, you can't relate it to your particular method in the


paper, right? Because your method is, I don't know, like my faster convergence rate of SGD.


But, and so what you do is you just go one level up, you go up the levels. So it's like,


optimization can be used for good and for bad. I mean, that's still kind of a bit vague. And then


you go up further, well, optimization can do more machine learning. And machine learning can be used


to do good and bad, for example, face recognition and things like this. So you just go up the levels.


And that's what they essentially do here. And that's what, you know, most people have defaulted


to. It's like, okay, so you know, our model here is, you know, we, it basically one can train more


efficient models, and then they simply highlight what more efficient models can do. efficient


models require less compute, if it is modeled by rerun on the end device. If models are more


efficient than large scale research is not limited to wealthier institutions. By the way, I also the


broader impact statement, I believe should be the impact on society, and not really on the research


community itself. So I also, this, this is a bit shaky with respect to I'm really regarding what


the broader impact statement should be. This is not my opinion. I'm, I'm trying to reflect


everything I've read of guidance about what the broader impact statement should be. By the way,


there is also method, method three, which is to simply tell me more about your paper in the


broader impact statement, which I guess is the smart method, because the broader impact statement


can be before the before the references. So it's in the main part, and people are required to read


it, not like the appendix reviewers are not required to read the appendix reviewers are


required to read the broader impact statement. So I guess the smart authors will just try to cloak


more information about their model in terms of a broader impact statement, I guess, well, whether


that's smart is a different discussion. But here, they just, it's already defaulting, right? These,


it's already the default people simply go level up, level up, level up until we can, you know,


say something generic. And we will also highlight and discuss the negative consequences of models,


which can efficiently learn many tasks and efficient models in general, when models are more


efficient, they're also more available, and less subject to regularization as a study of result.


For instance, when a high impact model is released, an institution will hopefully be accompanied by


a model card, analyzing the bias and intended use of the model. By contrast, if anyone is able to


train a powerful model, this may no longer be the case, resulting in a proliferation of model with


harmful biases or intended use. Taking the United States, for instance, bias can be harmful as


models show disproportionately more errors for already marginalized groups furthering existing


deeply rooted structural racism. This, this is like, well, technology, this is basically a


statement about technology. And so why, why do I have a particular not issue, but why do I pick


this broader impact statement, they even read this here, this is this gender shades paper, right,


where people went and they looked at these commercial API's for face recognition, I think


that's the paper. Yeah, gender shades. So if you have a face recognizer, they realize they divided


people up by I think, gender and race. So, you know, like they built four groups, or I haven't,


I haven't, I've just looked at the paper, but in my understanding that they divided people up into


groups, which I find arbitrary to have the these two axes, race and gender, but okay, you can do


that. And they discovered that these commercial API's have different accuracy for the different


groups, right. And that basically our point is that, you know, these commercial API's, if they're


offered for all humans, they should work equally well for all humans. Now, now you maybe see what


it has to do with this paper. Well, this paper is in the business of doing multi task learning. So


it is very viable to actually frame the the task for example, like this is an example. If you frame


the task of multi task learning, like face recognition on different groups of people as a


multi task learning problem, you have, you know, group, group one, right here, group two, group


three, and then if at inference time, so you can build, you know, good models for each of the group


at inference time, you're given an image and you're trying to first which group is that from and then


take the appropriate classifier, that would be, you know, that would be a good hypothetical


classifier for this thing. Now, what do we know about this thing? This thing is fails if the


tasks aren't equally hard. Also, in specifically, if, if for one group, let's say for group three,


the task is way harder because you have less data. I guess the one of the main problems there is that


the data sets are not equally balanced. If you have less data for that, then the task becomes


de facto harder, and the model is less sure about the task, which means that it's a double whammy.


So not only is the model itself less accurate, but these the input data point if the person is


actually of group three is less likely to be classified correctly into the correct model at


to begin with. So, you know, for all the for all, I've had my share of comments on the video I made,


and I still maintain that societal bias can comes about by data set. But for all the people saying


there are models that exaggerate existing biases in models, this would be like if there is any


ever any applicability of these broader impact statement guidelines, this would be the paper,


right? It's this right here is an actual system that if I have different classifiers and I combine


them with this method, it will double punish the classifier that is less sure that is less accurate,


because that is also going to be the one with the higher entropy, therefore not as much selected


if I give a data point of that particular task. And so this is like a I'm not criticizing the


method here, like by all means, like this is a cool method, where you can recognize that this


happens and try to calibrate accordingly. But if there was ever any straight ball for a broader


impact statement, I would, you know, this is it. And this, I'm not I'm not trying, I'm not saying


that these these authors didn't do that for a reason. I believe that, look, it's been whatever,


not even half a year since we've started with these general broader impact statements. And


everybody is already defaulting to simply say technology, good technology, bad. That's that's


the people aren't even thinking. And so this right, this is one of the reasons why I simply


find these broader impact statements to be not that like not a good idea, because there is a


default answer. And people are just putting it here, even in like when there is an error,


when there is an actual obvious, immensely obvious thing that they even they even cited,


like the basis for that. So, you know, that's sort of my take on this. I again, I enjoy this paper,


the code is is available, everything is good about this, this paper, I'm not even the fact that these


are, you know, I think these are kind of two separate ideas, they're combined, cool, they're


analyzed formally, in theory, there's intuition given all good. So don't get me wrong. This is not


like trashing this paper. It's just, I felt I had some thing more to say. And I think that was it.


So yeah, I'll see you next time with a new paper.


Okay, so our goal here is going to be to change this code to not use masks as mixtures, but


actually use neural networks with real weights as as mixtures and in superposition with each other.


Okay, so what we're going to do is we're going to train the different neural networks, and then use


this kind of superposition trick to figure out which task a data point came from. So,


let's have a look at the code right here. And there's a bunch of helper code. And if we go down


through everything, you'll see that this is the MNIST permuted data set. So each each task is


basically a random permutation of MNIST. And if you execute, I believe this here, and then you


train the model, and right now it's for five tasks, but I guess that's going to be enough for now.


Yeah, so if we get some good signal here, I guess it's a matter of of doing kind of engineering and


plumbing and tuning if until you get it up to whatever 200 or 2000 tasks, though, I might be


wrong there. So this is training and I short certainly sort of had a look at the code, but I


haven't actually tried this yet. So the thing the model is built here, you see this this multitask


fully connected, which has these different layers right here. And it's built by these multitask


mask linear models. Now the multitask mask linear models are defined right here. So it's basically


a linear model, as you can see, it's derived from a linear from a linear module. And it has a


parameter called num tasks, and then it has a parameter scores, which I guess is are these


these masks right here. And the scores, I'm going to guess are always going to be multiplied by the


weights here in the forward. So you can see the in forward, you get the weights from the alphas.


Yeah, this is the superimposed. Alright, so if we know the task ID down here, we get this subnet,


and we are going to multiply it with the weights. If we don't know the task ID, we want to get these


alphas. So the alphas are going to be one over the number of tasks at the beginning.


We're then going to multiply each of the alphas with the weights. And with that,


we're going to get this subnet mask right here. So we need to know what this self dot stacked is. So


the self dot stacked is getting right here in this cache mask, or simply stacking this this get


subnet for all of the things. So our plan is going to be that this subnet right here is going to be


the actual weights of the neural network, okay, and not just the weights of the neural network,


okay, and not just the not just the mask, and then we don't need to actually multiply it with


the weight, we can just just forget about the weight, honestly, and just train the subnet. So


for the subnet, as you can see here, you have this get subnet thing. And that's an autograd function,


which basically means in the forward pass, you want to discretize it and in the backward pass,


this is a straight through estimator. So our first task is going and this here should be done now my


laptop has stopped breathing. So we've trained five tasks. And now we can run inference on that.


So this is when the task is given real quick, you can see task one 92% 92% 92% 92%. So we have a an


overall performance of 92.44%. Then when the task is not given, we have two things to evaluate


whether or not basically how good we are overall, and whether or not we get the tasks correct.


Of course, the tasks are at this pre requirement. So we have 100% task inference accuracy. Okay,


so we don't we don't okay, we can we could evaluate this here, but you can already see the output from


last time, there is like no difference from the performance of the when the task is given, it's


always being able to infer the task, we want to check out the same thing. So we want to change


first of all this get subnet, this is where it's the scores are discretized. Now, given that these


given that these scores are going to be n to end up being our actual weights, we won't we don't do


that we simply return the scores. Now this is this is pretty pointless right now, but we'll keep it


just to be as close as possible to the to that. Now, mask in it. This is where we initialize the


mask. Now, this is climbing uniform, and it has some thing but we want probably we want to train


the neural network to be initialized, you know, as we know it. So let's try what what are other


initialize function. So init dot, what do we have here? Do we have what's usually, I don't even know


normal, that sounds about right. That sounds about right. All right. All right. So scores, and yeah,


let's try this. This could this could like break everything right if you initialize wrongly you get


like dumb results. So okay, signed constant, yada, yada, yada. Where is that used? Huh? Okay,


that's also initializing something. So we calculate the gain and then okay, this doesn't seem good.


This doesn't seem good. We'll just keep it. Hey, why not? Why not? Why not just keep it at that?


All right. So cool. Oh, yeah, this is for the weight. Anyway, we won't use the weight at all


of this layer. We'll just use our own weights. So here we have these stacked. Okay, we get the


scores. That's all good. Like I'm pretty happy with that. I'm pretty happy with this mask in it


that we make our parameters. So these are going to be our different neural networks that we train.


This all looks good. The alphas look good. Now the only thing we want to do honestly, is just to


do have not the weight times the subnet here, but the subnet as such like this. Is this it? Do we now


train actual neural networks? I have my doubts, honestly. Like there should be no, this should be


it. Yeah, yeah. Let's just try it. Like we're gonna get a mistake somewhere like a crash. Nope,


nope. Okay. All right. Actually training. So for real, like these scores, right here, the fact what


made them a mask is that we discretize them right here. So we made them into a mask right here. We're


not doing that anymore. So we're just training floats. And then we're also not multiplying it


by the weight. We are just using those floats, which means that we are using the a basically a


neural network. And then here the bias, I was worried about the bias, but the bias is always


zero, as you can see here. So the bias is always false. Yeah, so we're training five different


neural networks for five different tasks. And, you know, according to my hypothesis, these mask


things are just kind of crude, quantized ways of training neural networks. And if, if my hypothesis


is correct, this here is going to turn out probably even better than this masked thing.


Okay, so last task training right here, laptop starting to breathe. Good laptop, fast laptop.


Very nice. Come on, come on, come on.


And we're done. So again, we have an average top one performance of 92. Is this even did I even Oh,


no, I ran this right here. Okay. Like, that's the exact same number it was last time. So we need to


run inference again. And if we're given the task ID, then we are at 93.9%. So we increase slightly.


Which might just be due to the fact that we initialize terribly, terribly. Okay. So what does


it say about our task inference accuracy? Maybe there's some mask here. Set model task, the alphas


are to one. Nope. No, we're good. We're good. Task inference accuracy 100%. And I'm going to guess,


well with the task inference accuracy being 100%. I'm going to guess this here will give us the


exact same number. I like the 93 point some percent. So yeah, 93.9%. So I'm, you know, I'm


going to say right here that the on the super masks and the superposition really are two separate


ideas, right? You it's, it's because the paper is like, it sounds cool and all with the supermask


and superposition. But this inference using the superposition, and then the entropy to decide


is really one idea. And training different supermat, the, the advantage in using supermask is


of course that the model is way smaller. So you can remember it much more easily. But also, you


know, that it's really different. It there's, there's nothing to do with the superposition. Yeah.


All right. So I'm going, I'm going to guess this also works for, you know, 200 tasks and whatnot,


Synthesizer: Rethinking Self-Attention in Transformer Models (Paper Explained)

Hi there, today we're looking at synthesizer rethinking self attention in transformer models


by Yi Tai, Dara Bari, Donald Metzler, Da Cheng Chuan, Che Zhao and Che Cheng.


These people are of Google research and on a high level, they're trying to replace the


self attention mechanism, which is currently a dot product mechanism in a transformer by


a sort of a learned attention mechanism, therefore eliminating this expensive dot product and


they test the model and conclude that it sometimes works a bit.


So the results are sort of inconclusive, but that's the paper on a high level and it's


fairly cool to go through and as always, if you like content like this, consider subscribing


and sharing it out.


Alright, so they say the dot product self attention is known to be central and indispensable


to state of the art transformer models.


So if you don't know what a transformer is, it's best I made a video on the attention


is all you need paper and that explains what a transformer is and what an attention mechanism


is in detail.


But they are right, of course, the attention mechanism that is via the dot product of queries


and keys is pretty much what makes transformers transformers.


And they hear ask, is it really required, which is a bold question in light of that,


right?


They say, they investigate whether or not you really need this.


And they say via expand extensive experiments, we find that first random alignment matrices


surprisingly perform quite competitively, and to learning attention weights from token


token, that means query key interactions, which is this dot product interaction is not


that important after all.


Okay, they proposed this new model called synthesizer, a model that learns synthetic


attention weights without token token interactions.


Our experimental results show that synthesizers competitive against even nilla transformer


models across a range of tasks.


Okay, so let's dive in.


So what is what is different here?


They're basically saying, look in each in each, each transformer layer boils down to


something like this, where you have an input sequence x right here, and you want to get


an output sequence y.


And in order to do that, you need some sort of this thing, which is the attention matrix,


multiplied by this thing, which are called the values.


And we'll explore that a bit deeper over here.


So in these transformers, it's always kind of helpful to visualize yourself, the input


sequence as sort of nodes.


And so this, this would be one layer.


We have a five length sequence, and we want to transform it into the next length five


sequence.


And maybe it even helps to label maybe like a, b, c, d, you can just imagine kind of these.


Of course, as you go up the layers, it doesn't necessarily always correspond to the same


input token.


But the position labeling them is still it's pretty helpful, I find especially for you


know, like things like Bert or something like this.


So you want to transform the sequence that's incoming here into a another sequence.


And the basic, the basic mechanism in a transformer, if you go up the layers is the routing of


information.


So you want to like route information around the sequence.


And basically, such that at the end, the whole sequence knows about every word in the sentence


knows about every other word that there is in the sentence knows about the associations


between the other words and so on such that you gain sort of, you start out with individual


words.


And at the end, what you want is sort of that every word has a pretty good idea of what's


going on with every other word.


Okay.


And that's why you continuously as you go up the layer route around this information.


Now the question is, how do you route this information?


How do you know which word goes to which other word?


And here, maybe the sentence starts with the word, let's call Sarah, okay, Sarah.


And then it goes on and some point it says she, okay.


So she is the pronoun.


And we can also label these Sarah, she.


So if we want to, if we think, how do how do we route information, it would be beneficial


for us if the information that there is a word Sarah here in the sentence would be routed


to the word she because the word she it's a pronoun, it knows I'm a pronoun.


And if there is like a person in the sentence, that would be valuable information for me


like to know what's kind of going on and to understand myself better.


Basically, every word wants to understand itself.


And it kind of calls out for information from the other words.


In a transformer, this is done via what this paper calls this dot product attention.


And that's the follows every word, every token emits what is called a key and a value.


And the key and the value are just two vectors.


So every word is going to emit two vectors, I'm going to draw one at the bottom here.


And I'm going to draw one at the top.


Okay.


Like that.


So you can imagine the key as sort of the word advertising what it is to other words.


And so these are the keys down here.


And you sorry, the top, I think I called it value, that's wrong.


It's called a query.


You can imagine the query as a word asking describing what it wants from others to know.


So in that case, you'll see that the vector here and the vector here, the these are now


routed by dot product.


So the ones that align in the dot product in the angle, they will be routed to each


other.


So this would be routed here.


And maybe, you know, okay, I drew this, this one would be routed here and the others would


be kind of routed some a bit here, and a bit here, maybe.


Okay, it gets it gets fuzzy, but you get the concept.


But in order to do that, you basically need the to pull to put the dot product from every


single key with every value.


Sorry, query.


And that gives you basically this quadratic dot product that these transformers have.


And that's expensive.


Okay, so they have a little picture here.


This is what a vanilla transformer does.


Every input here emits two things, a query and a key.


And then there's the dot product attention to decide what's this attention matrix.


Okay.


Now, this attention matrix is then used to aggregate these values.


So actually, every token emits three things.


Also this value here, which is basically, it's not that important.


But this just describes the information that you want to pass on to the next layer.


And then it goes through this routing right here that routes the information to the correct


output places and you get your output.


Now what they propose is something different.


They propose this dense synthesizer right here, where instead of the dot product attention,


every single input here emits a basically a row of this matrix directly without having


to go through the dot product.


Now it helps a bit if you imagine it in our little framework here.


So let's draw this again.


And let's see what this synthesizer, by the way, they call this the dense synthesizer,


because they have another variant as well.


Okay, here's our here is our sequence, the lower layer, we want to transform it in the


upper layer.


Let's this is the Sarah node.


And this is the she node.


Sarah, she.


Okay.


So how do we route information now?


Wow.


Okay, I missed that.


In the dense synthesizer framework, every token just gets to output basically it already


gets to output where it wants information from.


So every single token here gets to output where it wants information to come from.


And by where, because in the original transformer, the where was basically defined by these inner


product.


Now the where is just defined by the position.


Okay, so it just says, I want information from position two, and three, or this node


here could say I want, I want information from position five, and three, okay.


And this is dependent on which token there is.


So each token looks at itself.


And in the case here of she, you can imagine this token says, Well, I'm a pronoun.


Therefore, I may be referring to a person.


And I know that in the English language, a person is often at the beginning of the sentence.


And therefore, I certainly want information from token from position zero, it doesn't


see that there is this word Sarah here, it simply can see only the positions.


Zero, one, two, three, four, okay, so it will output, it will basically output, each token


here will output an L dimensional vector and L here is the length of the sequence, and


L dimensional vector that already defines the distribution of how you want the information.


So I want lots of that, and then not much of that.


And maybe it wants a bit of that, and then not much of that.


Okay, so each word up here is going to emit this L dimensional vector.


Okay, so each word, each token decides for itself where it wants information to come


from pays based purely on what the token itself is, okay.


And of course, in the higher layers, this information, like the information of what


the token is, and what else is there gets aggregated.


And that's how computation happens.


But in a fundamental level, each node looks at itself and decides where do I want information


from just given what I am and not what others are.


So this results in a in you not having to do this dot product attention.


But of course, you lose the information of what's down here, you simply go on the positions


of the nodes.


And they they formalize this like this.


So they basically say, okay, each each transformer mechanism needs some sort of a a softmax over


this matrix B right here, which this is the this is this routing matrix.


And then g of x is just the values of x.


So G is often just a linear function.


And they say, well, this B here in the classic transformers computed via this dot product


attention, can we just simply have a function right here, that just outputs the B given


an x.


So and it's, you see here, x i refers to one row.


So x here is an L by D matrix.


And they say the sequence length is L. So if you imagine this, if you this is the sequence


length x, every x i is as is a vector here of dimension D, right, sort of like a word


embedding.


Okay.


Now, what you want to do is you want to take each individually, run it through the function


F, and then get out an L dimensional vector.


Sorry, an L dimensional vector, this is of L.


And if you do this with enough of the with all of the x's, you'll get an L by L matrix,


which basically is now your routing matrix, it tells you.


So this this thing tells you that this particular piece in the input sequence, wants how much


information from this particular piece in the output sequence or vice versa.


I see this is the problem here.


They don't really specify this B, how this B matrix will be composed from the in from


the bi, like is bi a column or a row of this B matrix, I don't know.


And therefore, it could actually be the other way around that the information is that it's


not the sort of the tokens deciding where they want information from, but it could be


the tokens deciding where they want to send information to.


But just from the notation, I can sort of guess that it's the way that I described right


here.


But I hope you see the difference here, right?


Before we had this dot product, and here, each of these columns basically is an independent


evaluation of this function that only considers x i, and doesn't consider any of the x j that


it wants information from it simply goes by position.


And they use for this, they use this, basically this two layer, one hidden layer neural network


right here, two weight matrices and a non linearity, a relu non linearity.


So they replace the dot product by simply learning what the attention pattern is going


to be per individual token.


Okay, now they do it one step further.


They say, Okay, so we've already lost the dependency basically on what the input sequence


tokens here are.


Can we also just kind of lose the dependency of what the output token sequences here are?


So what they propose in their second variant, this random synthesizer is the following is,


hey, why don't we just learn how the information is going to be routed irrespective of which


tokens come in, right?


We just we're just going to learn this.


And it's going to be one routing pattern for all of the possible input output sequences.


That's it's just going to be this routing pattern.


So they have actually two variants.


First one where it is really just random, like they just leave it random, and they don't


even train it and the second one where they train this thing, but these things now have


nothing to do with the tokens, they're just fixed.


And they are just global.


So this, this would here be this would directly be you learn this L by L matrix.


And yeah, so if this strikes you a bit in an odd way, because you kind of lose the dependency


of your data in this routing pattern, and it's not really routing anymore, right?


And if you think that you've seen this before somewhere, and you think, hey, that looks


like a feed forward layer from from like your very first MLP, then you would be absolutely


correct.


And I'm not sure why they don't point this out.


I'm like, I have a really hard time believing that they themselves trick them into such


a thing right here.


And so I can actually show it.


Right.


So the question is, how is how is this here, this dense synthesizer?


How is this still is this still different?


Or the same as a feed forward layer?


And is this different or the same as a feed forward layer?


So if you do the math, say, and you look at what a feed forward layer is, in a feed forward


layer y i, right, my i of entry in the output is going to be a sum over all the inputs x


j, multiplied by a weight i j.


Okay, so if whenever I can represent something in this fashion where I have a sum, and then


I have like a fixed weight, and that I learn, and that has nothing to do with x, that is


not dependent on x, multiplied by x, then it is a feed forward layer, basically, or


like a fancy feed forward layer.


So let's look at the dense synthesizer.


What does the dense synthesizer do?


The dense synthesizer says y i is equal to a sum.


Okay, we're starting off, we're starting off well.


And so it says g of x j of right, but this g usually is just a weight matrix where we


compute the values, right?


We said this was the values of x.


So g is usually just a matrix, let's call it V w v.


And then here, we have like some softmax thing, right?


We have some softmax and the softmax is going to be over this dense pattern right here,


that we described, sorry, here.


And this pattern is going to be f of x j, right?


So no x i, f of x i.


Is that correct?


x i and here is a j.


Maybe.


Yes, that's correct.


And f, okay, it's like two, it's two layers, but we can basically say it's like a weight


matrix, because ultimately, if we learn a neural network or a single layer, it doesn't


really matter to the discussion here.


So let's call this w b x i.


So you see right here, we do have a weighted sum over the x j's.


But the weight that the weighted sum is using is dependent on x i right here.


And therefore, you can't, you can't represent this as just a feed forward layer right here.


Of course, if you have the full dot product attention, then in here would actually be


a dot product between x j and x i right?


So x j transposed x i or something like this.


So what about what about this random synthesizer?


So the random synthesizer has y i has a weighted sum over this w v x i, that's the values softmax


over this matrix R, and R is simply this L by L matrix right here.


Now you can immediately see that this part right here is static, it doesn't depend on


any x and it is learned as in as a joint function, right?


So this if I just call this w, w i j, then I'm back to my formulation, right?


I'm sorry, i j, then I basically have my feed forward layer.


So the random synthesizer is just a fancy way of writing a feed forward layer.


Now of course, if you're going to have the softmax, you maybe have some different inductive


biases in learning it, but ultimately, it is a is a straightforward feed forward layer.


At least that's what it looks like to me.


I'm very open to be convinced otherwise.


Okay, so they have this drawing right here on the left, you see the vanilla transformer,


the dense synthesizer in the middle where you kind of learn how to produce this matrix


and then route the value through it, and on the right where you simply output this in


a learned or actually completely random fashion, and then route your values through that to


the output.


Okay, now the question, of course, is, okay, they also do factorize it.


But this is not really the this is more of a point where now you can actually if you


have such a matrix or you produce such a matrix, you can then factorize it into sort of lower


dimensional matrices.


And that is first of all, to save space.


And it is also a regularizer because what you're essentially saying is you're applying


an inductive prior to say, I think these matrices have like some low level structure to them.


And if you factorize them, that would that's a prior on that exactly.


So you can factorize the dense and the random synthesizer into smaller matrices.


And that will save you parameters.


And you can actually also mix too.


So you can, for example, mix the random and the dense synthesizer.


Now you have to pay attention, it's not like an interpolation.


If you mix random and dense, you will have to learn the parameters of the random end


of the dense synthesizer.


So that's going to be like strictly more powerful than either one alone.


They list everything here, where they say the standard dot product attention, what we


have is we have this formula right here, you can actually formulate it in their framework,


you condition on all the xj for any x i and and ah, see here, I wrote this as xj, I was


dumb should be the entire x.


And there is interaction between the tokens.


And it's going to cost you two D squared parameters.


Now parameters is different from computation, which if you don't, if you don't do the dot


product, you also save a bunch of computation.


But here they look at a number of parameters.


So in this random synthesizer, you simply output this matrix R, it's global, it there's


no interaction and you are you are it's cost you l squared memory in l squared parameters.


Now often in these models, L and D are actually pretty similar.


So L might be something like 512 tokens, the length and the dimension right here might


also be something like 512.


So per se, this is not really a saving in parameters.


Only when you go to the to the factorized models right here, can you bring in this k


and k is this lower dimensional factorization.


And if k is much, much smaller than L, then you save a bunch of parameters.


The dense synthesizer is formula is like this.


This is how you produce the attention matrix, you condition on x i but not x j, right?


For each y i you condition on x i, and you do not care about the x j's.


That is a it is local.


That means it depends on x i.


So the the routing actually depends on the information that goes through but there is


no interaction.


And you're going into D squared plus DL, which is also pretty much these 2d squared, right?


And or you go to this lower, lower number here if you choose a good k.


All right, now experiments.


So they apply this and we are absolutely stoked how this is going to turn out.


So they go on machine translation now.


Okay, for before we go into the results.


Do you think machine translation is a good or a bad task for this model?


Okay, I think it is a good task for this model is a very favorable task for this model.


Why Why is machine translation a favorite task?


Well, mostly in machine translation, if you think about how information is routed.


So I have I have a I have a sequence of German, let's call it or English, let's call it the


dog barks.


And I have a sequence of German there.


Come on, and belt now Okay, first of all, I know they're only talking about self attention.


So this example here actually makes little sense in the actual practical applications.


But I just want to demonstrate why machine translation specifically has.


So how would you route information here, if you have to route information between the


two things?


What you would do is pretty deterministically do this, right?


So in machine translation, what is very, very, very often the case is that mostly, you're


going to align the positions in the same way independent of the input.


Specifically here, you would always most of the time align the beginning with the beginning


the end with the end and so on.


Because for most languages, especially similar languages like English and German, the order


of sentences and number of words per thing you need to express is going to be roughly


the same.


So if you did not know about even about what the sequences were, or you only knew one of


them, like, you only knew there, and you have to guess where should information come from?


Well, I know in English, you also start with like this.


What's this called?


An article?


Yeah.


Yes, this is an article showing my linguistic skills here.


You would also you would also start with that, right?


You would say I want most information from position zero, obviously, I don't care what


they're what what is there.


So I and again, I know they'd only do it in self attention.


So it actually makes no sense that I have two different languages here.


But machine translation is probably a task that lends itself very much to sort of global


globally learned, or only partial the partial observably learned attention patterns because


just because of the nature of the task, right.


So let's keep that in mind.


And go to the go to the results right here.


Now, they first of all, what they do is they list the original transformer paper, and actually


have it here, because they have to they have this same experiment.


Now, this is the kind of transformer we're talking about right here.


And it is notable that this paper only proposes to replace the self attention, that means


the attention that would be within one of these two columns and not the attention that


goes across from the left to the right, right.


But still, you can see that the attention that goes from the left to the right, then


the next layer is going to end up as self attention information, right.


So my, I think my argument still counts in the in this case, in the machine translation


case.


All right.


So they have this same experiment right here, here.


Yes, they have English German translation.


And they their base model gets 27.3 and that's what they evaluate right here, they list this


But they also say, when we train it, we get a bit of a higher number 27.67.


And especially on English French, they get a higher number than that.


But let's stick to English German for now.


Now they also do language modeling, which the original paper didn't do, and record the


perplexity right here.


Okay, so the first thing they point out is, if we train the synthesizer with a fixed random


matrix, that means we just put a random routing, and we do not, we do not ever change it.


What's there to learn?


So if you want to learn something in the transformer, there's still many things to learn.


There's the feed forward layers, right?


There is the the value encoder, and so on.


So it is, it is reasonable to assume that the transformer could sort of learn to just


handle the attempt the routing pattern that is in place, and that the rest of the model


can sort of absorb that shock.


And interestingly, you get on to 23.9.


So almost 24 blue points.


And I mean, it seems they point out that it's fairly close.


If you look here, this 24 is actually pretty far away.


It's the worst, the worst baseline right here in the original paper, this bitenet had this


24 blue.


I mean, it's, I guess it's cool to point out that it works as such.


But you know, in these tasks, actually many things work right with with, if you distill


this down to some sort of a bag of words model, and so on, I'm pretty sure you can get pretty,


pretty good results as well.


And you can get, you know, fairly, you can go to 24 blue.


Actually, I have no clue of this field.


But I just want to point out just because the number is in the same ballpark doesn't


mean that it is very astonishing.


It's maybe just you have so many parameters that the rest of the model can sort of absorb


this the shock of not of not being able to learn this and can just handle whatever pattern


you put there, it can just kind of work with it.


Okay, that's many people have observed if you like, put just random junk in the lower


layers of a CNN, like random filters and never train them, you can still the rest of the


network can adapt.


So that's basically this effect right here.


I don't think it's a testament to we don't need the dot product attention.


It's more like this, this just happens in deep learning.


Then however, they say, if we now learn this one matrix, so we learn this routing, but


globally, we get into 27.27 blue.


And this already seems fairly close, right?


And you mainly need to compare with this number right here, because that's actually the same


training run and so on.


So but still, it is quite it is quite a bit away.


It's point four blue points away.


And that is a sort of significant difference, I think.


Then they go further, if they go to the dense synthesizer, you can see right here, the model


size is lower than this one.


And they get 27.43.


Now they get even closer, right?


And they are actually on par, if they mix random and dense right here.


And you can see that it's also almost the same amount of parameters.


And when they mix these random and vanilla, so what they now have is the dot product attention


plus a purely global feed forward, sort of like a bias of what to route where, then they


can outcompete this original model.


But also now they have more parameters, right?


So this model, you would expect it to be, you know, strictly better than either of the


two alone.


And it is, and it is actually astounding that the synthesizer that mixes the vanilla with


the dense, even though it has even more parameters, it does worse.


So with these sort of results, especially then you go fiddle with like point one, you


know, between this and that.


I know I said point four blow is a lot.


So point one must be something and it surely is.


But also, it's always the question of how many hyper parameter tunings you put into


something like this.


And generally, you should always sort of look at this if you were a researcher and had to


put the best possible numbers here, what would you do?


And then you correct for that in your mind for how much it might actually work if you


are to, if you are to, you know, go ahead and train that on your data.


But nevertheless, it gives some cool insights, right?


What I'm a bit confused by is that if you sort of look at the original paper, and you


look at the perplexities, and they have a table down here where they compare a bunch


of their instantiations of their model, and you compare, they compare also the perplexity


on a language modeling task.


And the perplexity here seems to correlate extremely well with the blue score, right?


Where as the perplexity here, if you look at the perplexities over here, they do correlate,


but I somehow, I have the feeling they don't really correlate as much here, which sort


of speaks to the fact that you're going to see in the rest of the paper that these models,


they tend to sometimes be able to do well, but then other times not.


And it's not really clear, super clear when.


So look at this, for example.


They now apply their models to summarization and dialogue generation, right?


These are two tasks where you need to output text, and you can see that the results are


all over the place.


So in this metric, Rouge 2, and Rouge is sort of an n-gram overlap metric between gold standards


and what you produce.


In this metric, the original transformer is best, but in Rouge 1, this synthesizer mix


here is the best, and in Rouge L, this one is the best, and in dialogue generation, all


of them are actually not as good as this one right here, where it's just the dense, which


is strictly less powerful than the ones on the bottom.


So as you can see, yeah, I think what you should take away from this is that it is interesting


that it sometimes works, but it seems to be a fair bit of shakiness to these results.


Okay, now they go on and they test this on Super Glue, and this is a benchmark.


So Glue and Super Glue, they consist of these different tasks right here.


And now we are out of the text generation game.


We are in the game of, for example, you have two sentences and you need to decide which


one is like, which one entails the other, or are they contradictory, or things like


this.


So it's more of a, let's say, a classification task, and people apply different models.


So it's no longer a text generation task, so they switch model.


Instead of the vanilla transformer from the Attention is All You Need paper, they now


go on to the T5, the text-to-text transformer.


And they change, they simply take the architecture and they change the attention in there with


their attention.


And you can see right here that the results are quite different than before.


So in every single case, either the T5 model, the base model with the dot product attention


is the best model, or the synthesizer but including V. So plus V means plus vanilla


means it also has the dot product attention plus this learned thing right here.


Okay, so R is now the learned, I think the learned, right?


I would be surprised if it was the random random, but it could also be.


But in any case, it's strictly better, right?


It's strictly more powerful model.


And the only way it can actually perform worse is when, you know, it's too many parameters


and so on, and they kind of take stuff from each other.


And there is effects where more parameters can hurt you.


But never is any model that doesn't have the dot product attention on top.


And these authors here argue that this can be largely attributed to the fact that the


encoder self attention in the T5 setting also functions as a cross sentence attention.


So what do they mean here?


If in the T5 is just as I understand it, this is just like an encoder like BERT.


So imagine, imagine maybe this is BERT, right?


What BERT is simply an encoder only transformer.


That means you here you put in your sequence, and out again comes a sequence and you have


like a special token that you use for classification and so on.


So this is less when you have to generate text, but more when you want to classify text


or things like this, find something in a text.


And what you would do if you have two sentences, you need to decide something about them.


You put the first sentence here.


And then you say you put like a separator token here, this usually called like a separator


token.


And then you put the second sentence here.


You just concatenate them.


And you let them go into the transformer.


And they argue that if you do self attention on this entire sequence, then you get attention


patterns like this.


And this is sort of like cross attention between sequences, right?


It's not really self attention.


And that's why their method doesn't work because it basically deals with self attention.


But I'm not really buying that argument.


I mean, if this is a sequence, it is if this is one sequence, this is self attention.


And if you are going to argue that out of the blue a a token in your case, like in your


original formulation can simply, you know, just by looking at itself know where where


which position it wants the information from.


And certainly here, this token could also learn that it wants information from over


here or from the first word here.


I don't I don't really see the difference.


Maybe maybe you need to somehow standardize where this separator token is, so that it's


always in the same place and that the second sentence always starts at the same place.


But if you have that, then I really don't see any difference in the argument you can


make here, that this shouldn't work as much as the others.


What I think is happening is that this task is simply involves more difficult reasoning,


involves more routing of information, like dynamic routing that's actually dependent


on what's in the tasks rather than something like machine translation, which most of the


time has some global routing bias, like some some pattern that works pretty well across


all.


All right.


So the last part here is where they kind of introspect the model.


And in the in the first thing they say, OK, we look at the distribution of weights.


So these are the weights, the weights in the decoder at the beginning of training.


And you can already see that the standard transformer weights are and the synthesizer


weights are different from the sorry, the dense synthesizer weights are different from


the random synthesizer weights.


And this probably is mostly due to the fact of how you initialize like these deep learning


frameworks.


If you have a matrix, they will look at what's this dimension, what's this dimension and


look calculate how they have to initialize it such that sort of the the total norm of


a random vector that goes through stays the same.


Sorry, the vector would go through like it would go in here and out there.


So you see, if it changes dimension, then if you just randomly initialize with all the


same like every matrix with the same number, then like with the normal distribution, then


the in this case, the vector would gain in norm.


And to account for that, you initialize the matrices such that the vector norms approximately


stay the same.


And this is why there I guess why there are different initializations here.


And you can see this at the end of the training.


Now these in different layers right here, it's pretty much always the same pattern that


they say they just remark it.


So this is what I find weird.


They just say what the graphs show.


They don't interpret it like I would expect something like, oh, this pattern is exactly


what we would expect from our model because something, something, something right?


Like if they claim that this attention is being able to be learned.


I just don't see why they do this stuff.


They simply point out, oh yeah, this, this is higher here and this is higher here.


But I don't even see that as too interesting given that is this is how you initialize it.


Like if I shift everything to the left of it and you know, this is a wall.


So like this piles up here, then this is exactly what turns out.


I don't, I don't see, you know, what, what this is supposed to mean, especially since


they don't make any claim of what it is supposed to mean.


And the same here, they say the effect of the number of heads.


Okay.


We, we investigate the effect and the number of heads on the random synthesizer models,


you know, and they train the number of heads.


Now somewhere in the text they say, I remember they say, since you know, since we don't dynamically


route it is very important for our models, very crucial to have many attention heads,


right?


Such that basically you don't have one routing pattern.


You have many routing patterns that you learn globally.


So they say it's, it's very important for our model to have many attention heads.


And I guess that's what they're trying to demonstrate here.


But again, they simply say what's happening.


They don't interpret it and, and they don't compare it to anything.


They just, you know, put it here.


They just put the number and I don't like, is this good?


Is this, is this bad?


Can you compare it to something?


And also here in the, so here in the original paper, they do the same thing here.


As you can see, the number H is the heads and they do ablate this, but at the same time


they adjust the dimensions of the key and value vectors such that in total they have


the same amount of parameters, right?


So they can really investigate is one big attention head better or worse than many small


attention heads?


Is there a trade off?


And they find here that there is a bit of a trade off.


Like there is a sweet spot.


You don't want too many, you don't want too much because they get too small.


Something like this.


But first we like, we don't know whether or not have they simply changed the heads but


left every other parameter the same or have they also adjusted the dimensions?


Because if they haven't adjusted the dimensions, then this, this increase would be absolutely


expected because you now have more parameters.


And if they have adjusted, then can we compare this to, you know, something?


Because this here is the, this is the T5 small.


This is not the original transformer, like is this big, is this small?


And what does it say about the claim that you made that the number of heads is so important


for your model?


Can you validate this using this?


So it's just a bit of like this entire page here.


It's just, they just measure some things and then they state them here.


And you're somehow supposed to guess what they mean by stating that here.


Okay, but that was enough for me ranting.


So they give some supplementary material right here.


But in essence, what I like about the paper is sort of the thinking that goes into this,


thinking outside the box, asking the fundamental questions about these models.


Do we really need this?


What, what do they do?


I don't think it's super well investigated really from a scientific point, like the formulation


of hypotheses.


It simply trains these things and then make some claims, but the claims interact, you


know, with the number of parameters here and so on.


So and they're sort of noisy all around.


And of course, the fact that this thing here turns out to be a fully connected layer in


disguise is also pretty funny.


But I get it.


It's a fan.


It's like, it's, it's, it's not exactly the same thing, but it, you know, yeah.


All right.


So that was my take on this paper.


If you have a different one, let me know in the comments for sure.


I read all of them.


And at least I try and I've always succeeded so far.


I'll see you next time.


Creating a Chatbot with Deep Learning, Python, and TensorFlow p.1

What is going on everybody welcome to a new tutorial series where I show you how you can make your very own machine friend that you can talk to.


For example I have my friend here, let's see what they have to say.


So first we'll start off with some pretty pressing questions like do you like pancakes?


We'd all like to know.


I like turtles.


Okay, what about what is the color of the sky?


What is the color of the sky?


The color of the blue.


Okay, okay.


Did you vote for Hillary or Donald?


I voted for the president.


Let's ask some more pressing questions that we really need to know the answers for.


What is the best song ever?


Derrick, Sandstorm.


No arguments there.


What is the universe?


The universe is a lie.


At least I have my friend so you know are you my pal, buddy?


Are you my pal, buddy?


He is.


I'm not your pal, friend.


That really hurt my feelings.


Why would you hurt me?


I'm not your buddy.


I'm just a little bit of a dick.


So the initial purpose for me to make a chatbot was I was hoping to have some sort of chatbot


that could converse with viewers on the Twitch stream of the self-driving car AI Plays GTA


stream.


And in that basically it's a chatbot that you could pretty much use everywhere.


You wouldn't have to necessarily use the exact same source that I used.


For training data and all that, but the methodology pretty much stays the same.


And really just like any task with machine learning, the biggest thing is just getting


the data and manipulating the data to be useful.


So anyways, with that, as I started to look into, okay, how do we actually go about making


a chatbot in Python and TensorFlow, I found a bunch of examples, a bunch of like weak


ways to make chatbots, nothing that was going to be super robust.


Also, I had a little bit of trouble trying to find like a decent data set to actually


do it.


So one of the more common data sets that people tend to or seem to be using is like this,


this Cornell movie database, which contains 220,000 conversational exchanges between 10,300


pairs of movie characters.


So this one's actually like a super common one to use.


This one's actually like a super common one to be used for a variety of reasons.


You've got the different movies, you've also got different, as it says here, different


genders, different characters, stuff like that.


So we can kind of begin to pull these apart.


But I didn't really like the fact that we only have 220,000 exchanges.


I wanted to be able to find something that had even more than that.


That way, if you really did want to pare it down to really specific situations, you'd


like to have much more data.


So, anyways, I continued my search and I found myself on thinking like, well, what about


Reddit?


Because Reddit has basically, you know, just tons of comments on tons of topics.


It's all sorted for you and all of that.


So I thought, hey, I want to use Reddit.


So my first thought was I'll use the Python Reddit API.


I'll go through that and I'll just start parsing from Reddit over time.


Well, the Reddit API has a pretty strict API limitations.


I forget what they are now.


I mentioned it in this tutorial, but it's nothing that's going to let you be able to


even parse like millions of rows without a long period of time or violating the terms


of service.


So moving along, I ended up stumbling across this awesome data set post from two years


ago.


But basically, you can read the title.


It's I have every publicly available Reddit comment.


And basically, it's everyone.


I don't really know the start date of Reddit, to be honest, but I believe this goes all


the way back to 2007.


I can't remember.


But anyways, it goes quite a ways back.


We'll see in a little bit anyway.


But it's 1.7 billion comments, which is a lot.


That's a lot more than 220,000.


So basically, if you're not familiar with Reddit, welcome to the internet.


Basically, Reddit is a place of talks.


It's a place where people get to talk with each other.


Anyway, here we have a few options.


And for this tutorial, if you want to follow along, you have also a few options.


First of all, this is for just one month of comments.


That will be more than enough to create a chat bot that is pretty good, that will be


like the one that you saw in the beginning of this video.


So you can download the torrent for that and use your favorite torrent.


It doesn't really matter.


And then, or torrent program, rather.


And then there's also, this is the full torrent.


Now, I've downloaded this on multiple occasions, mostly because I tried to move this to a machine


in the cloud and I didn't want to upload it, so I just re-downloaded it.


And the first time I downloaded it, it actually went really quick, the full archive.


The second time I tried to download it, it was horribly slow.


It took days, which was unfortunate because I was paying for the machine.


So try it out, and if it's too slow, maybe try the one month because you really can get


by with just the one month, but it's cool to have all the data again for if you want


to go through it and be really specific.


So for example, you could train an algorithm to work purely on specific subreddits.


It was kind of funny when I found this, somebody somewhere was suggesting that someone even,


like they were excited that someone might do that in the future.


Anyway, it's somewhere in there, but interesting.


The other option that you have is this one right here.


It links to a comment about 1.7 loaded onto BigQuery, but it's actually even more than that.


So let me pull it up real quick.


Great.


So anyway, there's this table, but as you can see, it actually goes all the way to current.


So it's currently November, so it doesn't have the full table.


But anyways, these are all the comments, right?


So on BigQuery, unfortunately, I'm just an idiot and I apparently can't write good BigQuery queries.


And I found it to be very cost prohibitive to work off BigQuery.


But if one of you kind viewers happens to figure out how to write a really efficient BigQuery poll


so we can actually pull, I would love to be able to pull very recent tables.


The 1.7 billion comment download or the torrent only goes up to May of 2015.


So it has no idea, like for Kofife is not in his vocabulary, right?


So it would be cool to have a more current chatbot.


So yeah, if anybody knows how to do it, like I said, I couldn't get polls to be big enough to be like cost effective.


I think I pulled maybe, I don't know, less than 100,000 rows if I recall right and it ended up costing me $30.


I don't know. No clue how that happened, but it did.


So I stopped. Yeah, so you can get that data set there.


And like I said, you can either use BigQuery, you can download the torrent,


and really you can get by with just a month's worth of comments.


And really the further you come down, like there's going to be way more comments in the month of October of 2017


than the last one, which was May 2015. I think May of 2015 had like 60 million comments.


And then once you have them, if you actually go to Reddit,


just understand basically the structure of Reddit if you don't understand it.


So we can just go to like the homepage here.


And as you can see, you've got like parent comments and then every child is like a response to that parent.


And then you might have multiple responses and it kind of just comes out in this tree form.


And so what we need to do is actually pull these apart and start to pair them together in parent-child or comment and reply manner,


which can take a little bit of processing and all that, but it's not something that we can't do.


So anyways, that's going to be our goal, and this is going to be the data set that we want to use.


So I'm actually going to cut it here just simply because downloading the data set is probably going to take a while.


But once you have one of the examples, if you do use BigQuery, by the way,


your format is going to be probably different than what we're using here.


But I really do want to see a BigQuery poll. So if someone does it, please do share.


Anyways, once you have the data and you're ready to go, you'll be good for the next tutorial.


EfficientZero: Mastering Atari Games with Limited Data (Machine Learning Research Paper Explained)

Hi there, today we're going to look at mastering Atari games with limited data by Wajiru Ye,


Shaohua Liu, Tanahard Kuretaj, Pietro Biel and Yang Gao. This paper presents the efficient


zero model, which is a model that can do reinforcement learning with severely limited


data. So the paper tackles the Atari 100k benchmark, which means to learn Atari the Atari


benchmark as a reinforcement learning task, as for example, Deep Q networks did, but you only get


100k transitions. This is about it's about two days worth of real time data to work with. And


after that, the model supposedly be able to play Atari. So this is a variant on mu zero mu zero,


which is an insanely data intensive reinforcement learning algorithm. And it introduces various


tricks and various amendments to mu zero to make it more sample efficient. So when we look at this


paper, you can see the gist of it right here. If you do this Atari 100k benchmark, you can see that


a lot of the other reinforcement learning algorithm, they fail to even reach human level


performance, whereas this new algorithm out competes not only the other RL algorithms on in


this low data regime, but also the humans. Here they say, efficient zeros performance is close to


DQN performance at 200 million frames while we consume 500 times less data. Efficient zeros,


low sample complexity, and high performance can bring RL closer to real world applicability.


They even say we implement our algorithm in an easy to understand manner. And it is available


at this GitHub address. So this code is out there, especially if you want to do reinforcement


learning, but you don't have as much compute or time or money. This might be for you. So we'll go


through the paper, we'll see what the improvements are. There's not a single improvement. There are


many improvements, three big ones to be exact. And yeah, if you like content like this, don't hesitate


to subscribe and tell your friends and family and professors, I guess. Alright, so we'll first take


a small look at what mu zero does just as a recap, I have done a video on mu zero. But if you haven't


seen that, then here is a short, a very short introduction to mu zero to the algorithm. So in


a classic reinforcement learning setting, you have your your basic setup of you have the environment


and you have the actor and the environment gives the actor some sort of an observation at time step,


let's call it t. The actor uses that observation to come up with some sort of an action at time step


at time step t, and then the environment gives the actor back a reward for that time step,


and the next observation t plus one. And that goes on and on and on. So the question is,


how is the actor supposed to come up with this action right here, given the past observations


that it has seen from the environment in order to maximize all of the reward that it gets. Now,


in a regular reinforcement learning algorithm, or regular, let's say in the simpler reinforcement


learning algorithm, what people are doing is they're doing model free reinforcement learning,


which essentially means that they take the series of observation, observation, one observation, two,


and so on that they've seen so far, they take that they stick it in a big neural network,


and they train it to output some sort of an action. And they train the neural network in


order to maximize this reward right here, usually using some sort of policy gradient or something


like this. So this is a rather, rather direct way we call that model free reinforcement learning,


because you directly predict the action without without an explicit model of the world. Now,


when you have a model of the world, so when this environment here is well described,


for example, a chessboard, in a chessboard, you know the rules, you know, everything that's going


to happen in a chessboard, you can use a model of the chessboard. So what you can do is this,


you can take these observations, and these observations would correspond to some


defined state or let's let's say tic tac toe tic tac toe is a better example. So, you know,


with the observation, I can actually construct the board of tic tac toe that I'm in. And then


what I can do is I can actually search, I can try out, I can say, okay, what if I put, you know,


something here, or then my opponent's certainly going to do that right here. And then what if I


put something here, and then my opponent's going to do that, and then they win, right. So that is


one, that is one way to do it. And usually you visualize this as a tree. So you are here at a


root node, that's your state. And you have several options to do things. And in the several options,


your opponent has several options, or if it's a one player game, you have several options again,


and so on. So what you want to do is you want to search this tree for the best possible path. And


this is what things like alpha go alpha zero and so on did. They have these explicit model, and they


search through it. And now the neural networks no longer predict actions directly, the neural


network help you search through that tree, which means they they vote essentially on which part


paths of the tree to explore, because the tree quickly becomes too large to explore as a whole,


right? You can't, like if it's more than three moves ahead, the possibilities just get giant,


even like, especially in a game like go. So the neural networks are here to guide the tree search.


And that was, in general, the techniques of that a center around the Monte Carlo tree search,


because at some point, you abort the search, and you simply play one game to the end, as sort of


an approximation of what happens, and so on. I'm not going to go into that super duper right here.


But what mu zero does is mu zero says, well, this this whole tree search stuff essentially only


works if I have an explicit model of the world, such as the tic tac toe board is clearly defined


how it works, right? Also, I can I can, I can have a simulator for it, I can rewind, I can try again,


this doesn't happen when you're interacting with any sort of real world thing, let's say,


or even the Atari benchmark. So in Atari, I know there is there's hacks where you can save the ROM


and so on. But essentially, you're not supposed to go back in time or go forward in time, you're


not supposed to be able to try something out and then say, Well, now that didn't work, I'm going to


search for a different path in the tree instead. So what people do is they, they try to learn


a model of the environment. So in absence of the model of the environment, they try to learn one.


And there are many, many different ways of doing this. And what mu zero does is it learns a latent


model of the environment. So how does that look? So here you have the current observation,


observation T, what mu zero does is it uses a neural network, I think they call this h or


something to get this into a hidden state. So they map the current observation into a hidden state.


And then they plan using the hidden state. So they plan, they say, Okay, I'm not going to


predict what the next observation is going to be like in the tic tac toe board, I'm only going to


predict what is the next hidden state going to be t plus one, t plus one, like this is one, this is


two, this is three. So you know, depending on which action I do, which which is going what is


going to be the next hidden state of the environment? Sorry, of Yeah, of the environment,


what's going to be the next hidden state? And from that hidden state, I always going to predict


what's going what's going to be the reward for transitioning there? What's going to be my own


policy, which is a bit weird that you have to do this, but you have to, and which is going which


what's going to be sort of the value and the value is, what is going to be my future reward when I go


from here. So these are the sort of things that mu zero predicts. And with that, it is able to search


this latent tree. Note the addition to mu zero, sorry. Yeah, the addition, sorry, to alpha zero,


which is this run right here. So we might we might label this, this is something like


reinforce. This is alpha zero, and this is mu zero. So the difference to alpha zero being that


we no longer have an explicit model. So in order to do tree search, we have to learn a model. And


the model that mu zero learns is in the latent space purely, right? There is it doesn't predict


future observations. And it only learns all of this from the signal that it predicts the reward,


it predicts its own policy, and it predicts the future value. And those are the only learning


signals for the world model. That is good, because it focuses the algorithm on what's essential,


it is essential to get the maximum reward possible. And therefore, the learning, the more


the learning signals center around those concepts, the better. But that also means learning the


entire world model just from signals like the reward is extremely sparse. So it uses a lot of


data. And that is, that's essentially the catch right here. So we're not going to go into, you


know, how exactly mu zero does Monte Carlo tree search, they have a way of balancing exploration


and exploitation right here by essentially using an upper confidence bound formula that you can see


right here. But so, efficient zero goes and says there are three main weaknesses with mu zero.


First of all, they say, lack of supervision on the environment model. That's what I just


said. All the the model, the latent model of the environment is learned purely from the signals of


the end from the reward signal, the value signal, these are single numbers. And to ask the model to


learn a transition function for the environment model is a big ask, and of course needs a lot of


data just from that. The second one is hardness to deal with aleatoric uncertainty. I like I'm


I've given up on trying to remember which one is aleatoric and which one is what's the other one


epistemic? I have no idea. Okay, let's just read the paragraph. The predicted rewards have large


prediction errors. So if there is uncertainty in the environment, for example, the environment is


hard to model, the reward prediction errors will accumulate when expanding the Monte Carlo tree


search tree to a large depth, resulting in suboptimal performance in exploration and


evaluation. So what they mean is that if I predict if I'm if this reward right here,


has a bit of an error, and then I go on searching, right, these branches right here,


and then the reward I predict right here also has a bit of an error, and so on. And we go down the


tree. And every reward has a bit of an error, what I'll do in order to add, you know, at the end,


at the end, right here, I have a path, and I don't go to the end, I stop after a while, and I


add up the rewards that led me here. And that's sort of, you know, how valuable this notice plus


the value that I predict right here, that's going to be the, the value of this path is going to be


the sum of the rewards until I'm here, plus the value from here on out. And if all of these little


rewards have little errors on them, that quickly adds up to a big error. So that's their second


criticism right here. That's something we're going to have to solve. And thirdly, off policy issues


with multi step value. And that is a general, that is a general thing in these reinforcement


learning algorithms, the more distributed you make them, the more sort of what people usually do is


they have like a learner box in the middle learn. So there's a neural network there. But then they


have a lot of actors, actor machines, so they distribute training, and interacting with the


environment. And these send back data, there's usually a replay buffer right here somewhere.


And that means just that the neural network that is here at the learner is not the same that


generated the data, because the data is kind of old. And until you use the data to practice,


the neural network will have already learned from other data. And therefore you get an off policy


issue, even though it's an on policy algorithm. Now mu zero does a little bit to correct this.


But they say this has to be done more. So how are they? Now we tackle these these three things. So


the first thing they tackle is this lack of supervision on the environment model. So what


they do is they add a self supervised consistency loss, you remember that we map the observation at


time t to a state a hidden state at time t. And then we use our latent model to predict for a


given action, what's the state going to be at time t plus one. And that's an estimate, right? Now,


what this paper says is that wait a minute, if we simply look at what happens in the real world,


right, observation t plus one, and we send it through the same. So through this, through this


through this same encoding function, then that gives us the hidden state at time t plus one.


So technically, these two things here should be equal. So the hidden state at time t plus one,


and the estimated hidden state at time t plus one, they should be kind of the same. So what they do


is they use a self supervised consistency loss that they they not from sim cm. So sim cm is


a contrast of learning framework, or self supervised learning framework. And it's usually


used to have two images which have been differently augmented. So to make their representation equal,


so so the model learns to sort of ignore the data augmentation. That's how you train self supervised


image models. But here, we don't augment differently, what we do is we take an observation,


and we take the observation at time t plus one. And the first observation, we actually map it


through that function that is supposed to give us this estimation of the next state. And then


we use a similarity loss in order to pull those two things together. So this function that gives


us the next state, and the representation functions, they're now going to be trained


in order to make those two things the next hidden state, and the estimation of the next hidden state


similar to each other. In fact, the the left branch right here is the one that's trained,


but that includes the representation function and the next state function.


So you might you might ask, you know, this is kind of the first question that everyone in mu


zero has is like, why is this not done? Because this is, if you look at the loss of mu zero,


you can pretty easily see that that is possible. And I think the mu zero authors have deliberately


not introduced a loss like this, because they say, no, if we learn from just the reward signals,


that is going to be a better algorithm, even though it might use more data. But at the end,


it really trains for what is important for what is the end goal. And that's why they didn't


introduce a loss like this. introducing a loss like this clearly trades off the what's the actual


target is namely optimizing the reward, right? We actually don't care if anything's consistent,


we simply want a higher reward. So it trades that off for sample efficiency, because now,


the supervision signal here is much, much larger, because now we work with different hidden states,


which are entire vectors. So that's going to be a much better signal. So that's the first


improvement. The second improvement is what they say, end to end prediction of the value prefix.


So they make an example right here of saying, okay, what's what's the value? You know, if you


if you look at this, you have to predict sort of the future value, can you really predict?


What's it going to be like either the green player, let's say the ball flies in this direction,


the green player is going to catch the ball or not, right? And that makes a huge difference.


Now, you as a human at this point, you know, that it's not going to the green player is not going to


catch that ball. And at this time, you're kind of sure. But it's quite hard to predict at this time


right here. And it's even harder to predict when you know, at which step in time that player is


going to miss the ball. And that's an argument they make for essentially saying, if we add up


the rewards of our own predictions, they can introduce a lot of mistakes. And but that's


exactly what we do. If we look at the Q value that we use in this tree search, what we do is we add


up the rewards that we got in the path so far, and we add the value at that particular path. And


that is very error prone, because this sum right here accumulates all the little errors that that


that that happened in in prediction. And, you know, as I said, if, if we're not exactly sure at which


point, that is just one of the examples to show you how hard this problem is of predicting rewards


step by step, if you look into the future. So what they do is is pretty simple. They say instead of


adding up all the rewards, k steps into the future, what if we simply take the hidden states


that we predict k steps into the future, and just shove them into a neural network. And then that


neural network will output the sum of the rewards. So instead of summing the rewards directly,


we have a neural network output the total sum, much like we have a neural network that outputs


the value function at that looks ahead, this neural network right here, it will look sort of


back it will look into the past from the current state to the state the end state that we rolled


out in imagination, it will predict the entire value, they're using LSTM for that because it can


take an arbitrary number of states. And the LSTM has per step rich supervision because we have a


reward at each step. And therefore they say that works quite well. So that's the second thing. The


third thing is the model based off policy correction. So yeah, this one, this one is a


little bit more tricky. But essentially, we can see where is it, we can read a bit through it to


see what it does. This is an off policy correction mechanism. And they have two different mechanisms


to do off policy correction, I already said off policy correction, you have to do it because the


data that you get to learn from comes from your replay buffer comes from delay from the network


and so on, and is a little bit older than the network that you're learning. And that turns out


to be quite a big problem. So what we usually do is we sample a trajectory from the replay buffer


and we compute and we compute this target value z right here for the value function. The value


target sums from off poll, sorry suffers from off policy issues since the trajectory is rolled out


using an older policy, and thus the value target is no longer accurate. Now mu zero, reanalyze,


this is a particular version of mu zero already handles that a little bit in that it actually


recomputes the values, the scalar values with the current network before it learns from them. But


still the policy used to generate that data is from an old policy. And so they say,


so they say, when data is limited, we have to reuse the data sample from a much older policy,


thus exaggerating the inaccurate value target issue. So what they do is they say, well, instead


of using instead of using sort of the path, so we're, this is the state, right? And here is what


actually happened, right, we took some actions, that's what actually happened. And now, what we


would like to do is we would like to take this and learn from it. But the policy used to generate


that path is an old policy. So the current network might have done something entirely different, it


might have done a different action right here and got to a different point. And that is a problem,


because in an own policy method, we'd largely like to learn from actions that have been generated


with the current policy. So what they say is that we're simply going to not use the entire trajectory


for learning. But we're going to cut off at some point, because of course, the further out, the


more uncertain we get. And that cutoff point is going to be closer, the older the trajectory is.


So for a very recent trajectory, my cutoff towards the end, but for a very old trajectory,


we went caught off like, all the way here. And then what we do after the cutoff point is so we


take this, we cut it off at some point, we say, well, it's old, but you know, this part right here


is still sort of the uncertainty is, is not large enough for us to worry so much. And then what they


do is they use because they have a latent model for for the environment for the world, they use


that model to imagine a rollout. So much like something like dreamer or so they now train using


imaginary rollouts from the point where they cut off. So the the trajectories in the replay buffer


are more like seed values. And after that, they imagine rollouts using their latent model of the


world. Alright, so yeah, so I think that's it. We redo an MCTS search with the current policy on the


last state and compute the empirical mean value through Oh, yeah, so at the last, so at the last


node right here, they redo an MCTS search they in order to get a really good target value there


with the current policy. Yep, that's that's it. Okay, so these are the three improvements. Again,


they introduce a consistency loss on the hidden states to make their transition model better.


Second, they directly predict the value, what they call value prefix, this thing right here,


instead of summing up the rewards as they go along the tree search. And thirdly, they seed,


they use the collector trajectories as seed values, and then train essentially, in half


imagined, half imagined rollouts with the current policy. So that's it. So what does that give them,


it gives them very good performance on this Atari 100k benchmark, they do some additional,


they do some additional things right here, additional ablation studies, for example,


they try to reconstruct the observation from the hidden state. And they see that, for example,


if you don't have a consistency loss, this quickly fails. So this will be the original mu zero,


whereas with the consistency loss, you can see that kind of sort of there is an, you know,


there's something right there, that looks like the observation. Now here, I don't know if that is


after the 100k steps, because of course, mu zero after 100k steps also doesn't perform super duper


well. And therefore, you wouldn't be surprised like that this is, or it could be because their


reconstruction method is just kind of poor as well. But the difference is noticeable between


the two models, the one that has the consistency loss and the one that doesn't. They also analyze,


for example, the validation loss, if you have if you directly predict the rewards, or if you use


this value prefix prediction method, you can see during training, it's approximately the same.


However, at validation time, this loss is much, much lower. And lastly, lastly,


what they do a lot of ablations that that is it, what I was surprised or not surprised what I


noticed in the ablations, and this is pretty much in all the ablations, there is no consistent


ranking. So they have three improvements right here. And sometimes this improvement right here,


for example, will be the most valuable. So you can see that without the value prefix,


alien drops quite a bit. And in other times, you can see right here, this one will be the most


valuable. And yet in other times, some some other one, like the last one will be the most valuable,


don't see one right now. But I have, I've looked at it and that there is no consistent thing. So


that it means that there's not a single recipe to make this thing better. It's a conglomeration. And


for different Atari games, different things are important. And that sort of leads you to think,


you know, is this, this isn't the this isn't the method from let's say principle, this is they


have looked at what fails, and they fixed, essentially, one by one, the major mistakes


that they found. And that is that is a way to go about it. But it is also a danger that we sort of


over engineer to the benchmarks that we have. Because, you know, clearly, if I just put one


of these improvements, and some of the Atari games will improve by a lot, but others won't. And that,


to me is a little bit of the of the danger right here. And this is why I'm not, you know, like,


I can't, I can't tell you if this algorithm is going to be a staple algorithm for sample efficient


RL, or if it just works particularly well on this benchmark, they do do another benchmark,


they do do the DeepMind control, a benchmark, but I think there's going to be more evaluation