manifold view, because of course, the decision boundary would be sort of following the data


The Dimpled Manifold Model of Adversarial Examples in Machine Learning (Research Paper Explained)

manifold, curving around the data, and then following the image manifold again. So here would


be sort of the other data point going below that a little bit. Alright, so that is the view right


here. Now, what they're going to try to show you is that if you want to create an adversarial example


on the manifold, you have to walk much longer, for much longer, until you find an adversarial


example. And if you go off the manifold, if you go, yeah, and they're also going to show you that


if you're not constrained, if you can go anywhere you want with an adversarial example, then that


will be very similar to when you force the adversarial example to go off the manifold.


And this gives a bit of proof that, you know, if two things behave equally, they're, you know,


probably equal. So what they're going to do is they're going to try to make an adversarial attack,


first of all, a regular one, this one, we're going to say, okay, we're going to make an adversarial


attack, let's measure how far we have to go to cross the decision boundary. Second, they're going


to say, let's make the same thing. But let's force the attack to be on the manifold of natural images.


And let's measure that. And lastly, they're going to ask, okay, let's do the same thing, but force


it to be off the data manifold. And then they're going to measure how long these are, how long the


adversarial attacks are, what's their their norm. And they're going to find, of course, they're


going to want to find that these two are about similar norms and way smaller than the one that


is on the data manifold, sort of giving evidence to, you know, if you go perpendicular to the data


manifold, you have to go very, not very far. And that's what adversarial attacks do. Okay, yeah.


So how, first of all, how do they force the the adversarial attack to be on the manifold?


What they do is they do an auto encoder. So they train an auto encoder. So they an auto encoder is


a neural network that has sort of a bottleneck layer. And you try to just reconstruct the inputs


data, okay, you tried that these two are equal. However, in the middle here, you have a very low


dimensional representation. So where this is an n dimensional representation, this is a k


dimensional representation and a k much smaller than n. If you can reconstruct the images correctly,


that means that you sort of have captured the representation in these low dimensions right here.


So what they're going to do is they train an auto encoder, they take that low dimensional


representation, they linearize around it. And that's how they have a way to project onto the


image manifold by simply only moving around in this low dimensional manifold, right.


Here, we're always projecting onto it. First of all, it's a bit of a trouble because how you train


the auto encoder is like for these experiment, I think it's very relevant to how the this image


manifold is going to look like if you train it with l two, you sort of already make some claims


about what are important features and whatnot. But let's disregard this right here. Let's say


they have an accurate way of projecting onto the image manifold onto the manifold of natural data.


And here's what they find. Look, let's look at ImageNet. Okay, no constraint PGD. It this is the


norm, you know, it's some number, okay, so like point one, four. Now off manifold PGD is where


they deliberately project off the manifold. So they project on the manifold, they subtract that


they say you're not to do anything with the man of the image manifold. And that's point one, five,


two, which is slightly larger than the no constraint PGD, but essentially the same size.


Now on manifold PGD. Okay, here is a way bigger number like six times bigger number. So their


claim is look up up to six times more, you have to go on the manifold and off the manifold. And


that gives credence to their claims. Now, okay, so what I've done is they have, you know, they have


some descriptions of their experiments, specifically, they have descriptions of what


library they used to use advert torch. Okay, so I used advert torch to they used, you know,


l two PGD, I use that too. And they told me how much their low dimensional representation is so


the K here, how much that is, how much the end is. And so I was able to reproduce that experiment.


Now, what I've done is, I have done the same thing. And you can see right here, this is this


the panda image from ImageNet, they use an ImageNet classifier. And what they do is they do


it greedy. So they stop as soon as they cross the decision boundary, and then they measure the norm.


You can see right here, this is the perturbation. Now it's a soccer ball. And here is the size


point 7772. That's the norm of the original perturbation adversarial. What I now do is I


project onto the manifold, but I don't the differences, I don't project onto the image


manifold. What I do is here you see project onto K, I simply project onto any K dimensional


manifold. So I know what K is, K is 3500. So it's a very small number compared to the input number.


And so what they project is actually the gradient. So the gradient of the adversarial attack that


you use to update your image, that's what they project, they have the algorithm clearly lined


out. So what I do is I simply take you can see right here, I take a random set of dimensions.


Like of pixel coordinates in the gradient. And I denote the first, the first few, the first K as


the manifold, and the last K as not the manifold. This is not the image manifold, there's nothing to


do with the image manifold. This is simply a random K dimensional subspace of the pixel space.


And now when I project onto K, I simply take all the others in the gradient and I set them to zero.


That's how I project onto a K dimensional manifold. After that you normalize the gradient and so on.


So you proceed, you proceed as you would, right. So here you can see the project is used before


you normalize the gradient. So there's no issue with sort of the step size, you simply project


onto the manifold. And I have the same thing by the way, projecting off the manifold where I simply


take the K dimensions and set them to zero. Okay, so now let's look what happens if I project


onto the manifold. Oh wow, before it was point seven seven. And now it's 6.5. So about eight times


larger. And now let's look what happens if I project off the manifold. It's point seven seven


seven three instead of points seven seven seven two. So what they're seeing right here and you


know maybe okay, maybe I've done it modulo I've done it wrong and I completely don't understand


what's going on. What they have found is simply an effect of projecting onto any lower dimensional


space. Yet they claim that this is like in support of their hypothesis, which clearly, I have no clue


what the data manifold is, I've just projected onto a random manifold, and I got the same results.


So I see they have other experiments where they try to kind of convince you with all the types of


perturbations and so on. But you know, like, no, this these, they have other experiments, but


this is just one that I could try quickly. Again, maybe I've done it wrong. To me this Occam's Razor


is strong here like Occam's Razor in this work is quite a bit like there can be like there can be


many hypotheses that coincide with the results you're getting and with the phenomena. And it's


easy to think that stuff is in favor of your hypothesis is providing support for it.


When there are other explanations available. Oh, I almost forgot about Goodfellas claim that,


you know, they say belongs to the sort of old thinking that is now that is not a correct


thinking and the claim that when you make an adversarial examples, you somehow go towards the


centroid of a different class. And this in imagination, it's something like this on the on


the left right here. However, if you think about this in this space, okay, let's say you start out


here, and you go towards the centroid of the other class, right? The pro like where's the


center right here, approximately, like this. What happens in feature space, because of the stretchy


feature because of the different scales, okay, what happens in feature spaces, it pretty much


like the blue arrow here. So it's that in feature space, you go a long way. Actually, this is


probably I should have drawn this here to be square and this here to be super stretchy. Right?


Yeah. Yeah, I think so. Yeah, I was I was wrong in drawing this. So this here should be squares and


this here actually should be super duper stretchy, right? So the centroid, what was the centroid here


is like way up here, like way up here somewhere. Okay, so this gets super stretched. And then


you cross the boundary in this one feature, right? Like the fur feature. And yeah, so I think this


is it's still a correct claim, you go towards the centroid of another class. But because you go this


in input space, in the feature space, this results in sort of a dramatic shift in some features and


a not so dramatic shift in other features. So while in the input space, you have this


same feature in the other class, but you don't have the same feature in the other class. So


while in the input space, you go towards the centroid equally in all pixel directions,


you don't go towards the centroid equally in all pixel directions in the sorry, you know,


feature directions. So I think the claim the good fellow made is valid here still, and explains like


that's also kind of what maybe I can't read his mind, but maybe what he meant by that and not


necessarily this picture right here, not necessarily that actually, the entire picture is going to


change into the other class. Okay, that was the interjection. And back to the conclusion. But as


I said, make up your own mind, what do you what do you think of this? Go through the paper, they


it's a good paper, like it's written, it's written well, that it has a lot of experiments has quite


a lot of appendix, where they give you more results and so on. And it's not like, again,


it's not like it's in it's necessarily incompatible, right? It's not, I don't disagree with them. I


just think it's it's not as useful as they claim. And it's kind of insufficient. I don't disagree


with their their main claims. Yeah, and I think we already kind of knew a lot of those stuff and


our current mental models are explaining things, maybe a little, a little better. And yeah, if you


use the squishy feature, what do they call it? The stretchy feature model has a fancy name now.


But again, this is not mine. This is just kind of a a bringing together of what we what I think we


know of about adversarial examples. Safe to say there's going to be something that challenges this


and that's going to be exciting. Alright, thanks so much for being here listening. And I'll see you


Reward Is Enough (Machine Learning Research Paper Explained)

From the makers of is all you need and do we really need and is it even useful now comes


enough.


So today we're going to look at reward is enough by David silver Satinder Singh, Doina


Preckup and Richard S Sutton.


This paper is a more philosophical paper I feel though it presents itself as having practical


advice in it.


And the core hypothesis in this paper and they stated as a hypothesis is that maximizing


reward in an sufficiently complex environment is a sufficient condition for intelligence


to arise implicitly in service of maximizing that reward.


So the the example they give is like a squirrel who wants to get as many nuts as possible


has to learn to do all kinds of things in the environment.


In order to do that it needs to know how to perceive how to motor act in the world, it


needs to understand maybe the cycles of the year, it needs to be able to communicate and


fend away other squirrels and so on.


So a lot of these abilities naturally arise from something that just wants to maximize


a reward in a complex environment.


I do have my troubles with this hypothesis right here, especially how they present it.


But we'll go through the paper, look at the hypothesis at the reasoning.


And as always, tell me what you think about this work.


The conclusion of the work is that if this is correct, this sort of gives a straight


path to general intelligence, namely, let's just maximize reward in a sufficiently complex


environment.


Yeah.


And, as always, if you do like it, share it out, subscribe if you haven't, and we'll dive


into the paper.


So the abstract says, in this article, we hypothesize that intelligence and its associated


abilities can be understood as subserving the maximization of reward.


Accordingly, reward is enough to drive behavior that exhibits abilities studied in natural


and artificial intelligence, including knowledge, learning, perception, social intelligence,


language, generalization, and imitation.


This is in contrast to the view that specialized problem formulations are needed for each ability


based on other signals or objectives.


Furthermore, we suggest that agents learn through trial and error experience to maximize


reward could learn behavior that exhibits most if not all of these abilities.


So it's agents that learn through trial and error.


And therefore, that powerful reinforcement learning agents could constitute a solution


to artificial general intelligence.


Now this has sort of, this is kind of the deep mind ethos, right in a nutshell, it is


let's just build in not like mo the most powerful reward maximization agents specifically through


reinforcement learning that we can, and that will sort of get us to general intelligence


because in order to achieve anything in the world, you need to be intelligent if you want


to achieve it to a very, very high degree.


Now if that tickles you a bit in the wrong spot, so it does the same to me.


But so they contrast this here, they ask, how does intelligent intelligence arise?


How does it arise?


And how is it so bountiful and so varied, and has very different subsystems?


And how does this come about?


They say one possible answer is that each ability arises from the pursuit of a goal


that is designed specifically to elicit that ability.


So for example, the ability of social intelligence has often been framed as the Nash equilibrium


of a multi agent system.


And and they go through others.


In this paper, they say we consider an alternative hypothesis that the generic objective of maximizing


reward is enough to drive behavior that exhibits most if not all abilities that are studied


in natural and artificial intelligence.


So they give an example right here with the with the squirrel.


And so one example is a squirrel in sort of the natural world.


And the other example is a kitchen robot or a household robot, also in the natural world.


Now the one of the core points of this paper is that the environment needs to be let's


say, complex enough.


And I feel like they're only going to be satisfied with a particular environment and that is


the real world.


So if they say a complex environment, just think of the real world like be that, you


know, agents on the real internet in the real world, or be that squirrels in the actual


physical world, they think of environments that are sufficiently complex.


And that's sort of how this hypothesis draws their power.


So the description of this figure says the reward is enough hypothesis postulates that


intelligence yada yada yada.


For example, a squirrel acts as to maximize its consumption of food.


That's the at the top right here, which is the reward depicted by the acorn, the acorn


symbol or a kitchen robot acts as to maximize cleanliness.


To achieve these goals, complex behaviors are required that exhibit a wide variety of


abilities associated with intelligence.


Okay, so the squirrel must learn to perceive it must learn to climb it must learn to assess


the knots, it must learn to bury them, it must learn to remember where they are, and


so on. and the cleanliness robot must learn also to perceive to use its sort of movements,


must learn to wash.


And it might even decide, let's get pizza delivered instead of instead of cooking, because


that will be just cleaner, arguable.


But yeah, so in in this framework, you can see on the right here, they see all of these


different abilities such as memory, perception, planning, and so on, just arising from these


things because they say, well, in order for the squirrel to maximize knots, it needs to


be able to do all of these things.


Otherwise, the squirrel will just sort of die.


It can't it can't like without perceiving the knots, it can't go get the knots.


And the also the cleanliness robot, if it is actually good at maximizing its reward,


it needs to develop all these abilities, including right, like the social abilities in order


to get a pizza delivered or in order to work together with the human, maybe even to manipulate


the human to make less dirt.


So that's the that's essentially the hypothesis right here.


They do give some example.


So they I mean, this first part the introduction, I mean, you can read it for yourself.


But they, they say, they give these examples here they say, watching this through the lens


of reward maximization may, in fact, provide a deeper understanding since it explains why


such ability arises.


For example, avoidance of crocodiles, because you need you don't want to be eaten.


In contrast, when each ability is understood as the solution to its own specialized goals,


the why question is sidestepped in order to focus upon the what the ability does.


singular goal may provide a broader understanding, and it might even lead to new sort of new


forms of intelligence.


They give examples, of course, here the games of Go and chess, where just maximizing the


reward alpha zero was able to come up with very new, very new tactics, very new openings


and games and so on.


And we didn't teach it to do openings, we didn't teach it to do board control and whatnot,


or whatever they call in the things in Go, we just asked it to maximize reward.


And it came up with all of these sort of sub abilities by itself, right.


Now they formalize this here, the reinforcement learning problem, they formalize it as an


agent interacting with the environment.


So here, the agent is just the decision making process.


So in the squirrel, actually, only the squirrel brain would be the agent and the squirrel


body is already part of the environment.


So if you're in a sort of multi agent system, all the other agents are part of the environment


in this framework.


And the environment, you interact with it, and you get a reward signal, right, reward


signal, and then maximizing that reward signal, that is what you call reward maximization.


And the core hypothesis of this paper, as I already said, right here, is the reward


is enough hypothesis.


And the hypothesis itself says, intelligence, and its associated abilities can be understood


as subserving the maximization of reward by an agent acting in its environment.


The the, it's a bit better stated above, I think that they say that the main different


forms of intelligence can be understood as subserving the maximization of reward and


that the many abilities associated with each form of intelligence may rise implicitly from


the pursuit of those rewards taken to its limit, we hypothesize that all intelligence


and associated abilities may be understood in this manner.


Now they do strengthen it.


They do strengthen this hypothesis, because what you might be thinking of what I was thinking


of first is that, oh, you know, you can just formulate any goal as reward.


And that's what they say here, they say the reward hypothesis, which is different from


their hypothesis, speculates that all goals of interest in studying natural or building


artificial agents may be represented by rewards.


This should not be confused with our reward is enough hypothesis, which considers the


abilities that arise from the pursuit of any such any one such goal.


Okay, so it's different than just saying, well, you can learn to perceive by doing reinforcement


learning, or well, you can learn to acquire knowledge by reinforcement learning.


Now this is stronger.


This says that the hypothesis here is intended to be much stronger, that intelligence and


associated abilities will implicitly arise in the service of maximizing one of many possible


reward signals corresponding to the many pragmatic goals towards which natural or artificial


intelligence may be directed.


So their idea is that there is a world and that world is sort of complex enough, right?


Maybe there's a tree.


And you know, there's a house.


So there is humans in it.


And you have your little squirrel, whatever here, squirrel has a bushy tail and the head.


I don't I don't know how the squirrel looks just this is a head.


And given in this environment, you pick any reward you can think of like any any reward


signal and then maximize such as like how many how much hunger do you have, you get


that as a negative reward, and then maximizing that reward will lead implicitly to the squirrel


having to develop intelligence having to develop perception having to develop the acquisition


of knowledge and even interacting with other squirrels or the humans in this world.


This is a strong hypothesis.


And as I said, I do have my problems with it.


First though, they go through a bunch of things they say, Well, let's explore how we let's


explore some abilities that people naturally associate with intelligence.


And let's explore how they might arise implicitly from reward maximization.


Okay, so again, think of the squirrel wanting to get as many nuts as possible.


Or like, I don't know, a human wanting to survive and live and thrive in the real world,


how something like intelligence may arise, just as a product of maximizing that reward.


And so here they go over a bunch of them.


The first one is knowledge and learning.


And the the arguments made here are always they're always pretty simple.


They're they're giving you an example and saying, Well, in order to maximize your reward


in the real world, it's useful to have knowledge.


And also because you don't have infinite memory or whatnot.


It's useful to learn things and to to abstract things right to to gather knowledge and so


on.


And then when you hear when they go for perception, they say, Well, in order to maximize your


reward to thrive, you need to perceive.


Okay, so, you know, naturally, it's like almost a tautology.


Okay, so they say, Well, a reward maximization agent can reward maximize better if it perceives


rather than if it doesn't perceive.


So it's, it's sort of an social intelligence.


Yes, so if you're a human, you want to thrive in the world, it's better if you are socially


intelligent.


In fact, it's better if you know language, because you can maximize reward by communicating.


So language, if, if you know, might just be a byproduct of reward maximization, generalization,


well, it's better if you generalize and imitation.


Yes, it's better if you imitate general intelligence.


Well, if you want to reward maximize, you need to be able to instant sort of switch


around between different sub goals in order to reward maximize and sort of solve new problems


really easily, that would be really good in order for you to maximize your reward.


And therefore general intelligence is might be, you know, if an if an agent maximize its


reward, general intelligence will help.


And I hope you've seen a little bit the trend here through all of these things.


And I think especially in the last thing, in this general intelligence, the flaw here,


what I think is the flaw becomes rather obvious.


Because I mean, so reward is enough for for general intelligence.


Essentially, you're saying, well, if we build something that's intelligent, right, then


we have then intelligence is a byproduct of that.


So if, if you if you postulate your reward maximization, as being intelligent, then yes,


intelligence arises as a byproduct.


Where their whole notion here is that if you have this complex environment, and you want


to do anything, you need to be intelligent.


And that's how they see the environment itself.


The big question here is, of course, what is this environment?


And what is the reward?


And they have a discussion at the end where they say, well, as long as the environment


is complex enough, we don't actually care, right?


If it's complex enough, you know, that any and and and also for the reward, like any reward


signal, any goal will do, you can, and they say, well, what if you if your if your goal


is to collect pebbles in the real world?


Okay, so, you know, there is a pebble, there is a pebble, there is a pebble.


So one agent might just learn to collect pebbles.


But the other agent might learn to sort of use the internet and buy pebble collectors


off of Amazon and then launch a political campaign and influence all the humans to also


collect pebbles for itself and then influence everything and get rich and buy more pebbles.


And that would necessitate intelligence.


So just maximizing getting pebbles would sort of lead to intelligence.


And I'm, I follow this way, but you know, again, this is sort of saying, if you're intelligent,


then you're intelligent.


And on the other hand, what if a agent could simply chemically transform anything it finds


into pebbles or anything that's even possible?


There's this, this meme, right with the distribution, where here is the new guy.


So the here, here you have like, here we have this guy with this hair, and with the teeth,


and this goes, collect, collect pebbles.


And then here you have the, I don't know, here's the smart person usually.


And this person is like, well, influence all the people and buy things with money and do


this and do that and do this and do that.


And over here, I just imagine the Zen, so there's usually the person in the hoodie,


right?


The Zen person, well, that's a terrible hoodie.


The Zen person again going collect pebbles.


Like you, you don't know this, it's, I think this is such a, this is such, it's just kind


of looking out at the world, and then abstracting that into what they consider a reward of the


environment.


And then naturally tautologically, what will arise is that if you sort of maximize that,


then intelligence will arise.


And that's not even the end of it, right?


Because a lot of things such as survival in the world and thriving in different environments


are done without intelligence.


If you think of bacteria, for example, bacteria, so I don't know, so here's the world.


And there's like a tiny sliver where humans can live in about one fourth or so of that


sliver, yet bacteria, they're everywhere, okay, they thrive much more than humans.


So if the if the goal is survival and fitness, I mean, bacteria solve that problem completely


without any intelligence.


So I disagree that just reward maximization is enough.


But then these people would say, well, the environment is not the same.


The environment for a bacteria is not the same as for a human.


Like if you are a human, clearly, your approach cannot be to just replicate.


So if you're a bacteria, you know, here's here your bacteria, what do you do?


You simply split.


Cool.


Don't need intelligence can colonize the entire planet.


However, if you're a human, that is not an option.


If you're a human, you need to be intelligent, right?


Your environment is different.


So your environment is much more what they would say complex, though I disagree, I think


that bacteria environment is incredibly complex.


But the human environment, they would say is so complex that you as a human need intelligence


in order to thrive that environment.


Now again, there is a fallacy here, in my opinion, right in my opinion, what do I know,


this is rich Sutton.


But in my opinion, there is a fallacy here, namely, so there is the environment, right?


And you're, you're the human right here, you're in the environment.


And in order to maximize your reward as a human, because you can't split because there


are other humans around, you need intelligence, right?


Intelligence needs to be right here in the human in order to survive and thrive in the


human environment.


However, that environment only exists because there is already intelligence, right?


So first of all, you as a human, you don't acquire intelligence because you need it in


your environment, you have it built into you, you do a bit of fine tuning during your life,


but not like the no one doubts that a that intelligence is present, even in a baby, okay,


like it, it might not be able to, to act it out.


But the all of the ingredients like the learning the the ability to absorb knowledge, and so


on that, like the ability to perceive and to learn language that is all present already.


So I disagree that humans acquire and have to acquire intelligence in order to thrive.


Now they, people would say, well, evolution, the evolutionary pressure on humans required


intelligence and that might be true.


But the individual human only needs intelligence because intelligence is already present in


the environment, or if you want to call it differently.


So here is your world, and you can go into different niches, right?


And one of the niches is the bacteria niche, where you simply you simply split.


Okay, another niche, another environmental niche is the niche where in fact, you need


intelligence in order to survive.


But that is determined.


That is just this niche, right?


And you need intelligence because the other humans have intelligence, and because you


were you're only born as a human, because you're because the, the environment has, or


the evolutionary direction has pushed you into that direction.


So it is not that the maximization of any reward be that fitness has led to intelligence


because the maximization of that same reward has also not led to intelligence.


It's simply that intelligence is present in this particular niche of the evolutionary


process, right?


I see this as a clear distinction, like I feel humans, first of all, they have innate


intelligence.


And second of all, the intelligent, the environment is only such that intelligence is necessary


because other humans before you also had intelligence.


Nowhere in this process is the environment determinist or a driver of the development


of intelligence, because at the beginning, right here, the environment wasn't such that


intelligence was necessary.


Okay, so the environments and the intelligence, they evolve together, they're sorry, they


the the environment that requires intelligence, and the intelligent beings evolve together


at no point did you have an environment that required intelligence because of maximization


of reward.


And you had an object in that environment, not having intelligence and then having to


acquire it.


It's simply one niche.


And there are other niches that don't require it.


So that's, that's, that's my one of the largest things that I criticize right here, I disagree


that reward maximization is enough for intelligence, because clearly, the same reward maximization


wasn't enough in other cases.


Also I think that there is no such, like if they think of the real world, and agents with


intelligence in it, those agents only exist, because intelligence exists, not the other


way around.


The agents don't make intelligence, they already are intelligent for the most part.


Okay.


And the last thing right here is, I just want to point to you here, that reward is enough


for knowledge and learning.


Okay, so now, they call learning one of these abilities that is associated with intelligence.


And now we go to the next part.


And the next part is where they ask themselves, well, given that we postulate that maximizing


reward might be enough for intelligence, how should we achieve that?


So it the the hypothesis of maximization of reward is fully agnostic to the nature of


the agent itself.


This leaves open the important question on how to construct an agent that maximizes reward.


So that's the question, right?


How do you construct an agent that maximizes reward?


Until now, we've heard no, of course, the answer is going to be reinforcement learning.


But until now, we have actually not heard much of that except in examples.


So they still leave it open how you would achieve such an agent.


But now they're going to say, reinforcement learning.


But first they say, in this section, we suggest that this question may also large may also


be largely answered by reward maximization.


Now I don't I don't actually know whether this intended here but how to construct an


agent that maximizes reward is largely answered by reward maximization.


Like is this intended?


Like is this an intended back reference saying like, how do we construct x?


Well, x, like, is this?


I'm not sure.


Is this an intended, like a little bit of a slight of like a little bit of a joke or


something?


I might just be too dumb, right?


Specifically, we consider agents with the general ability to learn how to maximize their


reward from their ongoing experience of interacting with the environment.


Such agents we will refer to as reinforcement learning agents provide several advantages.


So here they go into, you know, if you you don't want to pre program you like you don't


want to have the designers knowledge of the environment be in there because the designer


doesn't know everything you want to actually let the agents learn themselves.


And if the environment is sufficiently complex, and the in the reinforcement learning agent


is sufficiently powerful, then it will like the richness of experience of a complex environment


will provide enough signal for the agent, you know, disregard its practical implementation


and sample complexity.


Technically the whole richness of experience will provide enough of a signal to learn all


of this.


But I don't know, did you?


There's another thing right here.


We consider agents with a general ability to learn how to maximize reward.


So how do we build reward maximization agents, which, if successful, will give rise to intelligence,


right?


Well, by learning, okay, however, learning up here, learning is a product of intelligence


or an ability that comes with intelligence, right?


So like we need, we need learning in like learning comes with Intel learning is one


of the abilities that indicates intelligence.


So a little bit it's like learning.


Gens.


So intelligence, if something is intelligent, right?


Then then it will learn but also, in order to achieve this intelligence through reward


maximization.


That's how we achieve intelligence.


But then in order to do reward maximization, we need a learning algorithm.


But if the learning algorithm is not yet intelligent, right, then how is this happening?


So I guess you can make a split and saying, well, this learning that we use for reward


maximization, that's sort of a learning that we design or something like this.


But even if we design it, intelligence gives like if we design the learning algorithm,


that's again, this, this way in a sneaky backdoor way.


Or you can say, well, the type of learning for the reward maximization is a different


one than the learning we mean here, here, we mean the acquisition of knowledge, but


I'm pretty sure the acquisition of knowledge is part of reward maximization.


So a little bit of a closed loop there.


Honestly.


Yeah.


So I'm not I'm not sure.


But here they make the case and of course, like I agree with all of this, I agree that


RL, you know, reward maximization, if you have a powerful enough algorithm, it will


sort of discover these sub tasks and it will have to acquire these abilities and so on.


It might not be super sample efficient.


And certainly it's a better way to general and to general intelligence than like supervised


learning or or just prediction itself, like future prediction and so on.


That is, and that online learning is better than offline learning.


I agree with all of this, right.


And here in the discussion, by the way, they also say, which environment, right?


And then they say, well, it can be any as long as it's, I guess, complex enough, which


reward signal and here they also they give this this pebble example, where they say,


well, even a simple goal in the complex environment can can give rise or will give rise to intelligence.


And yeah, so I agree with most of it.


But this this core, the core thing, I'm just very skeptical about.


And my last comment here is that they, they so presuppose that this reward exists, right?


They so presuppose that there is an environment that gives you a reward.


And I also disagree with that, right?


So if you look at the only process that we know that actually has produced artificial,


or not artificial, natural general intelligence, or at least human like intelligence is evolution,


the evolutionary process has given rise to all the intelligence that we see be that in


humans or other animals today, or, or even like social systems, if you consider them


to be intelligent corporations.


The evolutionary process is the main backbone of this.


And even though you can maybe formulate the evolutionary process as maximizing fitness,


I don't like there is no for evolution, there is I don't think that's a good way of framing


it, it's rather that there is an absence of reward.


And it is not a single reward that's given right that the reward creates itself as evolution


goes into a new niche.


And it is not a a, you can consider it in the abstract as a reward maximization algorithm


if you consider fitness to be your reward.


But I do, I do, in my mind, there is a difference between this and what these people have in


mind right here of giving rise to social or to intelligence, in that I would consider


the evolutionary method to have no reward in the sense that they describe it right here.


And yeah, so that is, to me, the last sort of the kind of more baseline flaw in this


paper, in that it assumes that there even is such a thing as a reward in the environment


where I'm absolutely not sure that this is even the case.


And the only example that we have of general, or at least good intelligence, human like


intelligence, however you want to call it, is a process that arguably has no reward.


And it just does what works.


And some of the stuff that works is intelligent, but other stuff with the exact same apparent


reward signal is absolutely not intelligent.


And that leads me to believe to say, no, I don't think reward is, as you say, enough.


But that's my opinion.


Please give me your opinion in the comments.


I still read the comments.


And I will do so in the future.


So if you're watching this in 2040, and if you're an AI, then please write in the comment,


did you did you just use reward?


Or did you use something else?


I'd be very interested.


Also, please, please spare me.


Yeah, I'll see you next time.


If I'm still here.


Linear Transformers Are Secretly Fast Weight Memory Systems (Machine Learning Paper Explained)

Hi there, today we'll look at linear transformers are secretly fast-weight memory systems by


Immanuel Schlag, Kazuki Airy and Jürgen Schmiduba.


On a high level, this paper makes a connection between linear transformers, which are transformers


that linearize the attention mechanism, such as the performer, and fast-weight memory systems,


which is a bit of an older concept, where fast-weight refers to one mechanism producing


weights for another mechanism.


So like a neural network producing weights for another neural network, the first neural


network will be called the slow-weight and the produced weights would be called the fast


weights.


So the paper makes a connection between specifically autoregressive linearized transformers and


these fast-weight memory systems and looks at it in terms of how much memory are they


able to store in these weight matrices, and it analyzes it and proposes a new update mechanism


for autoregressive transformers, and then demonstrates kind of the the effect of that


in experiments.


We'll go through the connection they make and look at their new method, their new proposed


linearized attention, and we'll look at the experiments, and that will be the paper.


So if you like content like this, please share it out to all your friends and enemies, because


love is okay, I'm becoming Lex Friedman.


So what are fast-weight systems?


Fast-weight systems, as I already said, is when one neural network or one mechanism produces


weights of another neural network, so the fast network would not be learned per se,


but it would get its weights from the slow neural network.


And this here is an example of that.


By the way, new new new recording setup, thank you for your feedback very much, so I have


extended the screen here to cover the entire area.


Please more feedback, I know this is still pixel-ish.


If anyone knows how to make one node not do pixel-ish PDFs, please tell me.


Right, so here is one of these fast-weights mechanisms.


So a slow net with slow weights continuously generates fast weights for a fast net, making


the fast weight effectively dependent on the context.


Simply put, the slow net learns to program its fast net.


And here, in these papers by Schmidhuber, he proposes these outer product fast-weight systems.


And here is how it works.


So imagine you have a sequential input, so xi is going to be x over time.


Remember we're in the autoregressive setting.


So the autoregressive setting is where you have a sequence as inputs, and then you're


from that sequence, you're trying to produce the next element of the sequence.


For example, in language modeling, and then in the next steps, you take that next element


into your context, and you produce the next next element, and so on.


So that goes on.


And that is the autoregressive setting.


So we are wondering how do systems produce in these autoregressive systems produce their


outputs.


And one way is this fast-weight system.


So imagine you have these x's here, which are the input sequence.


So we're going terms of an input sequence.


How do we produce the y that is, so this is the, how do we produce the next input or specifically


in a more general setting, we have an input sequence and an output sequence.


And at each step, we kind of want to produce the corresponding output.


So in the first step, this, and then the second step, we already have two inputs, and we produce


this output.


And in the third step, we have three inputs, we produce the third output, sorry, we have


three inputs.


And then the fourth step, we have all four, we produce the fourth output.


Of course, in the autoregressive setting, we would every time take the output and plug


it in here at inference time, not at training time.


All right.


So we have input sequence and output sequence.


How each, how does each step look such that we produce the corresponding output?


Well, here's what we do.


We have these specifically, we have these matrices called w.


And the w matrices are these fast weights.


And you can see, the output is simply produced by taking the current input and multiplying


it in a linear fashion by the fast weight matrix.


Okay.


And right now, if you just look at this, this is simply a linear transformation.


The magic happens if you consider how these weights here come to be.


So these weights are now going to contain the entire context of the past inside the


weights.


So other than it is a bit like a recurrent neural network where you have a hidden state,


except here, the weights themselves are the hidden state.


So how do you generate the hidden the weights here, these fast weights, well, these fast


weights are produced by updating the fast weights of the last step you can see right


here and here is where the recurrence comes in.


So the fast weights of the current step, that's not supposed to happen, the fast weights of


the current step are produced by adding on top of the fast weights of the last step,


there is a non linearity involved right here.


But essentially, you take the last fast weights and add something to it.


Now what is that something that something is here, this outer product of a and of these


vectors a and b, which are themselves constructed by taking the input and running them through


their own neural networks or just their own linear transformations right here.


You can see that this mechanism will continuously produce weights.


So there is a few few intricacies here, like why do this is the outer product between the


vectors.


And that's needed because in every step, you want to produce a valid weight matrix, right.


And this is how you produce a valid weight matrix by taking the outer product.


If now you accumulate those outer products essentially in these fast weights, which has


some other interesting properties, and the paper is getting to those properties later


here when he talks about tensor product representation theory.


But essentially, this is how you have people store information inside of matrices is a


bit of magic.


But imagine you have keys and values, and you want to store those keys and values like


in a database, but you want to do it in kind of a continuous manner.


So this comes from a time when people were trying to bridge the symbolic world to the


neural network world, let's say so they were trying to put discrete things or objects and


symbols into distributed representations like vectors.


So if we want to build a database, what we have to do is we're going to have to have


keys and values that we store right, key one, value one, key two, value two.


This goes all into a database, key three, value three.


And if we then come and we query the database with one of the keys, like, okay, I have now


key two is my query, I define my query as key two.


And I go to the database, the database better give me value two.


How can we implement this as a distributed representation database?


So first of all, imagine we are going to have keys and values, they're all going to be vectors.


So the keys are going to be represented as vectors and the values are going to be represented


as vectors.


Okay, the key, maybe this, this vector and this vector here, and the values, this vector,


this vector and this vector.


It's we can we can do symbols to vectors by doing embeddings.


So we know how to obtain that.


But now how do we implement the database?


Well, if I'm just going to show you what I can do, how do I build the database, I'm going


to build the database as follows.


I'm going to take key one, and I'm going to do the outer product to that's, that's a plus,


I want to do the outer product between key one and value one.


And then I'm going to add to that the outer product between key two and value two.


And I'm going to add to that key three, value three.


Okay.


So why, why does that give us a database?


So that gives us a database.


And what we want to do is we want that if, if we go to the database, and we query it


with the query, and this is going to be a matrix multiplication, right, the database


is going to be a matrix, we want, and let's say the query is key two, we want that we


get value two, it's magic, right?


I can just add these things to the database with a plus.


And you can see, I can also update that in the future by simply adding to the database,


one of these outer products.


And we want this, it seems a bit like magic.


But here is how it works.


And the condition is that all of the keys are orthogonal to one another.


If the keys are orthogonal to one another, this is going to work because imagine we now


go to the database, and we multiply by q, what does that do?


That is going to be key one, we can write this as a sum, right?


We have this sum over the i of key i, value outer product with value i times q.


Now that we can pull in the q, so we're going to have the sum of i.


And here, we're going to have the key times the value.


And this all times q.


Now q is going to be, as we said, q is one of the keys, because we query the database


with one of the keys.


So here, it's going to be key number two, with key i, and this is an inner product right


here.


And this is an outer product with the value i.


Now if the keys are orthogonal, you're going to see pretty quickly that if i is equal to


j, sorry, to two, then this is going to be just the number one, if they are orthogonal


and normalized.


If the keys, however, are not equal, so if i is anything else than two, this is going


to be zero.


And magically, all of the things drop away, all of the sum elements drop away except the


one that contains vi or v2.


So this is going to get v2.


So magic.


And as we said, the conditions are that the keys are orthogonal to one another and normalized


if you want.


But this gives you now the flexibility if your embeddings are meaningful, meaning that


the latent space is meaningful, you can also query your queue can be kind of a superposition


of keys or something in between the keys and what you'll retrieve is an interpolation of


the values.


And this is very, very similar to the attention mechanisms we have nowadays, right, these


queries and the keys and the values.


And this paper is going to establish how exactly this is similar.


Another similarity, by the way, to attention mechanism is exactly this fast weight principle,


I've always said that an attention layer is essentially a fully connected layer, but the


weights aren't learned, the weights are dynamically produced by another mechanism, depending on


the input.


And this is exactly this fast weight concept.


So it makes total sense that there is a connection.


And it also obviously makes total sense that someone already invented this in the 90s,


as I think that's a meme by now.


Right?


So how do we make the connection between attention mechanism and these fast weight modules?


So here's how we do it.


First, this is the attention mechanism as we know it, it's just written a bit differently


in the specific context of autoregressive transformers or autoregressive attention mechanisms.


So we don't care about how we do all the queries, keys and values, we care about how do we produce


the queries, keys and values of the very last step, because in autoregressive transformers,


what you have as a limitation is this causal attention.


So if you have your sequence, and in a self attention, or in a, let's say non autoregressive


setting, you would have attention from each element to each element.


So all the queries can attend to all the keys.


However, in a causal attention layer, let's just build a causal attention layer on top


here of the non causal attention, which makes absolutely no sense.


Every single query can only attend to keys that are in the past.


So this can attend to here and here.


And I'm drawing the arrows in a different direction.


But you see what I mean, you can only attend to things that are in the past.


And technically, that is not technically, it is not, it is too much of a constraint.


Because if you have multiple layers, and you think of what is what does it mean to be autoregressive?


What it means to be autoregressive is that you want to produce the next element.


So if you have a stack of layers, you want to produce this element right here, it is


perfectly conceivable that the information in your network can flow from this element,


which is maybe the the noun in the sentence, to the verb of the sentence here, to the subject


of the sentence here, and then to the front again, or to here again, as long as you don't


draw information from from over here from the future, you're good, right.


But technically, within one context window, it is technically allowed to send information


around like this.


Now, the problem with this is we can't easily parallel isably train things like this.


So what we do is we simply restrict in each layer, the attention to only attend to things


in the past, which means that we end up with kind of these, these attention, sort of like


cones, where you can only send information forward, and not backward, even within a layer,


even though it's technically allowed.


So this restriction is also encapsulated in this formulation.


So we're going to ask ourselves, how do we produce the current output?


Why I the current output is going to be produced by simply looking at the current query, because


all the past queries we've already computed in the last steps, right.


So we simply need the current query.


And but we need all the values and all the keys, right?


The V and the K being capital here means that they are the accumulation of everything in


the past.


This is exactly what we've said, you can in fact, attend to your own to all the past,


but not the future.


So the current output is going to be produced by the current query attending to all of the


past.


The past here is constructed, you can see in each time step, what we're going to do


is we're going to compute the current key and value.


And we're going to concatenate that with the past keys and values that we've already computed,


there's no need to compute things twice here.


So that's, you know, in each time step, we simply need to compute the current queries,


keys and values and the keys and values, we're going to accumulate into these matrices by


concatenating them.


Now, if we slide, usually this extends the sequence like this, right, we extend and extend


and extend and extend, transformers have a limited size window.


So eventually, these things here are going to drop away, in which case these matrices


here are going to not be concatenated, but kind of shifted towards the right.


But you know, that's, that is a minor detail.


And the queries, keys and values are simply going to be produced by the learned matrices


here like this is so this is very standard transformer, or very standard attention mechanism.


Okay, now they say, look here.


So here we have the softmax.


And the softmax is pretty intrinsic to the attention mechanism, because otherwise, it


would just be a linear transformation.


So the softmax, what the softmax is going to do, once the query attends to all the keys,


once the query attends to all the keys, we're going to normalize that using a softmax, which


basically gives you a distribution over the over the input sequence.


So you don't want to know, where should I you want to know where should I attend in


proportion to everywhere else.


So there is a normalization involved.


And of course, also the non linearity and the softmax, but the real bottleneck is the


normalization.


So first, they say, what happens if we just leave away the softmax?


And this is this is a rederivation from other papers, by the way, this is they just building


their case here.


So what happens if we leave away the softmax, if we leave away the softmax, we simply have


here is the key query, here is the attention, and that is going to be multiplied by the


values.


Now we can rewrite this a bit actually comes from here, that's here, here is the here is


the attention matrix, this is the attention matrix for the current time step i, right,


just for the last query.


And that's going to be multiplied by the values.


And that gives you your output.


So the attention matrix tells you how you need to aggregate the values tells it tell


you what the value of the things you aggregate or and you do a weighted accumulation, it


gives you your output.


If you rewrite this a little bit, you can clearly see that instead of an inner product


between the keys and the queries, then being multiplied by the values, you can as well


write this as an outer product between the values and the keys, and then an multiplication


by the query.


And this should, you know, be familiar to you by now.


So here, you can write this as an outer product of the individual keys and values of the past,


and then the queries.


And this here is exactly this database we talked about actually with the sum, including


the sum.


So this is the database of the past.


And now you can see the connection to these to these fast weight algorithms.


And it means it's it looks exactly the same, except it has the fast weight also had this


kind of sigmoid in it.


But essentially, you're building this matrix, this, so the matrix is going to be multiplied


not by x directly, but by q, which is a linear transformation of x.


So that's pretty similar.


This is this is what they call w, wi.


And your output is simply going to be a linear function of the input, so to say.


And it is also going to be a query into this distributed database.


So they say, we can further rewrite these equations such that they directly relate to


these fast weight equations.


So you can build this up step by step, instead of building the whole sum, what you can do


is you can simply write this wi here as a decomposition into the wi from the last step,


simply add the current outer product to it between values and keys.


And then you have your current fast weights, your current database that you then query


by q.


So this relates it to the fast weight algorithm.


Now we made a crucial step in that we left away the softmax, right, and that now we're


going to have to fix that.


So this has already been done, like, we've already come this far.


And I've made a video about the performer.


So the performer reaches this point, and then they say, Okay, now, instead of leaving away


the softmax, we can generalize, we can generalize the softmax by writing it as a sort of kernel


by writing the softmax explicitly, equation seven can be written as so this is the full


equation, equation seven is the full with the softmax attention can be written as this.


And this is a bit tricky.


So k is the current is a kernel.


And the kernel, in this case, is the exponential function, the softmax is going to be this


part right here, so it involves this, and it's going to be normalized, right, the softmax


has the exponential function, and it has the normalization.


So this is going to be the softmax part, and then simply multiplied by the values over


here and aggregated.


Okay, so you can write it as such.


And then you can think about, okay, what kind of kernel could we substitute to approximate


the softmax, but without having, you know, kind of the pesky nonlinear things.


So if you know anything about kernels, which I don't, but there is a good street talk episode,


which I'll link where we were, I got to ask all the dumb questions about kernels, I hope


that helps, but every kernel represents an inner product in some kind of, in some kind


of space.


So every kernel can be implicitly written or explicitly written as this inner product


in some kind of space.


And phi here is the function that maps you to that space.


And the performer thought, can we find?


So the performer explicitly showed which phi you have to choose in order such that if you


plug it in to this kernel, it gives you back the softmax.


And that turned out to be an infinitely large space.


So an infinite, like a non computable function, but then they ask themselves, can we substitute?


Can we approximate that kernel with a finite function phi right here?


And that is the performer paper is very theoretically grounded, but it has some problems and they


discuss the problems here.


But first, see, if you write the kernel as such an inner product and which you could


actually compute, you can then you can see here, this bracket is the problem, this and


this.


Since the kernel is nonlinear, you cannot just pull these things apart.


However, if you write the kernel as the inner product, if you know what the phi is, you


can write it as such and pull it apart.


And then you can do the same transformations as here.


So you can see that here, it's an inner product.


But if this is linear, you can also see this as first the outer product of the key mapped


through the phi function with the value.


So there's an outer product, and only then multiplied by the query.


And you can as well see the normalization as an accumulation of these keys.


And only then you multiply the query in here.


So this gives you the benefit that in not in each step, you have to compute these things.


In fact, you can accumulate these things across the time steps.


They make this explicit here, write it as an explicit outer product, you can see it


is the same thing again, where you can build this database from the past.


So it's not value times key, but it's value times phi of the key.


And for the normalization, you can equally build up this didn't this accumulator on the


bottom right here.


So that's going to be your Z variable.


You can see that this pretty much results in the same algorithm, except that we also


keep track of the normalization here, which we can do just as we build the fast weights,


we can accumulate the normalization.


I believe this was already also discussed in the performer paper, but it's pretty cool


to see here that everything leads to the same path.


So first, we went from fast weights, then we looked at transformers without the softmax.


And we said, Oh, if this is linear, then there is a clear connection to fast weights.


And now we say, okay, if it's not linear, but if the kernel if we can find an explicit


kernel, then we can write it as a linearly decomposable thing.


And then it's also a fast weight algorithm modulo the normalization down here, which


I guess would still count as a fast weight, a fast weight algorithm.


So they say essentially, these linear transformers are fast weight algorithms is specifically


in the autoregressive case, right?


Always think that this is in the autoregressive case, because the specific constraint of how


we train autoregressive models with the causal attention mask gives rise to being able to


write the algorithm like they do here.


So they discuss this capacity limitation now, while the softmax is super nonlinear and normalizes


and all of that, it sort of has is not subject to these capacity limitations, but it is subject


to other capacity limitations, but if this is linear, if this is now a linear algorithm,


they say, endlessly adding new associations to a memory, that's the database of finite


size and as in equation 17, inevitably will reach a limit.


In linear attention information is stored in a matrix and is retrieved using matrix


multiplication.


As a consequence to prevent associations from interfering with each other upon retrieval,


their respective keys need to be orthogonal.


Otherwise the dot product will attend to more than one key and return a linear combination


of values.


With keys embedded in a D dot space, the dot here is the that's the inner the space of


the inner product.


There cannot be more than D dot orthogonal vectors that is storing more than the dot


associations will result in a retrieval error in linear transformers.


When the length of the sequence is longer than the dot, the model might be in such an


overcapacity regime.


So now they say, since these linear transformers are all fast weight algorithms are, they have


these capacity limitations, right?


They built this linear database with outer products.


So technically they can only store a finite and finite given by the dimensionality amount


of distinct data points.


Now this is a very special way of looking at these things.


And we're going to see later what they do.


So in their experiments, I can tell you right now in their experiments, what they do is


they have a sequence of random keys together with constructed values.


So the values are kind of orthogonal unit vectors, but the keys have to be learned,


but they are so let them be fixed set of keys, sorry, not the keys have to be learned, the


embeddings have to be learned.


Let them be finite and fixed sets of keys and values.


And they are sampled randomly.


So they're going to produce key value pairs randomly with random keys and fixed values.


And they see whether or not they can store and then retrieve an arbitrary one from that


database.


Q is randomly chosen to be one of the L keys.


So we store L elements that we sample at random, and then we see, can we retrieve one of them?


Now this isn't, this isn't exactly what we want in transformers.


This is very special way.


It's a very computational way of looking at things like, okay, what's the memory capacity


here?


How many distinct things can we store?


What we want in transformers is more, we're not interested in storing everything accurately.


But I think we explicitly want this interpolation in transformers.


It is very useful to look at these mechanisms from this kind of synthetic setting where


we really test the memory capacity.


But it's important to keep in mind that that is not ultimately what we want.


Ultimately, we explicitly want those superpositions to occur.


Because in NLP, we have synonyms, like we have same information from different words,


we have words in between other words, and so on.


So it is not exactly, you know, the criticism here is valid, but it is not exactly on, you


know, in the wound of what's hurting in transformers.


Nevertheless, they say, can we improve?


Can we improve this update rule?


They say, linear transformers can end up in this overcapacity regime, where they need


to store more things than their dimensionality allows.


If the sequence length L exceeds the dimension of the keys.


Once an in overcapacity, an ideal memory model should dynamically interact with the memory


contents and selectively determine which associations to remember and to forget.


So they criticize transformers here in saying, with this update rule, where we only ever


we only ever concatenate, right, we have the key, and we concatenate the new key, right


here, and so on.


Now, irrespective of whether we limit the sequence length right here, if the sequence


length, and you know, we drop things here, if the sequence length we consider is higher


than the dimensionality, we're bound to have keys that conflict with each other.


And so they say, when you add a new key, you know, given that you are bound to override


each other, you should be able to sort of dynamically, dynamically add keys and not


only concatenate to a fixed set.


Now, what they're going to do is actually not change the keys, but they're going to


change the values.


And this is something I quite find pretty cool, because they also, you also concatenate


the value onto this.


But what they're going to say is that instead of just appending the keys and the values,


what we're going to do is since this key is going to conflict with one key that's in here,


at least let's say it's going to conflict with one key.


What we're going to do is we're simply going, we're not going to store the actual value


to this key, we're going to store the diff in value between this key and the key that


it's conflicting with, you know, maybe they're not fully overlapping, maybe this key is a


little bit off that key, but mostly, so, you know, if we enter this key, and we would just


store naively the value, we would also retrieve the value associated with the other key because


we overlap and then we'd get like a superposition of the two values and so on.


So what we should do is instead of storing the value, we should store the diff between


the value, the old value, and the new value.


And then when we retrieve and inevitably overlap, we're going to retrieve, right, we're going


to retrieve the old value.


And we're going to retrieve the new value.


But now that's the diff.


So plus, okay, other way around.


So we're going to store this plus V. And since we store the diff, this cancels out.


And we only have the new value.


That's pretty cool.


Yeah, so instead of actually storing the diff, they say, you know, the network should be


able to say how much it wants to update that value.


So the network is going to also output a number beta, that is, as you can see, are computed


from the input by a little one layer neural network.


And what you're going to do is you're going to first retrieve the value that is associated


with the key that you want to put in.


So this this value here is, that's the old value, because this key probably overlaps


with something.


So you're going to use that key as a query into the database, retrieve the value that's


associated before, then you're going to interpolate the old value and the new value.


And that's what you're going to store.


And that turns out to be like this.


So you generate the new database from the old database, plus here, the diff, that's


the diff between the values, weighted by a factor saying how much really you want to


update that because of course, also, when you input the old key, you're going to retrieve


the new value.


So you might be, you know, you might not want to just slam in the new value, because of


course, the old value isn't updated yet.


So you know, this this gives you sort of a handle on that.


All right.


And then, of course, you simply retrieve the new thing with the query.


And now if the query is a key that's overlapping, you're going to retrieve the old value and


you're going to retrieve this weighted update on top of that.


Very cool.


They also discuss different normalization strategies.


So one normalization strategy, because we, we also have this denominator in the softmax,


right.


And if they simply do these accumulations, as we saw on top, right, if they simply compute


this, and they compute this, using the accumulation technique, like an accumulators, they are


bound to sort of explode because also these kernels, they map things to positive space.


So things explode.


So what they say is we should change our phi here to be the phi divided by sort of the


sum of the entries.


So this is an easy normalization you can do independent of anything else.


And it keeps the values in check.


The last thing they do is they now suggest a, they suggest a phi.


So you know, given that they've criticized things, they say, okay, let's look at the


phi's that are already around that would meet our requirements.


So we're looking for a function that acts as a mapping to the space of inner products


that is going to replace the kernel.


So one suggestion here is to use elu plus one, which is fairly easy, but it has some


disadvantages, namely, importantly, as an element wise function, preserves the dimension


of the input key vector without modifying the memory capacity as discussed.


So this, not only is this not the softmax, it also doesn't, you know, is actually problematic


because you have no handle on the memory capacity.


The reasoning here is that if you want to go from nonlinear with, you know, technically


infinite capacity or whatever nonlinear bound, if you want to go to linear, which has a clear


upper bound on the capacity, you need to have kind of a hyper parameter where you can artificially


increase that capacity to make up for the fact that you're going to linear space.


This doesn't have it even though it's super easy.


On the other hand, favor plus, which is the algorithm from the performer has that but


it relies on kind of random sampling from a normal distribution and it also relies on


kind of complicated, it's not super complicated, but it is mathematically actually rigorous.


If you go into enough dimensions, you will accurately approximate the softmax, but you


need random features for that.


And these random features can, you know, either hurt your performance, it can hurt your performance


if you happen to sample them in a bad way and you sample them once per training run


which or per model, which so you don't have do overs in that, I guess you can train again,


but you know, so they suggest a thing that is easy and you have a handle on the dimensionality.


So they say we consider four different keys, right?


If we have four different keys in R2, they are going to, so the keys are in two dimensions,


what they're going to do is they're going to construct a mapping into four dimensions


such that they have the highest possible chance of if two keys are different, they're going


to be orthogonal to each other in that higher space.


Now they're going to do this as this, so these are the four dimensions of the mapping, these


are these, this is going to be a vector at the end of these five functions and the R


is ReLU.


So what they're going to do if they, they're going to take a key and they're going to multiply


simply the positive part of the dimensions, the negative parts and the cross parts right


here to get the four features, which means that a given key can only be non-zero in one


of those four things, right?


Like either your first coordinate is positive or negative or your second coordinate is also


positive or negative, that gives you four possibilities and the construction here makes


it such that only one of those four entries is non-zero depending on which section you


are.


You can see that right here, these are the four sections, so if your vector is right


here, it's going to be non-zero in the blue component but not in the green, orange or


purple components.


So they say this gives you kind of maximal, if two keys are in the same quadrant, yes,


they're going to overlap in that higher dimensional space but if two keys are in different quadrants,


they're going to be guaranteed orthogonal.


They extend this to here, so they're going to say we're going to choose this parameter


new here, which that is going to be the handle on our dimensionality.


So setting new is upgrading your dimensionality of the mapping.


If new is equal to one, you keep the dimensionality of your key, actually you double it, but you


can set it to two or actually they only ever go to three.


Three is as high as they go, so they make the intrinsic dimension three times higher


than the original dimension at maximum.


So what are they going to do?


They're simply going to take the vector here of positive and negative elements of your


key and they're going to, so for entry i, they're going to choose the entry i and they're


going to multiply that with again the ReLU of some other coordinate of the same key.


So you're simply taking two coordinates, take the ReLU of them, you multiply them together.


If you include the negative parts of the vector, that gives you exactly what we've seen up


here and the new gives you saying like how many different coordinates do you want to


multiply.


So if new is one, you simply multiply coordinates one and two and then two and three and then


three and four, four and five and so on until you're once around.


If new is two, you do all of that, but also you concatenate that with one and three, two


and four, three and five and so on.


Now at the end they wrap around, like the last one would be like 10 and 1.


They say they have code for this, it's pretty easy, you simply kind of roll around the vector


and then ReLU it and then multiply it or first concatenate the positive and negative parts,


ReLU that and roll and then multiply.


They say this gives you in this upper dimension two times the dimensionality of the key, two


because you have the positive and negative elements, times the dimensionality of the


key, times new.


Now this only works, actually so this is wrong, I believe this is wrong right here.


Here they say you can choose new to be any of these values, which is not correct because


if new is higher than I believe D, what's D key, two divided by two, so if it's higher


than D key, then you're going to have duplicate elements because you sort, if you consider


this here and you view it as a matrix that you later unroll, right, as the projection


up, you have I and you have I, sorry you have new here and what you can have is at maximum


sorry this is I plus new.


Right, you can have I attending, you can have one attending to two, you can have one attending


to two and three, you can have one attending to two, three and four, but at some point


if you know, and then you have to have two attending to, so you can have one attending


to this, this, this, this, this, this, this, this. 2 cannot attend to 2, but it can attend to 3, 4, 5,


or attend to. It can be multiplied with this. 3 can be multiplied by 4, 5, 6, and so on. And since


you roll around, well, the code actually rolls around, so it goes around here, you can easily see


that now if nu is equal to the full 2 minus 1, to the full dimensionality of the matrix here,


then this element is going to be the same as this element, because it's going to be,


the first one is going to be k1 and k2, and then in the second one, because you roll around,


it's going to be k2 and k1, which is going to be the same. So just a little mistake in how you can


choose. Nevertheless, they never get up there. They go 1, 2, or 3, and they never even get close


to that being a problem. All right, so I've already told you the experiments they do,


where they try to retrieve random values, and I've already tried what kind of problem I have


with that. Nevertheless, they show here that the linear, and I'm sorry, this is super pixelish,


I'm going to try to fix that in the future, the linear transformer, as you can see, it has a,


so here is the number of unique keys that you can store. The lower your curve, the better. So these


are the mistakes, this is the loss that you make. So the linear one, the dimensionality is 64, the,


of the keys. So you would expect that it can store up to 64 keys well, and then it can't store more,


it gets conflicts. And that's exactly what you see. So here you start off no loss, and then at


around 60, the loss shoots up because you get into conflicts. Interestingly, these favorite the


performer algorithm shoots up immediately. And that's, you know, probably because it's not built


for this specific purpose. They try it with quite a high number of random features. But it is,


it's pretty interesting to see, whereas their method, so if they choose new equals to one,


it goes for double, which you would exactly expect. So if new is equal to one, the dimensionality of


their algorithm is two times the dimensionality of the keys. So after 120, some of the loss shoots


up. If you choose new to be two, then after, wait, then after, you can see right here, after 240 sum,


you shoot up and if you choose new equals to three, after 360, while the softmax, it gets,


you know, it gets into the error rates here, but this is a different regime of bounds, we cannot


analyze this with the linear bounds we derive because this is the highly, highly nonlinear,


highly infinite dimensional implicitly softmax. This is pretty cool. As I said, even though it's,


it's not exactly what we want from our attention mechanisms, but it's cool to look at them in this


way. They do a bunch of other experiments and they actually do language modeling. So they do machine


translation and machine translation. It's not, it's not really an autoregressive problem per se.


I mean, it is in, but you always have the input sentence and then you have the output sentence


and only the output sentence is autoregressive and not the input sentence. But still, you can


actually formulate it as an autoregressive problem. And if you only do causal attention in this part,


I don't know how much that hurts you, but technically you don't need to. The original


transformer, I think didn't do that. It did full attention in the input and then causal attention


in the output. So here they show that in the intermediate dimensions, they outperform the


performer. But if you go to higher dimensions, the performer outperforms them. However,


in language model experiment, so this is perplexity, so lower is better. In language


model experiment, no, sorry, they, they here they compare update rules, like they compare update


rules, plugging it in into the different transformers. So they show that their update


rule is better than just the sum update rule in the linear transformer and in the in the performer.


So here you can see the number of trainable parameters via yada in our update rule,


respectively for the small and medium configurations. So interestingly enough,


also, there's yet more evidence that you might not need position encodings if you have an


autoregressive models, which is quite astonishing. But if it's autoregressive, I can sort of


understand it because it kind of acts like an RNN and an RNN can intrinsically build a counter in


the they build a counter in inside the update mechanism. So I don't want to go too much into


the experiments right here, you can look at them, they are, let's say they they're promising in terms


of real applications. And it's definitely worth checking this out. If you are in an autoregressive


problems, though where it really shines is where you really have kind of a sequential task and need


to remember symbolic information might not necessarily be super important. But if you're


doing a really complex task, you might have to remember that symbolic information might not


necessarily be super applicable to language that has it's not really distinct symbols, right,


Neural Networks from Scratch - P.1 Intro and Neuron Code

What's going on everybody and welcome to the Neural Networks from Scratch series where


we will be creating neural networks end to end.


So from the neurons, the layers, activation functions, optimization, back propagation,


all this stuff, we're going to be coding that from scratch in Python.


Now everything we do we're going to show first in truly raw Python.


No libraries, or no third party libraries.


And then we're going to use NumPy for multiple reasons.


NumPy just makes a lot of sense here.


It's an extremely useful library.


And it'll cut our lines of our full application down a ton.


It'll make it much faster.


And NumPy is a great thing to learn.


So we will show everything from scratch in Python first and then we're going to use NumPy.


Now why would anybody want to do this to themselves?


All right.


Well first of all it's very interesting.


The idea is not, even though effectively we are going to be programming our own neural


networks framework, that's not really the purpose here.


The purpose is to actually learn how neural networks work at a very deep level so that


when we go back to whatever framework we actually use, be it PyTorch or TensorFlow Keras or


maybe some library that doesn't even exist yet, we actually understand what we're doing.


So for myself at least when I learned deep learning it was all, yes it was hard, but


at the same time everything was kind of solved for me.


How many layers, how many nodes per layer, what activation functions to use, all that


stuff.


And so when it was just fed to me I just kind of, okay, and then I just sort of memorized


like here's the activation function you use for this sort of task, here's the one.


But I didn't understand why.


And this became a problem when I tried to solve problems that had not yet been solved


for me.


So classifying images of handwritten digits, pretty darn simple.


Taking that a step further and classifying images of cats and dogs, pretty simple.


But then taking that just a tiny step further and classifying images that were instead frames


from a video game and trying to map that to actions that I want to take in the video game,


suddenly I'm lost.


And that's no good.


And there's no way to really know where to go next.


You can see that there's a problem, but actually trying to solve more custom problems is going


to require you to have a deeper understanding of how things actually work.


Unfortunately though, deeper understanding of deep learning can be very, very complicated.


So even just a simple forward pass of a neural network looks extremely daunting.


So looking at the calculation for a neural network, it looks pretty confusing fairly


quick.


You've got your input data and every unique input and every unique neuron, the information


coming through has a unique weight associated with it.


Those get summed together per neuron plus a bias, run through an activation function,


and then we do that for every single layer, giving us the output information.


From there we want to calculate a loss, which is a calculation of how wrong the neural network


is so that hopefully we can fix it.


And at the end of that, even though that was just the forward pass, that looks already


extremely, extremely daunting.


Now let's take a look at the exact same formula just in code version.


You have all your inputs times your weights and just you don't have to follow along perfectly


here, but just look at each element and see if any of these elements look like they're


over your head.


They really shouldn't be.


So inputs times weights, you can make this even simpler by doing just a dot product,


so np.dot.


Rectified linear is actually just the max of zero or the output.


We do this for all of the layers.


Then we have our softmax activation function at the very end.


And then the cross-entropy loss actually turns out to be a negative log loss, just due to


the nature of neural networks.


So this is the entire forward pass and calculation of loss formula.


I encourage you to look at all of the functions and things that we're doing here to determine


if any of this is really over your head because it shouldn't be, right?


We've got very simple functions going on here.


We're calculating a log, sum.


If you don't know what log means, by the way, we're going to explain it.


Sum you should know what that means.


Exponential again, if you don't know what that means, we're going to explain it, but


it's very simple.


Dot product again, if you don't know what that means, no worries.


We'll explain it.


Very simple.


Maximum.


This is just the max of whatever two values you pass here.


Again, very simple.


Some more dot products.


A transpose.


If you don't know it, we will be explaining it anyways.


Some sums and yeah, that's it.


None of this is over your head.


I promise you.


As far as prerequisites go, the only expectation I have for viewers is that you understand


programming and object oriented programming.


Otherwise you're going to feel kind of lost.


If you're coming from a different programming language, you'll probably find Python is very


simple.


You can probably use the programming language that you're familiar with.


Everything is so low level here that you should be able to follow along in any other language


that you want.


So feel free to do that if you want to do that.


Otherwise, if you want to follow along in Python, make sure you know the basics in object


oriented programming.


I'll put links in the description for both of those if you need to brush up.


Next is the version of Python.


Again, if you're following along in Python, we're going to do things like use f strings


for example.


So I'm going to be on Python 3.7.


The NumPy version, I can't think of any reason why it would matter.


I'll put that information in the description though just in case maybe some function does


change or something.


But like I said, everything is so low level, this series should be good for like 10 years.


Let's hope for that.


So yeah, those are the prerequisites.


It's really not much.


I don't expect anybody to have any background knowledge of deep learning.


So if you do know things about deep learning, yes, we're going to cover hopefully quickly


as possible the fundamentals just so people understand like what exactly are we aiming


for here.


And then the bulk of your understanding of how neural networks work is going to come


from us just building these neural networks.


So I don't plan to spend too much time on that.


And if things feel a little fuzzy to you, that's probably normal to be honest.


I think once you build a neural network from scratch, that is all the understanding you're


going to need.


So you're not expected to know math or anything like that.


For me, for example, in college, the only math class I took was called math fundamentals.


And I don't think we did any math calculation at all in that class.


It was definitely a joke.


So if I can do this, I know you can do this too.


If you do want to brush up on math, like the Khan Academy, like linear algebra and the


calculus stuff could prove useful.


But I wouldn't even suggest you go through the full series on either of those topics.


You can use those just to kind of like spot check issues that you still find confusing.


With that in mind, this series is also provided in conjunction with the Neural Networks from


Scratch book.


We are going to be covering the same material for the most part.


The book might be a little more verbose.


But the series is obviously free.


The book has various prices depending on if you want the e-book, softcover, hardcover.


We ship everywhere in the world.


Access to the book gives you access to the e-book.


So whichever version of the book you buy, you always have access to the e-book.


That gives you the opportunity to access the Google Docs draft.


Currently it's in draft form.


At some point it won't be a draft anymore.


But you can highlight, post comments, ask questions, stuff like that in line with the


text.


Also, if you're impatient you can get access basically even though it's in draft right


now it's complete from basically end to end of training in the neural network and we're


doing testing right now.


So we're already obviously quite a bit ahead of where the videos are.


So if you're impatient you can also access information a little earlier with the book.


Also if you just, I would use the book as either review or the videos as reviews.


I would maybe read the book before you watch the video and then use the video as kind of


a review of what you learned in the book or vice versa.


Watch the video, follow along and then read the book.


Use that as kind of a way to solidify what you learned.


So this is a topic that you're not going to blow through this in a weekend.


It's going to require multiple sittings, multiple environments and ideally multiple mediums.


So if you're interested in the book you can get that at nnfs.io.


So we call these neural networks because they look visually like a network.


You've got your neurons which in this case are the blue circles.


They're connected via those orange lines and in this case we have basically you've got


your input layer, two hidden layers of four neurons each and then your output layer.


Now data is going to get passed forward through this so it starts at your input layer.


So in this case we only have two pieces of data that are going to come in.


That gets passed forward to that first hidden layer then that hidden layer passes data to


the second layer and then finally to the output layer where we hope that it will output something


that we want.


So for example based on some sensor data maybe we want to predict failure or not failure.


You could either have one neuron or in this case we have two neurons so the top neuron


might be a failure neuron, the bottom neuron is a not failure neuron and depending on which


one has the higher value that's the supposed prediction.


Now the end goal of neural networks like most machine learning is to take some input data


and produce output data that is desired.


In this case we've got images of cats and dogs.


We hope that we can pass it through in pixel form to our neural network and if it's a dog


then that final output neuron on top is going to be the strongest.


If it's a cat then that final output neuron on the bottom will be the strongest.


And we can do this by tuning the weights and biases so all those unique weights and biases


we do that by tuning those and that is the actual training process is tuning those in


such a way so that hopefully we can take data that this neural network has never seen, give


it pictures of cats and dogs that it's never seen and have it accurately predict those.


So how and why do neural networks work?


Well if you just look at them and really consider what's going on every neuron is connected


to the subsequent layer of neurons in full so each of those orange lines that connection


that's a unique weight and then every neuron is a unique bias.


So what this ends up giving us is a huge number of uniquely tunable parameters that go into


this gigantic function.


So for example with 64 by 3 hidden layers here we have 9,164 tunable parameters in this


gigantic function and each of those parameters impacts the outputs of the next neurons and


so on.


And so what we end up having are these very complex relationships that can in theory be


mapped so to me the impressive thing of neural networks is not necessarily all of those connections


it's not really complex to understand.


The hard part of neural networks and deep learning is figuring out how to tune such


a thing.


Alright since I think it would be lame to not post any code at all in this first video


we are going to begin to code our first neuron but first I want to go over the version numbers


real quick because quite frankly I'm going to forget to put it in the description.


So I'm using Python 3.7.7, NumPy 1.18.2 and Matplotlib 3.2.1.


Again all of this should work very far into the future but just in case not there are


the exact versions if anybody needs to have the exact version to follow along.


Now let's go ahead and begin to code.


Just to say it I am using Sublime Text you can use whatever editor you want everyone's


got a really strong opinion on editors.


I'm going to be using Sublime Text there might be a time we use Jupyter Notebooks who knows


I'm going to use whatever makes the most sense to me at the time which may not make sense


to you or me later.


So let's pretend we are coding a neuron that is somewhere in this densely connected feed


forward multi-layer perceptron model using all the big words.


Don't worry about knowing what those words all mean by the end of this you will know


all of those words but part of the problem with learning deep learning is people use


the same three, four plus words for the exact same thing and it can be very daunting.


So for now we are just going to call this a neuron it's somewhere in our neural network.


Now in this fully connected neural network every neuron has a connection a unique connection


to every single previous neuron.


So let's say there's three neurons that are feeding into this neuron that we are going


to build.


So we don't know much about those neurons but we know that they are outputting some


values.


So first their outputs become the neuron that we are coding inputs.


So we are just going to make up some numbers we are just going to say 1.2, 5.1 and 2.1


those are the unique inputs so these are outputs from one, two, three neurons in the previous


layer.


Every input so every unique input is also going to have a unique weight associated with


it so we are going to say weights and you should already know how many weights are we


going to have.


Well since we have three inputs we know we are going to have three weights so 3.1, 2.1


these I am just making up these numbers.


It's just for beginning to code how neurons are going to work.


So you have your inputs, your weights and then every unique neuron has a unique bias.


So we are going to say bias equals 3.


So now the first step for a neuron is to add up all the inputs times the weights plus the


bias.


So this is relatively simple.


So in very very raw python no loops required at this stage.


We are just going to say basically output so far for this neuron is going to be inputs


0 times the weights at the 0th index plus and then I am just going to copy this I am


going to paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste,


paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste, paste,


paste.


And then we just need to add the bias.


And you can feel free to make that on new lines if you want, if that is bothering you.


But that is basically so far there will be other things that will happen soon enough


but so far that is the output to our neural network.


So we can just print the output run it 35.7.


If you are new to sublime text for some reason control B to run but you will have to set


up to run with python whatever version of python you want and all that.


I do kind of again expect that people know how to work with the programming language


but if not please again feel free to either comment below, join us in discord that is


discord.gg slash syntax and we will be happy to help.


Anyway that is it for now.


If you have got questions, comments, concerns, whatever moving forward feel free to ask them


below otherwise we are just going to keep slowly chipping away at neural networks from


scratch.


As you can see this was pretty darn simple so far and for the most part it is just adding


a bunch of as we have broken it down so far we are going to break it down to the point


where every little additional step is going to look an awful lot like that.


There is a couple of points that may be a little more challenging but the goal is to


break it down so much that it is painfully simple.


SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)

Hi there, today we'll look at SpineNet learning scale permuted backbone for recognition and


localization by Xianze Du et al of Google Research. On a high level, this paper proposes


to take current recognition and localization networks which have a CNN backbone, usually


something like a ResNet, and switch up the order of the blocks in the ResNet and cross


connect them in a different way, such that they reach a higher accuracy with the new