finite, then any sequence generated by the iteration equation three converges to some fixed


Hopfield Networks is All You Need (Paper Explained)

point. So, you know, basically saying that here we have this is not the loss, I'm sorry. No, this is


the domain. Never mind. I'm an idiot. This is basically saying that this algorithm will converge.


And they define here what it means for a pattern to be stored and retrieved. And that's for


establishing what the kind of storage capacity of a Hopfield network is. So we've established


that the update rule minimizes the appropriate energy, and the update rule will converge at


some point, which means that we can, you know, if it converges, we can retrieve the pattern that it


converges to. So now we define how many patterns can we actually store. For that, we need to know


what does it mean for a pattern to be stored. So we assume that we have patterns and these patterns


are called x, okay, x i, we have n different patterns, each one is called x with a subscript.


We assume that around every pattern, a sphere is given. So how do we imagine this? We have these


patterns, and this is this is just a space that they consider patterns of the on the sphere, but


we'll just conceptualize it as this will have a space, there are patterns we want to store, okay.


And we'll say around every pattern, there is a sphere. Okay, sphere like this. And naturally,


the patterns are going to be, there's going to be a notion of well separated patterns. And you can


imagine this a little bit like these spheres won't be touching each other. If these spheres aren't


touching each other, that means that the patterns are kind of well separated. And that means that


if we initialize the query, remember, the query here is a vector that kind of sort of looks like


a pattern. And that means the query is kind of close to the pattern in some notion of distance,


so if we initialize the query somewhere in that sphere, then it might if it converges to that


sphere to that pattern, then we retrieve the pattern. Okay. Now it gets a bit more complicated


than this, but not much more. We say a pattern is stored. If there is a single fixed point inside


the sphere, to which all points that start inside the sphere converge, and none of the spheres


intersect, so the sphere of point i doesn't intersect with the sphere of point j. So that's


where we say all these spheres are non intersecting. We say xi is retrieved if the iteration equation


three converged to the single fixed point in that sphere, the retrieval error is the distance. So


you'll notice you have two things you have xi, this is the actual pattern, and you have xi star,


this is the retrieved pattern. So these hopfers, they don't always have to give you the same thing


that you stored. That's part of the nature of continuous neural networks whatnot. So for every


sphere, we say there is a pattern, there is a sphere. Now, we as pattern is stored, if every


I can start wherever I want in this sphere, okay, wherever I want, it will always converge to a


point that's inside the sphere. Okay. And maybe that point isn't the pattern that I stored, but


actually this point right here, but wherever I start, I will always converge to that particular


point. If that's the case, then I have stored this particular pattern. Now, the fact is, I don't


retrieve this particular pattern, I retrieve the blue thing. But I can then store it in the


blue thing. But I can then define the error of retrieval. And the error of retrieval is simply


the distance between the two things. Ideally, this distance is very small, right? But you know,


we can't, can't guarantee it. Now, there are going to be theorems that deal exactly with this


retrieval error. But first, you can see that here, if, if these spheres become larger, you can't


accurately store a pattern anymore. So this is the kind of ideal situation. But there are also


situations where these spheres, you know, if I have these patterns right here, these spheres are


so large, kind of the, the attractions of the patterns are so large that if I start, let's say


here, then I don't converge to either of these two patterns, I converge to like something in the


middle, I converge to maybe this point right here. And that's going to be one of these meta stable


states, okay, we're going to encounter situations like this, but we're also going to encounter


situations like this. And the bottom thing isn't necessarily bad. And that's what you have to keep


in mind. And yeah, as I said, we'll get to it, but just keep this kind of sphere image in mind. Okay.


So first, we'll just deal with the, you know, the up the top situation where we store patterns and


then retrieve patterns. So we'll assume a failure probability, which is p, and p is going to be,


you know, pretty, pretty low for their example. So they have p equals 0.001, you know, like a


0.1% error probability of retrieving your pattern, things like this.


And randomly chosen patterns on the sphere with radius m, we define some constants,


yada, yada, yada, then with probability, one minus p, the number of random patterns that


can be stored and stored in the sense of having these spheres around them so that you can retrieve


them accurately, or at least you can retrieve something that's close to them is bounded lower


bounded by this quantity right here. So there's the square root of p, there is this constant c,


but then you see that d is in the exponent right here. So that means it's exponential in the number


of dimensions. So that's, that's pretty cool. So if you add a dimension, you exponentially increase


the number of the number of patterns you can store. And no, that's, that is a kind of, I mean,


it's, it's been known for modern Hopfield networks with binary strings. So it's not uber surprising,


but if you have, you know, it's not what you would imagine, like that. Okay, so they may give a few


examples of these, you have to set these constants, you know, in a particular fashion, such that this


is given and so on. But they say, examples here are where c is something like three, and d is 20. So


if you were to add a 21st dimension, then your, I guess, storage capacity would increase by a factor


of three, which is pretty cool. Alright, so this is how many that we can store infinitely,


not sorry, exponentially many patterns in these networks. Now, they deal, they say,


the next theorem states that the update rule typically converges after one update, if the


patterns are well separated. Okay, so if we're in a situation where these patterns are well separated,


which is kind of like this, but you can also imagine this in terms of dot products, because


we operate in the space of dot products. So if the patterns are well separated, that sort of means


that they all kind of sort of point away from each other. And this notion of separation is going to


be captured by this quantity right here. This is the separation of example of pattern i, which is


just the inner product with itself, minus the maximum inner product with any other pattern.


And this quantity is going to be large when no other pattern is close to it. So when the separation


is large, then the update rule, the retrieval rule of calculating, you know, if a query calculate


the inner product with all of those, then I reweigh all of the patterns by that inner product


by the softmax, then I use that new thing as a query again, and so on, as we discussed, it will


converge to the closest pattern. But this theorem says it actually converges pretty fast. And here,


I have my problems with saying that it converges after one step, typically converges after one


update, because that, you know, generally depends on a lot of constants, as we'll see, but it does


converge exponentially fast in this separation constant. As a theorem for says, with query Xi,


after one update, the distance of the new point to the fixed point is exponentially small in the


separation delta i, the precise bound using the Jacobian and its value in the mean value theorem


are the following. So here you can see, this is the distance between the updated Xi,


the updated Xi after one step, and the and the fixed point right here, this is what it converges


to is going to be the distance as it was before times this thing right here. So you can see,


since this is a this is a multiplicative update. And in this Jacobian, so this is expanded down


here, this is this. You can see here, you have the you have this, sorry, yeah, this is this. So this


is bounded by that, you have the exponent, the exponential function, negative, this separation


right here. So the higher the separation, the faster this algorithm converges, okay. To say


that it converges after one step is, you know, might be a bit of bragging. I don't know if this


is a common thing, if you have like an exponential convergence that you are allowed to say, it's


after one step, I'm not sure, especially what I'm not sure about is that you have n here as linear


constants in that factor, okay. So if you if you and that's what they do in their code. So if you


look at their code and the codes available, which is pretty cool, it's implemented in PyTorch as a


general module that can you can just drop in. So this is not only for transformers, this is for


you can replace like LSTMs, you can replace pooling mechanisms, you can, you know, do a whole


bunch of stuff in their paper, in the accompanying paper, they do this multi instance learning with


giant sets on using these Hopfield layers. So pretty, pretty cool. This code is definitely


worth kind of checking out and maybe you want to replace some stuff with you. But the question is,


how many of these update steps should you do, right? Because we looked at the diagram, at least


in the attention mechanism, it seems like you have attention layers, right, you have a transformer,


and the transformer consists of you know, you have this input right here, and you go through layer,


layer, layer, layer, layer. And in each layer, there's contained in it, and one of these attention


mechanism, right, this entire thing is in this layer, okay. And now if you interpret this as a


Hopfield network, and you want to do multiple steps, that means you go this branch right here.


So in each layer, potentially, you do multiple steps of these things. So for whatever computational


constraints transformers had already, this will certainly make it worse. But also you need to


decide how many steps you want to do. Now you can hard code that, of course, but they say,


you should do these steps until this norm here, until the norm between the old and the new is


small enough. So where is that? So you can't measure how close you are to the convergence


points, right, because you don't know in practice. But you can measure how far you're away, you can


measure, where did we have it, you can measure this quantity right here, that's something you


can measure how far two iterates are apart. So what you'll simply do is you'll measure that. And


if that is small enough, then you'll, you'll stop. But that, I guess, is very related to this. So how


if you, we've already proven it converges to this x star. So I guess, we can approximate this


quantity right here with the quantity above. And that tells you how many updates you need to do.


And that quantity is linear, not only linear, but actually here quadratic in n, I don't care,


you know, yes, it's exponential in the separation. But it's quadratic in n. And if I've learned


anything from kind of my fast code courses, is that constants actually matter when you're not


dealing with infinity with an infinite number of steps. So the number of the number of steps you


need to do, I guess, will depend on the sequence length in a quadratic fashion. So I'm not sure


you can always claim this is converges in one step. Now I might be super mistaken here. And


none of this will can, none of this actually makes a difference in the in the light of the


exponential decay here. But I would just, I'm just a bit worried saying this usually converges in one


step. It's clear, I guess, why they do it, right? Because the attention mechanism in transformers


is a one step application of this rule. And this here is kind of a theoretical justification


for interpreting this precisely as a Hopfield network, because you'd say, well, in a Hopfield


network, you would do multiple steps. But wait, wait, we can actually prove that even if you interpret


it as a Hopfield network, you it can it usually converges after one step. So what you're actually


doing in a transformer is applying a Hopfield network update rule to convergence. So yeah,


I'm not Yeah, I might be bickering on a high level here luxury problems. Theorem five then


says so theorem four is how fast does this converge? Theorem five, the last theorem right here,


says that the retrieval error of a pattern, then so this is the this is what you converge to. And


this is what you've stored is bounded by again, something that's exponential in the separation


right here, as you can see. Okay, so that was the theorem. So if we go quickly through them, again,


theorems one and two deal with the convergence of this algorithm and the fact that it actually


minimizes the proposed energy, then theorem three says you can store exponentially many patterns


in terms of the dimension of your space. And theorems four and five say that this update rule


will converge exponentially fast after after one step if you believe that, and the retrieval error


will also go down exponentially fast with the number of update steps that you do. Okay, that


sounds pretty, pretty, pretty good. But we've heard it, it's very dependent on how well


separated these patterns are. And it turns out that it's, you know, at least in transformers,


they aren't always well separated. And that might be on purpose. Remember, the the states here,


the patterns aren't pre stored, like in a classic Hopfield network, but the patterns,


if you interpret an attention mechanism as this, are also generated by the network itself. So the


pattern matrix that you retrieve from, and the query are generated by the attention mechanism.


In this case, as I said, this is applicable to many, many more domains than just this. But yeah,


so there's another slight modification that you have to do to make this actually equivalent to an


attention mechanism. And that is, you'll have to recast the value, because usually what you'll do


is you have some sort of input, and then you make queries, keys and values from that using different


heads. The only thing to make it formally equivalent is you have to make the values


generated from the keys. So the keys give rise to the values, as you can see right here,


you first multiply with the key matrix, and then with the value matrix, I think that's,


you know, that I don't, I doubt that this will change anything. If you, the only way that could


really change anything is if this matrix here would be super low rank, like collapse the space


of into like very few dimensions, which the value matrix wouldn't do. So, you know, but just letting


you know that the technical equality requires this slight modification. Okay, now we said that


it might not, you know, be that this is always super well separate and you retrieve a single


pattern. And that's what they research here in a pre trained BERT model. So they take a pre trained


BERT model from I guess from Hugging Face, and they run, they just run a data set through it.


And what they do is, so for each, for each query, and sorry, for each attention head, because you've


multiple ones of these attention heads, right? In each layer, so each layer, you have multiple ones


of these heads for each head, they look at over the course of the whole data set, how do these


softmax distributions look like? So when you believe that this is a Hopfield network, and you


believe that this converges in one step, then if the patterns are well separated, what we would


expect is a distribution, as we said, like this, okay, there will be one dominant pattern that you


retrieve, you know, that's what you want to retrieve. That's what comes out, but a bang.


You retrieve that accurate pattern, anything else would mean that the Hopfield network sort of


failed, right? It wouldn't give you back one particular pattern. So they have, I think that's


a pretty, it's a pretty smart experiment, they look how many bars do we need to add? How many of


these bars in the softmax distribution? Do we need to add to reach 90%? It depends a bit on the


temperature of the softmax, which is hard coded in attention mechanism, bdb is one, this squared over


D. So they say, how many do we need to add to get to 0.9 to 90% of the mass of this distribution?


And if this is the Hopfield network, where you retrieve one pattern, then one will be enough,


right? One of these bars will probably be, I don't know, like, a, a, a, a, a, a, a, a, a,


99%. Okay, but there are other cases, imagine the case where the patterns and the query you retrieve,


the spheres that it gives rise to are all like overlapping, okay? So what that will do is,


it won't converge to any particular pattern, but the attractor space in this kind. So you can


imagine if you have two spheres that are apart from each other, the update rule converges


either. So if it's closer to here, it converge here, if it's closer to here, it will converge


here. But if they are overlapping, like this, the energy landscape will actually make it such that


it will neither if it starts somewhere, it will neither converge to here nor to here, it will


actually converge to somewhere in the middle, okay, into the mean of the stored patterns.


And if we take that to the extreme, what could be is it could be that the softmax distribution looks


completely uniform, okay, which would basically mean that, you know, I don't care where my


information comes from just average. And this has its applications. So if you, for example,


want to make a sentiment classifier, very cheap way to do that is to simply take pre trained word


embeddings like glove or word2vec, you know, assign each word word embedding, and then just


average the word embeddings, okay. And you count on the fact if there are a lot of kind of negative


words in there, like bad, sad, angry, the word embedding kind of will, you know, reflect that,


and the average word embedding will point more into the bad direction. And if there's a lot of


happy words, the average will point into the happy direction. Okay, so there are applications of


averaging information, not caring particularly where it comes from. And in that case, what we'd


expect is that this number and we'll call that, so we'll call that the number k. In this case,


it equals one, but in this case, k equals, I guess, n, the number of inputs, okay, because we


need, well, not maybe n, but, you know, approximately, we need almost all of them to reach the 90%. Okay,


and there is an in between, and these are called these meta stable states, where in the in between


is something like, you'd have a couple of patterns here, a couple here, and a couple, maybe here.


It's almost like a clustering, like, and these overlap, and these overlap, and these overlap,


but they don't overlap with each other, which means that if you start somewhere here, you would


converge to the mean, but not to the mean of all the patterns, but just to the mean of these


patterns, and here, here, and here, here. So this, this is like a clustering in latent space, right?


So you can interpret these Hopfield update rules as somehow, you know, getting not going to a


particular pattern, but going to sort of a cluster. And this is, if you ask something like, hey, is


there any adjective around, right? And all of these patterns, they kind of overlap in that space, in


that query space of adjective, they overlap, and therefore, the update rule would converge to sort


of the mean, which would basically say, yes, there is an adjective here, right? And the information


would not be routed, so that the distribution, if we start here, right, and we converge to this,


the distribution would look something like small, small, small, and then you'd have a couple of


large ones, right? You'd have like, maybe two or three or four of large ones, and these would


exactly correspond to the patterns here. So the information will be routed from all of those in


that cluster to this particular note that asks the query. Okay, these are these are what's called


these meta stable states. And what they do is they calculate over the entire data set this number k,


and here they show you the distribution. So in these plots, what you'll see is over the entire


data set, k goes into that direction. So I guess, let's go to tis here, this is the tis


here, this, this seems pretty easy. So k is in this direction. And this is simply the amount of


like how, so in each, you let a data point run through it, you measure k for that particular


layer one, you see this is layer one, head four, okay, this is one layer one attention head.


And then you can see that the number k is distributed like this. Okay, so contrast this to


this head right here, where it's a lot of weight on the number one or like very few numbers, okay.


So these blue ones would be these are your typical like when you retrieve one particular pattern. So


this attention head, we can sort of conclude, in this particular attention head, this is very


specific, it looks at its input, they look at its token, and it decides what information do I want


and it retrieves one particular thing from the other nodes. Okay. Whereas here, it's more like


kind of an, an averaging is more like, I want this kind of information. And on average, I don't even


know what the sequence length is here, I guess it's maybe 512. So of the 512, the median, this


number is always the median, and median, it collects information from 231 of them. Okay, so


you can see that this corresponds these green and orange ones correspond to these meta stable states,


where there's kind of an implicit clustering done in the in this space of attention, whereas the


blue ones, they correspond to attention has that ask for particular information retrieve one


particular maybe few patterns, and happy with that. And the red ones here, you can see that


they often just average, they just, you know, because K is so high means that I need all of the,


I need all of these bars to get to the 90%, or I need almost all of them, which basically means


it's a uniform distribution, right? So it's like, I don't care where information comes from just


average, whatever average, I just want the average in some particular space. And, as we said, that


also has its uses. Interesting how this translate through. So this here is as we go down the BERT


model on the bottom, you have layer one, you see there are a lot of these averaging operations


going on. So a lot of the heads are simply doing averaging. And as you go up the layers, the heads


get more and more specific in the types of information they seek. But then again, in the


last layers, interestingly, you get into a lot of these meta stable states again, which, I guess,


again, interpret this as you as you want. I'm going to leave this up to you. But sort of says


like here, you want kind of general patterns at the bottom. And then the middle layers are kind of


the logical workhorses. So you look for very specific things in the input. This is, I guess,


this is where I guess this is where the thinking happens. So this is sort of pre processing.


I'm just making stuff up here, by the way, this is this must be a no way true. This is maybe thinking.


And this, this here, this might already be output again, because you know, after that,


you have language modeling or classification. So this might already be like aggregating types of


information. This is how I sort of interpret it. Okay. Yeah, so so this these these experiments are


pretty, pretty, pretty interesting. And here they have, they do. These are the last experiments for


this paper. They do an interesting experiment where they actually replace the attention heads


replace the attention heads by simply an average mechanism. And later, they actually replace them


by Gaussians. But in this case, they simply average. And they show that look, if I replace


layer one with this averaging, the perplexity doesn't rise that much. So it's pretty good.


Even if I replace an entire layer here with averaging, it perplexity goes more up. And you


can see the corresponds if you remember the previous plot, the correspondence is pretty


one to one with how much blue and green heads there are as in contrast to how much red and


orange ones there are. So here you have lots of blue ones. And you can see that the error


kind of goes up. And interestingly, here, you have more meta stable states at the end, but still,


the perplexity goes up more. So I guess you can only really replace the red ones with the averaging.


So this is always averaging in one particular layer. And they go into more detail here,


where they say, look, this is this is layer six, and this is layer 12. So this is one particular


attention head from layer six and layer 12. And the updates don't be confused, it goes in this


direction. Okay, I was confused at first. And you can see right here, this number k at first, you


know, it's kind of spread out. But then it pretty quickly converges to a very small number. And


there is this kind of point right here, I don't know if the learning rates decrease, I don't


think so. I think that's just kind of a, a phase transition right here. This is the blue line,


by the way, the blue training line, a phase transition, where all of a sudden, these just


these attention heads, they somehow decide, okay, this is the thing I want to specialize in. This


is the type of task I like a sub task of linguistic sub task on specialize in and then they concentrate


on one particular pattern per input. So they are really specializing. Whereas in the last layer,


you see here, that even during training, they are sort of continuously learning. So first,


they also do this averaging, then they go into this meta stable region, right? This is this


meta stable region, k isn't one, but also k isn't a very high number. So they continuously learned.


And it's even indicative of this training might not be done here, first of all. And second of all,


it would be really interesting to see how this works out with, you know, sizes of transformers,


and like, especially these, these huge transformers, just the fact that they can


keep learning, the more we train them might be, you know, be interpreted in the light of


what kind of states they converge to. And the fact that their attention heads, I don't know,


how does this go on? Do they stay in the meta stable states? Because it makes sense to have


meta stable states, as I said, it makes sense to kind of cluster things? Or are they simply


into is this simply an intermediate step? And if you go really far down, they would actually also


converge to the k equals one where they really specialize? Or maybe do we need more attention


heads for this? I don't know. It's just, I think this is just the the beginning of kind of research


in this direction. I think just this kind of number k, how it's how it's made, it's pretty simple.


And apparently, it's pretty, pretty revealing. So, you know, that's pretty cool. So that was the


paper. And it's experiments. It's it's a pretty sizable paper, as I said, even the paper itself


is 10 pages. And then there is this immune repertoire classification, which I will like


spend one minute looking at it. So you have you have these set classifications. So for each human,


you obtain a set of immune receptors. And you simply obtain one label whether that human is


immune to a particular disease or not. And your task is kind of and then a different human has a


different set, you have no idea which one of these things is responsible for it being for the human


being for the human being immune or not. In fact, there is a, you can't even decide based on these,


you can only decide based on like sub sequences of these and they might be in combination with each


other. So there might not be a single one responsible, but like a combination, but you don't


have labels for the individual ones. And you have different ones per human, then they are different


length. All of this is just a giant, giant task. And you have many of them, you have 10s of thousands


of thousands per human, right. So they build a system here where first they do this 1d convolutions


to process the inside sequences. And then they do this hopfield attention mechanism, all with with


learned queries over these things. And then they train on the output label. And surprisingly,


that actually works even with 10s of 1000s of inside sequences and only one label for all of


them. And so they achieve, I guess, favorable results compared to other baselines on this task


using these hopfield network, which is pretty interesting. But I'll let you look at that paper


yourself. So I hope this somehow made it a bit clear what happens here. And it would actually be


pretty interesting if we, you know, to see what happens if we just do maybe two rounds of these


updates. Is this even desirable, right? Is it desirable to run this to convergence? Is there


something good about not running it to convergence? Or does it actually not matter because it


actually does converge in one step? I don't know. But have a look at the code. It's pretty cool. And


I hope you enjoyed this video. I'm sure you have many open questions as do I. Don't hesitate to


ask me in the comments or join our Discord as I said, there are lots of helpful people


Thinking While Moving: Deep Reinforcement Learning with Concurrent Control

Hi there. So if you look at these two robots, the left one labeled blocking, the


right one labeled concurrent, the blocking robot as you can see always has


these little pauses in its movement where it does nothing and then it kind


of continues with its motion while the one on the right is one continuous


motion that it does. So the reasoning here is that the robot has a camera and


the camera takes some time to register what's going on and then also the robot


has a computer inside and the computer also takes some time to decide what to


do based on what the camera saw and while all of this is happening the robot


on the left just freezes so it performs an action and then it freezes because it


takes time to register a state and compute a new action whereas the robot


on the right it takes the same amount of time to do these things right it also


takes a time to register the state and to compute an action but it does that as


it is executing the last action so it does this in parallel and then it


executes the action once it has computed a new action it executes that new action


right on top of the old action and that gives this one big fluid motion so this


requires a new formulation of reinforcement learning and that's what


this paper does thinking while moving deep reinforcement learning with


concurrent control by people from Google Brain UC Berkeley and X so I they have a


nice diagram here in the supplementary material to show you what is going on in


their framework so in classic reinforcement learning right here in


classic reinforcement learning you have this dichotomy between agent and


environment right so the agent and the environment now the agent is supposed to


kind of act in the environment in the following manner the environment will


send an observation to the agent the observation in this case is the picture


of the of the camera so it sends an observation and the agent will think


what to do with the observation which is called a policy the policy pie will take


an observation and then output an action of what to do and will send back the


action to that environment right so in classic RL you assume that this part


here is kind of freezes time so the environment will output an observation


and as and to process of registering the observation of computing the action and


of sending the action back is happens in zero time of course it doesn't happen in


zero time but in our reinforcement learning problems for example the open


AI gym the environment stops until it gets the next action and then it


performs the action right it performs the action in the environment and by


that the environment changes and time happens and then it stops again as we


think of the next action right so this is we usually call this one step in the


in the kind of classic formulation of RL the only point that time happens is


when the action is executed no time happens when the state is registered or


when the action is computed and that's what you see here on the left so in blue


you have the state registration this is for example the camera the camera has


some time in order to register and store the image that it has taken right maybe


post process it a little bit so that's what the camera does but in our classic


formulation as you can see here if this is time it happens instantaneously all


at the same time and also this is the policy this is thinking what to do right


this is your evaluation of your neural network if this is a neural network


drawing a small neural network here this happens instantaneously in these


formulations and only as the action is executed time happens right and then


until here time freezes again and only once the action is determined time


happens again in the new formulation now as you've already seen what's what we


have here is this this kind of continuous framework where let's say


you're here it actually takes time for the camera to post process the image it


takes more time for you to think about what to do and then once you decide on


an action that action is going to happen right but we can we can say for example


at this point you tell the camera to take a new picture of the state right


but that takes time and while that's happening the old action is still


ongoing right so you don't even even have to say the action is still ongoing


but the world is still moving right the world is still changing while you think


while you post process and while you evaluate your policy the world is


thinking and only after some time right after this lag time here have you


decided on a new action and then you can break that old action and kind of


perform your new action and all of this is happening in time so this this is the


new framework now you see the problem here the problem is that you base your


decisions on the state and time t on time sorry time h here you base your


decisions on a state as it was at that time right that's what you used to think


right here that's what you store and think about but you perform so you


perform the action at this point in time so there is a considerable difference


here because the world has now changed so you see the problem the action you


perform is based on an old knowledge of the world and you have basically no way


of making the action dependent on the current state of the world because that


would require you to capture the current state and that takes time and in that


time the world has already shifted again so the agent is kind of required to


think ahead about the action that it is currently performing and how the world


changes according to that so this new formulation of reinforcement learning


formulates this in a in a formal way it formulates it in a formal way and will


quickly go through that yes so they go into the very basics here will quickly


quickly go through them so they introduce these quantities like the


policy pi the transition distribution the reward and the Q and value function


now we'll just quickly go over these so you have the agent sorry about that you


have the agent and you have the environment hello so if you think of the


agent and the environment the environment has this transition function


transition function the transition function it takes it says okay I'm in


this state and the agent does this action and here's the probability


distribution over the next state right so it says that your little spaceship


is here right and the meteors are here and then you push the button if if you


push the button for shoot then you'll be in the same place the meteors will still


be here but you'll have a little shot coming out of your spaceship that's what


the environment does right so you give it a state and action and it will give


you the next state it will also give you a reward right the reward in the same


thing reward here will be a second output here that tells you either let's


say negative one if you're you die or zero if nothing happens or plus one if


you shoot a meteor that's the reward so this you can think of as the real world


so these two quantities are in the real world in the environment so that's how


you model the environment then the agent has these quantities called the policy


pi so pi what pi does is much like the transition but pi takes in a state and


gives you an action right so this is now the agent deciding this is thinking the


policy takes in a state and gives an action and this this contains various


forms but it's just a function for now the agent also has a Q and a V function


and these are quite quite similar so the Q function what the Q function will do


if if you are in a state and you have several options of what to do right you


have action one action two and action three right you're in state s the Q


function of s and a one superscript pi would tell you the following it would


tell you what's my expected reward if I'm in state s that's here and perform


action a so a one so if I now take this path and after this path I follow the


policy pi right the policy pi for each of the following so it's like right now


I take action a one I don't care about my policy but after that I follow the


policy pi what is my expected reward going to be until the end of the episode


that's the Q function and the value function here very similar but it only


cares about the state it says if I'm in state s and I just follow the policy pi


even in the first step right I just follow this policy pi what is my


expected reward going to be over the course of the episode that is the Q and


the value functions you can see why Q learning is popular if you have a good


Q function and the Q and the value function these are the things that you


actually want to learn right if you have a good Q function you can simply always


plug in every action into your Q function and then simply take the maximum


the action that has the maximum Q value because that will can't that will give


you the best reward if if your policy pi right it's kind of self-referential if


your policy to is to always take the maximum Q value then taking the maximum


Q value with the policy given that you take the maximum Q value will be optimal


all right this was very convoluted but all right so let's let's start off with


modeling the environment in this continuous framework so instead of


having the next state be determined by the current state and action in the


continuous framework they do this via differential equation so the DS is how


does the environment change this is the change in the environment that is


determined by two functions F and G so F is your classic environment function it


takes in a state and an action at time T right these are now functions and it


will it will output how the state changes and the G here is this is a


Wiener process is to introduce stochasticity as I understand it because


in the classic formulation the transition model gives you a probability up here a


probability distribution so this Wiener process is responsible for


introducing that probabilistic nature into this differential equation but


ultimately it simply tells you how does the state change depending on my state


current state and action that I perform so the reward function now is also


pretty simple the tau here is a trajectory and the trajectory is simply


the state and action over time so if I integrate from time zero to infinity or


to the end of the episode my reward function at each point in time right so


I go through my episode and I get higher reward not so high and so on so the


integral under this curve will be my total reward just like we sum up the


reward of individual steps in the discrete case in the continuous case you


can think of each in infinitesimal time step giving you a tiny bit of reward so


the entire reward is just an integral then we go on the value function for a


given state at time T right so think about what this is the value function


for a state means what reward can I expect starting in this particular state


and then following policy pi until the end of the episode and that here is the


expectation over all trajectories that come from my policy of the reward in


that trajectory so I can you know if I'm here my policy now is also a


distribution it can go multiple trajectories right and I want to I want


to have the expected value of the reward so each one of these has a reward the


expected value of the reward over all trajectories starting from state st


right and again here you say that that is the integral over the now here I have


a bit of a problem because here they say T equals zero going from here and here


but here the T is already here so I believe this should be this should be T


equals T prime and then T prime T prime T prime up here and T minus T prime or


something like this in any case I think it should actually start from this state


here and not from time zero but I might be missing something I'm not the biggest


integrator in the world so you know all right then you have the Q function now


think of it what the Q function is in the discrete case the Q function tells


you if I'm in state s and perform action a what is my expected reward going to be


that I have to introduce some different things here they say if I'm in state s


and I act action a at time T until time H right now you have to say how long


you're gonna perform the action for until you perform the next action right


so H is your your lag time here until you perform the next action so this now


I actually agree with this formulation with the integral here so this is going


to be the integral from time T to time T plus H that's how long you perform the


action your reward of performing that action right given the state plus the


value function at the end of that so you're here you're an ST and you


perform action a right and then this is your state at time t plus H and then


you're here and from there on you could perform many many many actions right but


in the original notion of the Q function the Q function tells you if I'm here and


I perform this action and after that I act according to pot to policy pi what


is my what is my expected reward and there's a classic recurrence relation in


reinforcement learning where you can say the Q function in ST given to a is the


reward that I get from performing a in state s plus the value function at state


s at this at the next state right because the value function is exactly


the reward that you would get by following policy pi in that next state


and the Q function means I perform a now and after that I perform pi so this is


the continuous analog that's why you have this part here where you perform


the action for H time and after H time you just go after go with your policy


and that will be the value function so this is the continuous formulation of


the of the problem right and now they can introduce these these lagging times


so in their diagram up here they define these notions so you have your state st


right here then after this time you capture the new state right so after


that time you capture the new state and decide on an action and then you perform


it for H time is that correct until here so the the the I minus 1th action is


performed at this time and the I've action is performed at this time no


that makes no sense so let's read it ah so this is when you capture the state


and you need the time to perform to to think right this is thinking and then


you perform this action at that time this is the lag time now and you


perform this action you want to know you want to know if I perform this action


until this time here what is what is happening so this is the new Q function


takes into account this thing it tells you if I'm in state s and I think this


is thinking leads me to here this is the old action right this is the old action


that's still happening while I observe this state right so it means if I do


this right now and after thinking I do this right so I'm at state I'm at time t


and this is still happening and then after I think thinking leads me here t


plus TAS I perform this new action I'm out of colors I perform this new action


at that point until time H what's my Q function so my Q function is going to be


the integral time T where I start observing the state and start thinking


until T plus TAS that's when I still perform the old action right so this is


going to be the reward in the state given the old action and then at that


time I switch over to the new action right so at that time until time H now I


perform the new action so this entire part here this part until here is taking


the place of this first part here in the Q function of this first part right so


because before it was simply executing one action we didn't have this


concurrency yet so executing the action and after that it's going to be the


value function and now it's executing two actions first execute the old action


then once you're done thinking execute the new action and then it's the value


function from there on I hope this is clear it wasn't clear until just now as


well all right so they define the Monte Carlo estimator where you can do this


with just samples of the of trajectories instead of expectations and then they


define the Bellman operator the Bellman backup operator now the Bellman backup


operator is an important quantity in in value-based reinforcement learning


because the Bellman backup operator is basically what I talked about before


it's it tells you that if your policy is the to always select the maximum the


action with the maximum Q value right that's what's down here after you do


this action then the then the policy you arrive at and you can give certain


optimality guarantees but in essence this is so called a contraction so if


you always do that and you calculate your Q function that way it will mean


that in the contraction is defined as if you have an operator if you have two


things that are X 1 and X 2 that are some apart from each other then after


you apply the operator this T here X 1 minus T X 2 they will be closer together


which basically means that the Q to Q functions of the individual states will


be closer together and you'll converge to a single Q function right so given


enough time and enough data you'll converge on one Q function there's one


fixed point Q function that you'll converge to and you can show under


assumption in classic RL that this is going to be the the optimal Q function


the true let's say Q function so they first prove this and then they prove a


now they go back to discrete time so now they were in continuous time they go


back to discrete time but now they have a discrete time formulation with this


lag here and also they prove that that Bellman operator is a contraction so the


contraction part basically means that if you perform Q learning you're going to


arrive at a solution that's what this means to be contraction but now


obviously that solution in classic RL is going to be the the optimal Q function


but here actually don't know all right so they try this out and they introduce


one last important concept here what they call vector to go which basically


means that at the point where they start thinking where is a good thing to show


this at the point where they start thinking they give a they give the the


last action with so at this point right here where they sorry where they capture


the state they also sort of the state contains a information about what part of


the action that you started here is still outstanding so maybe your action


was and they illustrate this down here maybe your action was to move your robot


arm from down here to up here right that was your planned action at this point in


time now if you are at step if you perform the action here and here you


start capturing the next state then you would also give this particular vector


here to the to the to the agent so not only will you tell it hey by the way my


last action was a t-minus one as you would need in the Q value you will also


say and this much is outstanding this is much is where as I still have to do that


much so basically you're saying I wanted to move my arm right here and I still


have to do this part of the action now you can see while the algorithm is able


to learn much better given that information because otherwise it has it


would have to basically infer that vector from kind of differencing the


action minus the what probably happened in the meantime right so they tell you


test this out and what results is the robot videos you've seen before where


they say they can recover the original the original Q learning in this


continuous framework so here on the left side you have blocking actions and it


says when it says yes here it is kind of the old old framework you see the grasp


success at like 92% where as if you go to non blocking actions but do none of


the none of the concurrent information the grasp success suffers but you can


recover the grasp success if you if you give these concurrent information like


even introduce a time step penalty and you give this vector to go and the


information about the previous action and you can also see that the episode


duration here is much lower when you go for the continuous actions then when you


are in the old framework naturally because you don't need to pause right in


this is so this is the simulated robotics and the real world robotic


grasping results you see kind of similar results in that if you do have blocking


actions your grasp success is higher than if you don't but your duration of


your of your policy is cut in half so maybe this is a trade-off worth


considering I think this is a is pretty cool framework and I think there's going


to be a lot of work still outstanding here and I invite you to check out the


paper and look at their videos and their ablation studies of what's important and


Evolving Normalization-Activation Layers

Hi there. Today we're looking at evolving normalization activation layers by Hanjiao Liu,


Andrew Brock, Karen Simonian and Guo Wili. These are people from Google Brain and Google DeepMind.


The topic of this paper is, as you can see, it's, it's about normalization activation layers,


and we want to evolve them. I think the title says a lot, but let's go down here and see what


this is about. So we'll look at image neural networks. And current architectures are kind


of focused around the same principles. What they'll have is so ever since ResNet, you'll,


these neural networks will be composed of these kind of blocks that come one after another. So


there will be a block up here, and then the signal will propagate and there will be another block


down here. Now these blocks usually consist of what's called a skip connection out here. This


is the fundamental ingredient of ResNets that made ResNets so, so effective seem to be the introduction


of this skip connection. You see, all of these have the skip connection here. So these are variants


on ResNets. And then we see that these are always mixed between convolutional layers,


and then other things that here are called Evo norm. Now in a classic ResNet, you would have


something like a convolution layer, then you would have a batch normalization. And then you would


have a non linearity, for example, a ReLU. And then you would go on to the next convolutional


layer. So you see that the paper mainly cares about these two layers here, the batch norm and


the ReLU. And it combines them into what it's called a calls an Evo norm. So the Evo norm layers


here are supposed to replace the normalization and the activation layers, combine them and make


them better. And how does it do that through evolutionary search? Right, so these three models


here are the ResNet, MobileNet and EfficientNet architectures. They're all image classifier


architectures. Alright, so let's see how it does that. What it does is it evolves these layers


from single primitives. Now, if you've seen the batch normalization paper, then you know that


these the batch normalization is is just kind of a formula you can write down, right? And these other


normalization methods. So people have developed other ones than batch norm. For example, this is


group norm with a ReLU activation function. And you can write these two layers down as this mathematical


expression. So it features things like this is the input signal, I think this is the mean across some


groups. This is a bias term that you can train, this is the standard deviation across the same


groups, and so on. So and this here is the the ReLU term. So you can, you can kind of write this


down as a combination of these primitives. And the so you can write it in a graph. Now this graph here


is actually an activation function that this paper has found. It's called Evo norm s zero. And the


mathematical equation is the thing down here. It's not that different, you can see from previous


activations, it also has the input signal, it has the this variance or standard deviation across


groups, it has a non linearity here. And the graph here is simply a graph of mathematical operations


made out of primitives, right? So this paper takes all of the primitives that they can think of, and


puts them in this table, and they say, okay, well, our search space for these layers, so we want to


evolve these layers, our search space is a combination of any of these primitives. So you can


see here, you have something like addition, multiplication, negate, like negation. So you can,


you can subtract things, you can take the log of things, you can take the square root, and so on.


So here you have a max, which is the relu activation function. But if you if you put zero as


one of them, you have the sigmoid, which is another non linearity. Then you can also do something like,


I want to compute the batch mean, or I want to compute a group standard deviation, pretty much


anything that current activation functions that have been handcrafted use are available as primitives


across this search. So how does this method search? This is the process of how the method searches,


it does this in an evolutionary way. And evolutionary protocols, it means that you don't


develop one layer like you would do if you if you're to do something like gradient descent,


you develop a whole population of layers. So these can be maybe a couple of hundred or a couple of


thousand different arc layer architectures that you develop at the same time, right? What you do


each time you put them into a tournament, which basically means you want to sample a couple of


them, I don't think they do all at the same time, they sample a couple of them, right? And then they


train on what they call a proxy task. Their proxy task is CIFAR-10. So CIFAR-10 is a fairly small


image classification task. And they train on CIFAR-10 because it's pretty fast, right? You can


train something on CIFAR-10 in like a couple of minutes or an hour or so, and get a pretty good


feeling for how good the final accuracy will be. You can get that pretty fast. So this is a fast


classification task, because they need to do this a lot, right? The population is large, and they


need to repeat this over time, right? In any case, they take a sample, they train it on CIFAR-10,


and then the winner, the winning layer is picked from this sample. And only the winning layer is


allowed to mutate, right? So the winning layer is mutated then, and mutation means you kind of


change it a bit. Now you don't change it in an informed way, you just change it at random. And


of course, the, and then you put the mutated layers back into the population. Of course,


the hope is that by repeating this process, right, you repeat and repeat and repeat, that the, simply


by picking the winning layers over and over and over again is a selective pressure, such that


through the random mutations, but the tournament style evaluation and picking of the winner,


that over time, the best performing models in your population, right, the best scoring model here,


will get better and better and better, right? So the assumption is that this isn't like a pure


combinatorial optimization, or like a pure random function, is that if I take something that works


well, there are ways that I can perturb it that make it work even better, right? Even if most of


the perturbations are worse, there are some, and the tournament style will always, will always find


these ones for me that perform better. And then I can modify these again, at random. And then among


these, I can again find the ones that perform even better. So that is the method. And so the


question, there are two questions, how do you mutate a layer, right? And mutation, I believe,


is done in sort of different ways here. But if you look at this here, at this expression, so what


you have here is the input is this signal here, right? And you always start out, I believe, with


the input with a layer that just emits the number one with a layer that just emits the number zero,


or a component. And then you have two trainable vectors that you can include. And you just start


out with these four things. And then every time you mutate, you add one of these blocks. And I


believe there's also a method like a randomness of changing the individual things or of actually


starting from scratch. Again, it's pretty important. Otherwise, you just grow bigger and


bigger monsters. And but the way you mutate is the following. You add a new block, let's say I


add one here, and you decide on one of the primitives from the table, right? Here, I'm going


to simply decide on a minus operation, so a subtraction operation. And then once you've done


that, you choose two children, sorry, two parents, however you see it, you choose two parents,


because the minus operation needs two parents, you choose two of the parents at random here. So I'm


going to choose this thing here to be a parent. And I'm going to choose this thing here to be a


parent at random, right. And then this new node will become the new output of the layer. So you


see that this was the previous output here, this multiplication node between this and this. Now,


this is no longer the output. Now this is obsolete, right? This is no longer part of the final


mathematical expression here. So you see all the gray nodes here were actually sort of


obsolete nodes, but they are still kept because in subsequent steps, you can choose them as parents,


and then they become part of the expression again. You can see here this tanh node, it was just a


node that was sort of a dead end in the expression before, but now with a new mutation, it is again


included in the expression because I have randomly selected it as a parent. But then this node here,


and that was we said this node here, they are now obsolete nodes, because they are no longer part of


the expression, the expression in this case would go from here to here, right, including this node.


And it would go from here over here, right? So these nodes are now part of the expression. So


this is how you mutate, and as I said, you can also mutate such that you start from scratch.


And so that's how you mutate. The second part in this thing is how do you exactly determine the


winner? And what is the tournament? So how do you do that? The tournament exactly is what we've seen


before, when we looked at the different layers. So we said we train on CIFAR-10. And what we do


is we train these three architectures on CIFAR-10. So the ResNet, the MobileNet, and the EfficientNet.


We train these three architectures on CIFAR-10 with the Evo norm layer instantiated by, you know,


that particular sample from the population. And then we look at their accuracies, and we do,


we determine what is called the Pareto frontier of the population. So I think this is further up. Oh,


right here. Okay. So the dots here, the red and the gray dots, would be our sample. So all of this


would be our samples. And their performance, here it's on actually on two models, but in practice,


we have three, just to graph it better. So we plot them here, and we determine the Pareto frontier.


Now here, A, B, and C are part of the Pareto frontier, because A outperforms everything else


on model one, C outperforms everything else on model two, and B outperforms C on model one,


but also outperforms A on model two. This is what's called the Pareto frontier. And we pick


one of those as the winner. So they all are kind of one third winners here. So this is how you do


the tournament, you pick the winner like this, and then you allow the winner to mutate. The last part


that is not drawn in here actually, is somewhere here-ish, which is called the rejection step. So


the rejection step is important, because what they want to do is they say, hi, we have these mutated


layers, but some of the mutations are probably going to be just terrible, like destroying


everything, not trainable layers, just, you know, horrible, horrible, such that the layer is useless.


They don't want to keep, they don't want to put them back here into the population,


because that might either deteriorate or severely slow down this progress here. So they want to stop,


stop them, and only the good ones, only the ones that are somewhat fairly okay, get back to the


population, right? They don't always have to improve, but they have to be minimally useful.


So the rejection step, they describe down here, in the rejection protocols. They have two criteria


for rejecting mutated architectures. First, they have a quality criterion. Say we discard layers


that achieve less than 20% validation accuracy in 100 training steps on any of the three anchor


architectures, right? So the reasoning behind this is if you have 100 training steps, and you achieve


less than 20% validation accuracy, you're not going anywhere, right? You're just, because 10%


is already random performance. If you are less than 20% after 100 steps, your layer is pretty


useless and can be discarded, right? So they say this simple mechanism ensures the compute resources


to concentrate on the full training process of a small subset of promising candidates on, sorry,


yeah. So the 100 training steps, of course, is not enough to train fully, but you can see after 100


training steps whether or not the layer even does something. So you reject those. So this makes


pretty much sense, right? The second criterion is what they call stability, right? They say,


we reject layers that are subject to numerical instability, right? And how do they find numerical


instability? They define it like this. So what they do is they take the parameters, so the layers,


and this is an architecture. Yeah, so the model, these are the convolutional weights are the theta,


right? And the G is the computation graph, which is the Evo norm in this case. And there is a loss


defined across them, of course, right? This is the loss of the neural network on the samples,


right? So these are the, these are the convolutional layers, and these are the normalization layers.


Now, what we want to do is we want to see how does this loss change when we change the convolutional


layers. So you have to imagine, here are the convolutional layers, and then there are these


weird normalization layers. And then again, there are the convolutional layers. Now, we want to see


how does the loss change if we change the weights of the convolution by a little bit, right? We just


change it a little bit and see how does the loss change. This is the gradient of the weights,


basically. This is how you train. This entire thing here is how you train the neural network,


right? So you want to see how large is this gradient, and you kind of want to do this in


an adversarial way. So you want to find the maximum perturbation you can achieve, right? You say,


okay, if I change this a little bit in the worst direction I possibly can, how large is the


is the perturbation going to be? And that's how they define numerical instability. So it basically


means if this is very high, then the network might be doing well right where it is, but just a little


bit changing it will make it terrible. So they say we ascend the value on this direction for


100 steps and layer with the worst case gradient norm greater than 10 to the eighth are rejected.


In addition, so as a reasoning, this seems pretty strange, right? This quality criterion,


it it made sense, but the stability criterion, it kind of seems, I mean, reasonable, but strange


in here. The reason now, so the two tests are complimentary with each other. For example,


we found a layer like this is able to achieve reasonable accuracies on c410 across all the


anchor architectures. So it passes the quality criterion above, but its gradients quickly


explode on ImageNet, possibly due to the absence of normalization operations. So and then you see,


aha, okay, so what probably happened is the following. They did their experiment without


this right just with this quality criteria, which I guess makes sense. They did this right,


they train on c410. That's how they do their evolutionary research. Then they took their


best performing things among them is this one. And they went to ImageNet. And they said, let's


test these now on ImageNet classifiers, like we found these new architectures, let's see. And then


they got exploding gradients, right? And then they went back into their original problem formulation,


okay, how what can we build in to the evolution, such that this won't happen. And here you already


see kind of the problem with this, what you would like to have is kind of an algorithm that is


general, such as to not depend on the architectures and so on that is used. But but you see already


here that the authors, they don't direct the search, right? The search is evolution. But they


guide the evolution is very much guided by what these rejection protocols are. And you you see


here the authors tailoring their rejection protocols to the specific data sets and architectures that


they use and the specific problems they experienced when trying to to apply their method. And that I


think weakens a bit the the application of this method, because it seems that this particular form


of protocols of this particular form of rejection protocols is very much tailored to this. Let's do


these three architectures on CIFAR-10 and then go to ImageNet. And that tells me if I want to do this


in a very different domain that I would have to couldn't, it's not very clear that I could just to


plop whatever they found works in and it would just work just as outperformingly of the others


as as in their experiment. It tells me that there is like a somewhat large dependence on the specifics


here. Yeah, so but that being said, these are the rejection criteria. So they reject each step here,


the worst ones, and they go back into the population and then that process repeats and


repeats and repeats and then at the end, you hopefully end up with some very good normalization


layers. Now I have to see here, if you compare now these these found normalization layers with


the classic variant, so the classic thing here is this red line, this is batch norm and relu,


this is a classic activation normalization combo you put in a neural network and you see that these


methods outperform this on a very kind of a stable basis, right? So that's pretty cool, but that is,


as we said, on CIFAR-10, that is on the exact thing that they search over, right? So it's not really a


surprise that if I, you know, search a bunch of combinations and always get the best ones, I would


outperform just one of them. The interesting thing is what happens now if we take what we found and


put them into a different architecture for a different data set. Now here the architecture


isn't really different because it's kind of the same, but they do evaluate it on ImageNet, right?


ImageNet, different data set than CIFAR-10, much larger and so they put their architectures, which


here EvoNorm, into ImageNet and evaluated and you can see that it has fairly competitive results


across, right? So I find that to be to be fairly cool that the best performing ones on CIFAR-10


would also perform better than the corresponding ones on ImageNet. But you already see as well that


it's not super high, right? So the differences here are, I would say, it is improving, but sometimes,


you know, it's the same, sometimes it's actually worse. It doesn't appear to, no, to me,


that those kind of things are not super convincing, especially because this is the paper that suggests


these methods, so they're naturally going to present them in the best possible way. So it


seems like the massive outperformance on CIFAR translates only marginally to ImageNet and these


are the same architectures, right? The ResNet-50 and MobileNet and FissionNet, these were already


the architectures that they searched over, so my trust that this new normalization layer put into


an actual different architecture is less still. Now they do actually do some experiments on that


as well, but I just, this is my thoughts when reading this. And as well, and this I find very


interesting, this column here are random search, so if you just do a random search, which means you


just produce random layers, then it doesn't work at all, right? So you take the best ones of the


random ones you found and it doesn't transfer at all. But interestingly, if you do random search


plus rejection, so the same rejection that they do, just you don't do this tournament evolution


mutation style, you simply random search and then do rejection, that gives you fairly competitive


numbers, right? And in some cases even, see here, it does, it outperforms some of the classic


methods. So just that will give you fairly decent results, right? And that is, to me, that seems to


be even more a sign of, okay, this, what this method is mostly doing is just searching like mad


for something that works on these particular architectures. And of course you can find things


that work better if you search like mad, but then what do you do with it? Like, what does it mean?


It, can we generalize? Now they do two additional tasks to show that it does generalize two other


architecture and tasks. So first of all, they do object detection and instance segmentation,


right? On Coco. So this is a very different task. This is a mask or CNN, right? And they just put in


their layer there. And you can see here that they generally outperform the baseline. I don't, I can't


speak to how much this is, this outperformance is here. It seems like the numbers are fairly close


together, but they are consistently better. And again, I don't, I don't necessarily trust these


kind of experiments too much because who knows how much effort you can spend on making your method


better. But in any case, they show that they are better, which is already something. But again,


here the R50 indicates that we're again dealing with like ResNet-50, ResNet-101 architectures,


which are fairly similar to the ones that we, that the method was searching over. So the second


thing is they say we generalize to GAN training. So they take a big GAN, a big GAN deep, and they


show that their method will outperform these other methods on the IS and FID metrics. I don't even


know what, Inception score and Freshlet Inception distance, yay. So it will outperform them, but in


kind of a weird way, okay, here it outperforms them consistently, but then in the Inception score,


this batch norm plus relu still seems to be like a lot higher than this evonorm V0. And then this


thing here that was performing worse in the ImageNet is now performing somewhat better. It


just, so it is a cool result and definitely cool that you can pop this in here. I just think that


the things that turn out here, that they are tuned to very specific architectures to very specific


tasks. So I think the big GAN deep, the kind of architectures will always be kind of the same,


it will always be kind of ResNet-ish style neural networks, and the tasks here will always be sort


of CIFAR ImageNet style things. And therefore, I believe, with the results we've seen the fact


that it outperforms so much on CIFAR-10, but then the gains on ImageNet become more marginal,


I think that indicates that the gains here most probably don't translate the further away you go.


So I'm not sure that the evonorm that they find, like that this particular thing here,


will remain the best thing across tasks. I think they just found this to work well in their


particular setting here. And if I run the same thing with the slightly different architectures


and slightly different tasks, I will come up with a different best thing. Yeah. Alright,


so these were my comments. They do some interesting experiments where they show that if they just do


random layers, it's not as performant, which I can believe if you just jumble these things randomly,


probably not as good. So you need some kind of search criterion. And yeah, that was my thoughts


on this paper. I invite you to read it, look at it, look at the additional experiment. It is a


Growing Neural Cellular Automata

Hi there. Today I thought we would be looking at growing neural cellular automata, which


Growing Neural Cellular Automata

is an article on distil.pub, which I found pretty neat. So this is kind of an interactive


Growing Neural Cellular Automata

article. If you don't know distil.pub, check it out. It is a cool new concept as an alternative


Growing Neural Cellular Automata

to the classical journals or the the conference system. So what it allows you to do is to


Growing Neural Cellular Automata

kind of write articles that are a bit more interactive, a bit more in like engaging and


Growing Neural Cellular Automata

don't have the like, there's no PDFs, there's no pages, there are animations, and so on.


Growing Neural Cellular Automata

So I thought we'd be looking at this article today, which is kind of growing neural cellular


Growing Neural Cellular Automata

automata. So if you don't know what cellular automata are, this is a very kind of old concept.


Growing Neural Cellular Automata

The most famous one is called the game of life where you have these cells here, you


Growing Neural Cellular Automata

can see every pixel is a cell, and they follow some kind of update rule. And usually it's


Growing Neural Cellular Automata

the update rule, something like if my neighbor is alive, I'm going to be alive as well in


Growing Neural Cellular Automata

the next time step. And if or if enough neighbors are alive, and if only very few neighbors


Growing Neural Cellular Automata

are alive, I'm going to die. So this gives rise to these kind of patterns. And here is


Growing Neural Cellular Automata

the same is done with color. And the update rules are a bit more complicated. So basically,


Growing Neural Cellular Automata

here, traveler, oh, nice. Okay. So in the game of life, if you play it, the most prestigious


Growing Neural Cellular Automata

thing to get is are these kind of travelers, I have not. This is the first time I've managed


Growing Neural Cellular Automata

to do this in this thing. So what does it do? So each pixel here is kind of an autonomous


Growing Neural Cellular Automata

thing that is only allowed to look at its neighbors in order to decide whether or not


Growing Neural Cellular Automata

in the next time step it is going to be alive. Look, it's like incorporating again. So So


Growing Neural Cellular Automata

each each cell looks at its neighbors, and then decides what its next state will be.


Growing Neural Cellular Automata

And here, it's not only alive or dead. Dead would be white. And alive would be anything


Growing Neural Cellular Automata

else. But it is also, I guess this white isn't, it is also the color. So each cell decides


Growing Neural Cellular Automata

on what color it should have. And then this is a live thing. So it kind of reproduces,


Growing Neural Cellular Automata

right? You can see if I started new, if you double click here, it grows from somewhere


Growing Neural Cellular Automata

else. And this is completely local. So these cells really only look at their neighbors.


Growing Neural Cellular Automata

That's the special part, right? They don't look at the global structure. It's not like


Growing Neural Cellular Automata

a GAN that can look at the entire picture and decide what what's still missing. What


Growing Neural Cellular Automata

these can also do if you destroy part of it, they can kind of grow back just again, just


Growing Neural Cellular Automata

out of local update rules at the level of the individual cells and their neighbors.


Growing Neural Cellular Automata

They're trained to do these big structures. So let's look at how they do it. So basically,


Growing Neural Cellular Automata

this is how they model a cell. And let's go over here. So each cell, as I said, is made


Growing Neural Cellular Automata

up of 16 channels. And here it's modeled as three by three. But I think each cell is really


Growing Neural Cellular Automata

one pixel. And each cell is allowed to look at its eight neighbors, right? So each cell


Growing Neural Cellular Automata

is allowed to look at its eight neighbors across 16 different channels. And the 16 channels


Growing Neural Cellular Automata

here mean the first three are RGB. So this is the actual color that is seen. Then there


Growing Neural Cellular Automata

is an alive or dead channel. So when what they call an alpha channel. So if this channel


Growing Neural Cellular Automata

is high, the cell is considered alive. Otherwise, it is considered dead and not part of the


Growing Neural Cellular Automata

pattern. So a cell can come alive or die, depending on its neighbors. And then the rest,


Growing Neural Cellular Automata

the rest 12 channels are what they call hidden channels. So the cell is allowed to encode


Growing Neural Cellular Automata

some hidden state there. So there's each cell is represented by the 16 dimensional vector,


Growing Neural Cellular Automata

which is not much right. And then each cell is allowed to look at three things. So from


Growing Neural Cellular Automata

the bottom here, it's allowed to look at its own state, so at its own 16 dimensional vectors.


Growing Neural Cellular Automata

And it is allowed to look at its neighbors. And it does this by doing a convolution with


Growing Neural Cellular Automata

a sobel filter. And the sobel filter is simply a fixed filter that you do a three by three


Growing Neural Cellular Automata

convolution with, as you can see here, is basically a gradient filter. So basically,


Growing Neural Cellular Automata

there's the difference between what's to the left of the cell and what's to the right of


Growing Neural Cellular Automata

the cell. And here in the sobel y direction, the same in the y direction. So it's basically


Growing Neural Cellular Automata

allowed to look at gradients in states of its neighbors. This is modeled after real


Growing Neural Cellular Automata

cells kind of looking at chemical gradients in their neighborhoods. So and this is all


Growing Neural Cellular Automata

this, this is all that the cell has to decide what it's supposed to do next, right. And


Growing Neural Cellular Automata

what we want is we want that each individual cell only looking at its neighbors produces


Growing Neural Cellular Automata

in total, they will produce these kind of very complex pattern. So the update rule is


Growing Neural Cellular Automata

the following, you convolute with the sobel filters, and you take the cell identity, you


Growing Neural Cellular Automata

put this all into a vector, you put it through a very, very small neural network. So this


Growing Neural Cellular Automata

is one dense layer, one relu, and then another dense layer to get the next 16 dimensional


Growing Neural Cellular Automata

vector, which is the next state. And that defines your update rules, that doesn't really


Growing Neural Cellular Automata

define the next state that defines the delta to the next state, kind of like a residual


Growing Neural Cellular Automata

neural network. So basically, which cells need to come alive in the next time step,


Growing Neural Cellular Automata

which cells need to die, and how are they to change their colors, right. And then you


Growing Neural Cellular Automata

get the output of the next step, right. So that's, that's basically the entire thing.


Growing Neural Cellular Automata

So all that is learned here is the the update rule of the neural network, right. So basically,


Growing Neural Cellular Automata

the neural network decides, it looks at a cell and its neighbors and decides what the


Growing Neural Cellular Automata

information in the cell in the next step should be, right. And you do this for multiple time


Growing Neural Cellular Automata

steps. That's I actually want to go down here, you do this for multiple time steps, the initial


Growing Neural Cellular Automata

state is simply one cell that is alive here in the middle, everything else is dead, this


Growing Neural Cellular Automata

cell is alive and black, you do this for many steps, right. And then at some point, you


Growing Neural Cellular Automata

get an output. And you compare the output to your desired output, you compute a loss


Growing Neural Cellular Automata

that is differentiable. And because your update rule is differentiable, and your losses differentiable,


Growing Neural Cellular Automata

you can back prop through time to the original to the original pattern here. And you can


Growing Neural Cellular Automata

basically learn this update rule by back propping through time, this is a bit like an LSTM.


Growing Neural Cellular Automata

And if you see in the architecture here, I think this residual connection is really the


Growing Neural Cellular Automata

key to making this work over time. Because usually, I would not expect something like


Growing Neural Cellular Automata

this to easily emerge over time, because you have the problem of vanishing and exploding


Growing Neural Cellular Automata

gradients. And you have no way of mitigating this problem here in this simple neural network.


Growing Neural Cellular Automata

But in any case, they back prop through time here. So each of these update steps, which,


Growing Neural Cellular Automata

again, this isn't one neural network with many layers, this is the same neural network


Growing Neural Cellular Automata

applied over and over and over and over again. And then there is a loss computed. So basically,


Growing Neural Cellular Automata

the gradients will accumulate over these steps, basically tell the network what it needs to


Growing Neural Cellular Automata

adjust to go from this one single black pixel to this final desired state. If you do this


Growing Neural Cellular Automata

over and over again, you learn things, you learn a update rule that will give rise to


Growing Neural Cellular Automata

that pattern, hopefully. Now, here is a kind of an illustration of this alive and dead


Growing Neural Cellular Automata

thing. So what they do is they consider cells that have an alpha channel, this one of these


Growing Neural Cellular Automata

channels called alpha, they have an alpha channel above 0.1, it's considered alive,


Growing Neural Cellular Automata

right, and part of the of the loss, right. Then the neighbors, the neighbors of these


Growing Neural Cellular Automata

cells that are below 0.1, but are neighboring a cell that is mature, alive, they're called


Growing Neural Cellular Automata

growing, they're also part of the loss, right. So simply by being close to something, someone


Growing Neural Cellular Automata

that is alive, a cell that is alive, you are considered alive as well. But your neighbors


Growing Neural Cellular Automata

aren't right, you like only the neighbors of really alive. So there's really alive kind


Growing Neural Cellular Automata

of alive. And then there is dead and dead, that the meaning of dead here at the gray


Growing Neural Cellular Automata

ones is that they're not they won't become part of the pattern part of the loss, right,


Growing Neural Cellular Automata

they're dead. All right, so what will this get you initially? So here is an animation.


Growing Neural Cellular Automata

If they train this just like that, just back up through time with a target pattern, and


Growing Neural Cellular Automata

then they let it run, you see these patterns actually emerge. So that's pretty cool. But


Growing Neural Cellular Automata

then if you let him run for longer than they've been trained, you basically have no guarantees


Growing Neural Cellular Automata

on what's going to happen. Like these update rules are simply trained to achieve the pattern


Growing Neural Cellular Automata

within a certain number of steps, right? If you run for more than that, and apply the


Growing Neural Cellular Automata

update rules for longer than that, you you have, like there's little like you have no


Growing Neural Cellular Automata

guarantee what's going to happen, these update rules will simply continue, as you can see


Growing Neural Cellular Automata

here, and produce some weird stuff. So they are trying to fix this. So what they do is


Growing Neural Cellular Automata

basically they train for longer, but they do it in a kind of different way. So at each


Growing Neural Cellular Automata

at each step of training, and a step, I mean, a batch over these number of time steps. So


Growing Neural Cellular Automata

they, they sample a batch initially, it's just all black pixels, right, as we see above.


Growing Neural Cellular Automata

And then they optimize for these number of time steps. And then they're at the end. So


Growing Neural Cellular Automata

what they do is they don't always start from the black pixel. But sometimes they also start


Growing Neural Cellular Automata

from a previously seen end state. So basically, they take the end state of a previous training


Growing Neural Cellular Automata

zone. And then they just continue from that instead of starting from the initial point.


Growing Neural Cellular Automata

And you see after some training, they get better and better. So initially, you see the


Growing Neural Cellular Automata

thing on the left here, the thing on the left here being a starting state. And then it progressively


Growing Neural Cellular Automata

gets better. So basically, by starting from end states of other things, you will learn


Growing Neural Cellular Automata

to so if the end state of the other thing isn't very good, you basically learn to go


Growing Neural Cellular Automata

to the good pattern to the pattern you want. But of course, over time, there's going to


Growing Neural Cellular Automata

be more and more of these end states that you train from that are already pretty close


Growing Neural Cellular Automata

to the pattern you want. And so then what that means is you learn to reproduce the pattern.


Growing Neural Cellular Automata

So you are already at a good point, you learn to stay at that good point. And then that


Growing Neural Cellular Automata

enables you to basically learn update rules that if you're not at the pattern you want,


Growing Neural Cellular Automata

they go towards the pattern you want. But also, if you run for longer, if you're already


Growing Neural Cellular Automata

are at the pattern you want, then you stay at the pattern you want. So that's what we


Growing Neural Cellular Automata

basically saw in the very initial demonstration where you could this is a live demonstration


Growing Neural Cellular Automata

like this thing up here. This is a live this is running, right. And you see the update


Growing Neural Cellular Automata

rules data, they are continuously applied, they basically stay at the pattern where they


Growing Neural Cellular Automata

are. And that is also that is learned because of this protocol that you train from end states


Growing Neural Cellular Automata

as well as from beginning states. So the next thing is what I'm doing here is I can destroy


Growing Neural Cellular Automata

part of the pattern, and it will kind of regrow right, you see that here. So this is also


Growing Neural Cellular Automata

a part. So for now, we've only learned to go from a single pixel like here from a black


Growing Neural Cellular Automata

pixel to the pattern. But now we also want to learn to go to regrow when destroyed because


Growing Neural Cellular Automata

that is this you can see this is modeled after kind of live tissue. So here you can see the


Growing Neural Cellular Automata

their parts are cut away, and then the the cells try to regrow. So this is I think, initially,


Growing Neural Cellular Automata

initially when you just train them, they exhibit some of that property, but not like very satisfying


Growing Neural Cellular Automata

in some cases. So what they do is they train not only do they use end states, like we saw


Growing Neural Cellular Automata

before, but also some of their training samples are simply the pattern destroyed a bit. So


Growing Neural Cellular Automata

as you can see in some of these samples, like these here, they in each sample, they kind


Growing Neural Cellular Automata

of cut out part of the sample and they train the update rules to regrow that part. That


Growing Neural Cellular Automata

gives you that now gives you the ability to if you damage to pretty consistently regrow


Growing Neural Cellular Automata

the pattern, as you can see here. And they also train for rotation, which is non trivial


Growing Neural Cellular Automata

if you have these kind of pixel based, pixel based models. But I want to jump that because


Growing Neural Cellular Automata

I want to keep it kind of short here. So the the entire goal of this is to kind of model


Growing Neural Cellular Automata

the behavior of that of natural cells, because the natural cells, they don't have an overarching


Growing Neural Cellular Automata

view, they only have the view of their neighbors, right, and they are able to grow into very


Growing Neural Cellular Automata

complex structures. I invite you to give this a try. The distilled up pub journal is very


Growing Neural Cellular Automata

cool. It's very interactive. You can play around with it, you can reproduce things in


Growing Neural Cellular Automata

a collab. And yeah, shout out to the authors here. Alexander more than seven, a Torah or


Growing Neural Cellular Automata

that so sorry, Randall. So Evan Nicholson and Michael Levin. Yep, that was it for me.


Growing Neural Cellular Automata


[ML News] AI Threatens Biological Arms Race

DeepMind uses deep learning to restore ancient Greek texts.


A drug discovery system has been abused to create thousands and thousands of super toxic


compounds and Gary Marcus claims deep learning is hitting a wall.


Welcome to ML News.


It's Monday.


Video's GTC conference goes into its next iteration.


Now GTC is a company conference.


Like all of the big companies, they present all of their newest stuff there.


But they also have a host of external speakers and all kinds of people that just give education


and talks about how they use deep learning for various things.


Now all of it is obviously NVIDIA themed, but I can promise you the talks are interesting


by themselves as well.


The highlight of the conference is obviously the keynote by Jensen Huang and depending


on when you're watching this video, the conference is going on probably right now.


And the best part is if you use my link, that's why kilcher.com slash GTC and you use that


to sign up for the conference, you can win a 3090 that has been hand signed by Jensen


Huang.


So the same person that is giving the keynote will have signed your GPU if you win it.


Now this is pretty cool.


All you have to do is sign up using my link and then attend at least one session and why


not attend the keynote.


The keynote will go into all of the upcoming things of NVIDIA.


For example, is there going to be something like a 4090?


How does it look like?


Why do they increase the first digit of 3090 and not just make it the 3091?


All the biggest questions of humanity.


Now other than new architectures coming up, there will also be a lot of talks on the topics


of accelerated computing, autonomous driving, anything to do with computer vision, rendering,


cybersecurity, NVIDIA hardware now powers almost all of deep learning advances apart


from some specialized vendors.


So this is definitely a good place to look.


Another thing I want to highlight is the NVIDIA Omniverse platform, which is a high performance


and really good simulation, physics and rendering engine.


This includes Pixar's universal scene description technology and can be used to do accurate


renderings.


And since synthetic data is such a big deal in recent times, this could really be something


to accelerate your research if you are into simulated data transferring to the real world.


It's pretty cool and a lot of things can be done with it.


And know the Omniverse isn't the metaverse per se, but there is a session that you can


attend in GTC that talks about the metaverse and how to build virtual connected worlds.


And as you can see, one of the speakers is the VP of Omniverse.


So in the end, everything's somehow going to be together.


There are even sessions called connect with the experts where you get one on one time


with experts in a certain area.


For example, GPU performance analysis and optimization.


This is first come first serve.


So hurry up.


As I said, besides the keynote, there is an entire plethora of sessions that you can attend.


These go from building large language models to next generation rendering to using AI for


cybersecurity or understanding how newest technologies can help your business.


There's also more specialized tracks such as focuses on healthcare, autonomous driving


and other areas.


Registration is free and you can put together your own little calendar that reminds you


whenever these sessions are coming up.


Again, use my link to sign up in order to win a 3090.


There's one caveat you need to be in EMEA which is Europe, Middle East or Africa in


order to qualify for the 3090 raffle.


However, I've decided that anyone living outside of these areas can also participate in another


raffle that I sponsor and that will just give you some some merch.


So inside EMEA, you can participate for the 3090 outside EMEA you can participate for


the merch.


Now if you are in either bucket and you want to be in the other bucket, I'm sure we're


going to do stuff in the future where you can win to your heart's content.


But for now, this seems the most fairest allocation of resources.


And remember, you have to attend a session in GTC in order to qualify for the 3090.


DeepMind has released a new blog post called predicting the past with Ithaca.


Now this is a system that restores ancient texts, namely ancient texts from the Greeks.


So throughout the years, a lot of these inscriptions in stone have gone missing have been damaged,


and therefore historians, they need to tease out what things could mean.


Now this is obviously a good application for something like a language model.


So what Ithaca does is it takes in whatever is undamaged, and a few hints of where it


needs to fill in missing characters.


And it tries to reconstruct these things.


Not only will it give an output that restores the missing pieces of text, but it will also


determine a probability distribution over the geographical origins of this piece of text,


as well as a chronological attribution, meaning it will estimate when the text was written.


Now it's interesting to me, as you can see right here, the input is just plain text,


I would have guessed that they would use some sort of computer visiony things as well as


maybe the Greeks would have written down some stuff in certain ways in certain order, but


I'm not too educated in ancient Greek.


So this might not have been the case.


After all, what is cool, though, is that the blog post goes into a lot of detail, not only


about the system itself, and how good it is, which it undoubtedly is, but how the combination


of humans and machines together can outperform anyone alone.


They talk a lot about how to build tools in order for historians to be able to effectively


interface with the system and that it has really accelerated their research.


Now this isn't only good for ancient Greek texts, but the more we learn and how we can


use AI in order to accelerate other fields, I think the better the success rates for all


of science.


This goes along with an open access paper in nature that you can read, the code is online,


you can try it out for yourself.


And they even have a website with a little demo application, where you can try it out


yourself in just in case you happen to have some ancient Greek block laying around with


some damages in it, just enter it here, it will it will do it, it will predict it.


Overall, I think it's a pretty cool trend what DeepMind is doing interfacing with lots


of experts in adjacent and even non adjacent fields, and using AI in order to come up with


accelerations in those fields.


I think it's a neat application and it benefits everyone.