network that has the same amount of parameters or almost the same amount of parameters. They


SpineNet: Learning Scale-Permuted Backbone for Recognition and Localization (Paper Explained)

then further modify this network such that it reaches that higher accuracy with less


compute than the original network. So if you want to know how it's done, you know, stick


around. You can help me by sharing out this video if you liked it. If you didn't like


it, leave a comment and tell me what you didn't like. Otherwise, I have no chance of improving.


So that's the deal. Okay, cool. So the task here is a recognition and localization as


you can see here, which basically means that you have an image and there's stuff on the


image. Maybe there's a cat right here. And maybe there is some kind of a house right


here. And the task, these tasks come in various forms. But some of the tasks are to say what's


on the image. So in this case, cat and house. And also, where is it? Now, this could be


a point, this could be a bounding box, or this could actually be a pixel segmentation.


All of this sort of tasks exists in various forms. What usually is done in these tasks


is you want to go in some way through a neural network, and the neural network will output


the same image again, or the same shape. So it will output a image that is of the same


shape that if this is your input image, I'm going to, I'm just going to quickly redraw


without the labels. If this is your input image, then the output image, let's say we're


doing bounding boxes, the output would say something like, here are bounding boxes. And


also the output would be cat and house. So these are the two outputs that the neural


network would generate. This is some sort of a convolutional neural network because


they deal with images fairly well. Now, usually, when we do image processing, and we know this


from, for example, image classification. So if we just have image classification, just


to classify here, if we just want the outputs cat or house, or even just one single thing,


like an image net, our convolutional neural networks have a particular architecture, namely,


what we do is we have the first convolutional neural net, the first layers will take the


image and run these convolutional filters across them, which gives you the same shape


image back. But then, with time, we scale it down, we have a max pooling or a convolution


with stride two, so that the image is only this big anymore. And then we have a bunch


of further layers, and then we scale it down again, and so on. Now, as we scale it down,


the number of channels goes up, of course, at the beginning, you have three channels


for the three colors. But then after the first convolution, you might have whatever 32 channels


right here, this is no longer the original image, this now is, of course, for each pixel,


you have a stack of features, right, you have a stack of features right here, because that's


what your convolutional layer does. And then when you scale it down, you have even more


feature maps. So we tend to scale down the resolution of our feature maps. But we tend


to increase the number of feature maps right here. The reasoning behind this is, if you


look at these bounding boxes, they, they don't really so not enough, sorry, if you look at


the labels right here, the fact that there's a cat on the image shouldn't depend on the


exact pixel location of the cat, right. So even if I scale this down a bit, I'll still


recognize that there's a cat somewhere, it can still aggregate that information. In fact,


I could deal with a single, right, I could deal with scaling this down successively,


up to a single pixel. And that's ultimately what an image classifier does. You simply


have a single vector at the end with the features in it. And from that you classify. So the


reasoning is that as you go through the network, you pick up the low level features first,


like here, you pick up the edges and the kind of low level shapes. As you go higher through


the network, your features become more abstract, but less localized, which means that it's


less important where they are. And that's why you look at this image through a coarser


and coarser segmentation. And at the end, your segmentation might be something like


this. Okay, so we we have had a lot of success building image classifiers with this reasoning.


And this is sort of a human heuristic that just hasn't worked well. Now, when we do something


like this, bounding box classification, or even per pixel classification, all of a sudden,


it is very important where the things are, right, it is very important that it's this


pixel and this pixel and this pixel and this pixel forming the bounding box. Because the


more accurate you are, the better your bounding box classifier, you still have this right


here, this recognition, but the localization part, we can't just scale down anymore, because


we need to output something that's of the same size. So what people have done is they've


gone from this kind of from architecture that scales down, because we know that works well,


we know the downscaling works well. So we take that, and then we scale up again. And


there is some reasoning behind this, right? So that's what we can do. Because we know


this part works very well for extracting high level features that are not that localized.


So our reasoning is going to be something like, okay, we'll force the network through


this kind of bottleneck right here, we'll force it to learn some high level features.


Because otherwise, you can just, you know, kind of remember the individual pixels, and


that won't work as well, we'll force it to remember the high level, we'll force it to


remember what a cat is. And then it will help in the pixel segmentation to know what a cat


is. This is very valid assumption, but it doesn't need to be the case. And so there's


one additional thing that these networks usually do is that they have like some skip connections


here from the layers that are of the same size to the layers that are of the same size


right here to here, in order to kind of recover these high level features. Because if you


only look at an image through the lens of this right here, and you're you have to segment


the ear of the cat, you know, you can only either color an entire pixel or not. So you


want to gain back some of that some of those high level features. And that's what you do


with skip connections. And that's why these networks usually look like this. Now, in this


work, the authors sort of criticize this, they say, why, why are we doing it like this?


Isn't there a better way to do it? Specifically, we want to look at this part right here, which


is called the backbone. So we assume that we have these these output layers that give


you at different scales, different features. And what we have to do is we have to construct


a backbone that somehow feeds features either, you know, through this direct way or through


these connections right here, and feeds features to this, these ultimate classifiers. So these


classifiers will then be used to classify the bounding boxes and classify the output


classes for recognition and localization. This is an illustration of this. On the left,


you see a typical backbone. And they call this a scale decreased network. So an example


of scale decreased network on the left versus a scale permuted network on the right, the


width of the block indicates feature resolution, and the height indicates feature dimension,


dotted arrows represent connections from two blocks not plotted. Okay, so on the left,


you have the typical architecture, you see that the the width, so this is the resolution


is very high. And as you go through the layers, that resolution gets smaller and smaller and


smaller. But the number of features indicated by the height gets higher and higher and higher


as you go through. This is your typical architecture, we are looking into, let's say that this is


not the only one, what we can do is we can build any sort of backbone. And here they


restrict themselves, they say, okay, in order to make it comparable, in order to be, you


know, scientifically a bit more rigorous than just building anything, what we restrict ourselves


are simply permutations of this. So we only allow us to permute these things. So all the,


you know, this goes here, and this goes here, as you see, and that ensures that you still


have roughly the same amount of parameters. Now, there is a sort of a parameter difference,


because these connections here, you need to up and down sample the images. And sometimes


that introduces parameters. But in essence, you have the rough same amount of parameters.


And then you can really research what can we improve a network simply by rearranging


its blocks, because that would give evidence that this scaling down architecture isn't


really the best one. Okay, so here, you can see an example of this. This is what they


call a scale permuted network. Sorry, this scale permuted network right here. So in a


scale permuted network, what you're allowed to do is you're allowed you have these blocks


on the left, and you're allowed to put them anywhere you want in in any in any sort of


I don't want to say order, but yes, in any sort of order. Yes, it's an order actually.


So it goes from here down it this is 12345. Any block for any block you first place you


first place this block, you're allowed to connect it to any other block before it. Now


here we don't see but you can see there's two incoming connections right here. So we


make use of more than one connection on the left, you see there's always one connection


between the blocks. And on the right, you see, we allow up to two blocks to connect


to a given block. Okay, then you're done with this block, you place the next block this


one here. Also, you're allowed to have two incoming connections, this one here, and this


one here. And you place the next block and so on. Now how you make this you also see


that there doesn't need to be like a straight linear path, because there is no connection


right here, if you can see that. So you might be wondering, how do I decide which block


goes where? And how do I decide on which connections connect where? And that is going to be the


idea here to use neural architecture search. So neural architecture search, right now is


still a fancy way of saying, let's try stuff out. And so what you'll do is you will initialize


a reinforcement learning controller that decides on the ordering and on the connections. And


it has some action space, and you basically let it run. So it, it, you know, proposes


a couple of architectures. And, and then you measure all of them, you train all of these


architectures and you see how well they fare. And then you go back to the controller, that's


the reward signal. And so we can draw so you have an agent, which builds the building plan.


So the agent in the agent will emit as an action, a building plan, like big, small,


big, small, big with connections like this, and like this, and like this. And then that


will go to the environment, the environment here simply takes the architecture and trains,


train the architecture. And then the, let's say the eval loss or the validation loss,


the validation accuracy is equal is going to be your reward signal. So you simply train


a reinforcement learning agent to solve this particular problem, which is training this


in recognition and localization on the particular data set, as well as possible to basically


come up with the best architecture, which, you know, it's, it's fancy, and it's a bit


better than trying everything out. But it's not much better right now. And it takes a


lot of compute to run these experiments, because it takes a lot of iterations of this. And


every iteration consists of training one of these networks fully once, and you can do


something with like early stopping and stuff. But so you get the idea. This is what they


what they propose. And this is, you know, how they get better. So there are a number


of challenges in this. Namely, we've, we said, okay, when you input a signal, for example,


when you input a signal from this layer to this layer, you can see that you have to shrink


the resolution, and you have to up the number of features. And this was already sort of


solved in the resonant original resonant paper, but they reiterate how they do this here.


Basically, you have, you have this layer, and it is connected to these two layers, we


said every layer can receive inputs from two layers, you see, at the very end, these are


just added together. Okay, so we have two things. First of all, the number of features


is different, you can see right over here, the number of features, the number of channels


is different than the number of channels in the output image, let's say, right here. So


those are different. And in fact, they're different in both inputs. And we have this


method of one by one convolutions that was introduced in the original resonant paper.


If we do one by one convolutions, it's basically a learned transformation from a number of


input channels to a number of output channels without chain without doing any actual convolution


operation. This is simply linear operation, upscaling or up, up, upping the number of


feature maps, you can see these one by one convolutions are employed here in various


ways. So because this is fairly compute intensive, or so they claim, what they do first is they


always first, go to less features. So here we have a number of features, which is maybe,


let's say this is f or sorry, this is c zero. You can see very small here, maybe that there


is this first we go to alpha times c zero and alpha is I think in the default setting,


it's one half. So first, we always go to one half the number of features. Before we do


this switch here, and then we have two options, either. So you go to one half the features


and at the end, you go to the number of target features. So it could be if the target features


are more than you currently have, it could be that you first go to less features. And


then you go to even more features, right? As if you the current one has more features


than the end, it's probably not as bad because you first go to less features and to even


less features. This is probably one of the things they did to save computation. But which


you can imagine that it hurts because here you simply have to you have to basically throw


away half the features or you have to like linearly combine them in every step where


you connect two layers to each other. Okay, so there's two situations. First situation,


your current resolution here is higher than the target resolution. In that case, we can


simply do a convolution with a bigger stride than one, right? If you have an image and


you do a convolution, usually you have this overlapping convolution, such that the result


is the same size as you started with. But you can also do a bigger stride. And I'm a


bit overdrawing this here. But you can do a bigger stride such that the final resolution


is smaller. And you can also do this max pooling right here. So the max pooling is also a way


to reduce the number of of the resolution of the image. So if we're bigger, we can do


that. If we are currently smaller than the target, what we can do is we can up sample


and up sample you can do by doing nearest neighbor or things like this. We can also


do a learned up sample. There are various ways I believe here they do a nearest neighbor.


But I'm not sure anymore. Actually. Let's check it out. That's here somewhere. resampling


and cross scale connections, yada yada yada yada yada yada yada. It's important to keep


the computational resampling low, we introduce a scaling factor alpha, we had that, then


we use a nearest neighbor interpolation for up sampling, or a stripe to three by three


convolution. Okay, so it's nearest neighbor by up sampling. All right. So that's, that's


how they up and down sample the feature maps to the correct shapes, either using nearest


neighbor up sampling, or using multi stride convolution followed by max pooling. So what


does that give them? Now, they do several different steps in this. So the first architecture,


this resnet 50 is the original architecture. And remember, we're only talking about the


backbone right here. Now in the original resnet 50 architecture, we have this resnet 50, FPN.


And this FPN is these are called the output layers. This is what then goes and classifies


the bounding boxes and the labels and so on. Now, here you can see the resnet 50 is continuously


getting smaller and more features. They do an intermediate step. So this, this right


here is their final thing where they let this let this algorithm go wild. And you can see


that it's pretty, pretty fuzzy. So this RL controller finds this architecture to be the


best architecture. And you can see it's continuously down and up and down and sorry, and up and


down. And there's considerable cross connections between all of these things. And then here


you have the you have the different output layers built into the network rather than


next to the network, right. So these are the ones that are now the red border ones are


now the features that are used for going in classifying. As an intermediate step, they


also consider this architecture, where they basically built a smaller resnet right here,


and then let the algorithm decide on the rest right here. So it still has the same amount


of parameters roughly, but they can investigate what happens if we go to these to this lower


if we have this structure at the beginning, but then part of it we can do with our algorithm.


And lastly, they also consider this architecture. Now this architecture, again, their algorithm


has control over the whole network. But there is an additional thing that the algorithm


can do the algorithm can also decide to change the number of features and to change the type


of block. So here you can see these are all residual blocks. And these are these called


bottleneck blocks, they're simply a different way of of doing a residual block, it wasn't


introduced in the original resnet paper. But the controller can simply switch to that.


And that can save some computation if you go through these bottleneck blocks. So what


does that give you, you can see below that the resnet 50 is at 37.8% average precision.


If you liberate the top part to leave it to the algorithm, it's at 39. If you liberate


the entire network, it's at 40.7. And remember, these are like roughly the same amount of


parameters. And then if you if you also let the network control a bit of the feature size


and the type of block, you get a 40.8, which is the same as before. But now this one, I


believe has about Oh, yeah, here we go with 10% fewer flops. Okay, so that's, that's pretty


cool. Though, remember that the left thing this is this is made by humans, this is just


our heuristic. And the right things they are made by RL. And they are no for these particular


data sets, though they do find that generally this also transfers to ImageNet classification.


But still, this is sort of a, it works well for the type of data we work with, and so


on. So I don't know how much I would trust it how far we should go of building SpineNet


49 as our new backbone for every image task that we have. It remains to be seen, I believe.


Before actually, we go to the experiments before we go to the experiments, I want to


state my idea right here. So you get the general gist here. And so another kind of coral I


have with this is that, you know, in here, you always have these single connections.


And here you always have these these double connections. And I've looked through the experiment,


it seems like nowhere do they ablate or anything what what it means to only have single connections,


or if they so if they let the ResNet run with double connections, so if their controller


could not switch the order, but only introduce the connections, they might have done this,


they have a lot of experiments where they do the different ablations. So I would be


interested what happens when you let it run on the ResNet, but let it have two connections


per per layer. Is it then better or not? So here the importance, I'll get to my idea later.


The importance of scale permutation, that's where they investigate how important is it


that you permute the layers and that turns out to be fairly important. Then the importance


of cross scale connections. That's how they investigate here. So these are these connections,


they say, the cross scale connections play a crucial role in fusing features at different


resolutions throughout the scale permuted network. So that's the reasoning behind it.


We we take features from different kinds of resolutions, and we can also scale up again


and then scale down again, to gain some additional features from the from the higher resolutions.


We study it's important by graph damage. So either they remove the short term connections,


or they remove the long range connections, or they remove both and then connect one block


to the previous block via a sequential connection. So this is only this is only in the things


that they learned, right. So this model is where they fully give their model control


over the ordering and connections, you can see that as this 40.7%. Now, if they delete


the short range connections, they drop to 35. If they delete the only the long range,


they drop to even more. So here, you can see that these long range connections, which I


guess are connections that are going across multiple blocks, skipping multiple blocks,


these tend to be very important. So you can make the case that it might be very important


to fuse these things from different layers to fuse the features from different resolutions,


because these long range connections tend to be important, though, it's one thing to


say that if we just leave them away with our model, if we just damage it and then let it


run, it it it drops in accuracy. It's not entirely the same thing as to say that these


are important because you don't really know what happens like if you train without them,


maybe you could if you train without them, you could reach as good an accuracy. So this


graph damage investigation, it has something but not I wouldn't trust it too much. And


yeah, I think they haven't investigated what happens if they keep the ResNet order but


let the connections be twice. But you get the general the general idea of the paper


right here of what they do. Now, they do this with architecture search right here. But here's


an idea. Okay, I propose the following, you have an image right here. And we are wondering


here, should we let it go through a layer that's wide and with less features? Should


we let it go through a layer that's, you know, very many features, but not as wide. But we


have to downscale the image? Or should we let it go first through something intermediate?


Let's see it like this. Okay, so we're wondering, how should we order these blocks? Why can't


we do all of at the same time? Why can't we do this, this and this? Okay. And then in


the next layer, again, do all of them at the same time. And you, you can already see where


this is going, I hope, I hope you can see where this is going. So you have a routing


right here. And how do we do routing in modern times in deep learning with attention? So


I propose you have layers with different attention. Let's say these are these are now your sequences


or you can also make them as attention heads. Okay, these are your these and the lower level


features are routed to the higher level features with an attention mechanism. And you do this


layer by layer by layer. So you let because what's the problem here? The problem here


is that the same data point has to go, you know, you find these good connections, but


the all the data points have to go through the same connections. And it might actually


be that you need different routing depending on the data point, it might be that what this


is, this is good for the average data point, but it will be much better if whenever there's


a cat, you take one path, and whenever there's a dog, you take a different path. So this


will allow for that, you basically have the routing parameterized by an attention mechanism.


This I have no clue how much compute this would take. It doesn't seem that outrageous.


Because what's your sequence length here, your sequence length is going to be the number


of layers maybe and maybe times the number of feature maps, maybe have different attention


heads. So you maybe want to replicate some of those here. But ultimately, I would guess


the attention mechanism itself isn't that much of an overhead. Maybe it's an overhead


that you have so many in parallel. Yeah, but you know, it remains to be seen. That's, that's


the idea. Yeah, you heard it here first. Okay, so they have more experiments. So they also


build here is where they say, Okay, we have the spine net 49. Now and we found this to


work. We found this to work really well. This is our spine net 49 architecture. Cool. And


you want to make it bigger. But I guess they didn't have the computational resources to


run the neural architecture search for bigger networks. This is now as about as big as a


resonant 50. Right. But what if you want to go to resonant 100 or resonant 150? There


you you don't have the computational resources do neural architectures. Imagine this Google


hasn't doesn't have the computational resources to do neural architecture search on this thing.


So this must be expensive, or I'm just, I have no idea. But what they do is they kind


of do a trick. So here they take the the spine net 49. And they say we build a spine net


6096 by simply repeating each block twice. So all the incoming connections would go to


the first block, and all the outgoing connections would come from the second block right here,


you had two in and maybe there's actually no limit to how many outgoing connections


you can have. And also, you can also do this three times, which I think is a bit of a cheap


way. And it kind of defeats the entire purpose, right? Couldn't you make the exact same argument


again here that maybe it's helpful to route from this block right here? Or maybe it's


helpful that these don't have the same scale, right after one another. It just seems, but


okay, so they say we found this good structure and we simply duplicate each block. I'm not


that big of a fan, in any case. So they train this and it of course outperforms everything


else. If you compare with kind of models of the same size. So here you compare this spine


net 49 to the resnet 50. And you can see there's about the same number of parameters, how about


it outperforms the resnet 50, pretty much. And as you go up the number of parameters


here, the performance goes up yet again. And I believe these dagger ones here are simply


trained with a special schedule with here with applying stochastic depth and swish activation


for a longer training schedule. So you can see that not only do are the spine nets, sorry,


the number of parameters is here. Not only are the spine nets slightly smaller than the


resnets, they do require less flops. And they reach better accuracy. So you know, every


everything is a win here. Yeah, so they apply this to these data sets, I don't want to go


too much into into that. But in the last part, they also apply this to image net. So there's


image classification, where they basically say, Okay, we can just go to our architecture


and we can just add up all the output blocks, we scale them appropriately, and add up all


the output blocks right here, because these are good features for localization and so


on. And we can train it to do image classification. So all of these go into a big combination


classifier that does the 1000 classes of image net image classification. And that also works


pretty well with this network. So they basically argue what they found is sort of a better


image image processing network than the resnet 50. And I guess they would argue that from


now on, you should take this as your backbone for image classification and recognition and


so on, which is entirely possible that this works better. There's no particular reason


why the resnet 50 should work at all, right? It's just a heuristic. But I guess it remains


to be to be seen, whether that's generally true or just in the things they considered.


So you can see right here, the spine net generally improving over the image net, which is not


stated right here, but it does generally improve. And you can see, as you go higher and higher


spine net, the numbers tend to improve as well. And this is already pretty respectable,


respectable number for image net, right? All right. So this was it for this paper for this


particular paper. They do have, you know, two different of these object detection recognition


data sets. And I invite you to check out the experiments more closely if you're interested


in that sort of thing. I was mainly interested in the method of doing and arranging these


layers and so on. It seems like it's a cool engineering project, cool investigative project,


the experiments are done well. And in the end, they reach a better, you know, they achieve,


they get a better model out of that. And if it turns out that this model is a good model,


the entire community will be better off. Unfortunately, there's no broader impact statement to tell


us that also the terrorists will be able to use this for purposes. But you can imagine


that yourself. All right, that was it for me. Again, leave a comment if you want me


to change anything or have suggestions. Leave a like if you liked the video, share it out.


Attention Is All You Need

Hi there. Today we're looking at Attention is all you need by Google. Just to declare,


Attention Is All You Need

I don't work for Google just because we've been looking at Google papers lately. But


Attention Is All You Need

it's just an interesting paper and we're going to see what's the deal with it. So basically


Attention Is All You Need

what the authors are saying is we should kind of get away from basically RNNs. So traditionally


Attention Is All You Need

what you would do, and these authors particularly are interested in NLP, natural language processing.


Attention Is All You Need

So traditionally when you have like a language task, like the cat eats the mouse and you'd


Attention Is All You Need

like to translate this to say any other language like let's say German or whatever. What you


Attention Is All You Need

would do is you would try to encode this sentence into a representation and then decode it again.


Attention Is All You Need

So somehow, somehow this sentence needs to all go into say one vector and then this one


Attention Is All You Need

vector needs to somehow be transformed into the target language. So these are traditionally


Attention Is All You Need

called sec to sec tasks and they have been solved so far using recurrent neural networks.


Attention Is All You Need

You might know the LSTM networks that are very popular for these tasks. What basically


Attention Is All You Need

happens in an RNN is that you go over the say source sentence here one by one. Here


Attention Is All You Need

you take the word the, you kind of encode it maybe with a word vector if you know what


Attention Is All You Need

that is. So you turn it into like a vector, a word vector and then you use a neural network


Attention Is All You Need

to turn this vector into what we call a hidden state. So this H0 is a hidden state. You then


Attention Is All You Need

take the second token here cat, you again take its word vector because you need to represent


Attention Is All You Need

it with numbers somehow. So use word vectors for that. You turn this into, you put it through


Attention Is All You Need

the same function. So here is what's like a little e for encoder. You turn it into the


Attention Is All You Need

same function but this time this hidden state also gets plugged in here. So the word vector


Attention Is All You Need

hidden state. You can actually think of having like a start hidden state here H start. Usually


Attention Is All You Need

people either learn this or just initialize with zeros that kind of goes in to the encoder


Attention Is All You Need

function. So it's always really the same function. And from the previous hidden state and the


Attention Is All You Need

current word vector the encoder again predicts another hidden state H1 and so on. So you


Attention Is All You Need

take the next token, you turn it into a word vector, you put it through this little e encoder


Attention Is All You Need

function and of course this is a lot more complicated in actual like say an LSTM but


Attention Is All You Need

it's the basic principle behind it. So you end up with H2 and here you'd have H3, H4


Attention Is All You Need

and the last hidden state H4 here you would use this in kind of exactly the same fashion.


Attention Is All You Need

You would plug it into like a decoder, little e decoder which would output you a word d


Attention Is All You Need

and it would also output you a next hidden state so H5. Let's just go on with the listing


Attention Is All You Need

of the states and this H5 would again go into the decoder which would output kapse. So that's


Attention Is All You Need

how you would decode. Basically these RNNs what they do is they kind of take, if you


Attention Is All You Need

look on top here they take an input, a current input and they take the last hidden state


Attention Is All You Need

and they compute a new hidden state. In the case of the decoder they take the hidden state


Attention Is All You Need

and they take kind of the previous, usually the previous word that you output, you also


Attention Is All You Need

feed this back into the decoder and they will output the next word. It kind of makes sense.


Attention Is All You Need

So you would guess that the hidden state kind of encodes what the sentence means and the


Attention Is All You Need

last word that you output you need this because maybe for grammar, right? You know what you've


Attention Is All You Need

just output so kind of the next word should be based on that. Of course you don't have


Attention Is All You Need

to do it exactly this way but that's kind of what RNNs did. So attention is a mechanism


Attention Is All You Need

here to basically increase the performance of the RNNs. So what attention would do is


Attention Is All You Need

in this particular case if we look at the decoder here, if it's trying to predict this


Attention Is All You Need

word for cat then or the next word here, say here it wants the next word and in essence


Attention Is All You Need

the only, the only, H6, the only information it really has is what the last word was, German


Attention Is All You Need

word for cat and what the hidden state is. So if we look at what word it actually should


Attention Is All You Need

output in the input sentence it's this here, eats, right? And if we look at kind of the


Attention Is All You Need

information flow that this word has to travel so first it needs to encode into a word vector.


Attention Is All You Need

It needs to go through this encoder. That's the same function for all the words so nothing


Attention Is All You Need

specific can be learned to the word eats here, right? It needs to go through this hidden


Attention Is All You Need

state, traverse again into another step, this hidden state because we have two more tokens


Attention Is All You Need

and then the next hidden state and then it goes all the way to the decoder where the


Attention Is All You Need

first two words are decoded and still so this H6, this hidden state somehow still needs


Attention Is All You Need

to retain the information that now the word eats somehow is kind of the word to be translated


Attention Is All You Need

and that the decoder should find the German word for that. So that's of course very, a


Attention Is All You Need

very long path. There's a lot of transformations involved over these, all of these hidden states


Attention Is All You Need

and the hidden states not only do they need to remember this particular word but all of


Attention Is All You Need

the words and the order and so on and the grammar, okay the grammar you can actually


Attention Is All You Need

learn with the decoders themselves but kind of the meaning and the structure of the sentence


Attention Is All You Need

so it's very hard for an RNN to learn all of this what we call long range dependencies


Attention Is All You Need

and so naturally you actually think well why can't we just decode the first word to the


Attention Is All You Need

first word, second word to the second word. It actually works pretty well in this example,


Attention Is All You Need

right, like the, the, cat, eats, we could just decode one by one but of course that's


Attention Is All You Need

not how translation works in translations the sentences can become rearranged in the


Attention Is All You Need

target language like one word can become many words or it could even be an entirely different


Attention Is All You Need

expression so attention is a mechanism by which this decoder here in the step that we're


Attention Is All You Need

looking at can actually decide to go back and look at particular parts of the input


Attention Is All You Need

especially what it would do in like popular attention mechanisms is that this decoder


Attention Is All You Need

here would, can decide to attend to the hidden states of the input sentence. What that means


Attention Is All You Need

is in this particular case we would like to teach the decoder somehow that aha look here


Attention Is All You Need

I need to pay close attention to this step here because that was the step when the word


Attention Is All You Need

eats was just encoded so it probably has a lot of information about what I would like


Attention Is All You Need

to do right now namely translate this word eats so this mechanism allows if you look


Attention Is All You Need

at the information flow it simply goes through this word vector goes through one encoding


Attention Is All You Need

step and then is at the hidden state and then the decoder can look directly at that so that


Attention Is All You Need

the path length of information is much shorter than going through all the hidden states in


Attention Is All You Need

a traditional way so that's where attention helps and the way that the decoder decides


Attention Is All You Need

what to look at is like a kind of an addressing scheme you may know it from neural Turing


Attention Is All You Need

machines or kind of other other kind of neural algorithms things so what the decoder would


Attention Is All You Need

do is in each step it would output a bunch of keys oops sorry about that that's my hand


Attention Is All You Need

being trippy so what it would output is a bunch of keys so k1 through k and what these


Attention Is All You Need

keys would do is they would index these hidden kind of hidden states via a kind of a softmax


Attention Is All You Need

architecture and we're going to look at this I think in the actual paper we're discussing


Attention Is All You Need

because it's going to become more clear but just kind of notice that the decoder here


Attention Is All You Need

can decide to attend to the input sentence and kind of draw information directly from


Attention Is All You Need

there instead of having to go just to the hidden state it's provided with so if we go


Attention Is All You Need

to the paper here what do these authors propose and the thing is they ditch the RNNs they


Attention Is All You Need

basically say attention is all you need you don't need the entire recurrent things basically


Attention Is All You Need

in every step of this decode of this and basically of the decoding so you want to produce the


Attention Is All You Need

target sentence so in this step in this step in this step you can basically you don't need


Attention Is All You Need

the recurrence you can just kind of do attention over everything and it will be fine namely


Attention Is All You Need

what they do is they propose this transformer architecture so what does it do it has two


Attention Is All You Need

parts what's called an encoder and a decoder but don't kind of be confused because this


Attention Is All You Need

all happens at once so this is not an RNN it all happens at once every all the source


Attention Is All You Need

sentence so if we again have the cat oops that doesn't work as easy let's just do this


Attention Is All You Need

this is a source sentence and then we also have a target sentence that maybe we've produced


Attention Is All You Need

two words and we want to produce this third word here I want to produce this so we would


Attention Is All You Need

feed the entire source sentence and also the target sentence we've produced so far to this


Attention Is All You Need

network namely the source sentence would go into this part and the target that we've produced


Attention Is All You Need

so far would go into this part and this is then all combined and at the end we get an


Attention Is All You Need

output here at the output probabilities that kind of tells us the probabilities for the


Attention Is All You Need

next word so we can choose the top probability and then repeat the entire process so basically


Attention Is All You Need

every step in production is one training sample every step in producing sentence here before


Attention Is All You Need

with the RNNs the entire sentence to sentence translation is one sample because we need


Attention Is All You Need

to back propagate through all of these RNN steps because they all happen kind of in sequence


Attention Is All You Need

here basically output of one single token is one sample and then the computation is


Attention Is All You Need

finished the back prop happens through everything only for this one step there is no multi-step


Attention Is All You Need

kind of back propagation as an RNN and this is kind of a paradigm shift in sequence processing


Attention Is All You Need

because people were always convinced that you kind of need these recurrent things in


Attention Is All You Need

order to to make good to learn these dependencies but here they basically say no no no we can


Attention Is All You Need

just do attention over everything and it'll actually be fine if we just do one step predictions


Attention Is All You Need

so let's go one by one so here we have an input embedding and say an output embedding


Attention Is All You Need

these these are symmetrical so basically the tokens just get embedded with say word vectors


Attention Is All You Need

again then there's a positional encoding this is kind of a special thing where because you


Attention Is All You Need

now lose this kind of sequence nature of your algorithm you kind of need to encode where


Attention Is All You Need

the words are that you push through the network so the network kind of ah this is a word at


Attention Is All You Need

the beginning of the sentence or is the word towards the end of the sentence or that it


Attention Is All You Need

can compare two words like which one comes first which one comes second and you do this


Attention Is All You Need

it's pretty easy for the networks if you do it with kind of these trigonometric functioning


Attention Is All You Need

embeddings so if I draw you a sine wave and I draw you a sine wave of that is double as


Attention Is All You Need

fast and I draw you a sine wave that is even faster maybe this one actually sync one two


Attention Is All You Need

three four five nah it doesn't matter you know what I mean so I can encode the first


Attention Is All You Need

word I can encode the first position with all down and then the second position is kind


Attention Is All You Need

of down down up and the third position is kind of up down and so on so this is kind


Attention Is All You Need

of a continuous way of binary encoding of position so if I want to compare two words


Attention Is All You Need

I can just look at all the scales of these things and I know how this word one word has


Attention Is All You Need

a high here and the other word is low here so they must be pretty far away like one must


Attention Is All You Need

be at the beginning and one must be at the end if they happen to match in this long wave


Attention Is All You Need

and they also are both kind of low on this wave and then I can look in this way for like


Attention Is All You Need

oh maybe they're close together but here I really get the information which one's first


Attention Is All You Need

which one's second so these are kind of positional encodings they're not critical to this but


Attention Is All You Need

algorithm but they just encode where the words are which of course that is important and


Attention Is All You Need

it gives the network a significant boost in performance but it's not like it's not the


Attention Is All You Need

meat of the thing the meat of the thing is that now that these encodings go into the


Attention Is All You Need

networks they simply do what they call attention here attention here and attention here so


Attention Is All You Need

there's kind of three kinds of attention so basically the first attention on the bottom


Attention Is All You Need

left is simply attention as you can see over the input sentence so that if I told you before


Attention Is All You Need

you need to take this input sentence if you look over here and you somehow need to encode


Attention Is All You Need

it into a hidden representation and this now looks much more like the picture I drew here


Attention Is All You Need

and the picture I drew right at the beginning is that all at once you kind of put together


Attention Is All You Need

this hidden representation and all you do is you use attention over the input sequence


Attention Is All You Need

which basically means you kind of pick and choose which words you look at more or less


Attention Is All You Need

so with the bottom right so in the output sentence that you've produced so far you simply


Attention Is All You Need

encode it into kind of a hidden state and then the third on the top right that's the


Attention Is All You Need

I think that sorry I got interrupted so as I was saying the top right is the most interesting


Attention Is All You Need

part of the attention mechanism here where basically it unites the kind of encoder part


Attention Is All You Need

with the kind of decoder that's not it combines the source sentence with the target sentence


Attention Is All You Need

that you've produced so far so as you can see maybe here I can just slightly annoying


Attention Is All You Need

but I'm just going to remove these kind of circles here so if you can see here there's


Attention Is All You Need

an output going from the part that encodes the source sentence and it goes into this


Attention Is All You Need

multi-head attention there's two connections and there's also one connection coming from


Attention Is All You Need

the encoded output so far here and so there's three connections going into this and we're


Attention Is All You Need

going to take a look at what these three connections are so the three connections here basically


Attention Is All You Need

are the keys values and queries if you see here the values and the keys are what is output


Attention Is All You Need

by the encoding part of the source sentence and the query is output by the encoding part


Attention Is All You Need

of the target sentence and these are not only one value key and query so there are many


Attention Is All You Need

and this kind of multi-head attention fashion so there are just many of them instead of


Attention Is All You Need

one but you can think of and as just kind of sets so the attention computed here is


Attention Is All You Need

what does it do so first of all it calculates a adult product of the keys and the queries


Attention Is All You Need

and then it does a softmax over this and then it multiplies it by the value so what does


Attention Is All You Need

this do if you thought product the keys and the queries what you would get is so as you


Attention Is All You Need

know if you have two vectors and the dot their dot product basically gives you the angle


Attention Is All You Need

between the vectors with especially in high dimensions most vectors are going to be of


Attention Is All You Need

kind of a 90 degree kind of I know the Americans do the little the little square so most vectors


Attention Is All You Need

are going to be not aligned very well so their dot product will kind of be zero ish but if


Attention Is All You Need

a key in the query actually align with each other like if they point into the same directions


Attention Is All You Need

their dot product will actually be large so what you can think of this as the keys are


Attention Is All You Need

kind of here the keys are just a bunch of vectors in space and each key has an associated


Attention Is All You Need

value so each key there is kind of a table value one value two value three this is really


Attention Is All You Need

annoying if I do this over text right so again here so we have a bunch of keys right in space


Attention Is All You Need

and we have a table with values and each key here corresponds to value value one value


Attention Is All You Need

two value three value four and so each key is associated with one of these values and


Attention Is All You Need

then when we introduce a query what we can do so query will be a vector like this and


Attention Is All You Need

we simply compute these so this is Q this is the query we compute the dot product with


Attention Is All You Need

each of the keys and and then we compute a softmax over this which means that one key


Attention Is All You Need

will basically be selected so in this case it would be probably this blue key here that


Attention Is All You Need

has the biggest dot product with the query so this is key two in this in this case and


Attention Is All You Need

softmax so if you if you don't know what a softmax is you have you have like X 1 to X


Attention Is All You Need

and B like some numbers then you simply do you map them to the exponential function each


Attention Is All You Need

one of them and but also each one of them you divide by the sum of over over I of e


Attention Is All You Need

to the X I so basically this is a renormalization basically you you do the exponential function


Attention Is All You Need

of the numbers which of course this makes the kind of big numbers even bigger so basically


Attention Is All You Need

what you end up with is one of these numbers X 1 through X n will become very big compared


Attention Is All You Need

to the others and then you renormalize so basically one of them will be almost one and


Attention Is All You Need

the other ones will be almost zero simply the the maximum function you can think of


Attention Is All You Need

in a differentiable way and you should just want to select the biggest entry in this case


Attention Is All You Need

here we kind of select the key that aligns most with the query which in this case will


Attention Is All You Need

be key two and then we when we multiply this softmax thing with the with the values and


Attention Is All You Need

so this query this this inner product if we multiply Q with K 2 as an inner product and


Attention Is All You Need

we take the softmax over it softmax what we'll do is I'm going to draw it upwards here we're


Attention Is All You Need

going to induce a distribution like this and if we multiply this by the value it will basically


Attention Is All You Need

select value 2 so this is this is kind of an indexing scheme into this memory of values


Attention Is All You Need

and this is what then the network uses to compute further things using so you see the


Attention Is All You Need

output here goes into kind of more layers of the neural network upwards so basically


Attention Is All You Need

what what you can think what does this mean you can think of here's the whoops deep I


Attention Is All You Need

want to delete this you can think of this as basically the encoder of the source sentence


Attention Is All You Need

right here discovers interesting things that's that looks ugly it discovers interesting things


Attention Is All You Need

about the about the the source sentence and it builds key value pairs and then the encoder


Attention Is All You Need

of the target sentence builds the queries and together they give you kind of the next


Attention Is All You Need

to next signal so it means that the network basically says here's a bunch of things here's


Attention Is All You Need

a here's a bunch of things about the source sentence that you might find interesting that's


Attention Is All You Need

the values and the keys are ways to index the values so it says here's a bunch of things


Attention Is All You Need

that are interesting which are the values and here is how you would address these things


Attention Is All You Need

which is the keys and then the other part of the network builds the queries it says


Attention Is All You Need

I would like to know certain things so think of the values like attributes like here here


Attention Is All You Need

is the name and the kind of tallness and the weight of a person right and the keys are


Attention Is All You Need

like the actual indexes like name height weight and then the the other part of the network


Attention Is All You Need

can decide what do I want I actually want the name so my query is the name it will be


Attention Is All You Need

aligned with the key name and the corresponding value would be the name of the person you


Attention Is All You Need

would like to describe so that's how kind of these networks work together and I think


Attention Is All You Need

it's a it's a pretty ingenious it's not entirely new because it has been done of course before


Attention Is All You Need

with all the differentiable Turing machines and whatnot but it's pretty cool that this


Attention Is All You Need

actually works and actually works kind of better than RNNs if you simply do this so


Attention Is All You Need

they describe a bunch of other things here I don't think they're too important basically


Attention Is All You Need

that the point that they make about this attention is that it reduces path lengths and kind of


Attention Is All You Need

that's the the main reason why it should work better with this entire attention mechanism


Attention Is All You Need

you reduce the amount of computation steps that information has to flow from one point


Attention Is All You Need

in the network to another and that's what brings the major improvement because all the


Attention Is All You Need

computation steps can make you lose information and you don't want that you want short path


Attention Is All You Need

lengths and so that's that's what this method achieves and they claim that's why it's better


Attention Is All You Need

and it works so well so they have experiments you can look at them they're really good at


Attention Is All You Need

everything of course of course here was a state of the art and I think I will conclude


Attention Is All You Need

here if you want to check it out yourself they have extensive code on github where you


Attention Is All You Need


Fourier Neural Operator for Parametric Partial Differential Equations (Paper Explained)

AI has cracked a key mathematical puzzle for understanding our world. This just in from MIT


technology review and look at this puzzle right here. It's got the bumps, it's got the valleys,


the surfaces, it's got the braille, it's got the bits, the ones and the zeros, not only going up


and down like in the matrix, but going in circles. It's got it all this puzzle is really hard,


as you can see, and AI has just cracked it. I'm being a bit hyperbolic, of course, this is actually


about a new paper that can solve numerically solve a particular type of partial differential


equations way faster than any thing before it. So this is about this, this new paper, and we'll


get into the paper in a second. It's pretty cool. But as you can see, MC Hammer, the infamous MC


Hammer has tweeted this out. And, and he is actually a pretty cool Twitter feed on where,


where he regularly tweets about scientific papers and so on. So pretty cool cross domain overlap,


I recommend that. So we'll get into the paper, we'll get into the code a little bit as well,


because I think it helps to understand what's going on. And I want to start out by this is the


the blog post by one of the authors. And it's pretty good to get a basic overview of the paper.


And here is the motivational example. So the motivational example is the Navier-Stokes equation,


which is an equation in fluid dynamics. So you're trying to predict how a fluid evolves over time,


given a certain parameters like its viscosity, and a forcing function. So basically how sticky it is,


and how hard you stir it. And then you want to know how it evolves over time. And you can see on


the left is given an initial condition. And I think on the right is sort of a rollout after


the 10th time step until the 50th time step. And the ground truth is obtained with a sort of classic


numerical solver where you do little time steps and you calculate the interactions. And then this


takes a lot of time and compute. And on the right is the prediction of this new Fourier neural


operator that this paper develops. And you can see it's almost equal. And the gist of it is that


the thing on the right simply takes one forward propagation through a neural network. So it takes


like super like point zero, zero, something of a second to compute the thing on the right,


whereas the thing on the left is quite hard to compute and as I understand can take minutes. So


you here you see the motivational example, these things are described by partial differential


equations, which are sort of linearized, linearized ways of describing how the system evolves over one


time step. And it'd be cool if we could solve these faster, because this is applications in


aerodynamics and other types of engineering fields. Alright, so let's jump into the paper.


And as always, if you like content like this, consider sharing it out, telling your friends


about it and subscribing, of course. So the paper is called Fourier neural operator for parametric


partial differential equations. And it's by Tsong Li, Nicola Kovatsky, Kamya Aziza-Denesheli,


Buregere Liu, Kaushik Bhattacharya, Andrew Stewart and Anima Anandkumar of Caltech and Purdue


University. So I feel I feel the paper is both very cool and a bit overhyped. So we're going to


see what it does. It's it's for a particular type of PDEs. And it has a lot of, let's say,


engineering choices that make it possible to solve with neural networks, but also that limit its


applicability to where the classical methods would be applicable where this thing isn't. So there are


trade offs, definitely to reach the sort of speed up that they reach. But we'll get into this. First,


I actually want to scroll down right here all the way because there is something that you don't often


see in the sort of machine learning field. And that is here in the acknowledgments section. And I


just, you know, I just find this I just find it interesting. Don't don't don't regard this as


anyone. But here we are supported by the LWLL grants, which I understand is DARPA beyond limits,


which is like a makes soft or makes AI or systems for things like gas and oil and so on with British


Petroleum as a main sponsor, Raytheon, which of course, is a giant military manufacturer, we have


the Army Research Laboratory, and so on. So. So you can see that this, this is kind of, I don't know,


I don't see this often. This is sort of a good bouquet of sponsorships. Of course, there's also


Microsoft, Google, and so on. Yeah, but it's just it's just interesting to see that that the the


army is pretty heavily into these things. And of course, they would be I mean, rockets need to fly,


and they need to be aerodynamic and so on. So yeah, not saying this is bad or good. I just


thought it was, it was interesting that you know, Raytheon would would be a sponsor of this. Alright,


so let's dive in. As we said, we're interested in these types of problems right here, where you


have this thing called so the there's this quantity called the vorticity, which as I understand is a


derivation of the viscosity. So it sort of tells you how the fluid is is moving right now. And so


this this state right here, and then you apply a sort of constant forcing function. And you want to


know how that evolves over time. So you can see at time step 15, you get sort of this picture. So


these these move past each other and see this moves here, this moves here. And then at time step 20,


you can see they are fairly moved. Okay, this blue thing moves in here as well. And they just sort of


mix and there's this, there are certain parameters that make the fluid more sticky or not so sticky.


And the interesting regimes is I guess when it's not very sticky, so not not too sticky, but also


not not sticky enough. And then these really complicated patterns occur. And to predict them


would be very, very valuable. So you want something that takes in this initial state right here, and


outputs all of these these future states. And usually this is done by these classical numerical


solvers. So the Navier-Stokes equation is described by a set of partial differential equations. And


you can see this down here. So Navier-Stokes equation is described by this set of equations


right here. Is there? Yep. And you can see that the that this this is fairly complex. It includes


partial derivatives, gradients, and so on. So this is the this is this vorticity. And it includes


that on on both sides. And this is this the Yeah, this is two derivatives, maybe, or is it just the


delta? I don't even know. I'm I'm not an expert in partial differential equations by any means. So


anything coming from that direction, don't take me for granted, I'm going to give you sort of the


under the thing of what I understand from this paper. And so with respect to that entire area,


I'm not an expert, I just can understand that this is fairly complex. And what you usually do is


you take the initial state, and you just evolve it in time. So you take this time parameter, and


you do you go one little, little time step. And then you calculate because these are all sort of


linear linear equations, you calculate this one little time step into the future, you update your


state, right? It's it's sort of like, you know, your points here and how they move, and how they


move is given by their gradients. So these are all sort of linearized things. Now, you don't want to


move them too much per time step. Because ultimately, if this thing moves, and this thing


moves, then the movement of this arrow will change because this thing over here moves, right? So you


want to compute this one little time step into the future, like to here and this to here. And then


you want to recompute all of these arrows. So maybe now that points a little bit more here, and


that points a little bit more here. And then you want to update it again. So you have these sort of


these these numerical solvers that go little tiny time step by little tiny time step. It's not even


this if here if you see t equals 20, or something, it's not 20 times that for these solvers, but


these usually go like 1000 or 100 steps per time step that is here, or something like this, they


need to take very tiny steps to be accurate. And that takes a long time. So the idea is, can we


simply can't we simply simply input this, let's say this thing or like something at time 15, and


directly predict the thing at time 30. And that's exactly what this paper does. And a lot of papers


have done this before, but without much success. So this paper proposes to do this in the Fourier


domain, and we'll see the path that they take right there. So they go into the will shortly go


into sort of the basics right here. So what you want, what you're looking for, is a function g,


that takes an A and gives a U. So what are A and U, A and U are both function spaces. So a, a and


U here are functions. So a is a function, as you can see, a is a function, and U is a function,


but you can characterize them as data points. So in this, in this way, there is a functions and data


points are sort of interchangeable, you can see an image like this as a data point, where it's an


image, but you can also see it as a function where every x and y coordinate is mapped to a value,


right. So when when they talk about functions, very often they talk about this type of function,


where you have x, y and t. So t is also t is zero here, x, so the function would x, y, t, map that


to some value, right, we hear the vorticity. And you want to transform this function. So this


function would be a, a would be the function at time, let's say zero or something, or times zero


to 15, you would want to map that to the function, the function u, that also takes an x and a y,


let's leave t out for the moment, also takes an x and a y and let's say t, but t is set to 30. And


maps that to a vorticity, right? So you want to input a function and output a function, but it's


the same as inputting an image and outputting an image in as it for from an engineering perspective,


of course, from a math perspective, it's a little bit different. But other than that, it's a fairly


standard machine learning problem. So you have this, these sets A and U, and you're looking for


this function g that maps a to u. Okay, so we studied maps, which maps g, which arises the


solution operators of parametric PDEs. Suppose we have observations, where a is an iid sequence from


probability measure mu supported on i and u is the a transported by g, it is possibly corrupted with


noise, we aim to build an approximation of g by constructing a parametric map, this g right here.


So it's a bit of a mathy way of saying, we have a bunch of data points, where we were a, this is


the initial state goes to u, which is the state at some point in time. And we know that there is


a function g, this is this g with this inverse cross, we know that there is a true function that


maps any a to u. So a single function g, that can if I input the initial state can give me the output


state. And what I want to do is I want to approximate this by a parametric version. So these


here are the parameters. And of course, as you can guess, by now, g is going to be this g right here


is going to be a neural network that is parameterized by theta. So these would be the


layers of the neural network. And we're going to input a into the neural network, and we're going


to get out u. So that's basically that there is quite a bit of math right here. And the math here


is to derive what they call a neural operator. So here is one layer of this neural network. As we


said, we're going to input a. Now a first thing that we do a is going to be, let's say up projected.


So a is going to be made into a latent representation v zero. So this is let's call that


here, p. So there is a function p, which is going to be a little layer of neural network.


And it is going to produce this v zero. So v zero is going to be a latent state of the neural


network. And then there is going to be a number of these layers that transform this to v one v two,


v three. And we've I think there are four layers of these in their particular implementation,


but there don't need to be four layers, you can choose that as you can choose any depth of neural


network. And then at the end, you're going to project that down to whatever output you want.


So you okay, so this function here is called q. And these are just going to be neural networks. So p


and q are going to be your very, very classic up projections and down projections of data point


we'll get into, actually, we'll get into sampling. Let's go actually right now. So so


one thing right here, and they stress this is that they work in function space, right? They they


don't they don't work on the let's say they don't map the data point to the data point. What you


could do is simply have like a convolutional neural network, an image to image network, and so on. But


what is the problem with that? So if you have your a, which is your initial state, and it has, you


know, it has these bunch of fluid things right here. And what you do when you have an image is


you sample this, right? You sample this a different sorry, maybe a regular grid. I am terrible at


regular. So you sample this into a certain amount of pixels, and your neural network will operate


on this, right, this will give you some kind of a tensor, which is, let's say we have a so this is


a seven by seven grid, okay, so your neural network is going to expect this as an input dimension.


And whatever you is, of course, so you map this to you, which is also going to be some sort of


image, okay, where you need to output pixels. So again, you have some set resolution, and your


neural network can only operate at that particular resolution. What they're doing right here is


the cool thing about is it can operate at any resolution. So once you've learned the network,


you can input higher resolution images, or you can output higher resolution images, any, any sort of


you can deal with more resolution, less resolution, sampled irregularly, you can deal with a lot of


things once the neural network is their neural network is learned that how do they do it? They


do it by only ever acting point wise in the spatial domain. So what they're going to do is


they're going to take this a and now we get into the more critical things. So here a and do aren't


just the beginning state and the end state. In fact, in this Navier-Stokes example, a is a tensor


like this. So a is going to be a tensor with slices. And each slice describes one time step


up to a given time. So this here could be t equals zero. So there is kind of the initial


distribution, and then t equals one and so on, up until t equals like 10, let's say, I think they do


10. So it they let this thing evolve for 10 time steps, and I'm going to guess they do it using one


of these classical methods. And that's the input. So the input isn't just the initial state, the


input is actually here is what happened in the first time 10 time steps, and then the output


isn't just the output at some particular time, but the output is actually also a slice right here.


Sorry, a sliced tensor. So each slice here describes the output at a particular time. So


this would be t equals 11, up until t equals 50. Okay, so this is this is you. So the top one is


sort of the conceptual thing. But the bottom one is what really happens. So they input 10 time steps


and they get out the 40 subsequent time steps, they predict them all at once. Okay, so and now


you can see that in this particular case, how I can understand this is at each pixel here, I want


to know what what is that pixels value after what after like, certain amount of time steps, okay,


like 11 or 50 right here or 40. And, of course, the result is going to not only depend on the time


zero, but on the entire evolution of time zero to time 10. So this here is an entire column for that


pixel. And this is akin to that particular pixel having this many channels. So here I can see


Well, these are technically 10 channels or 11 or something like this, I've probably screwed up,


this should be t equals zero to nine, and then 10 to 49. But so this is, this is an entire stack,


this is, we can interpret this as input channels right here. And we can interpret these as output


channels. Okay, so ultimately, one pixel is going to have input channels, all the time steps that


happened up until the point where we want to predict, and the output channels are going to be


at the same time, all the time steps of what we want to predict. Okay. So this is the input


of what we want to predict. Okay. So these projections now coming back to this, they simply


work in the channels. So these p and q, they are one by one convolutions. And the one by one


convolution simply up project and down project. These features, you see, these are one by one


convolutions, actually, they could be dense layers, let's check that in the code later. But for sure,


what they do is they only work point wise. So they don't, they don't mix the individual pixels


together. In here, you simply get at like a D by D grid with each has 10 channels. And then you simply


up project that to so here you have D by D times 10. And then you up project that using p to D by D


times and here is a parameter that you choose. So this is sort of your latent dimension,


okay. And you are going to transform this tensor keeping it in this D by D by W dimensionality,


until you back projected using q to D by D by in this case, 40. Okay, so, but this, this and this,


they only work point wise. And that means there is no particular dependence on the D right here. So


the next data point could actually have a different D as long as this pipeline right here can handle


different dimensions, because the p and q only act point wise, you're good. So what do what do


these magic layers here do? So these are these Fourier neural operators, okay, they transform one


hidden state into the next note that we have four of these layers. So they don't need to be the same


as the number of time steps we're trying to predict, you see. And it's pretty clear from here.


So we, these four hidden layers, they're simply transforming this entire volume right here,


this entire input volume, they are transforming this as a sequence of latent states, and then


outputting this entire volume. So this down here has nothing to do with the time steps that we're


trying to predict, it is simply a sequence of computations of latent computations. And you know


that in a neural network, the deeper you make it, the sort of more complicated functions arise,


even though of course, the universal approximation theorem says that with one hidden layer, you can


do anything. But in general, if you have deeper neural networks, the more you can kind of make


more complicated things. And so four seems to be a good number of complicated for these particular


problems. So here's what one of these layers does. It is very much like a residual network. So here


you have v, the v is the hidden representation at t plus one and t plus one is not, as I said,


is not the time step in the sim in the Navier-Stokes sense of time evolution of the PDE. This is simply


the layer t plus one. So I don't know why they maybe Yeah, maybe t here makes still makes sense,


is it not because it's large t. Yeah, so they have large t right here. Okay, maybe but in the


engineering sense, it is not it's simply the layer. And you can see it's formulated as a function.


But again, don't be like the x right here. This is simply the x and y and t coordinates. So this,


this, all of this here can be represented as one big tensor, x, y, t, or x, y channels or something


like this. Okay. Don't so don't, don't be confused by the fact that these are formulated as functions.


So what we want to do is we have two different things. So one neural, this is one neural network


layer, as you can see, at the very end is a non linearity. This is a pointwise non linearity. And


this is in the original pixel space or in the original spatial space, the d by d space, each of


the things gets a nonlinear function slapped on top, as is normal, then this part is normal as


well. This is simply a linear transformation of the input. Again, this is pointwise. Okay, so


this is a linear transformation. Okay, so so far, so good. We have a linear transformation of the


input and a non linearity. The important part is this thing here. So what this thing is, this is a


kernel function that depends on the initial condition. So not only on the last hidden state,


but the initial condition, and sort of is then multiplied by the last hidden representation,


like, like here. And then only x is applied. So notice the difference right here. This is at a


point x, we're getting this function value, which means we're getting the entry of that tensor. And


then we're applying the linear transformation. Okay, this makes it pointwise. So this is a


linear transformation. This makes it pointwise. Here, first, we compute this function by this by


applying this kernel to the input function, so to the entire input tensor. And only then we are


looking for the particular entry. So that means this thing here is a pointwise transformation of


the input tensor. And then here, it takes in the whole tensor and outputs a sort of new tensor. So


this is going to be the magic here where k it goes, you can see it goes from from u space to u


space maps to bounded linear operators on you. And is parameterized by theta. Maybe what's this? I


don't know. So the this this kernel, we choose this to be a kernel integral transformation


parameterized by neural network. So they define the kernel integral operator as this. And you can


see this is an integral over the D, D is the input space of u and a actually. So this is a function


that's dependent not only on where you are in the tensor, but on the initial input this a and then


that's convolved. So this here is a a integral over the entire space. So that's convolved with


V, you can see that this is a convolution. And it's fairly complicated. So this alone tells you


nothing. But luckily, they say that they restrict this. So it's a bit annoying, when things always


depend on this a. That means that each of these functions right here, each of these arrows right


here, these are the neural operators, actually, let's go here. Each of these Fourier neural


operators right here, they would always also depend on this a here, like this, and like this,


and like this. This is a bit annoying for deep learning, because we sort of want one layer's


representation to go into the next one. So they simply make an engineering choice and say,


Nope, nope, nope. So. So they say, we impose, right, we impose, if we remove the dependence


on the function a, we impose that the kernel is simply a function of x, not only x and w, but


only x minus w. So now you have a sort of proper kernel function in there that we can handle.


We obtain that four is a convolution operator. Okay, it wasn't the convolution before it was


just an integral. But now if you restrict your kernel functions to this, you get a convolution,


we exploit the fact in the following section by parameterizing K directly in Fourier space and


using the fast Fourier transform to efficiently compute for this leads to fast architecture,


which obtains state of the art results for PD problems. So there's quite a bit of math right


here. To finally arrive at this thing here. So what is all this math for? This math is for saying


saying what we want, we want to build our neural network like this, okay. And what we do is we


simplify and specify this kernel thing, until the kernel looks something like this. So we restrict


the kernel to be a convolution. And since a convolution in Fourier space is just a multiplication,


what we can do is, instead of taking the function v and convolving it with this kernel,


what we can do is we take the Fourier transform of the function v, then multiply it in Fourier space


by this thing. And this thing is now simply a matrix that's learned in as as a bunch of parameters.


And then we do the inverse Fourier transform. Now, you might ask, why is this relevant? Why can't we


just why can't we just do a convolution like we do normally? And the reason is, so when you do a


Fourier transform, what do you do, you have a some some kind of signal, like, and so on, you have a


signal, and you transform this into Fourier space. And here, we just go like one vector. So here,


as you know, in Fourier space, you have these basis functions, which are sort of these different


parameterization of sine waves, or you can do it with cosine waves, and they get faster and faster


and so on. So you know that you can decompose any signal into its basis functions in this kind of


periodic function space. So this function right here, it might have, you know, one times this


function, plus 0.1 times this function, plus two times this function minus five times this function,


and so on. So you can describe any any of that. Now for these type of PDEs that we're looking for,


the special thing about them is they are fairly well described if you simply cut away the sort of


top Fourier modes and only work with these because they are, you know, sort of the the individual


tiny ripples, you might not want to take into account. So you can truncate the lower Fourier


modes. And that's what they do exactly here. And they learn. So instead of transforming this signal


directly into the next hidden representation, they go to Fourier space, cut the top Fourier modes,


they have a way of making the next representation in Fourier space. And this is this r here. And


that is simply a weight matrix that they multiply with. And that is you can you can prove that that


is the same as convolving in or within the original space. So multiplying in Fourier space


is the same as convolving in the original space. And so they multiply the green numbers right here


by r, then you get something out. So I should maybe this is way too much. So the green numbers


you multiply by r to obtain new green numbers. So maybe r is the is two, two, four. So the new


green numbers would be two 0.4. Then you do the inverse Fourier transform. So you get back to a


signal now with two times this, so it might be bigger, and point four times so I can't even draw


but you sort of get the idea, you put it into Fourier space, you apply the function r, which is


a multiplying by a matrix that you learn in Fourier space, you get new Fourier coefficients, you map


them back. And there you have your next layers representation, almost okay. So this is this


Fourier neural operator and is described right here. What you do is you take your representation,


your hidden representation, put it through a Fourier transform, which you can do in a


differentiable fashion, you get these Fourier modes, which describes how to decompose the signal


into these periodic functions, you throw away the top modes, which is your sort of regularization.


You apply r, which is in a dense layer of neural net, not even that it's a multiplication, okay,


by a weight matrix. And then you obtain this, these new Fourier modes, you do the inverse,


and then you have the next representation almost what you do is we saw this before a point wise


transformation in the original pixel space. So this is very much like a residual network, right?


Residual networks, they also have this, they have the implemented as one by one convolutions. So


and then at the end, you apply the non linearity. What is good about this? Two things. First of all,


throwing away the top Fourier modes is very advantageous to these types of problems that


we have right here. You can see that the little the little jiggles right here, they


will be sort of sorted out by the larger scale movements of the fluid. So throwing away the top


modes is a sort of a regularization, it helps with generalization. And it's very easy and Fourier


space. So these things other than natural images are described well by these Fourier spaces. And


that, again, is an engineering choice. So you cannot not apply these things to everything, you


can apply them to where this type of assumption holds. Second of all, this is now fully independent


of the discretization of the input, okay? Because when I take a picture and I sample it in a three


by three gate, I can do a Fourier transform. And I'll get all of these numbers right here. Okay,


it's just, you know, the Fourier transform does a good job as possible. But the problem is that


it's not possible when I sample it in a seven by seven grid. Like I sample it super densely,


I do the same for transform, I get the same numbers right here, okay. And it's not exactly


the same. So they always claim it's the same. It's not exactly the same. Of course, if you don't


sample densely enough, your Fourier transform isn't going to be as accurate, let's say. So ideally,


you can sample this in a three by three grid, but it's not going to be as accurate as the


Fourier transform of the real signal of the real underlying signal. But since you sample this,


you can't have this. So there is a bit of a difference, but it is independent. So that's


true. So the function are that you learn, simply operates on these Fourier modes. And these are


fairly independent. Yeah, so so that's, that's good. So if you if you have what they're going to


do is they're going to have something like the three by three during training and then sample


more densely during during inference, which is something you can do but understand that this is


just it's just a form of interpolation, right? So the inverse Fourier transform simply gives you


whatever you want interpolating using the Fourier modes it has. And of course, given a certain


number of Fourier modes, which is quite small for them, I think it's something like eight or 12.


Higher resolution at some point doesn't help you anymore, because you've cut off the high


resolution Fourier modes, I guess what can help you is this, this thing right here, but this thing


here only acts point wise. So you see, this is now fully independent of the discretization of the


signal, which is a cool thing. So the two cool things about this entire stuff is that first of


all, independent of discretization. Second of all, these types of problems that we are having here


lend themselves very well to be described in Fourier space. Yeah, so that's why I'm saying,


this is for a particular type of problem. And also, there are a bunch of other things you can see


right here, you have this entire input tensor right here, and this entire output tensor right here.


And these can be fairly large, right? And all the intermediate representations have to be kind of


d by d by w. So this is, you can't go infinite time right here, like you could with a classic


solver, like a numerical solver, all you need is the last time step, right? You go, what's the t


equals one, then t equals 1.11. point two, and so on, you just count up, and you just go always from


the last time step to the next time step here. Since it's a neural network, during training,


you need to keep all of these tensors the intermediate things, I guess you can do gradient


checkpointing, but this is engineering wise, you predict all the future time steps at the same time.


So you can't really go infinite in time. And how do you train this thing? You train it by simply


giving it one of these a right, you have a you have a bunch of a's. So you have a bunch of these


input tensors, a data set. And where you always say here is a one of these Navier-Stokes equation,


sorry, type of problems. I've sampled it somehow. And I've let it run for 10 times steps.


And then I've let it run for longer, you. So I let it run for longer. And here are time steps


of this t equals zero to t equals nine or 10. Let's go 10. And here is t equals 11 to t equals


50. Okay, so you have a data set. And this data set is fully computed by a classic forward solver.


So you can't replace the forward solvers right yet, because you need them for generating training


data, right? So this becomes your training data, this becomes generally your x and this becomes


your y. And now you're learning this neural network, this entire thing to give you x to y.


So you see, you still need the classic solvers to produce the training data. That's the first thing.


The second thing is you can pretty clearly see that the good thing is that now we can input any


a so the classic solvers, you need to rerun them for each initial condition. Now we simply train


with a bunch of initial conditions trained in neural network to predict what happens then. And


then it can generalize to other initial conditions. But you know about generalization that the


problem is we can, we can only trust our neural network, if the problem we're considering is very


similar to what we had in the data set, it doesn't arbitrarily generalize, okay. So that is, you


know, it's something to remember. So I said all of these things have trade offs trade off one there


is you have to predict all time steps at the same time, which is hard on your memory, right? It limits


the size of things you can do. trade off to you can only really trust your neural network if the


problem you're considering is within your data set vicinity. There are other problems that we've


mentioned problem three, we've made very specific choices with respect to how our kernel looks that


it's only ever dependent on x minus y. So therefore it is a convolution. There's all these


these channels, you know, engineering choice, more you cut off the top Fourier modes, which limits


the types of signals you can analyze. The next choice is the number of intermediate computation


steps right here, which limits the complexity you can assume and so on. So there are just, I'm not


saying you don't have choices in the other numerical solvers, you probably do. But just to


remember there, that that this is the case. So someone might say, well, can't you can't you just,


if you want to predict for longer time steps, you could make this t equals 11. And then simply, you


know, not not go in slices of one, but maybe going slices of 100. So this could be t equals 111. This


could be t equals 211, and so on. And that is completely, completely valid. What they actually


do is they subdivide the space further. So instead of doing like 40 time steps, they are doing like


80 time steps, but still times 11 to 50, I believe. The problem with extrapolating like like this and


leaving away time steps is that see here, you have a supervision signal in your training for each of


the times. And it, it might be that the fact that so you know, time step 15 looks something like


this. And I know I'm trying to end this time step 16 is just like a small evolution, like this from


right, it's it's like a small difference. And it could be that the neural networks because they


don't have internal dynamics, right, they don't internally, like dynamically simulate this physical


system, they simply learn to map things to things. And if, if they are still related to each other a


lot, then sort of they can make sense of it. So if one slice, so this could be the slice 15. And


this could be slice 16. If, if these are sort of related, you know, it can, it can make sense,


there is a relation between them. Also, you can implement this as an RNN. And then also, from one


step to the next, it's sort of makes sense, you don't need an internal dynamic simulation. However,


if you jump from time step 15 directly to time step 115, right, then it might look like it might


look nothing like it, right, because it has evolved so much. And there can be quite chaotic dynamics.


And that's the entire problem with PD is that the dynamics can be super complicated, and not easily


predictable. So here, you don't really have a relation, right. And so since the neural network


doesn't do internal dynamic simulation, it probably wouldn't, I'm going to guess something like this


wouldn't work too well, I could be wrong. But I'm going to guess classical solvers are still needed


for this type of situation. So that's the other limiting factor is that you sort of are bound to


data samples that can be statistically correlatively predicted from one another,


without having to do these physical, the real physical underlying simulations, though, I've been


proven wrong in the past. Alright, so they talk a bit about how the fast Fourier transform plays


into this. And there is actually an interesting thing, which we'll see at the code. And then they


have three examples, like the Darcy flow burgers equation and Navy Navier Stokes equation. And they


also do these Bayesian inverse problems, where I believe the what here what you have is sort of a


thing at time step, you have the bottom thing given at some time step, and then you want to find out


the original thing. And what you do is you have like an algorithm that is simply guessing. So you


have a U given and you want to find out the A, so the A is unknown. So you simply start with a zero


and guess what you is going to be from that a zero. So you evolve your state a to you. And then


if it's not entirely correct, you try again, you try a one, okay, what does that give me now? You


see, you kind of play a game of guessing and you have an algorithm that does this guessing kind of


smartly. So it says, Oh, no, that's not the direction I want to go to. It's sort of a reinforcement


learning algorithm a little bit. And the important part is it needs to do a lot of these forward


evaluation, right, it needs to change a little bit, and then evaluate and see if the U that comes out


is the same as the U that you want. So you want to find the initial state of any given evolved state.


And if you need a lot of forward evaluations, it's going to be a problem if the if the forward


evaluation is really slow, like these classical simulators. So these neural networks can really


help right here. And I think they bring it down, they bring down the time it takes from 18 hours


or so to two and a half minutes for this entire evaluation. So that's pretty cool. And they also


outperform actually, in terms of error, they outperform these these kind of baseline methods.


So this is pretty cool as well. So not only are they faster, they also are less error prone. All


of this pretty cool. Now let's just spend like a short time to dive into the code. The code is still


quite a bit quite hacky. But that's research. So deal with it. So here you can see that


that the top class is what this called this net 2d. So net 2d. I always I like to look at the forward


pass before I look at the how the network is made, because you understand how things flow. So in the


forward pass, you simply have this conv this this convolution right here. What's called conv one,


it's not really a convolution, right? This is this is simply an instance of this simple block and x


is just passed through it. So this simple block right here, by the way, the data is prepared. As


you can see, there is quite a bit of preparation going on. So you have a and you have u. So a as


you can see, is prepared as an s by s, that's the discretization of the grid by t in. So this is


your d by d by 10. Like this is 10 input time steps. And it is already expanded to a t tensor.


So the t is going to be the output steps that we're going to consider. So here, a is going to be


transformed repeatedly into a a tensor that ultimately will have t output time steps. You can


see you have to hold one of these things in memory for each training sample. And then you annotate


actually x and y and t. These are like positional encodings. For if you know transformer positional


encodings, these are simply linear positional encodings for x, y and t, you concatenate those


and off you go. So where were we? x was forward passed through this simple block 2d. What's the


simple block 2d? The simple block 2d is this thing right here. So again, let's look at the forward


pass. So first of all, we're going to FC zero, which what looks like a fully connected layer,


we're going to permute the axis, then we're going to through conv zero, w zero, a batch norm, and


a relu. So you can see this right here is what we saw in the diagram, x one and x two are the


different paths through the network. This is the top path. If I go back to the paper quickly,


this is the top path in this diagram. Okay. And the bottom path is this thing right here.


And then there, the two are added. And then there's a batch norm, which is not in the diagram.


And then there is a relu. Okay, so the bottom path is pretty simple. And you can see right here, by


the way they restructure it, that this is going to be point wise. So this is not going to be in


pixel space, this is going to be a point wise, only in the channel transformation. So these w's


are implemented as one, one by one convolution, you see it's a 1d convolution, and the kernel size is


one. So all these does is for each point, for each point in the grid space in the pixel space for


each pixel, they're going to take this all of this pixels channels and transform this into a new


vector of the same amount of channels. So you can see the input channels and output channels are


always the same dimensions. So actually, this entire network right here operates on this width,


which is this latent dimension. It's only the first layer that transforms this from 13, which


is 10 plus the three positional encodings to this latent dimension. And then the last network,


this transforms it from the hidden dimension to 128, for some reason, and then 128 to one,


which is each pixel has a one dimensional output, which is this vorticity that you're trying to


predict. And by pixel here, I mean an x, y, t entry. Okay. All right. So yeah, so exactly. So


this goes from 13 to one, and then it is reshaped again, of course, to the to the appropriate size


to give you all of the outputs. Okay, so you can see this is the input, this is the output down


here. In between, we have four blocks of this upper path and lower path. So the upper path,


sorry, the lower path we just saw is a one by one convolution. And the upper path is this conv zero.


So this conv zero is this spectral conv 3d fast, okay. And it's parameterized by these modes. So