the modes is how many of these Fourier modes you want to retain, we saw we throw away the top


Fourier Neural Operator for Parametric Partial Differential Equations (Paper Explained)

Fourier modes, whatever they are. And the modes here is whatever you want to retain in this case


is set to four, which is actually eight, if you work it out, and we'll see why. So the spectral


conv 3d fast, again, let's look at the forward pass. So what does the forward pass do, it does


a Fourier transform, a fast Fourier transform. And at the end, it does an inverse Fourier transform.


Okay, so this is certainly certainly we are now in the top part right here, Fourier transform,


and at the end, inverse Fourier transform. And now these are in the middle is implemented a bit


weirdly, because of how the fast Fourier transform works, what you get, basically, you get an image


out of it, not get actually a 3d thing, but you get an image. And the important Fourier modes are


not like at the bottom or at the top, the important Fourier modes are actually in the corners right


here. So what you what you want to cut away is all of this, all of this middle part if you want to


throw away. So this is equivalent to throwing away these high frequency things right here. So that's


why this is implemented. So weirdly, you can see that here. First, we are going up to the modes in


each of the x, y and t direction. But then we're also going from here, we're going to the last modes


in this direction with all the others. This is corner, this is corner one, this is corner two,


this is corner three, and this is corner four, sorry, the bottom two right here is corner four.


It's a bit weird. And we don't have to actually do this with eight corners, which you might have


guessed, because why don't we do it with modes three, you see modes one and two, they always


appear negative and positive. And you would guess we'd need to do the same thing again, with negative


modes three, but we don't because this thing here is one sided, which because this is con con,


because this is a has a property of, of conjugacy. A lot of these entries of the Fourier transform


would actually be sort of symmetric. And the one sided only gives you one part of the symmetries


such that it doesn't waste memory. And it does so for the last dimension. So this dimension right


here doesn't have this corner property. It's a bit weird. And you need to know the exact


implementation of the Fourier transforms. But you know, that's what it is. So you can see that this


mole 3d here is a, it's compo mole 3d, it simply multiplies the input, which is the signal right


here, by these weights, the weights, as you can see, is simply a weight matrix, that is in channels,


out channels, modes, modes, modes, and two, two, because it's complex numbers. And you see in this


multiplication, that the this is a complex number multiplication. So the real parts,


and the real part is this, the imaginary part is this. And the operator is an Einstein operator,


I just thought this was funny. It says, bixies, yoxes, boxes, just as though I challenge everyone


to make Einstein, Einstein, some notation that spell cool words, bixies, yoxes, boxes.


But the the important part here is, so a is going to be the signal, which is going to be a batch


in channel and then x, y, t, b is going to be the weight that comes in the weight matrix,


which is in channel, out channels, x, y, t. And you can see pretty clearly in the Einstein notation


are also here, that the input channels are multiplied away. So these are summed over,


and what results is the output channels. So this is basically a matrix multiplication for each of


the samples in the batch. And for each location, x, y, z, it's a multiplication summing over the


input channels resulting in the output channels. This is pretty standard, pretty standard transform,


mapping vectors to vectors. It's complex, it's in Fourier space. But ultimately, it's just a


multiplication. So this is the code, they simply do four of these layers, going to Fourier space,


and then back again to Fourier space, and then back again, why do they do this? Because, as we


saw, they throw away these higher modes right here. And that also limits severely this applicability.


So if you only throw away the higher modes, if you just do everything in Fourier space, you


severely limit yourself. In fact, these Fourier methods, they are already not really good for


problems that have like non periodic boundary conditions. So the periodic boundary conditions


case is, as I understand, one of the easiest cases. And so the applicability would be limited.


And the authors hope that by sort of doing this in the real space all the time, and also having


these encoder and decoder networks, that they can retain sort of this information and be applicable


to more than just periodic boundary conditions. Yeah, exactly. And, and that's basically it.


I was ranting for so long. I think we are through to this paper. So maybe a quick summary, because


this was a bit bit of a rant, right? So you want to predict these types of things. These types of


things are well described by by their Fourier analysis. So transformations in the Fourier domain


actually make more sense, because the evolutions of these thing is more or less kind of these


global signals. It's not localized like natural images, like there's the cat and there's something


these these this pattern right here, it will repeat, you know, as you go into infinity,


these these sort of patterns will repeat and repeat. So the sort of global interactions between


these periodic signal is much more important. That's why it makes sense to go to Fourier space


to transform that in Fourier space, you can regularize by throwing away the higher modes,


and you get the additional benefit that you are discretization independent. So you learn the


function once and then you can input differently discretized signals as you choose and the function


stays the same because the Fourier transform, it will do as well as it can with the discretization


that you give it. Once you're in Fourier space, you simply have a multiplication. And it's actually


interesting, the filters here, the author shows some of the filters that are learned. So on top,


you see filters in a CNN. And on the bottom, you see these filters, these Fourier filters learn,


these are actually, as I understand it, these are transported back to the pixel space, so we can


understand them. So you can see that the global kinds of patterns that these Fourier operators are


sensitive to, compared to the CNN filters, which just have like localize a certain pattern. So this


is, this is quite interesting. So it makes sense to go into Fourier space, there are a number of


trade offs you have to do, you specifically you have memory requirements, and you can only predict


signals that are similar to what you've seen in the training data set. And you could only solve


things with periodic boundary conditions, but by means of architecture of these encoder and decoder


networks at the beginning, like the P and the Q, and the fact that you always carry through


and residual way, the pixel space signal makes it such that you might get around this, you might,


right? It's not it's not a proof, but there is a possibility that you might get around this in


total, this thing is way faster and more accurate than baselines, and has applicabilities and is


sponsored by the nice people at the military. Alright, so this was long, I realize, but I invite


you to check it out. The paper is technical, but well written. If you stick this kind of math part


Recurrent Neural Networks (RNN) - Deep Learning w/ Python, TensorFlow & Keras p.7

What is going on everybody and welcome to part 7 of the deep learning with


Python TensorFlow and Keras tutorial series. In this part what we're going to be


talking about is the recurrent neural network. So the idea of a recurrent neural


network is that the order of some data carries significance and importance. So


two areas where this tends to be the case is in like time series data where


data is organized temporally or chronologically. Otherwise you might have


this is important in natural language so the order of something in a sentence


carries a lot of importance as far as the meaning of that sentence. So for


example if you said you know some people made a neural network what happens when


you feed it through some sort of natural language processing algorithm let's say


that was just a deep neural network what usually tends to be the case is we're


going to tokenize data which just means let's say in this case we're going to


split it by word so each word is its own unique feature. But the problem with a


deep neural network is this sentence would have the same you know meaning as


a neural a neural network made some people and obviously these two sentences


have widely varying meanings and impacts and all that. So it's important the order


of things in these sentences. So that's the idea of a recurrent neural network.


So how do they actually work and I've got some beautiful beautiful drawings I


don't know who drew those but man they're beautiful. Anyways so the idea


here is that let's say these green boxes here these are recurrent cells so you


could have a basic recurrent cell very rarely do these get used usually it's


like an LSTM which is short for long short-term memory there's also the GRU


or gated recurrent unit but again I just pretty much see everybody use LSTM cells


in all honesty. So the way these work is you know you've got your sequential data


here and the sequential data is passed in let's say to one of these cells and


then the cell does two outputs to two locations it outputs one to the next


layer but two and the next layer could be the next you know it could be input


to the next recurrent type of layer or it could be the output layer or a dense


layer who knows but anyways each cell is going to output somewhere to another


layer but also down to the next node in the in that the specific recurrent layer


now where these output to it doesn't necessarily have to be to the next thing


down and then it also doesn't have to only go in one direction you can have


bi-directional recurrent layers for example and you can get really crazy


with the with the order of things and where data gets passed and stuff but in


general this is a super basic one so a another way to look at it is like this


is one cell and so so let's say you've got basically let's say this is like the


second one in the layer so let's say it's like we're looking at this cell


right here so this green box is this cell here okay so basically some some


data has come from the previous cell it's wrapped around here it's coming in


and the LSTM cell that we're looking at here it chooses okay what do we want to


forget from the previous node then there's input data coming in okay based


on this input data what do we want to add to this bundle of info that we've


got and then finally okay what based on this bundle of info what do we want to


output to both the next layer and the next node that we want to output to so


that's the task of the LSTM cell now obviously that's pretty complicated


especially when you know all these things like the whole point is to


eventually just pass along like a scalar value so how do we do that very


challenging stuff if you want to learn more about how LSTM cells work


specifically there is a great timeless write-up on the LSTM cell I linked to it


in the text-based version of the tutorial so definitely go check that out


if you want to otherwise let's get into writing our own basic recurrent neural


networks so what we're gonna do in this video is just do a really simple one


just so we can get the idea of like what really how easy a recurrent neural


network itself is but the really the hard part with recurrent neural networks


is getting your data set structured in such a way because the type of data that


you're going to be using typically doesn't have targets it has it's just


time series it's just informal kind of data and we don't really have you know


what are we actually trying to point this to so there's usually a huge amount


of pre-processing involved in when you work with recurrent neural networks so


what we're going to do here is just use a simple MNIST example and in the next


tutorial we'll do a realistic example with some time series data specifically


with cryptocurrency prices so yeah anyway I'm gonna minimize this and let's


begin with some imports so import TensorFlow as T jerk always does this to


me in part tensor flow is TF fix that then we're gonna go from tensor tensor


flow dot Keras dot models we're going to import the sequential type of model from


tensor flow dot Keras dot layers we're going to import the dense layer one


because the output layer itself needs to be a dense layer but also we are going


to end with another like a realistic you know a dense layer before the output


layer that's a pretty common operation to do then we're gonna have some dropout


just so nothing gets overly weighted and the LSTM cell I am going to comment here


if you are on the GPU version of tensor flow check out the cuDNN LSTM cell so


even if you are on the GPU version of tensor flow this LSTM cell will indeed


run on your GPU but the cuDNN LSTM cell is even more optimized and I it's like


five plus times faster even though both run on your GPU if you're on the CPU


version of tensor flow well then you know you're gonna this tutorial like


this model is going to take like five hours to train or something either way


okay so so those are the things that we need and now we need the data set so


we're just going to grab MNIST real quick so MNIST is equal to TF dot Keras dot


data sets dot MNIST and then we just need to unpack it so it's just these two


tuples and then it's X train Y train then it's X test Y test and that's MNIST


dot load underscore data so again my apologies for using MNIST but I think


it's better to do it make this one as basic as possible and then in the next


one we'll get in because honestly it would take us it's probably like the


next one's probably gonna take us two three maybe even four parts just to get


to okay now feed it through the neural network and that's the easy part really


feeding it through is not too hard so so for example let's do let's just do a


print X train dot shape and then we should be able to deduce this from the


above but X train and let's just get the first sample dot shape just so we get an


idea what we're dealing with here so so we can see we got 60,000 examples of 28


by 28 images and of course the zeroth element is just 28 by 28 so take a


minute pause and think about okay so this is our training data and it's


already in sequences for us right and what are the sequences well it's in


sequences like what is this 28 by 28 well it's 28 rows of 28 pixels per row so we


could in theory say each each row is a part of a sequence and if you were to


take the first row and then kind of scan downwards could you figure out what the


number is probably so the so the hope is that a recurrent neural network could


actually do this and maybe even more so we should see that it it could plausibly


do it better than a deep neural network but we'll see the more important thing


is just to show the recurrent net anyways but yeah so so that's the


sequence it's a sequence of rows of pixels that we're that we're going to be


feeding through this neural network so now we want to build the actual model


itself so we're gonna say model equals I'm gonna give us some space here model


equals and it's just gonna be that sequential type of model now we're gonna


do model dot add and we're going to add in an LSTM cell how many or at LSTM layer


right how many cells do we want to have we're just gonna use 128 for now we're


gonna specify the input shape and that is 28 by 28 we already kind of know that


but we can make this somewhat dynamic by doing X train dot shape and then one


colon so then we can pass an activation function and we'll use our good friend


rectified linear and again do check out cu cu DNN LSTM I'm just not going to do


that yet I'll probably change it before we run it because it is really really


fast but for now we'll just keep it whoops didn't mean to move that we'll


just keep it here okay so the next thing that we want to do is one more parameter


which is return underscore sequences and that's gonna be true so do we want this


layer to return sequences like it was input like sequences were input or do we


want this layer to return something flat right so if we're going to a dense layer


we wouldn't want to return sequences because the dense layer is not going to


understand what the heck's going on but if we're going to another recurrent


layer we definitely want to return those sequences this is just a long line


anyway okay so so that's one of our layers and then we're just gonna do


model that add and we'll throw in a dropout because that's what you do so


we'll do a 20% dropout there and then we'll do another layer so model dot add


we'll add an LSTM layer we'll do another 128 we don't need to specify input


shape in fact I don't know why we never had to and then we will pass an


activation function here activation equals rectified linear we'll do a


dropout I'm just gonna copy and paste copy paste and then let's do a model dot


add we'll add a dense layer we'll make it 32 nodes and then activation gosh can


you guys think of what activation oh yes that's right rectified linear then we'll


do a dropout and then we'll do we'll go to the final dense layer so I'm just


gonna copy this paste and I'm just gonna hard code in 10 that's just it's just


how many classes do you have so we're just gonna go with 10 the activation


shouldn't be rectified linear it should be softmax at this point and okay I


think we're done there with like at least with our you know model structure


now all we need to do is do the compile and then the fitment so to do the


compile we're gonna have an optimizer compile compile anyways tf dot keras dot


optimizers optimizers dot Adam and we'll just specify a learning rate of 1e


negative 3 and then we'll do a decay well we haven't really talked about


decay but we'll do a decay 1e negative 6 or something probably that's too small


maybe 1e negative 5 so a decay is just over time we'd probably like you want to


start with a larger learning rate to take those large steps that you want to


take and then but over time you'll kind of find this like local spot but


probably you're too if you took smaller steps you could do even better so if you


keep taking those big steps you're gonna keep bouncing around that little u-shape


but if you take a smaller step you could plausibly do better so anyways that's


what decays about it's just over time every batch it's gonna decay the


learning rate a little bit so it's gonna keep shrinking that learning rate and so


you'll take smaller steps anyways model dot compile and we'll say loss


equals sparse categorical cross entropy I just want to say they did add a


shorthand for mean squared error squared here use you know that's one way you can


measure loss and they said it was it's now you can use MSE I'd really like it


if they added shorthand for all of them someone tell me below if if they have


and I just don't know about it but you know I'd like to be able to say s c c


that'd be nice I hate typing that out okay so we got our loss the optimizer


equals opt really simple there and then the metrics that we're gonna track here


we'll just go with accuracy and then finally model out fit x train y train


epochs will do three epochs validation data so we can see it over time will be


x test y test okay so that should be good to go hopefully I can run this and


not crash the computer because I know sublime likes to really get me so we'll


go ahead and run that now I'll probably just go through maybe one epoch up let's


see invalid syntax at the dropout what did we and maybe not what line is that


14 do we not fully close this off we didn't try again


with fruit okay things are looking good so far all right and we are off we are


learning the accuracy does not look very good but you can see how slow this is


like this is extremely slow I would expect accuracy to be a little better


even by now I wonder why this is going so painfully slow I wonder if I've made


a mistake even so I don't even see loss going down and almost looks like loss is


going up 14 3 9 3 8 hmm


no first I saw this I was like oh that's our problem no we've got our final dense


layer activation is soft max what did we screw up we have to have screwed up


something it should be learning much more by now so this isn't it oh I don't


know that we've ever showed this but I brought it up at least what aren't we


doing here we are not normalizing this data so we could use the Keras


normalization or we can just say X train equals X train divided by 255 X test


equals X test divided by 255 I'm gonna save that the other thing I'm going to


do while I have you guys is show you cuDNN LSTM so let's go ahead and import


that as well I'm gonna change this to cuDNN LSTM in this one and then you you


can't the cuDNN LSTM uses a tanh activation function which is you can


google it if you want to see like the shape of it but you should be able to


with rectified linear you should still be able to get a good accuracy here but


the problem was def I almost kind of I almost kind of want to show you guys


because I know some people aren't gonna be able to run this if they're on CPU


and they just want to learn so I guess I'll just rerun it but we can even see


so like this did not train and hopefully that was the problem wouldn't be really


sad if that's not the problem but the re almost certainly the reason this didn't


train is because we didn't scale between zero and one so let's see the example


yeah I mean look how much faster it's learning now all we did was scale it


between zero and one that's how huge of an impact that makes so yeah it's super


important okay so I'm gonna go ahead and break well I guess I don't have to break


it but I'm gonna show you guys now kudian and LSTM because it's so fast


and it's paste paste and then I'm gonna delete this and delete this whoops we do


want to have that though so see we're already at like 41 40 I'm sure we're


gonna get into like the 90s or something but let me eta to three minutes so now


let's run this one hopefully I'm not gonna crash the GPU sublime is pretty


bad I probably should be running this in a console instead but whatever look at


this look how fast this is the best the best thing ever what a great little


addition I mean recurrent nets are so slow this kudian and LSTM cell is just


glorious so we're already at an accuracy of you know 96% and before epochs we're


taking I don't know we were pretty deep into that one and it was like two two


minutes and 30 seconds to go whereas these entire epochs are taking 13


seconds that's that's crazy okay so I think we're only gonna run for three


epochs so I guess I'll just let this one finish good done so validation so


accuracy here 97.73 validation actually did better 98.11 I don't know if I can


get back to the other layers or not I did want to show like on this so like on


something like this I think the reason why validation accuracy is higher than


accuracy is because this is an average for the entire epoch whereas this is


like at the end of the epoch we run it so and that's also why like this is at


the this is an average for the epoch so then after the next epoch we can see a


lot of times our average will jump quite a bit but even in this case it's still


lower than the validation accuracy which tells me we actually probably could have


continued to train because by the end of the epoch we were much much better but


we could also look at an intensivboard is loss still falling and all that so


anyways that is all for now in the next tutorial we will go over a far more


complex example of doing this on like a realistic time series data set because


that just takes so much more work but I didn't want to bog us down with the


pre-processing without this because actually you know feeding it through a


recurrent net itself as long as you have data that's already in sequential form


and already has a target it's actually pretty simple so anyways that's it for


now a quick shout out to their most recent channel members Taj Jagdish Dana


Larson and Juliano sorry if I mispronounced anybody's names here it's


pretty crazy how many like people's like different names that I've never heard


before and I have no idea how to pronounce so I'm really sorry but it's


just crazy like my audience just comes from from everywhere so it's really cool


to see and again thank you guys so much for your support it's really appreciated


so anyway that's all for now if you have questions comments concerns whatever


feel free to leave them below if I did screw something up in general there's a


red flag if validation accuracy is higher than accuracy but I do believe my


explanation before is why that's the case also I'm very confident that their


data is prepared correctly but anyways if that ever does happen to you that


should always be a massive red flag keep training and make sure that that is not


the case forever anyways that's it for now questions


comments concerns whatever feel free to leave them below if you've got any


requests for things that you'd like to see in this series let me know otherwise


[ML News] DeepMind AlphaCode | OpenAI math prover | Meta battles harmful content with AI

DeepMinds AlphaCode solves programming challenges, OpenAI language models solve math problems,


and Eleuther AI releases a 20 billion parameter language model open source. Welcome to ML News.


Before the rest of the video, this video is sponsored by Weights and Biases. Weights and


Biases builds developer tools for machine learning for researchers for practitioners


for juniors for seniors, whatever your favorite flavor of yogurt is, they don't care, they build


products for you except cherry who likes cherry. Today, I want to talk to you about a feature


called artifacts. So artifacts essentially are files in the cloud. But you're probably going


to use them mostly for two things, data and models. Both of these things are notoriously


tricky to work with data set is too large to check into get we need to keep it up to date,


we may have different versions of it and models even more, we want to save the outputs of our runs


into models that we can then use later, maybe introspect. And these things are also versioned,


and we want to depend on them. So when I did this, I had to save the model to some special folder.


And then I had to go grab it from that folder, put it on all the machines in a correct folder,


and then reference that folder from all my scripts that would then consume this model


with artifacts, this gets a lot easier. So we first uploaded the original data set to an artifact. Now


we're going to consume that artifact, split the data into train validation and test data,


and then emit those things as artifacts. So if there's a new version of the raw data available,


I can simply run the same script depending on the same thing. And it will create new versions


of the train validation and test data. You can make this arbitrarily complex, but I hope you can


see the point here. The same goes for models. If your run outputs and saves some kind of a model,


you can log that as an artifact. And from then on, you can consume that model in all subsequent runs.


Here's one of my models. It's a CNN, you can see it's already version 116 of that model. But you


can see all I have to do to use this model in any code in any script in the future, I simply call


the download method on the artifact and it will be available locally. And as I told you, you can do


this with any file. But since this is a model of a deep learning framework, weights and biases


understands it and gives me a neat viewer where I can actually introspect the model and look at the


shapes and even at the weights of my CNN. So I think this is incredibly powerful. These things


quickly get complicated with versions and scripts building upon other scripts. And the artifact


framework really helps you to make sense of all of it. There's even the possibility that the data


stays in specific private buckets with access controls. So not everyone in your team has access


to all of the data. Of course, artifacts are only one of the features of weights and biases.


If you're interested, please check them out. Free accounts are free, academic accounts are free


enterprise accounts cost a bit. And that's it for this week's sponsor spot. Thanks a lot to


weights and biases. Let's get into the video. Hello, and welcome to ML news. How's everyone


doing? We'll jump into our first story, which is that DeepMind has released alpha code, which is a


model that can take a programming challenge description. You might know these descriptions


if you've ever done competitive programming or had a programming exam or something like this.


So we have one right here given two strings s and t both consisting of lowercase English letters.


This is kind of overly formal about but it kind of details a procedure. So you can press the back


space button. And as you type the string s and then the character is deleted. And the question


is, can you get from the string s to the string t by pressing the backspace button at the appropriate


time. So for example, here is the input, you have four inputs, the first string is a b, a, b, a,


that's s and b, a is t. The question is, can you type s and you always have the choice of typing


the button of the letter or the backspace button, and you have to end up at t. So we'll try this.


For example, first type a backspace, right? Then there's nothing then we'll type b, a, and then


we'll type b, and then a backspace. And all of that should result in b, a. So we are going to have to


write a program that figures out if it is possible to get to t from s. And they have a bunch of these


example inputs right here, they have some notes. And as you can see, this is a text description,


this is the problem, you feed this to a neural network, the neural network will output a program,


a an actual piece of code, in this case, Python code that actually reads the input from the


provided file, not only these, by the way, so there's going to be other test cases, not just


the ones that they have, as an example, right here, implements the algorithm all by itself,


there's no human in the loop right here, and then prints out the correct answer, according to the


specification in the textual description of the problem. This is, let's say, quite challenging,


this is a hard problem, especially given the description is a natural language, and alpha code


solves this. So they have submitted alpha code to programming challenge competitions, and scored at


about a 50th percentile of humans. Now, that is not super duper good, as lots of these programming


challenge competitors are maybe students, people who get into programming who kind of want to


prepare for an interview, or hone their skills a little bit. So it is at an intermediate level


right now. But it is definitely more than I would have expected. So the way the model works is that


kind of like codecs, it is pre trained on GitHub problems, but then it is fine tuned to solve


exactly these code challenge data sets. So there exists data sets given problems in natural language


description and solutions. And the solutions are programs, obviously. So DeepMind takes their


pre trained model and then fine tunes it on these pairs of problem description and solution. Now,


when it comes to actually solving a problem at inference time, they take that problem description,


they feed it to the network, but they don't just output whatever the most likely output of the


model is, they actually sample a giant amount of possible samples, which means possible programs


that the model suggests. Now, a lot of them are going to be wrong. So what they do is they filter


those programs based on the small subset of provided solutions that you get in the problem


descriptions. In this case, here, they have four different example inputs for different example


outputs that will filter out in the paper, they say that will filter out over 99% of possible


solutions very often. Now filtering alone isn't enough, as that still leaves them with a large


number of potential solutions. And very often, these coding competitions, they're limited to a


very small number of submissions. In this case, I believe it was 10 submissions. So in order to


achieve that they have a step on top of that where they cluster solutions. So they try to cluster


together programs that are textually different, but essentially don't do a different thing, like


maybe the variable names are different, maybe the same algorithm is implemented in a slightly


different way. So they have a clustering algorithm that lumps those together. And that brings them


down to the 10 submissions that they're going to make. These are not the only parts of the system


by any means, there is a large number of components to the system that really brings up the system to


the level of the average human where it currently stands. Now there's a website where you can


explore the solutions given by the model. And you can look at sort of the attention heads of


different models, like what they pay attention to along the different types and things they do. So


on the left here, you see the description of the exact problem we saw before, this is pure text


with natural language. And on the right, you see the solution. So as you hover over this right here,


it shows you token probabilities, and it shows you according to what this token is decided upon. So


for example, when I say when I hover over the line s is the input right here, you can see that on the


left, it focuses on this text right here on the first line of each test contains the string s


when I focus on T, it focuses mostly on the line below where it describes whatever T is, the


attention is not only to the problem description, but also within the program that was already


generated. And it's generally pretty cool to explore, I recommend you give it a try. As I said,


there is a detailed paper with this where they describe exactly what the components of the system


are and so on. Give it a read. It is quite a lengthy paper, I believe it has its own table


of contents. Yes, it does about 30 pages, so not too long. So my question is a little bit when I


think back at like AlphaGo, AlphaZero, and so on. Those models also didn't start out world class,


but they were able to quickly get there and beyond simply by doing more self play. In this case,


it seems the data set is a limiting factor. So there's only a finite amount of these human


generated programming competition data points. The question would be, is there a way that we could


come up with synthetic data like synthetically produced code samples? And is there a way that


we could make them progressively harder and harder and harder in a self play kind of style? Because


if that's the case, and if we really get this data generation part right, it could also be that the


coding AI here will become you know, like, good beyond limits. But I am kind of skeptical about


that. We also have some different voices giving their opinions on this. One of these, for example,


is Jimitri Badanao, who is a competitive programmer has done this for a while apparently,


and puts it a little bit into perspective saying it is impressive. Yes, but he says human level


is still light years away mentions again that 50th percentile in these competitions doesn't


necessarily mean that it's particularly good that a human challenge is often not only the


difficulties of the problems, but also the limited time you have available for them and the disparity


between humans and the machine of the approach, namely that 99% of all programs that alpha code


outputs are wrong, whereas a human will maybe make a mistake in the first try of the implementation,


but doesn't need to generate 1000s and 1000s of hypotheses until they get a correct one.


And certainly, they don't evaluate all of these hypotheses by filtering them using the tiny,


tiny amount of examples they have. So humans and machines, they seem to have a sort of fundamentally


different approach for now to solving these problems. Yet, I can definitely see a version


of alpha code that more iteratively takes into account sort of partial programs, and more does


a more guided search for the rest. And ultimately, yeah, humans also they run their program on the


small test examples. And if that doesn't work out, they're like, wait, something's wrong. So


this is an exciting field. I'm very curious where it goes.


Next news, open air releases a blog post called solving some formal math Olympiad problems. They


detail how a language model that was fine tuned is able to solve formal mathematics problems. This is


very, very cool. So other than in alpha code, these problems actually come with a formal


description, they are defined in a formal language that is amenable to be proven yet still to apply


language modeling to this problem, and then do some post processing, obviously, is quite a hard


task. So the reason they use language modeling right here is that other than in chess or anything


like this, the action space is huge, it's actually infinite in proving formal mathematics, because


you can just invent new things by yourself. They do have a set of tactics that the models kind of


allowed to apply, but still the action space is infinite. And the language modeling is very, very


important. So the action space is infinite. And the language model helps them to determine what


are the most likely next steps that they want to do if they want to solve this proof. The other thing


that differentiates them from games is what they call the lack of self play opportunity. There's


no reward to people playing against each other or anything like this, which usually serves as sort


of a curriculum method. As the agents play against each other, they sort of level each other up in


a way that they can't win. And so they have quite a smart data generation and sampling process,


where they start off with some hand provided samples of various difficulties of where they


want to go. And then they start with the lowest ones that they might be able to prove with the


current technique of language model plus proof search note that it is not only a language model


is combined actually with a proof searcher that is guided by language model. And as they prove more


they add those to the data set, which they then reuse to train the language model. So in this


case, the model automates its own curriculum by proving more and more statements. This isn't


obviously without challenge because math is full of trivial and nonsensical statements that you


can prove to be true. So choosing even what to prove becomes a hard task. But nevertheless,


using this approach, they're able to generate quite good proofs. In fact, they're able to


outperform pure proof search by quite a bit. They're also able to solve problems of the


International Math Olympiad, which is usually quite a hard problem. There is a paper to go


along with this, give it a read if you're interested. Luther AI announces GPT Neo x 20b


that is a 20 billion parameter model. And by the time you're watching this, the model is going to


be available for free, it's going to be kind of a pain to run it because it's so big, but you can


just download it. I've made an entire video interviewing Connor Leahy, who is one of the


co founders of Luther AI and has worked on this project about how this came to be about how they,


you know, got their hands on the hardware necessary and so on. So if you're interested,


check that out. Another new paper about style again, x l, the papers called scaling style


again to large diverse data sets. That is a hard thing to say scaling style again,


like try saying that over and over again, scaling style again. So the TLDR here is with the right


training strategy stylegan achieves state of the art on ImageNet. So if you remember stylegan


always used to be trained on very specific data sets. Stylegan is the thing that powers this


person does not exist.com this shoe does not exist.com this sneaker does not exist.com and so


on. But these are all very limited data sets often of the same thing. And approaches like biggan have


traditionally been better at modeling diverse data sets such as ImageNet, which has many different


things. The authors here show that with the right training protocol, namely projected GANs,


up sampling and so on progressive training, you can get these GANs to the level of ImageNet. This


is also built on stylegan v3, which means that it kind of retains it has these translation


invariance properties. I have reported on this on ML news previously. So go check that out if you


are interested. So they're able to generate images up until 1024 to 1024 resolution, which is quite


impressive. They can also invert images on the left, you actually see a real image. And on the


right is an inverted image where they have fed this into the GAN and then figured out the latent


codes and then they're able to edit the image on the right as they see fit. And as I said, it


retains the translation equivariance from stylegan three. If you're interested, check out their


website and check out their paper. Our five by its AR five IV that is a website it's ar5iv.org.


What it allows you to do it allows you to view archive articles as HTML five web pages. I'm not


exactly sure how it's pronounced. I was told it's pronounced r five. But then again, it should


probably be our five I've like the way it's written. I don't know. Also, the browser showed me


a warning when I went on this website asking me whether or not I have maybe confused it with


archive. So yeah, this might be just a giant phishing attack. But it is pretty cool. Here is


an example that they give now my browser is dark mode. So I don't know if that's available in light


mode. But you can see that the references are real true links that you can open as a pop up,


there are still some kind of artifacts right here. As you can see, equations are rendered nicely. And


also the side note, the footnotes here are rendered right beside the text. I don't know


what happens if I zoom in. Okay, they just are pop over also allows you to jump to equations and then


using the back button, jump back to where you were. This is like this is the greatest thing ever.


The amount of times I had not clicked on like an internal reference on a PDF, just because I was


like, No, I'm not going to scroll back to where I was. So thank you check out our five.


Okay, we have some helpful things this week. The first helpful thing is Yi Tai saying they've


released over 170 pre trained transformer checkpoints of many different shapes and sizes


as part of their paper. This is by Google research. Check out the scaling transformers paper, the


scaling transformers repo, and the models released. Thank you. FFCV is a library by the lab of


Alexander Madri that makes training machine learning models fast. If there's ever like a


buzzwordy title that says nothing, it's trained machine learning models fast. So they provide a


set of sort of throw in replacements, for example, for data loaders that will just kind of speed up


common use cases of training neural networks. They claim their code is hyper optimized removes


bottlenecks, it's super duper pipeline and parallel and all of that. So if speed is an issue for you,


maybe give this a try. OTT or optimal transport tools is a toolbox for all things Wasserstein,


as they call it, it is an optimal transport library for jacks. Sumit Chintala advertises


diet GPU, which is a lossless compression algorithm for Nvidia GPUs. The code of this


is available. It's authored by Geoff Johnson. And what it does is it can compress stuff and


uncompressed stuff on GPUs. So if you have a slow network, and you have a distributed training,


and you really care about making this fast and efficient, what you can do is you can compress


stuff that you need to send over the network on the GPUs, send it over then uncompress it,


this library will make the compression and uncompression part really fast and really


efficient. Alright, that was it for helpful things. I hope you got help.


User Bremen 79 on Reddit says the ICML 2022 conference is changing their review process


slightly. So now there are two phases in phase one, the reviewers just give a recommendation.


If there are two recommendations that are negative for a paper in phase one, it is already rejected,


I guess this is a goal to call down on the amount of papers that have to be seriously reviewed. It's


all the more important now that your paper makes a good first impression. So they say the meta


reviewer can reverse this outcome. Okay. And other changes that reviewers do not make accept or


reject recommendations in phase two, the meta reviewers will decide based on the reviews. So


I just write my review, and then the meta reviewer reads it and integrates it all instead of me


saying, well, this is a seven or this is a four, this is a strong accept or a weak accept. Now,


technically, it shouldn't make a difference, right? Because me voice, like my score that I would


usually put is just kind of a conglomeration of what I said before, but you know, tiny changes


like this, you know, because we're humans, and we're not consistent, and we're not, you know,


we're not attentive enough, tiny changes might like this might actually make a difference,


I'd be interested to see some statistical analysis after the fact of what this did.


If you're interested, the entire process is detailed in the ICML 2022 review form. Now,


it just occurred to me that the submission deadline was actually last week, which I should


know. So if your paper is not pretty, and doesn't make a good first impression, then you just you


just gotta gotta hope for that really good meta reviewer that recognizes its inner beauty.


This is a little bit older, but I hadn't seen it at the time, there is a blog post on meta


AI's research blog saying harmful content can evolve quickly, our new AI system adapts to


tackle it. So they describe a system that they call few shot learner, which essentially means


that it's a system that can monitor harmful content and adapt quickly to new harmful content,


because it's ever evolving, I find a few things interesting right here. First on a sort of a


scientific level, what is pretty interesting is that the model doesn't only consider training


data. So data that has been labeled as harmful or not harmful or borderline, or anything like this,


it does do that. But it also takes a description of the policy like a textual description of the


current policy. And by doing that, it's able to adapt to policies over time, having seen, you


know, with this policy, this stuff is okay. And then with this new policy, this other stuff is


okay. So the fine tuning process can potentially happen with less data, I found this pretty,


pretty interesting to actually provide the policy itself to the model. The other interesting thing


is just this video right here. So as you can see, the people here, they're interacting with with the


internet and they see harmful content. And they're like, like, they're like, Oh, no, I'm gonna log


all Oh, no, all this harmful content. And then, you know, there's the system they describe their


system. Yeah, whoa, okay. So now they you know, they filter all of this, this new harmful content.


And then at the end, look what happens. Oh, everyone's smile, like, look, they're smiling.


It's like, Oh, the artists just it is so awesome. Thank you. Thank you, meta. Thank you. The few


shot learner. Thank God all the harmful content was prevented from destroying smiles. Now, okay,


on a more serious note, it is a hard problem, right? There's no way you can monitor all the


content all the time. There's no way you can train a static system because sort of the meta of bad


content of bad language of people bullying each other and so on is always evolving. So props to


you know, meta for actually trying to tackle this problem. Because what I mean, what's the alternative


here? Shut down all communication? That's not going to happen. Tell people to be nice, like,


well, try but I see a bit too much complaining about this. And yeah, I do. I do like that they're


actually tackling this problem. And I find the approach to be cool. It's just the marketing


that's a bit cringey. But what am I saying? I'm wearing sunglasses indoors. Okay, last news for


today. MIT Technology Review says this company says it's developing a system that can recognize


your face from just your DNA. Now people have been extremely skeptical of statements like these. This


is a company that deals in broad language with law enforcement, searching people, security,


surveillance, and so on. And you know, you might debate the merits or unmerits of that in a separate


topic. But the particular question of can we actually get someone's facial features from their


DNA is highly debated. Just to be set, the company isn't only focused on that it's called core site,


and they have different plans. These are not systems that run right now. These are sort of


future plans to do things. One of them is this DNA to face thing. Now I do feel the criticisms


of this are often maybe overly skeptical, let's say now again, I don't mind the skepticism about


the applications of this, but the possibility that there's a reason that children often look like


their parents, your facial structure is in large part determined by your genetic material. Now,


the article points out that obviously, age and environmental influences also have big impacts


on that. So no doubt about that. And they make a good point in that they say the technology will


probably not be able to tell you the exact number of millimeters between the eyes or the ratios


between the eyes, nose and mouth. And those are some of the features that the current facial


recognition technologies rely upon. So since we can't get those features accurately from genetic


data, because there may be more environmentally determined, the current facial recognition


algorithms wouldn't work. However, I don't see the extrapolation discussed right here in that


I would think it might be absolutely possible to train facial recognition algorithms that only use


the features that we can read from the DNA, like the argument that the face reconstructions that


the DNA data gives us doesn't work with current facial recognition software is almost a moot


point by then question is obviously how accurate it's going to be. And again, whether or not you


even want to do this in the first place. But let me know what you think. Should this be done? Can


this be done? And would you want to do it? Let me know in the comments. I this was ML news. Thank


Flask Tutorial Web Development with Python 14 - Connecting to database

Hello everybody and welcome to part 14 of our practical flask tutorial series.


In the last video we created our users database and now we're actually ready to start at least


interacting sort of with that database.


So first what we're going to do is we're going to need to disconnect here.


I think we can just do quit.


Yeah.


So just type quit and it'll say bye if you're still connected.


So the next thing that we want to do is to interact with MySQL with Python.


We need MySQL DB.


I don't recall setting that up initially so what we're going to do is we're just going


to say pip install and I forget how you type this exactly but let's try pip install MySQL


DB see if that works.


Looks like it's not going to work.


It's probably going to tell us there's no package.


It's taking a long time to scour for packages.


This one's always hard.


There's a few that are kind of difficult to find.


Let's just do sudo apt-get install python-mysqldb like this and if that doesn't work we'll do


all lowercase but looks like it's working.


Fancy fancy.


Alrighty.


So it looks like we got it.


Let's just type python really quick.


We are using python 2.7 here.


MySQL DB.


So it's capital M, Y, capital S, capital Q, capital L, DB.


Make sure you can do that if you can.


Great.


Control D to exit there.


Only hit that once though otherwise you will be logged out of SSH.


Okay so we have MySQL DB now.


Now we want to create this dbconnect.py file that we can use for connecting to the database.


So let's go file, new, we'll do file here and we're going to call this literally dbconnect.py.


Here it is for us.


Let me make the font a little bigger so it's easy to see for everyone involved.


Okay.


So here's our dbconnect.py file.


We're ready to populate this.


First of all in order to connect to the database we need to import MySQL DB like that and now


we're going to define connection.


So this will be our connection function and if you follow along in any of my MySQL tutorials


this should all be very, very familiar.


If you're interested in learning more about MySQL with Python there's actually a lot of


things you can do.


Here all we're actually going to be doing is selecting and inserting data.


Eventually I may allow users to change passwords.


I have not even coded that myself.


It needs to be there.


And whenever I do that maybe I'll do an updated tutorial on how to do that since that would


be actually updating rows in a database as well.


Anyway define connection.


We're going to say CON for connection equals capital MySQL DB.connect and then we insert


our parameters.


So the first parameter is going to be host.


So this is who are we connecting to.


For now it's local host which is 127.0.0.1.


You can connect remotely to and from databases.


You can set that up.


You can have a database allow remote connections and you can actually allow remote, when you


allow remote connections you allow it from a specific IP for a specific user from that


IP.


So keep that in mind.


That is possible and you can do a lot of really cool stuff allowing remote connections.


Some people think that that's a security risk.


Honestly it's not.


It's, you know, it's actually one more password and username and password is required anyway.


But you will be transmitting the data in that password remotely.


So for say usernames and passwords that's probably a bad idea.


But for, I don't know, data for displaying a graph or something that is not sensitive


or security sensitive data it's fine.


Anyway host, local host.


That's who we're going to connect to.


The user is equal to root because we're using a root user.


The passwd for password equals, and this is the password for your root user on your MySQL


database.


My password was cookies.


And then, no that's not the password to the actual database just so, and then the db,


so the database that we want to actually connect to is, whoops, is python programming.


Okay, that's it.


That's our connection parameter.


Now we're going to specify c for cursor and that's going to be just con.cursor for short.


And then after all of that we just return c and con.


That's it.


Now we save dbconnect.py and close out of that.


Let's open up our init.py now and we'll come to the very, very, very, very top here and


let's do from dbconnect and for connection, connection, like that.


Then just to check our connection let's go down to, we don't have a register page.


I guess, let's just start building the register page real quick and then we'll worry about


connection.


So let's just take this, copy that, paste, come on down to it, and we're going to call


this register and then instead of login page it's the register page.


And then let's just return just a simple string for now, okay, and let's do ccon equals connection.


And that should be it.


ccon equals connection.


Cool.


So now let's go ahead and try that but if we fail for whatever reason, except exception


as the return string e, we'll save that.


Now if something goes wrong in the importing of dbconnect we are going to still have a


500 error so hopefully we'll be okay.


Where is, there it is, a restart.


Come over here, let's refresh.


No problem there.


Let's go ahead and sign up, register, okay.


So all is well.


So we come back over here so now we've got Python actually successfully connecting to


our user, well actually sorry, just connecting to the database.


It's actually not connecting to the user table just yet but that's what we're going to be


working on in the next video.


So anyways I think we'll cut it off here and in the next video we'll actually build this


register page and actually connect and all that and actually have someone register either


in the next one or the one after that.


So if you have any questions or comments please feel free to leave them below otherwise as


always thanks for watching, thanks for all the support and the subscriptions and until


Neural Networks from Scratch - P.7 Calculating Loss with Categorical Cross-Entropy

What is going on everybody and welcome to part 7 of the Neural Networks From Scratch


tutorial series.


In the previous video what we did was the Softmax activation function which is what


we apply to the output layer of a classifier neural network and of course the next step


that everybody wants to do is backpropagation and optimization but before we do backpropagation


and optimization what we need to have is some sort of metric for error.


How wrong is the model that we're trying to train?


The first kind of question or insight that you might have is well let's use accuracy


that's what we care about why don't we optimize for accuracy.


So if we only cared about the accuracy of a model we really would just feed in the binary


true or false to an optimizer and make the attempt to optimize the neural network that


way.


Now while that's I suppose in theory possible the problem is we're throwing away a lot of


actually useful information because our neural network doesn't actually output an actual


classification instead what it outputs is a probability distribution.


So when we pass data into our classifier neural network the output is a distribution a confidence


score for each of the classes.


Now while we do do a argmax to figure out what is the predicted class it should be fairly


obvious that imagine you've got the it predicts the correct class but one in one instance


it predicts with a 87 percent confidence and in another instance maybe it predicts with


a 50.2 percent confidence right.


Let's say there's a difference there and it would be very helpful if you were the optimizer


it would be very helpful to you that's informative information if you have to go in there and


make tweaks to weights and biases.


One simple example of a loss function is mean absolute error which is used with regression


so regression is just where we're trying to output a specific value rather than the probability


distribution that we're trying to do in our case but this should give you an idea of why


a loss function is a little more useful than simply doing accuracy.


So with mean absolute error all you're doing is just taking the difference of you're taking


the average of the distances of various points of prediction versus the target values and


as you can see we're just kind of showing some examples here but as you're more as you


get closer to the correct value the mean absolute error gets lower and lower and lower as you


get further away it grows and this is just a lot more informative than being right or


wrong.


So again rather than doing regression we're doing classification with a softmax activation


function on the output layer so our model is actually outputting a probability distribution


and then from this distribution we also know from the training data what was the intended


target value and then from here we want to come up with some sort of method for determining


how wrong is the model.


Now there are many ways that you could theoretically go about doing this but in general the loss


function of choice for classification at least classification is using softmax as the activation


function on the output layer is categorical cross entropy the formula for which is to


take the negative sum and basically what we're passing through to do categorical cross entropy


is two probability distributions.


So the formula for this is to take the negative sum of the target value multiplied by the


log of the predicted value for each of the values in the distribution.


So this formula winds up simplifying quite a bit to just being the negative log of the


target classes predicted value and the reason for that is because of one-hot encoding.


So a couple of things first off we will explain one-hot coding in just a moment and then also


we'll explain logarithm if you don't know what a log or logarithm is don't worry about


it and then also I'm sure some people are wondering well why the heck are we using that


confusing formula when there are lots of other you know maybe ideas floating around in your


head as far as simpler ways to calculate loss.


There are many ways that you can calculate loss and in fact many more challenging problems


are going to be likely solved by some sort of custom loss function entirely.


It just so happens that in classification this is one of the more popular ones one because


it's successful it works and then two it actually becomes fairly convenient if you if you can


believe it it becomes very convenient in the back propagation and optimization steps so


more on that when we get there but that is why.


So first off the first thing I want to explain is what is one-hot encoding.


So the idea of one-hot encoding is that you'll have a vector that is n classes long so however


many classes you have that's how long that vector is and that vector is filled with zeros


except at the index of the target class where you'll have a one.


So imagine a scenario you've got three classes that means you're going to have a vector that


is three values long if your label is zero at the zero-th index of your one-hot vector


you would have a one and then here are some examples of various numbers of classes the


intended target label and then the one-hot vector that would represent that.


Okay so what's all this about logs?


Generally speaking when someone mentions log without specifying some specific base number


they are referring to the natural log which also can be referred to as ln but you'll often


just see log and in programming just log and then you know some sort of parameter being


passed.


When this is the case this is the natural log and the natural log is log base e or Euler's


number which you've already seen and is somewhat of a hint of why this is a convenient thing


for us to use before we do backpropagation.


So I've heard that some people in certain studies consider the natural log to be base


10 but those people are wrong.


So for the purpose of this series and programming in general just assume that the log of something


means natural log and that natural log is base e.


Okay let's come down from the clouds for a brief brief moment and discuss what a log


actually is what does it do what's it calculating.


So a logarithm in general now obviously you can have different base values but a logarithm


in general is solving for x in the equation of e raised to the x equals b.


So the input to your logarithm is actually going to be this b value.


So to see this in action let's I'm going to follow the exact same example from the book


you can code along if you'd like you don't have to you can just watch and then also you


can use your own values and just kind of tinker with it just to get an idea for actually what


logarithm is doing.


So first of all we're going to import numpy as np that's what we're going to use for the


actual log part and what we're going to say is we're going to say b equals 5.2 again just


using the same value from the book and we're going to print np.log b.


So again what's what is this asking it's asking Euler's number raised to what equals 5.2 and


again Euler's number is something we've covered already so we'll go ahead and print that and


what we get is this 1.6486 and so on number.


Now we can check this how can we check this well we can quite literally just do Euler's


number times this and just make sure it's either 5.2 or very very close.


So for Euler's number we're going to import math and then we'll just come down here and


let's just print math.e raised to the power of that this output.


So we can go ahead and run that and sure enough what we get is very close to 5.2 the issue


is just a floating point precision issue.


So yeah that's that's what a log is and how it works.


So now that we have an understanding of the core components of the categorical cross entropy


function let's revisit the formula and how it would be applied to a single sample with


three classes where the target label is a zero which would make the one hot vector one


zero zero and this vector is the distribution that we'd compare to the prediction distribution


from the neural network.


So again to calculate this categorical cross entropy we start by taking the negative sum


of the target value times the log of the predicted value for each of the values in the distribution.


Now one thing that should be becoming more clear is that we have a few of these being


zero times something plus zero times something else which winds up essentially doing nothing


and this is why because of the one hot vectors when we calculate categorical cross entropy


with one hot vectors and neural networks it allows us to actually simplify things quite


a bit to just being the negative log of the predicted target class because even in that


case the value for the target distribution was a one so we don't even have to consider


it it just becomes negative log.


So you might wonder why don't we just call this negative log?


Well first of all categorical cross entropy just sounds cooler but also we are actually


doing categorical cross entropy it just so happens to be the case that it simplifies


down to being negative log of the predicted target class value.


So now that we know that let's go ahead and code the actual categorical cross entropy


solution on an example.


So I will just clear this out and we are going to be using the exact example from the book


which is on page 114 and to start we are going to import math we are going to be using math


for math.log in the previous example I used numpy.log but we are going to use math.log


because this is neural networks from scratch in python and numpy is kind of cheating so


this is how we get around that.


So we will use math in this example.


So first we will start with a softmax output and for this we are going to use and these


are just made up values we are going to say 0.7, 0.1, and 0.2 so what is this?


So again these are just example values that we are making up for the time being and these


are meant to be an example of an output that you might get from the softmax activation


function from your output layer of your neural network and what we want to do is calculate


the loss on this.


So imagine you have a scenario where the target class was 0.


If the target class is 0 what would that make the one hot vector?


Well that would mean at index 0 it is hot otherwise it is of n class length which is


3 so that would mean our target output distribution would be 1, 0, 0.


So I am going to get rid of this and now what we want to do is actually calculate the loss.


So we are going to say loss is equal to a negative and then we are going to sum together


the math.log of the predicted so this is going to be softmax output 0 and we are doing it


for each of the indices so it is the math.log of softmax output 0 times the target output


0 and then what we are going to do is we are going to add each of these for each index


so paste and then we will edit this in 1, 2 and then this would be index 1 and 2.


So let's remove that plus and now we can go ahead and print out the loss.


So in this case that is the loss that we got and if you recall from the animation one thing


to note is in the animation I think we actually flipped these two things around it was actually


target output times math.log but it doesn't actually matter pemdaz everybody.


So anyway another thing you might remember is the zeros so because we are essentially


multiplying these we got math.log and then what is target output at index 1?


Well that is a 0.


What about this one?


Well that is also a 0.


So this value zeros out.


This value also zeros out.


What happens here?


We do the math.log softmax output at 0 which is 0.7 and then what happens here?


Well it turns out this is 1 so it is actually just 1 times this so really this entire formula


becomes the same as doing loss equals negative math.log of the softmax output target class


being 0.


So let me actually let's just print loss and we should see that we do indeed get the same


exact number.


So that is how that very complex looking formula becomes as simple as doing this.


Now just for example let us I think what I will do is let's just take this and we will


print that and then rather than passing this let's just type 0.7 so sure enough we get


the exact same value and then let's say because this is the target class let's say instead


the prediction was 0.5.


So what you can see is and in fact just to make it abundantly clear to everybody let's


do this.


Now you can see that so in this case where the confidence was higher in the correct class


loss is lower and then where the confidence gets lower in the correct class the loss this


measurement of error gets higher.


So that is our loss function of choice here that's how it works that's how it simplifies.


In the next video we are going to be applying this to the actual you know the code that


we built up so far but make sure you have a good grasp on this because as we do that


we are going to be converting it to a class we are going to be essentially converting


because it becomes this you know negative log of the output at the index of the target


class we are going to be abstracting that away because we also want to be doing that


in batches so as we add the logic to that if you don't have a firm grasp of what is


intended to happen it can be very confusing when we go to write that code so make sure


you have a good idea of what's going on if you have any questions comments concerns whatever


on any of that feel free to leave them below we also have a discord server it's discord.gg


slash syntax you can come in there hang out ask questions all that good stuff.


Otherwise that is it I don't know if I mentioned it but the neural network from scratch book


is available it's on neural network or nnfs.io you can get the physical books hardcover softcover


Axial Attention & MetNet: A Neural Weather Model for Precipitation Forecasting

Hi there! So what you're looking at here is a weather forecast model. Specifically the very top row is a new weather forecast model called NetNet by Google Research.


So the goal of weather prediction is pretty simple. You want to know what the weather is going to be in the future. Specifically here you want to know precipitation rates.


And so this is a new work that uses neural network instead of physical models in order to predict precipitation.


So in the middle here you see this is the ground truth of what really happened at that particular time. You see precipitation rates in red here moving across the country.


Now the bottom there is a physical model. And as far as I understand it, physical models have been used so far to make weather predictions.


Which basically means that you simulate these rain clouds and the movement of them across the country.


And you do a physical simulation like a particle simulation type of thing and then that allows you to predict and then you run that maybe multiple times and you get an idea of the kind of distribution you're going to get.


Now what NetNet does is it simply uses a neural network to predict the outcome directly. So there is no physical simulation involved.


There is just a neural network that takes as input what's the situation now and maybe over a stretch of time.


And then you ask it please make a prediction in eight hours or something. And then the NetNet will make that prediction and it will just output it like snap.


No physical simulation needed. And you also see here that NetNet outputs things in kind of a cloud way, in a probabilistic way.


In one forward pass you don't need to run it multiple times. But we'll get to that.


On the bottom here you see the measurement. So the axis is F1. F1 is kind of the overlap of how well you're able to predict the precipitation.


And you see here the NetNet is above the HRR baseline for most of this time. Up to 480 minutes into the future, which is eight hours I believe.


All right. So the paper is the following. It's called NetNet, a neural weather model for precipitation forecasting.


And I'm not going to read all the names here. The main corresponding authors are Kasper K. Sonderby and Nelkalk Brenner.


And it's a team of Google research. So specifically they use the input of these two things here.


So one is this GOES-16, which is what you see here on the left. And the precipitation rates are here depicted on the right.


So you want to take these things as input into your model. Now how do you do that?


Of course we're going to build a neural network. And this is the architecture they come up with.


So on the bottom here, they feed in the data. And they feed in the data in 15 minute interval from 90 minutes into the past.


So you have to imagine it like this. So there's a timeline. I'm going to use a little bit of a finer thing.


So there is a timeline and you, let's say, are here. This is now. And then here in the future, this is maybe one hour into the future.


This is your target. This is you are here and you're looking out. You would like to know what's the precipitation going to be in one hour from now.


What MetNet does is it takes an input. And specifically it takes the last 90 minutes before now as an input.


And it samples it in frequencies of 15 minute intervals. So each one of these is going to be 15 minutes.


And each 15 minutes you get like a snapshot of this entire of the input region.


Now the input region, if I can jump back here to the website for a second, they show it what the input region is.


The input region, if you want to predict in the middle of this small square, the input region is actually the entire 1024 square kilometers around it.


So it's very big input, though the actual region you consider is the inside 64 square kilometers.


But you take in information from the big region. And the main point of the paper, I believe, is how to do that.


All right. So each 15 minutes you take in a snapshot. And these are these snapshots here on the bottom.


So these are, and you have to imagine in here, every 15 minutes there's a stack of these inputs.


So what are these inputs? These inputs are some kind of features that you have.


So there is the target time, which in this case would be this one hour here.


There is the month, day and hour, which is important for weather prediction.


Time of year, time of day and so on. Longitude, latitude is probably pretty important. Elevation map is probably pretty important.


So these you can see, these are all maps. Now sometimes, and this is how you encode things in these.


Since it's a neural network, you know, all of these things must be of the same dimensions here.


So if you have 256 dimensions here and probably 256 dimensions here, then all of these things must be of the same dimension.


And if you want to give a feature such as the target time, which in this case, let's say it's one hour.


So you just put here one hour. What's one hour? Let's say 60 minutes. So you just put the number 60 here.


60, 60, 60, 60, 60, 256 times and 256 times 250, 65, sorry, 265 times, no 56. I'm confusing with German.


So this is how you encode features. It's pretty primitive, but it turns out it works the best if you do it this way.


All right. So you have these planes and some, as I said, are just features such as the target time, month, day and hour and so on.


Elevation, I guess, is a map, is like an elevation map of the region you consider.


And this corresponds now to this 64 kilometers times 64 kilometers here.


And that's exactly what these center crops are here. So this center crop thing, that now, this thing here, this plane, sorry, is these 64 by 64 region.


That's this plane here. And also the, that's the precipitation and the GOES, that's this thing here.


Now, we also have these down sampled things, which these are the 1024 kilometers.


So this here and this here, these are the 1024 square kilometer patches, but they are down sampled.


So everything is down sampled, I guess, to 256 by 256 pixels.


So you don't really take into account every nuance of that very big, of that very big input, but you do down sample it.


So you kind of get the big picture of the outer frame and in the inner frame you take it in a much higher resolution in order to get the details.


All right. So you stack all of this up into a big tensor and then you feed it into here, into a spatial down sampler, which I guess, no, I have read is a, just a convolutional neural network.


Right. So this is your typical image processing pipeline. So you do this for each of these stacks. Right.


And then what you get out of it is a lower size representation right here.


So you get these representation and then you let a temporal encoder run over it.


What does a temporal encoder do? This in particular is a convolutional LSTM.


And if you already know what an LSTM is, a convolutional LSTM is nothing more than an LSTM that has as intermediate layers, convolutional layers.


So it's pretty suited to do, for example, videos or any sort of image processing that goes over time like this one.


So the temporal encoder simply starts out here with an initial state.


My pens are screwing me today. So it started out here with an initial state and then it simply inputs each of these representations, takes them one by one, runs across time.


Right. And each time producing a new intermediate representation of the, of the input until it finally reaches this here, final representation.


So this thing here is a single final representation of all of this input, right, of this entire, of this entire time span of all of these stacks here.


So you compress this into a single input with first a convolutional network to down sample each time point individually and then with a recurrent neural network, an LSTM, to integrate the information over time.


You end up with the single piece here. And then what you do, so you still, here you still retain kind of an image sort of thing.


So this, this, this representation here, you can see it in the background. Maybe I get a thumb a scribbles here.


This here is still sort of an image tensor, though I guess it's a hidden representation, so you couldn't really look at it.


But it is, it still has dimensions of images. So this here is still, I think, the same or corresponding to these dimensions here.


So this has still has some spatial information where this might be north, south here in this axis might be east and west. Right.


And then these are just the hidden channels, the channels of the hidden representations. Right.


So what you would like to do now is, is to basically encode information from the space around you.


If you look at, let's look at one of these, one of the big pictures, what you would like to do in weather prediction.


Let's say you are right here. What's a good color? You are right here. Right.


Now, if you want to know if this, this particular cloud over here is going to move to your direction.


What you want to know is, for example, is there a mountain range here? Right.


Because then it's more probable that this cloud is going maybe to move up there.


You would also want to know how this cloud here moves. Right.


If this cloud here moves somewhere here around, then it's probably this cloud down here might be pulled with it or something like this.


So you're very much, sorry, you're there.


You're very much kind of want to look out into each of the directions here.


And you want to incorporate kind of what's happening across the space.


We're already used to kind of convolutional networks being able to do this.


But in here, the authors use attention to do that.


So if you don't know what attention is, my most popular video is about attention.


And you can do attention for images.


So the way that works is that you have a series of images of stacked blocks of a neural network.


Let me draw this here. So you have an image here. And let's say it has just four pixels. Right.


So you have the next layer of these four pixels. Right. So you have layers of this.


So the next layers of the four pixels, they all emit what are called queries.


And queries are just vectors. So each pixel emits a single vector.


Let's say this, that, that, this. Right.


And each of the lower layers emits what is called a key. This, this, this, this.


And now the keys and the queries are routed together based on their inner product.


So these two would be routed together. This would probably be routed here. This as well. This would probably routed here.


So what in effect, each of the pixels of the higher layer can look at specific pixels of the lower layer.


Now you can imagine this is exactly what we want here in that if there is a mountain range here,


and we might be interested in that, so we'd be able from our, from our point here to specifically attend to that location.


Using, using attention. Right. So the authors here build basically a stacked model of attention layers.


And that's what's happening in the third part here.


This is, the attention is in order to incorporate long range dependencies.


As I made the example with the mountain range, this might be far away, but it might actually influence your weather very much.


So the attention is to incorporate these long range dependencies.


But the problem with attention is, is as you saw in the example,


each of these pixels can attend to each of the pixels in the lower layer.


So what you'd end up with, so each can attend to that, this can attend to each, this can attend to each.


You'll see you'll end up with 16 connections, can't even draw them, so you end up with 16 connections.


In general, if you have D here, you will end up with a D squared number of things you need to calculate. Right.


So if this here, and now of course we have images, so generally we'll think of D by D pixels.


Now we have D by D pixels and that thing squared number of things we need to calculate.


This quickly gets too much. So in, for example, MNIST you have a 28 by 28, you have 28 by 28 pixel images.


This is 780 or something, I don't quite remember.


But you have to calculate this squared many connections between things.


This is simply gets impossible pretty quickly, especially if you scale up the images and then have some channels in here as well.


So attention for image processing has been a bit lagging compared to natural language processing.


In natural language processing usually have maybe 500 tokens or something.


Images you have much more, so attention is much more expensive, so you can't really do it on current hardware.


Now this paper uses something called axial attention.


Axial attention is kind of the trick of how to make these, this tension happen for images.


And for that I want to switch over to this paper.


It's called Axial Attention in Multidimensional Transformers by some of the same authors.


So Jonathan Ho and Nelkout Brenner, also of Google Brain and UC Berkeley.


And they proposed this axial transformer, now they originally proposed axial attention for autoregressive models.


If you know transformers they also started by making autoregressive models, so language modeling and so on.


But we can decouple the axial attention from the autoregressivity of these models.


So I'm not going to talk about autoregressive model here, it's just axial attention.


So what is axial attention? It's pretty simple actually.


And I want to start by talking about convolutions.


So what does a convolution do? Let's just take a one-dimensional image, which is pretty boring.


But let's say it has these 8 pixels here, right?


So this is an image, it just has one row of 8 pixels.


What do I do when I run a convolutional filter across that?


This is the lower layer, and now this is the next layer that is produced by a convolution.


So for each of the pixels in the next layer, what I can do with the convolutional layer,


I can look at its neighbors in the lower layer, right?


So these three would be part of that.


And then I go on to this, and again I look at its neighbors, at these three, right?


I might have done this in a different color.


And then I look at this, and it can look at itself and its neighbors, right?


So a convolution is pretty smart.


And then of course in the next layer, that repeats.


Now if you think, what's the difference between doing this and a fully connected layer?


So if I have a fully connected layer, you know, classic neural network, fully connected layer,


then this pixel here would incorporate information from all of the pixels here, right?


And this pixel here would incorporate information from all the pixels.


Now why might this be better?


Because, you know, the information that I want here for this pixel might depend on this pixel over here.


So I might benefit from a connection over there, or it might benefit from, sorry, from this pixel here, right?


Which you can't reach.


And with a convolutional network, I can't do that.


Why are then convolutional networks preferable?


Because the convolutional network can do the same thing across multiple layers, right?


So let's assume again that this pixel here needs information from this pixel right here.


And as you can see, in just one layer, it can only get information from those, right?


But now take the next layer.


So the same pixel here, it can attend to these three, right?


Now, these three can each in turn attend to it their neighbors, right?


And I'm not going to draw everything, but the resolution field for this pixel here will end up being all of this, right?


Now we still don't have our desired pixel in here, but if we just go one layer more,


then this pixel right here, in a different color, this pixel right here, right?


The resolution field across the layers increases because it's always incorporating information from downstream.


And the downstream again incorporates information from the downstream.


So eventually you can aggregate the same information.


So instead of having a single layer with all of these connections, we have convolutional layers,


which seem like a worse idea because they can only do less things, attend to less things.


But across the layers, they actually can do the same thing, right?


And that turns out to be a huge advantage of these convolutional layers.


And that's why convolutional layers are used for image processing and not the multi-layer perceptrons.


So the same exact thing happens with axial attention, just in a different form.


It is a bit poorly drawn here, I believe, but this is how you have to imagine it.


As before, this pixel, the red pixel here, if I just have a normal transformer layer,


the red pixel can attend to all of the other pixels in the image, right?


That's basically, and each of the pixels can do that.


So that's your D squared computation right here.


Now, what we want to do is, in a convolutional layer, what we would do is, okay, you can only attend to your neighbors.


And then in the next layer, the neighbors can attend to their neighbors, and thereby you go out and out.


In axial attention, you say, okay, this thing can only attend to its row and its column, right?


That's it. You can only do attention to your row and your column.


And I believe they don't even do it at the same time.


So in one layer, you can tend to the row you're in, and in the other, you can attend to the column you're in.


Now, let's see how the same thing happens as for a convolutional layer.


So in the, basically, how then, if the red pixel needs access to information in this green pixel, how does it do that?


So in the first layer, it can attend to its row and its column, right?


And so can every other pixel, including, sorry, including, of course, the pixel where that, so let's say, this square here, can also attend to its row and its column.


And its row happens to be, including the green one, right?


So in layer one, this red square here gets information from the green square via row attention, right?


And then in layer two, now, this our red square of interest now can row attend to this other red square here, so they get connected in layer two.


I'm sorry, I don't want that.


So you see that within just two layers, we've transferred information from the green square via this red square to that red square.


So we can, in the same way as a convolution, you can replace the long range arbitrary dependencies between pixels by simply having multiple layers of restricted dependence.


And the same goes for this axial attention.


So you can replace the arbitrary attention in layers, right?


You can replace that by a two step process where you first transfer information via the column and then transfer it via the row.


It's a bit like, you know, in chess, you can have a queen that can move any direction, especially diagonally.


And then if you just have a rook, you kind of need to do two moves.


So in the queen is like the full attention and the rook is the multi layer axial attention.


They can achieve the same thing.


You just need more layers, but as a trade off, you get a super, super saving in requirement of memory and computation.


So they stress that, you know, kind of you can represent the same distributions with the axial attention.


And you know, the trade off is you just have to do multiple layers of it.


Right, so this is axial attention, and they are now able to incorporate this into their model right here.


So they have, I believe, eight blocks.


So four row attention, you see this right here, and four column attention blocks in their model.


And finally, they output this distribution here across their region of interest.


Now this, again, is your, I believe, this 64 by 64 resolution.


So you can see how they kind of aggregated information across the 64 using this axial attention.


And then that makes their prediction in this one hour.


So this is this.


All right, so this was a long way.


So recap, they have 15 minute snapshots of these input data along with some features.


They use a spatial downsampler, which is a CNN, on each of them individually.


Then they use a convolutional LSTM to encode this across time to end up with a single representation here at the end.


Then they use axial attention in order to aggregate information across the spatial dimensions.


They do this in multiple stages and at the end, they make a participation prediction, which is a distribution, as you can see here.


So as an output, you directly get a distribution of results.


Which is also cool because the physical simulation, you have to let it run many, many times in order to get a distribution of results.


And this neural network can simply give you a distribution right away.


That's what they say right here.


So they go a bit into the architecture compared to baseline.


I want to get back to what I showed you at the beginning, and this here is just the picture, kind of the picture book example.


So left is the ground truth, in the middle is MetNet, and on the right is a baseline method.


And this here is in, as you can see, in two hours, in four, six and eight.


So you can see the MetNet gives you as an output this distribution.


What I find interesting, for example, is this sample two right here.


So in this sample one, you can see there is a consistent difference, and this is the forecast time.


So how much in advance you want to know.


This would be a, was our one hour, but it can go up to eight hours.


So here is a consistent gap in F1, which means the MetNet does it better across this span of time, which is for the top sample right here.


For the bottom sample though, you can see here, there is a big gap at the beginning.


Again, there is a big gap at the beginning, and then this gap gets smaller and smaller and smaller.


And this, I think, might give you an indication of, let's say, the weakness of this approach, doing it with neural networks.


So with neural networks, you kind of rely on regularities.


You kind of rely on broad scale correct things that you can learn from the data.


And this might work well as long as things are regular, which, of course, across shorter time spans,


things tend to be more regular, right?


But if you go for longer time spans, I believe there is more of a chaos element to it, like whether it can be very dependent on very subtle things.


And the physics simulation that is really, you know, taking into account the actual physics, might be able to much, much better account for that.


And that's why I believe across time here, you'll see that the two models get closer together.


That being said, MetNet, of course, still on top here.


But it will be interesting to forecast for longer even, though I haven't actually dig through their results, through their numerical results.


But you can do that if you want.


All right, so this was it for MetNet and axial attention.


[ML News] DeepMind builds Gopher | Google builds GLaM | Suicide capsule uses AI to check access

DeepMind builds a dense language model with 280 billion parameters. Google builds a sparse


language model with over a trillion parameters. And Microsoft has a new dashboard. Welcome


to ML News.


Hey there, this video is sponsored by weights and biases. Me and weights and biases, we've


decided to take the next step in our relationship. And that means I now have my custom link,


Wanda B dot me slash Yannick for all your needs. I actually don't know what's I'm gonna


look it up after but there might be a surprise who knows what's behind that link. The only


way you're going to find out is by going to it. Anyway, today I want to tell you about


a new feature in weights and biases. So I've previously told you about tables tables is


this very cool thing in weights and biases that allows you to analyze your data, your


models, your results, your outputs in a table form, but the table is like interactive. So


the table can do anything from filter and group to display your plots, play little sound


files, play GIFs, and so on. And it's just an awesome way to look at your data from different


angles. They have now added a new feature to tables called the embedding projector.


So whenever I wanted to look at some sort of projection of my embeddings or data, I


had to do that within the experiment and then log that like as a picture to TensorBoard.


Now TensorBoard has also gained some projector view. But this here is really cool. So you


can take any table and any columns of those tables as long as they're in or floats. And


you can use these projections to map them to a two dimensional space and then look at


them in 2d. Now for that you have several algorithms at your disposal. On the left you


can see a PCA projection of the digits data set. And hovering over any given sample shows


you more information in this case, the sample itself. In the middle, you see a UMAP and


on the right is a t-SNE. You can interactively configure these projections, including their


parameters which columns are included, how the data is constructed and much, much more.


And these are interactive like you can do anything here that you would do in a regular


interactive plot. And as always, you can then pull those into reports and show them together


with data or with some explanation. And this is just a really cool tool to do data exploration


or exploration of the predictions of your model. You can see you have all the power


available here of regular weights and biases plots such as color coding, or intensity coding,


whatever you want. Look at that. Isn't that a data set? Oh t-SNE, what are you doing?


Now I absolutely invite you to go check out weights and biases not only for the embedding


projector but as you know, they have tons and tons of features for both practitioners


and researchers. It's completely free for personal use and academic use and no excuse


not to try it. Thanks again to weights and biases for sponsoring this video and let's


get into it.


DeepMind releases a blog post called language modeling at scale gopher ethical considerations


and retrieval that details not one but three new papers out of DeepMind. Gopher is a huge


language model and its biggest configuration is over 280 billion parameters. That is almost


twice the size of GPT-3. Now the authors here evaluate the model on 152 diverse tasks and


they achieve state of the art performance in the majority of them. The paper as you


can see is pretty long as it needs its own table of contents. But it's essentially a


big investigation into what these language models can do, what they cannot do, and how


they perform in the individual tasks. The main interest here is what happens if you


scale these models up? What can you do and what can't you do? And the author's notes


gains from scale are largest in areas such as reading comprehension, fact checking, and


the identification of toxic language, but logical and mathematical reasoning see less