commands and so on while keeping the policies fairly simple. So it's, I think it's quite


Author Interview: SayCan - Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

exciting to see how much further we can we can push that direction.


Yeah, I think representations have always been such a challenge for especially like


task representations are such a challenge for robotics. And I think language has provided


this like really nice interface to interact with the robot and then have the robot interact


with the world.


Excellent. Well, Carl, Brian, Fay, thank you very much for being here. This was a lot of


fun and I hope to see you again soon.


XLNet: Generalized Autoregressive Pretraining for Language Understanding

Hi there, today we're looking at XLNet,


generalized autoregressive pre-training for language understanding


by Jilin Yang and other people from Carnegie Mellon University as well as


Google Brain.


So this is kind of the elephant in the room


currently as XLNet is the first model


to beat BERT, which was the previous state of the art on a lot of NLP tasks.


To beat BERT at a lot of these same NLP tasks, so the


chief state of the art result on 18 of


20 tasks I believe.


Maybe they test more. They outperformed BERT on 20, the chief state of the art on


18, including things as question answering,


natural language inference, sentiment analysis, and so on.


So those are kind of remarkable results and even more remarkable is that the


architecture of the network is actually very


fairly similar to BERT. The kind of new introduction is


a pre-training, a different pre-training procedure


and we'll look into that. So


let's actually jump into their main points straight away.


What they go into is there are two kinds of currently used


pre-training methods for these NLP tasks and both


can be understood as kind of language modeling.


So language modeling for


those of you who don't know is predict the next word in a sequence.


So if I give you the sequence here, unsupervised representation learning has


been, and then I ask you what's next?


And then you're supposed to say highly.


This is language modeling in a nutshell.


So what they


differentiate are two kinds of language modeling. The first one


they say is autoregressive language modeling. Now what autoregressive language


modeling does is exactly what we've


looked at. I give you unsupervised learning has been,


you're supposed to predict highly. And then in the next step


I give you unsupervised representation learning has been highly,


and you're supposed to predict successful.


And so on. So in the next step I'm going to give you the entire sentence up until


here and you're supposed to


predict in. Autoregressive because


each token can look at the kind of previous


ones in the sequence. So when you,


sorry you can't see that, when you predict,


when you predict you can always kind of


autoregressively look at what the previous ones were


including what you've previously predicted.


Of course during training this is teacher first


as I said so you put the actual words there. This is


autoregressive modeling in contrast to


what they call autoencoding. And autoencoding


is what BERT does. And this is the following.


So in contrast to that let's say I


have the same sequence unsupervised representation learning has


been highly successful in


the domain of, yeah something.


And then I say okay, I give you the sequence but


I'm going to delete this and this.


And now I ask you to predict


these two. So you can see


the task is slightly different as you now have access to all


of the sequence basically except the ones that you're asked to predict.


But you're kind of asked to predict them


not in any order but you're asked to predict them


at the same time basically. So at the same time you're asked to predict


this word and this word. And


so the first


kind of this autoregressive language modeling has been used by transformer


models


until BERT. And then basically BERT


really pushed this autoencoding


language model pre-training which made it so successful.


And now this paper XLNet wants to


like combine the best of both of them.


And in order to understand what's the best of


of both of them, so what's good at BERT we've already seen


it can actually draw information from all of the context


of the words it's trying to predict. But what is the kind of


pitfall of BERT? And they actually put this really nicely


in an example they gave way further down


where they say comparison to BERT. I don't know why that is not


like also in the introduction but here they


have the sentence New York is a city. Right? New York is a city. This one.


And you're asked to predict these two words.


And if you now compare BERT to what XLNet does


if. So the context is a city and you're asked to predict New


York. What BERT does is it simply masks out the two words


and says here please fill in these two words.


Now this translates to the kind of objective being separated in the


two words such that the prediction of York here is


completely independent of the prediction of New.


So if you know of any other city that is made of two words for example


San Francisco or Los Angeles then these would be as valid and any mixture


would be as valid. So you might BERT might end up with


Los York is a city and that will be perfectly fine for BERT because


while it's predicting Los is a perfectly fine prediction for the first word of a


two-word city and York is a perfectly fine prediction for


the last word of a two-word city. Right? So


these are the kind of mistakes that BERT can get into


by not being autoregressive by basically predicting all of these tokens


at the same time independently of each other. Whereas XLNet


what it would do is it would specify an order. Let's say okay first I will


predict the word New for the first word


New something is a city and then when I predict


New York I will actually take into account that I previously have predicted


the word New. So that's the main


advantage that autoregressive training has


over auto encoding. Now what are the pitfalls? The pitfalls are


if you have this this sentence let's look at it


I'll write it down New York is a city.


Right? If you have this sentence and let's say actually you're not you're


not asked to predict New and York you're asked to predict the the word A


here A right you're asked to predict that


in autoregressive style or A city it's a better example


the two words A city in autoregressive style if you predict the word


A you can only ever look at what comes beforehand


whereas if Bert were to predict A just the word A


it would be able to look at all of it. Okay let's not predict city so you see


the the kind of autoregressive model is


bound to the order of the of the factorization of the sentence


that's why it's bound to the order in which it has to predict the tokens


so here if it's predicting A you can only look at stuff that comes before it


because it needs to do it in order right once it gets to city you can actually


look at the entire sentence here but before that it only


ever has partial information about the about the context


so actually it wouldn't be much better if I had said we're trying to predict


these two words is and A right and once I predict so


so Bert would actually have access to the word city here whereas the


autoregressive models only have access to the ones before it I hope that makes


it clearer so the main idea in ExcelNet is


where does this order dependence come from in the autoregressive model


the order dependence actually comes from the factorization


of the sentence of the of the language model


so in a language model we're actually trying to assess


the probability distribution of sentences here x is a sentence right


and this can be naturally factorized into a product over the words


where the probability of each word is only dependent on the words before it


so this is a this is an equal it's not an approximation this is an equality


the probability of a sequence can be decomposed


into a product of probabilities like this


exactly so this here is exactly what these autoregressive models implement


each word is predicted from the words before it right


there are other kinds of autoregressive models that also do the other direction


where here they say okay the probability of a sentence is a product and each word


is predicted from the words after it but it kind of is the same problem you


only ever have access into the one direction basically however


you define the order of decoding you only ever have access from a given


word to what was before it in the order so the


main idea of xlnet is they say hey


why don't we consider all possible orderings


right i mean that that's kind of a yeah it's it's an idea so let's go back


to our thing here they say why don't we consider all


possible orderings so basically what we will do is if this


sample comes up new york is a city all right um


what i can do is i can define an ordering let's say i always want to


predict two words so burt typically masks out about 15 of its input


um the to be predicted and here let's say we'll mask out 20 which is


two words so of this sequence we'll mask two words


and ask the model to predict it that's that'll be our our pre-training


objective um the first time the sample comes up


from the data set i might specify the order just classically right just


one two three four five all right i'll predict the last two


words i'll kind of mask them out right i give the


model new york is uh and then i let it


predict a and then in the next step i'll give it


new york is a and i'll let it predict city cool so now


the pitfall is the word a here only has access


to things before it and not to city itself


city has access to everything all right so but then


i continued training and the next step time


this sample right it's in my data set new york is a city


the next time it comes up i simply go for a different order


let's say one two three four five right so


now um again i'm asked i'm asking to predict


the last two tokens which here are uh city and york so in the


first step i would give it is a and new and i will ask it what's here


and i'll ask it to predict city and then in the second step i'll also give it


that and i'll ask it okay now what's here given all of that


right so new is a city all right you're asked to predict the


missing word so that that's pretty um so in the first


step it's new is a and you're asked to predict


the second and then the second step is new is the city and you're asked to


predict the first um


so now as you can see while predicting city here all of a sudden we didn't


no longer in this ordering we don't have access to the word


york so we'll have to learn to predict city from the rest of the context


now even more um even more if we now decide let's decide on


a different ordering again one two three


four five so now we'll actually first step is to ask


new york city please predict this thing here all right you model


might yeah you might train the model to predict


is and then the second step you say new york is city please predict this


now you see before before when we are we're asked to predict the word


a it only had access to things to the left of it in the very first


example but now it actually has access to the entire context


so the the um the idea is as we sample this data point multiple


times and each time we decide on a different ordering to decode


for each prediction of each token token sorry


will actually have seen many many parts many different variants


of the context and um in expectation we'll actually have seen


all of the context just like burt but we'll


always having have done it in an autoregressive way


so basically you get all the advantages of being autoregressive namely


that you are able to decode step by step while always um referring to


everything in front of you in the ordering


so the predictions are not independent but you also get the benefit of birth


that it's able to basically look at all of the rest of the context in


expectation in order to make this prediction so this


is this is the main idea of um of xlnet they


formalize this jump up again they formalize it in saying okay what


bird does here is it actually see it factorizes the


log probability of a sentence into this sum so the product in the


log becomes a sum into the sum of log probabilities of


um no sorry this is autoregressive confused


into the the words conditioned on everything in front of


them what bird does is it actually


approximately factorizes the log probability


into each word and then everything in the context everything that's not


masked in the context and this is only an approximate


factorization because now you're basically dropping away


all these masked tokens and um what they do


now is they do the same as the ar as the autoregressive models here


they decompose the log probability into a sum of log probabilities over


each of the words given all the words before it but now not


before it in the sequence but before it in an chosen permutation z


and z is sampled uniformly from the set of all


possible permutations so in expectation they'll see all of the


context so this is the this is the main thing they


show this here in a kind of a picture with so here is the neural network


this is the input layer and these are the hidden layers as the


attention layers go up and up here you're asked to


predict the the token so here you're always asked to predict


x3 so there is no there's never going to be any um


weight here since if you knew x3 you would be able trivially to


predict x3 one would hope um all right so in the in the first example


the factorization order chosen at random is three two four one


now you're asked to predict x3 and we know


okay we should only we should only do this with things that are before it in


the permutation order well here since x3 is the first in the


permutation order we actually don't um we don't have


anything to go on we basically ask to predict x3 from


scratch as if it were the start of the sentence so we'll


basically tell the model i have a sentence that goes


please predict the third all right it's a hard task yeah um by the


way you're always able to look at this memory thing here uh don't worry about


this for now this is just uh this is a an


augmentation they do on top of their idea this is not the core idea


so okay but now the second time this sample comes up from the training set we


decide on a different order so the order here is two four three


one now again we're asked to predict x3 and


we're allowed to look at everything before it so two and four as you see


here there are weights from x2 and x4


into this column that finally is then a ask to predict x3 so this is also


this is now an easier task right you're allowed to look at the word to the left


and to the right um if you have the following permutation


order one four two three you're actually allowed to look at all


of the other words because x3 is at the end of the


permutation order in order to produce x3 so all of these


four and the fourth thing is similar so all of


these four things will appear during training and you will learn


from them so in expectations you'll basically have seen all variants of


different uh of different versions of the context


which which helps a lot apparently


all right so


in the in order to achieve this they had to make some architectural changes


to the um to the model namely what you want to do is in a single


pass through the model here you not only want to predict


one token but you want to do many predictions this helps training a lot so


bird and naturally always does like the 15


we mess up 15 of the tokens or so what what's that like 40 50 tokens


so it masks them and it predicts them all at the same time


now you would like to do this here as well you would like to predict all at


the same time um the ones that you're asked to predict


but of course the problem is for here if you're asked if in this


factorization order 2431 if you're asked to predict x3


you're allowed to look at x2 and x4 if you're asked to predict x1 you're


allowed to look at x2 x4 and x3 so if you only have a single pass


through the model the question is do you now input x3


or do you not because the prediction of x3 is


not allowed to look at x3 while the prediction of x1


is allowed to look at x3 so they do an architectural change


in order to achieve both things so that you can have a single pass through the


through the model but the prediction of each token only


depends on the things uh in front of it in the permutation


order and they do this by having these kind of


two stream um these masked two stream attention where


they basically have not only one hidden representation like in classic


transformers but they have at each step two hidden representations


one they call h and one they call g um so the h's are initialized with the


embeddings of the tokens and the g's are just


initialized randomly and then they get transformed


the point is the h of the next layer is always able to look at everything in


front of it including its own its own h


basically it's one layer down its own position one layer down


while the g is only allowed to look at the it's allowed to look at the


h's but the h's from before right so


so all the g's here are only ever able to look at the h's


from before the current position whereas the h is always allowed


here to look at the same but also at the h


at the current position and now at the last layer


you simply ask the model to predict the token from just the g and you can


easily see that this results in this model only um


oh yeah only attending to things before it


um okay the g by the way can also look at the g


of the current layer so that's that's also


the thing but it cannot look at that at the h so there's never any information


flowing from the current um


from the current word embedding of the token you're trying to predict


to the prediction layer so basically that means the model can't just look


like you're not telling the model the answer yet you're still able to feed


to predict multiple things in a single pass through the model


uh formally this is described here in the attention layer


um so they divide how they produce the queries and how they produce the keys


and values usually the queries and the keys and values


are produced from the same hidden representation but here they produce the


keys and values from the h's in both cases


but to update the g's they produce the queries from


the last layer's g and to produce the h's they produce the queries from the


last layer h's and most importantly when they produce


the keys and values the h's they look at here to update


the the g you're only allowed to look at h's before you in the permutation order


but to update the h you're allowed to look at everything


before including the position you're currently


at so that's kind of the that's a it's an


engineering solution to the problem introduced by their


augmentation i think it's a pretty pretty neat solution


pretty cool um yeah


so the rest of the paper here is incorporating ideas from


transformer xl so transformer xl is one of these classic transformers that


that is like this um ar so this autoregressive style of transformer


but that has a few improvements over the classic


kind of vanilla transformer and they incorporate


a number of things here namely first of all they incorporate this


memory thing so the memory thing allows you to


input longer sequences let's say our our


transformer input length is maximum of five tokens


what the transformer xl allows you to do is


you input five tokens and then you save you do your transformer thing you


encode it and you save something into this memory and then when


you input the next five tokens your transformer is then allowed to


look at the memory of the last sequence right and also update it so that that's


kind of these these mem blocks you saw here so you're


always allowed to look at these mem blocks from last sequence and then


the hidden representations here of this sequence they will actually be


stored in the mem block for the next sequence so this is kind of


a trick um to to to carry over information


it's not the the the updating the memory part isn't learned


with the objective to make the next prediction better but it's just um


some information some kind of gradient free information to provide to the next


step and it apparently helps you can


incorporate longer sequences into this transformer excel so they take this over


and implement this into xl net they also do relative position encodings


relative segment encodings i won't go into this too much more


here because it's not the the main idea basically


so they do experiments and they um compare to bird architecture with the


same basically same architecture there's the same number of parameters


and or layers and um they beat burped in of these kind of nlp


tasks or most of i think they said in 20 they reach new state


of the art in 18 nlp tasks so apparently their method


works very well um so what they do here is


the last thing i find important is an ablation study of the effects of their


improvements um so they were because


kind of my problem is i never know like they have this new idea okay we do


these random permutations but then they also say oh and


also we include uh memory from xl net and we we do relative


position encodings and so on to me these kind of papers of course you


reach better numbers you get a new state of the art so it's kind of a landmark


paper but to me a paper should more be like a


single thing so whatever your idea is this


your idea is these orderings and whatever you need to do to make that


work okay fine but then why why the additional


transformer xl things um it's it's really then hard to


estimate how much of the improvement comes from your idea and how much of the


improvement simply comes from the fact that you already


put these other things that actually have nothing to do with it


so i appreciate these kind of analyses called ablation studies


where they kind of try to take away the memory and


and these things and kind of look at what it's doing


to the model and you you see here um kind of degrades down here as


for example this column degrades as you take stuff away


um while still being more kind of more successful than burt


um so that that i would say also yeah here is more unclear but also


kind of seems to degrade a bit and while being more


successful than burt so i appreciate this this um


kind of really trying to show that your gains really come


from your new idea and not from some other stuff


all right so the last thing i want to mention actually is


uh this thing so someone claiming or calculating that


it costs 245 000 to train the xl net model


um the way they describe it in the paper i'm sure that's going to be brought down


because it was brought down that like the time to


train was brought down with burt as well but this is just i mean


this is crazy this is just training it um it it kind of gives large


questions about the state of research and the


ability um for kind of let's say more academic players to


participate in research on the one hand of course we


like of course these companies should be able to do this and


on the other hand if it seems like currently in some fields just putting


more money on the table will get you a better result um


not this this actually like this paper is actually a cool idea but it's still


kind of prohibitively expensive to even reproduce it


yeah all right so that was that was that for this paper


The Odds are Odd: A Statistical Test for Detecting Adversarial Examples

Hello and welcome. Today we're looking at the odds are odd, a statistical test for detecting adversarial examples.


So, shameless self-promotion here, since this is me.


So this is an archive, and basically what we do is we're detecting adversarial examples.


For those who don't know what an adversarial example is, it's basically a way of fooling a classifier in order to kind of get it to do something weird.


Let's look at it. So maybe you have an image of a cat.


Cat.


And, yeah, I have no clue how a cat looks.


Alright, so you have an image of a cat, and you have a classifier. So the classifier takes this image as an input, kind of winds it down to some probabilities of classes.


Cat, dog, and so on.


And it then gives you an estimate of how likely each class is.


And so on.


So, what the adversarial example does is it changes this image, and it adds a noise.


So this is just very specific noise.


And you have kind of a multiplier here, gamma, which is super small.


So the noise is almost, you can't see it with the human eye, basically. It's so small.


But it's able to perturb this image in a way that the probabilities will change such that all of a sudden a different class now is the highest class.


So basically we're able to fool these classifiers by adding just very little bit of very, very specific noise.


So that's an adversarial example.


These have many implications in, let's say, security applications and also in understanding how these classifiers work.


Alright, so our task is to explain and detect them. Explain why they happen, and detect when they happen.


Alright, so what do we do?


Basically, let's just jump right into the thing here.


We view a classifier as an output.


So you have logits, what's called logits, L is this.


This here is your neural network up to the last layer.


Basically it can be something like a convolutional neural network and so on.


It gives you a feature representation.


So you extract from the image X a feature representation, which is this entire thing here.


And then you multiply this feature representation, so this is going to be some vector of dimension d.


You multiply this by this weight matrix, which is going to be something like,


okay, I've drawn it in the wrong direction here.


Let's draw W over here.


It's going to be d by, let's say, k, where k is the number of classes.


Okay, still wrong.


So it's going to be d by k, right, and output a vector of dimension k, which then is this cat, dog, and so on.


So these are the logits, and the logits get transformed to the probabilities by running it through a softmax layer.


But basically, we view a classifier as having a feature representation and a weight matrix.


This here is a matrix multiplication, a dot product by matrix.


So what we see basically is this is kind of where the adversarial examples happen.


So when we look at this weight matrix, right, again, we look at the d-dimensional feature vector here,


and we look at the weight matrix, what it does is it has columns, right.


Columns.


Let's say we have four classes here, right.


So it has these four columns, and each of them is d-dimensional.


So each of them is going to be multiplied by this thing and giving a score.


So the final score for a class is going to be the multiplication of a row,


w1, w2, w3, w4, by this feature vector.


Let's call the feature vector little f.


So your logit of class i is going to be the inner product of wi and f.


We'll leave away biases for now. We can introduce biases to make it a bit more complicated, but it changes nothing.


So your logit is going to be the inner product, and whichever logit is the highest wins.


So that's going to be the prediction of the classifier.


So in an adversarial example, what can you change?


You can change this feature vector here, this f.


By changing the x, you can change the output of the convolutional neural network, which is the feature vector.


And what you have to do in order to make a logit as high as possible,


basically make one class as high as possible, is you need to make this inner product as high as possible.


And what's an inner product?


If you look in a classic vector representation space, if this is wi and this is f,


what you want to do is you want to make f and w align as much as possible,


because the inner product is going to be basically dependent on the angle and the magnitude.


So you can stretch f for sure, but it's going to be kind of aligned with all the w's then more by stretching or more negatively, in whatever way you want it.


But basically you want to rotate, you want to align as much as possible the f with the wi.


So you want to kind of go into this direction with f.


So now not only do you want to kind of maximize f with a particular wi, what you want to do is be adversarial.


The adversarial task is often framed as just, it's either targeted, so i needs to be a particular other class,


or it's untargeted, which means just give me a perturbation that will make the classifier be fooled.


And be fooled means whatever it predicts right now, it should predict something different.


So what you ultimately want to do is you want this as high as possible for some i that is not the correct i,


and you want this other quantity, w y, let's call it w y, let's say the classifier is 100% correct.


So w y, y is the label of x, and w y is whatever column here is currently predicted.


So you want the sum column where i is not equal to y to have maximum inner product,


and so this is no longer li, we'll get to that, to have maximum inner product,


and you want this inner product with the correct class to be as small as possible,


which ultimately means you want this entire quantity maximized.


So it's a pretty simple idea, we'll call, let's say this is the logit i minus the logit y,


we have slightly different notation in the paper, I think we'll call this z, but never mind.


So you basically just want to make this as large as possible.


And so our point is, since this is not the only thing, you want to maximize this, but you have a constraint.


Namely your constraint is that your delta x can only be small, right?


Your delta x can only be small because the point of an adversarial example is that the perturbation is so small you can't see it.


And that means that you basically don't have much wiggle room to do these perturbations,


which means that we should be able to detect a pattern like this in the latent space.


So this here is the latent space feature vector, and if we can kind of detect a pattern in the latent space,


then we kind of get the adversarial example detector.


All right, so how do we do this? We measure exactly this.


What we do is we measure the alignment between the currently predicted class and between all other classes.


So in this graphic here, you see this. It's a 10-class classifier. This is CIFAR-10.


We only show six of the other classes, but we have the full graphic in this.


So this shows an adversarial example.


The axis going on top of each of the images is the alignment with the adversarial class.


So this has been an adversarial perturbed sample. So this shows the alignment with the adversarial class.


And of course, you see the bright red dot. If you just focus on that, that is the adversarial example projected into this.


So of course, the alignment is going to be very, very high with this class, since the classifier actually predicts this class.


The blue here is the sample that the adversarial sample was derived from, which means the original image kind of.


And you already see without looking at any of the other dots that the blue is around zero here, around zero here, around zero here, here, and here.


But here, it's very high in this axis.


So the axis to the right is for each of these plots here, it's one of the other classes, except for the currently predicted adversarial class.


So this axis is always the same axis, while the axis to the right for each plot is a different one.


And you can already see, and that's why we frame it in the green, this plot here is where the axis to the right corresponds to the original class of the classifier.


So don't look yet at the other plots. What you see here is basically the blue is really high in this class.


And the adversarial example procedure basically has driven it down this class and up this class, which is exactly saying it has made this inner product small and this inner product large.


So where do we go from here?


Let's actually jump this graphic a bit and go to this one that's way down here.


All right. So what we've done is we've taken an example just out of the data set, right?


And then we've taken an adversarial example. So let's say x is the example of the data set.


And then x hat is the adversarial example derived from this.


All right. In this plot to the right, x would be sitting about here.


I'm going to explain what the kind of meaning is.


And x hat would be sitting down here, right? It's about one third from the top, one third from the bottom.


Let me draw this more. All right.


So what this axis represents here is basically what we've done is we've gone from x to x hat in very small steps.


And at each step, we've asked the classifier, hey classifier, what's the probability of the class y?


So y is the class of x, right? And the class of x, x hat is some different, some other class, right?


Since it's an adversarial example.


So we've asked the classifier, what's the class of x?


And basically, no, basically we've asked what's the probability that y is the class of x?


And that's represented in white, right?


So the more white, the higher the classifier thinks the class y is probable.


All right. So the direction down is going into the direction of x hat minus x.


So it's going into the adversarial direction.


And then the direction across is we've taken some direction that's orthogonal to this direction


and then also went into tiny steps and asked the classifier, hey classifier, what do you think is the probability of y here?


So we've basically done this kind of grid sampling and at each point, we've asked the classifier,


what do you think, which, how probable is y?


And the classifier will always output some number and we plot it in white.


And so this direction again is always the same.


And in this direction, we basically randomize it and then aggregate it over lots and lots of different samples.


And we also aggregate this entire thing over the data set so we get a comprehensive view of what adversarial examples look like in the view of the classifier.


And what we find is pretty interesting.


So when you go from the original class, you basically in every direction here, in every direction,


the original class kind of decreases smoothly.


You see at the edges here, it kind of gets black.


So the further away you go from the original example, the more, the kind of shadier the classifier gets.


It's like, yeah, I'm not so sure anymore that this is the class.


But if you go into the direction of here, if you go into the direction of the adversarial example, the kind of drop off is, first of all, it's very steep.


So all of a sudden here, you're in very dark territory, which means the classifier doesn't think y is probable at all anymore.


And moreover, you get this kind of cone here.


So what we see is, what we think is happening is that given an example, there are these directions in image space, basically.


Straight directions that go to adversarial examples.


And we call these cones because they're kind of low dimensional directions in the space where the adversarial example lies.


And what's really interesting is, we have some plots here.


Do we have more?


So what's quite interesting is that, ah, here.


If you go, come on.


Well, this is kind of, okay, the quality of the plots is not very good.


So I'm going to, I may be able to draw this here.


So if you start here and you go here, what happens to the original class is you start out high, you go down rapidly, and you stay down.


Even if you go super far into this direction, this class will stay down.


Whereas, let's say this is y hat.


y hat will start low, go up, and then kind of fade.


So here is where the adversarial example would sit.


Sorry, at about this distance.


That's this distance here.


It means as you go towards the adversarial example, right here, the probability of the adversarial class rises and the probability of the original class drops.


Then as you go further, this is what's interesting kind of.


This probability here drops, which means the classifier is kind of like, yeah, okay, there's too much noise now.


I'm not so sure about this class anymore.


But this class here kind of stays low, very, very long, even if you go into this direction.


So this gives us kind of a hint that adversarial examples are characterized by specific directions that you go into, that you can go into, and kind of suppress the original class and pump the new class up, which is kind of exactly what we've claimed with this inner product alignment.


The next experiment we've done is we've taken this adversarial example here and said, well, if we go outside, if we go into random directions, it's just really this one direction that's problematic.


If we go into random directions, actually, we should be, you know, go back to the original class, right, since it's basically surrounded by the original class.


This is just one direction, and this here represents all the other directions there are.


And how many directions are there in pixel space? Like, a lot.


So we should be able to get back to the original class, but that's not the case.


That's, we found that's not the case.


And we also found why.


So I still want to go back to this plot here.


If you do this, if you add noise, and this is the noise magnitude here, what you'll see is the orange here is the adversarial class.


So orange will go down, down, down, down, down, down, right, as you increase the noise.


The blue is the source class, so the blue goes up, and it goes up faster.


You see it goes up faster than the green, which is the highest other class.


So green is whatever class is not the adversarial, not the source, but the highest class other than that.


So the source class goes up quickly, but before the source class can overpass the adversarial class, which happens back there,


the highest other class is already kind of taken over.


So the source class is basically too weak, and if you again look at this, this plot here,


if you go with an actual color picker, you see that the amount of white here and here is not high enough.


It's like 0.3 or something out of 1, or even lower.


So the kind of source class is not strong enough that by simply adding a bit of noise, you can go back.


But we thought, hey, if this is correct, we can actually detect this effect here, this rising of the source class faster.


So our plan is basically we add noise, a particular amount of noise, just a little bit actually,


and then we detect which basically which class falls and which class rises.


And the way we do this is we detect this exact alignment that I've described before under noise.


So we form this quantity here for all classes other than Y.


So Y is the class that's currently predicted.


And we look at it, what happens under it, under noise, right?


So and that's where we get to this graphic here.


So again, this axis is the adversarial class or the class that's currently predicted, right?


This axis here is all the other classes for each plot one.


And when we add noise, what do you see is the noise magnitude is encoded in the brightness of the dots.


So the darker the red dots, the more noise we've added.


Here is the original adversarial sample.


Then as we add noise, you see here, here, more noise, more noise, more noise.


Nothing's really happening for the if it's like one class that has nothing to do with the original class.


It simply kind of goes down, simply kind of gets less sure about this class, right?


But in case of the original class that the adversarial example was derived from, it really rises.


It really kind of at the same time that it drops, it rises into that direction.


So we're able to measure these deltas here under noise.


And we're able to devise basically statistics of what happens to these quantities under, like if it's not an adversarial sample versus what happens to these quantities if it's an adversarial sample.


So here you see pairings of basically source class and adversarial class samples.


So each of these histograms is collected from that.


And what you can see is in blue, the kind of alignments under noise of the source class.


Sorry, the alignments under noise of a non-perturbed sample.


And in orange, the alignments under noise of an adversarial sample.


And what's cool is that these alignments you can see in all of these cases are very different.


So there's a clear signature in the adversarial sample in these noise-induced alignments with the weight matrix rows that makes you able to basically build a detector.


You can say, all right, anything to the left is clean, anything to the right is adversarial.


And we can do this over many different types of noises and then build basically a voting mechanism on that.


And thereby detect adversarial examples.


So we have a bunch of experiments.


We mostly experiment on the C410 and on the ImageNet dataset.


And you can see, where do we see? Here.


So this is the main, kind of one of the main results, the detection rates of our statistical test.


So as you can see, our detection rate, this is on clean samples.


So on clean samples, you want the detection rate to be low.


On adversarial samples, you want the detection rate to be high.


And we achieve very large detection rates while having very low false positive rates, especially on ImageNet.


So it seems like the more tuned these models are, the better these models are, the better we are at detecting adversarial examples to it.


It's kind of a direct correlation to how well the models perform on accuracy in a clean setting.


And what we can do is now, since we can not only detect these things, but we can detect these things in a fashion.


So if you look at these things and you have a sample of a particular class that's predicted, let's say this class,


and you go and look at it at the position of the noise-induced features over each of them, so let's say here, here, here, here, here, here, here, here, here, right?


You can then clearly say, well, not only do I detect an adversarial example here, right?


I look at each of the classes that it could be derived from, right?


If all of them say it's a clean sample, then all right, it's a clean sample.


But if one of them says it's an adversarial sample, then not only do I know it's an adversarial sample, but I say, aha, this must be the source class, right?


This is the exact effect we saw here, right?


If we detect this pattern here, we can also back deduce, basically, aha, so this must be the original class that the adversarial example was derived from.


So we're basically able to build not only a detector, but we're basically able to reconstruct the original class.


And here you see for these models, let's say on CIFAR-10, we mentioned that it's a bit too large as a Vyat for our compute,


but on these models that have clean accuracies that are pretty high on CIFAR-10, plus this kind of toy network here,


we're able to reconstruct the original class, so basically this is defense against adversarial examples by getting to almost clean accuracy back.


So this is really surprising, actually, and kind of nice.


So we do a bunch of other experiments, including we defend against an attacker that's actually aware of this thing.


But the main point here is we don't say this is kind of the end-all method of defending against adversarial examples.


We simply want to kind of encourage the way of thinking of these kind of noise.


What if you noise-induce perturbations? How does your network react to that?


Can you detect these effects here? Can you detect effects like this?


And are these unavoidable, or are there architectures?


Are there architectures we can basically build such that adversarial examples have no chance except doing something like this,


which we can then easily detect?


Reinforcement Learning with Stable Baselines 3 - Introduction (P.1)

What is going on everybody and welcome to a reinforcement learning with stable baselines 3 tutorial series.


So stable baselines 3 is to reinforcement learning like scikit-learn is to general machine learning.


The idea here is to abstract away some of the more complex and nuances of writing and applying these reinforcement learning algorithms


and sort of homologize the entire experience such that once you've created your environment and you can essentially just try all these various algorithms really quickly, really easily


and just kind of get a lot of the code written for you that you're going to have to write every single time.


And so at least for me when I was embarking on a recent series for the simulating robots and doing reinforcement learning to try to teach the robot how to walk


I did indeed find this very challenging to apply these existing reinforcement learning algorithms to some environment


and then if you want to change it or anything like that it becomes just a massive, massive headache.


So anyways, I eventually stumbled into stable baselines and found it to be unbelievably easy to use so I highly recommend it.


So with that to use stable baselines 3 you're going to need PyTorch that is the backend that they use.


So you can come here and install PyTorch, this is PyTorch.org and you can get started.


And then also you will need stable baselines 3 which is just a pip install stable-baselines 3 and then the extra stuff.


I don't actually know if you will need the extra, I don't know what's included but grab it anyways.


So once you have that let's go ahead and get started.


So we will be using OpenAI Gym as well, I believe Gym is included when you install it but I don't actually see anything specific here


but you also might need to do a pip install Gym and then also possibly some of the extras but I think we're just going to be using the 2D environment


so a pip install Gym might be successful but if not check the description.


So I'll throw something in there, there's also a text based write up so if we have any problems I think it will be fine.


I think the only thing you'll need here, I'll copy and paste this in here, you'll need the Box2D so pip 3 install Gym Box2D


that's all we're going to use from the Gym and obviously eventually we'll use a custom environment.


I'll show you guys exactly how to do that as well but for now to keep things simple we're going to use the OpenAI Gym environments.


So before we get too deep into that I could spend a whole bunch of time on all the theoreticals of reinforcement learning.


I'm not going to do that, I think the best way to learn reinforcement learning is to jump in and if you find an algorithm that works really well


then you can dive in and learn more about that algorithm but I think you're going to learn by using much better.


So I won't get too deep into the weeds but what I will say is we are going to cover some terms that you must know before you get into this.


So first of all we have the environment, this should be fairly obvious but it's what you're trying to solve.


So Carpool, Lunar Lander, some other custom environment, if you're trying to make some AI play a video game, the game is the environment.


Then you're going to have model, so this is just basically what algorithm are you using, it's your specific trained algorithm is your model.


Then you have agent, this is the thing that's interacting with the environment and using that algorithm or model to do so.


Then you'll have observations or states, what's the state of the environment?


This could be like imagery or visuals or it could be other vector information like sensors and stuff like that.


You have an action, these are basically what is your agent going to do in that environment essentially per step and that's another term, it's just simply the step.


So when you step in environment you quite literally just take a step.


In general you pass your action to some step method, the environment performs that step and returns a new observation


and reward for you to continue in this kind of loop until you're done.


So if you think in terms of like frames per second, if you're playing a game at 30 frames per second then you have 30 steps that you would take per second.


It can get far more complicated than this, a step is just simply progressing in the environment.


And then finally when it comes to action spaces, you have discrete or continuous actions.


You can think of discrete like classifications, cart-pole for example has a discrete action space, you go left or you go right.


It's nothing in between, there's no like continuous values here, it's left or right, not somewhat left, somewhat right and so on, discrete.


Then you have continuous, you can think of continuous like regression, it's a range of nearly infinite possibilities.


The bipedal walker environment is a continuous action space as is much of robotics.


Because you're going to set the servo torque anywhere in a range between let's say in the example of bipedal walker, negative one and positive one or a servo position, it's not always going to be torque.


Typically a continuous environment is harder to learn but sometimes this is a requirement.


So again in robotics this tends to be more of a continuous type problem even though servos are actually discrete with something like 32,000 positions, that's far too many discrete positions to pass into your neural network.


So instead we treat this as a continuous problem.


But those can be converted into, often a continuous problem anyways, can be converted in various ways to a discrete problem.


And more on that later and why we want to do that, more on that later too.


So assuming you all have GIM and Stable Baselines 3 installed and PyTorch obviously, let's go ahead and run through some of the basics of GIM and these environments and how things are working.


So we'll import GIM and then env will be GIM.make and in this case we're going to make lunar lander v2.


As an aside I've been coding almost entirely with Copilot and it's so painful writing things out completely.


I've been super spoiled.


So every time you have an environment before you can actually step in that environment you have to do an env.reset.


So we'll go ahead and call that and then we can print sample action and we will do env.action space.sample.


So we're just going to see an example of what is an example action for this environment and then we can also print observation space shape.


And we will say env.observation space.shape.


So painful typing things out.


And we'll do a sample observation.


So sample observation and we'll make this env.observation space.sample.


So once we have those things let's go ahead and finally close this off with an env.close and let's go ahead and run that.


Get rid of this.


Cool.


So sample action you can see here is a zero.


If we run it again possibly we'll get something else.


Right.


So just an example there.


So your observations space shape is just a flat eight values and as you can see these are the eight values.


Honestly I don't even know what they are but it's just as you get into reinforcement learning this is what I mean by I think the harder part of reinforcement learning is less about the algorithm.


Yes you can tweak some of the hyperparameters in your algorithm but in general it's going to come down to your environment, your reward mechanism and your observations.


So like what you feed in to those algorithms those algorithms especially by now are pretty robust algorithms.


I mean they still take a very long time to train in that regard but if it's going to learn it will learn and changing those hyperparameters won't do too much.


At least you can screw it up but from the baseline values the default values I have found you know I can eke out maybe an extra 5-10 percent performance possibly by tweaking hyperparameters but for the most part the actual gains that you're going to make are going to come from algorithm choice but also like reward mechanism and yeah your observation space.


So anyways keep that in mind.


So yeah.


So these are just some examples of kind of what we're working with.


So in this case our observation is just yes this flat vector and then we have an action space of certain values that we can pass.


So and this is hopefully the fact that you know we don't even need to really know what these values are to kind of show you how robust I suppose stable baselines 3 really is because you'll see as we go to train a model on this environment.


It's not that stable baselines 3 just knows the lunar lander environment.


It's that that's what's so nice about stable baselines 3 is if you decide later you want to change something about your observation or your reward mechanism or anything like that you just make the change in your environment.


You don't have to touch your algorithm at all.


And that's what makes it.


It just it makes it really really helps.


It saves so much time.


So let's go ahead and see an example of our environment in action.


So if we go ahead and do four step in range 200 we'll do env dot render and then we will env dot step.


And in this case we're just going to pass a random action so env dot action space dot sample.


So we'll just do a random action here.


So we'll go ahead and run that.


And here we have our agent.


So just ran for 200 steps actually really good.


Good example let's do another one.


It's not the worst random usually it's like flies around.


Not bad.


OK.


So yeah.


So so we have that also that's really small.


Let me.


That looks better.


So hopefully we'll be able to actually see what's going on.


So kill that.


Hopefully that'll go away.


Cool.


So so that's just a random lunar lander.


Now what we one thing we can do is every time you take a step you actually are getting back an observation a reward whether or not the environment is done and any extra information that you care about you might care about.


So let's go ahead and print reward and then I'll change this by minute to 200 just so I could modify the recording.


So we'll run that and then we can kind of see here.


So these are just our rewards essentially and since it eventually crashed we got a negative 100.


But here you can see the rewards or whatever they are.


And again it's almost irrelevant as long as we can assume the environment's been been made reasonably well which we know the open I environments have when you go to create your own environment.


It's completely a different story but we'll get there.


So so now what we want to do is train an algorithm on this this environment.


So what are some of the things that we know about lunar lander.


Well the only thing we really need to know to know which algorithms we can work with is is it discrete or continuous.


And in some respects that can actually get even more complicated especially with like there's discrete and there's multi discrete.


So what does that mean.


So if you have discrete that would mean what we have here which is just discrete.


Your actions is you have one action choice per step to make whereas you could have multi discrete like.


So in the case of like a robot for example you're actually trying to get servo positions for let's say in my example with doing the the little robot dog there's eight individual servos all that need predictions to be made.


So that's a multi discrete problem for example if you're using a discrete algorithm anyways.


So so to know what we can use we can come back.


So to know what we can use we can go to essentially the home page here go to oral algorithms and they have a lovely little table.


It almost reminds me of scikit learn initially which which algorithm to use it was like this like walkway path thing.


I have many memories of like revisiting that over and over for various projects.


But anyways we can see here what things can we work with.


So in this case so box you can think of box as this is your continuous discrete is like the classifications and then multi discrete whether or not that's supported and then multi binary.


Essentially that would that's going to be the same as multi discrete.


It's just binary.


So so yeah so now what we're going to do is we can check out let's just do the first one on the list because we're just going to say OK we want to do discrete.


Cool. Which ones of the which ones of these which of these algorithms can support discrete.


And we can see there's there's a huge list that we can choose from.


So we'll just start with A2C to start.


So how do we do this.


So again imagine that this is like it doesn't have to be the lunar lander environment.


This could be your own custom environment which again we will get to soon.


So you have this environment and what you want to do is test one of these algorithms on your environment.


So to do that it's just so simple from stable baselines three we're going to import and we will first bring in A2C all caps.


So from here what we want to decide is essentially once you have your environment we'll go ahead and I'll reset there and then we'll come down here and then all we want to do is first define our model.


So we'll say our model is equal to in this in this case it's A2C.


This is a MLP policy so MLP police policy and we pass and and then verbose will set to one for and we'll we'll cover like tensor board later and stuff.


But for now this is pretty much I think I've changed a lot of algorithms and I'm pretty sure I haven't touched this.


Obviously multilayer perceptron.


I don't even know what the other policies are but obviously there are other ones.


I think that's just so it doesn't have to be fully connected.


I don't know maybe maybe that in a distant video but so far through my personal uses of stable baselines three I have not.


I don't think I've ever changed this line at all.


I think I'm just editing what algorithm we're using.


Anyways now to train model dot learn boom done.


So here we specify how many total times. How did you know that that's what I wanted already.


Interesting double check copilot total time steps 10,000 will set for now.


So this will train for 10,000 time steps.


It should go very very quick and then we will instead do.


I kind of want to change this code for 200 steps or do we want to just let it go.


I'll just let it go.


We'll say well the correct way to you would normally do this will just I'll just write it out episodes will set to 10 and then we're going to say so for up in range episodes episodes.


What do we want to do well rather than four step in range we're just going to say.


First we'll start with OBS equals and reset.


So other than step this is the reset method also returns an observation but nothing else there's no reward because you just reset the algorithm there's no info to pass or anything like that.


So it just will return an observation.


So we will grab that observation we're going to set done to be false and then we will just simply while not done.


We will do this.


I'm no longer going to print the reward I don't want to see it also stable baselines three will give us a little output of information so no need.


So I'll go ahead and save that and I think we can display it in here.


I think that'll be fine.


So let me make sure I did everything somewhat decently.


I guess we'll find out.


Go ahead and run it.


So first we'll start training we're already a thousand in thousand.


Let's make this a little bigger so we can actually see it.


So this is the length of the episodes of how long is the agent surviving sometimes that's a very useful metric to know and then episode reward being this is just telling us what is our typical average reward per episode at this time step.


There's other information here like frames per second iteration all this other stuff sometimes this will be important to you sometimes not.


So after ten thousand steps we can see this is our agent and actually given what I showed you before of the random agent this doesn't look too far off through an image but I assure you the random agent is worse than what we initially saw.


But as you can see it's just it's not very good.


So so the next thing that we could do is increase the time steps so we can go for a hundred thousand steps instead of just ten thousand steps.


And if you're wondering about saving and loading all that that is the subject of the very next video but for now let's train this for a hundred thousand steps.


OK. Moment of truth.


Here we go.


Honestly that's not looking.


Not very good at least the the episode reward you know average episode reward did come down.


But but that's still not that great.


That's that's not the best that I've ever seen.


Right. So yeah.


Going up.


Well it actually interestingly.


So in this case episode.


And then we come down here.


It almost looks like it got even worse over time.


So about I don't have the full history here unfortunately we will get to TensorBoard logging soon the next tutorial.


But for now as you can see it was decent at around seventy ish thousand steps and then eventually just kind of went bad.


So this is why in the very next tutorial is going to be saving and loading models and saving as you train models because this sort of thing is not uncommon.


But for just to show you now you've got A2C which is a you know not as robust algorithm.


We also have PPO.


So literally to try an entirely different reinforcement learning algorithm you just import it and then we change it right here.


And that's it.


Now we use PPO.


So again we'll do the exact same thing.


I'll just go ahead and run this and we'll run this again for one hundred thousand steps and we'll see what kind of difference do we get.


So we ended here at negative forty eight.


But at some point we did actually get a reward of like fifty.


Anyway I'll be back when this is done.


OK.


So we have our results.


And in this case again it's almost it's very hard to compare these two algorithms because they both look quite bad after only a hundred thousand steps.


I mean it's better than random but it is comical how my random example worked pretty well and then the trained examples did it.


In reality we are training these models for far too few steps so we need to train them longer.


But as you can see even like when we did the first ten thousand steps well when we decided OK let's try a hundred thousand steps.


Well we had to retrain the model all over again.


So what really we want to do is be able to save load and then track performance and all that.


So that's what we're going to be doing in the next tutorial.


In this case the only standout difference one is it trained maybe 10 percent faster than the A2C model did.


And then also it's staying alive a lot longer on average.


But again like I said neither of these models even remotely close to their actual training time.


So to put that in perspective you know generally you're going to do millions of steps.


So you know 10 to like a hundred million steps.


I can imagine there are some environments where you might even do more than that.


But anyways yeah so by this point hopefully you have a pretty general understanding of Stable Baselines 3,


reinforcement learning, kind of what are the main you know terminology like observation and actions and steps and all that kind of stuff.


And in the coming tutorials we'll hopefully show you how to actually start using this to train a model and kind of track its progress over time.


And then eventually how to apply this to a truly a custom environment, a custom problem.


And really get into the nuts and bolts of what I think makes reinforcement learning hard which is actually engineering your own environment entirely.


It has really nothing to do with these algorithms.


If you're interested in learning more about neural networks and how they work then you might want to check out Neural Networks From Scratch,


a book by myself and Daniel Kukiewa.


Inside of which you'll learn how to code neurons, activation functions, how to calculate loss, do optimization and back propagation,


and of course apply everything that you've just learned to a problem.


The book itself is in full color for graphs and code syntax highlighting.


Everything's on high quality pages with full font and a comfortable line spacing to make reading easy.


Finally, every concept including the math and coding is broken down into the truly atomic parts and then put back together step by step


to give a full understanding of neural networks and how they work including all of the math behind them.


By the end of the book you will have created your own neural network framework from scratch.


This is the book that I wish existed when I was learning deep learning and I'm extremely happy to be able to offer it to you.


If this sounds interesting to you then you can head over to nnfs.io to learn more and buy yourself a copy.


So anyways, that's all for now, thanks for watching and I will see you guys in another video.


Discovering Symbolic Models from Deep Learning with Inductive Biases (Paper Explained)

Hi there, today we're looking at discovering symbolic models from deep learning with inductive


biases by Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David


Spergel and Shirley Ho.


So this paper on a high level, it uses graph neural networks to fit a data set of observations


of a physical system.


And then it uses symbolic regression in order to parse out equations, symbolic equations


from the graph neural network.


And the symbolic equations that will are found such they describe the physical system.


And they do find they do recover some known equations, and they do find a new one in the


field of cosmology.


So we'll go through how they do it, what these two steps are, and why this might work better


than previous approaches.


So yeah, join me.


If you like content like this, as always, feel free to share it.


Subscribe if you haven't, if you want more content like this, and tell me what you think


in the comments.


Alright, so they claim, we develop a general approach to distill symbolic representation


of a learned deep model by introducing strong inductive biases.


And this, it doesn't really, it doesn't really say a whole lot.


But I think the the abstract doesn't say a whole lot.


So let me give you an example.


If you have three different, let's say planets, or stars, right, this is a, this three, three


body problem is a unsolved problem, I think still.


So if you have these three stars, and you just let the simulation run, they have gravity,


they attract each other, right, so they are going to move around somehow.


So this one's going to move here, this one's going to move like this, this one's going


to move like this, and then it turns around, and this one turns around, and so on.


So there is a fairly complex motions in already three different things that are somehow in


a physical system together.


This is a bigger problem than just stars.


So you have these systems, for example, when these are atoms, and there is like an electromagnetic


force between them or the strong force.


There can be these can be things where springs are attached to them, and so on.


So our goal is to derive equations that govern this behavior, right?


In the case of gravity, we know that these objects sort of pull on each other with the


with the force proportional to something like the mass of the first times the mass of the


second divided by the radius that they are apart squared.


Something like this times like this gravitational constant.


That's we know the equation that governs these interactions, we don't know the symbolic solution


to the whole problem, but we know the equation that governs the interaction, right?


Now, imagine if we didn't know the equation, what do we have to do?


Well, what did Newton do?


Ultimately, he sat down and just came up with an equation that seemed okay to him, and then


found out that the equation actually does predict very accurately the how the things


move.


So we're going to try to replicate that process in a in an AI system, the process of coming


up with an equation that governs this behavior.


So what we have is a data set, as I said, we let this stuff run.


So we let it run for one time step, and then this is here, maybe this is here, and this


is here, okay.


And then we let it run for the next time step, this goes here, this goes here, this goes


here, and so on.


So that will give us basically it will give us frame by frame, how the system evolves


frame by frame.


And that will give us a data set.


So this right here, if we let it run, and maybe we restarted a couple of times with


different initializations, we let it run, we get a data set.


So now we have a data set, right?


So our goal is to be to take that data set and come up with an equation like m one m


two divided by r squared, that governs this behavior.


Now previous approaches have resorted to symbolic regression, I think they they call this.


And that is basically, it's pretty simple.


Namely, what you do is you simply provide the system with a bunch of options, you tell


it, I have a list, and the list can include the mass of the first, it can include the


mass of the second and can include the x and the y position of the things it can include


the delta x and delta y, which basically means the speed of the objects, it can include any


constant a and b that you want, it can include the symbols plus minus division, multiplication,


square, maybe exponential function, and so on.


So you give it a bunch of options of what it could potentially use in an equation.


And then you simply let it make equations, and you see how well these equations describe


the data set.


And the way you do that is, you can do it naively by just searching and trying out,


or you can be a little bit smarter about it and use like evolutionary methods.


So you start with like some equations like this, you're gonna just okay, I'm gonna x


plus delta x minus a squared, okay, you see how that how that describes the data set,


you'll find not very well.


And then you go on and you say, okay, maybe I'll make a like a small mutation, I mutate


this to a minus and so on.


And if you do this with an entire population, as is common in these evolutionary methods,


you'll end up with something better at the end.


Now this works until a point.


So whenever the the space of things to explore, like this one here gets larger, and it doesn't


have to be super large to already exhaust the capabilities of these methods.


So these methods are very limited in the space they can search, and have proven not really


effective so far for this type of problem.


This paper right here goes a different route.


It uses graph neural networks in order to describe the data set.


So in between this step of collecting a data set and making the equation, it fits another


step.


So it says in between here, we fit another step and that other step is going to be we


have a graph neural network and you don't know yet you don't have to know yet what that


exactly is.


But it's like a type of neural network, and we're going to have that neural network learn


the data set.


Now as you know from neural networks, they can't do symbolic regression, they can't give


you an equation, they can simply predict numbers.


So what the network will do is it will simply predict like the motions or the accelerations,


whatever you're interested in, it will predict those things as numbers, not as equations,


as just you can plug in this situation right here, and it will tell you how the things


will move.


Okay, neural networks are pretty good at that.


And once you have a graph neural network that can describe the system in a numeric fashion,


then you parse out the equations from this graph neural network.


And we're going to go over why that is going to be much, much easier than parsing out the


equations directly from the physical system.


It's going to be because you engineer the graph neural network in a way that makes it


very congruent with physical reality, that makes it very adapt to parse out equations


like this, that makes the job of this evolutionary method much easier.


All right, so that's the that's basically the two step process here.


First step is to numerically regress a neural network to describe the system.


And then second step is going to be from that neural network parse out the equations.


So we have to talk about graph neural networks.


So here you see the entire process as they describe it.


So they have this data set right here of observations of these physical systems, right?


This is like, it's like, you know, any data set that you have in machine learning.


They predicted the dynamics, which means in a numeric fashion, with a graph neural network.


And then from the graph neural network, they extract the symbolic equation, as you can


see, right here.


And this here is going to be the equation that they figure out that was previously unknown.


They even said unknown dark matter over density equation.


Cool.


We haven't really done this on this channel so far.


And I'm not the like a big expert on graph neural networks.


But in they come in all shapes and forms.


In this particular paper, they use what they call a type of interaction network that's


called a graph network.


So graph network is something different than graph neural network.


I think graph network is a type of graph neural network.


And specifically here, they use a network that so a graph neural network has these things


called vertices, and then it has edges and edges connect vertices like in a graph.


Now we're going to build this graph neural network such that the number of vertices is


exactly equal to the number of particles in our system.


So in this paper, they consider systems with, I believe, four or eight particles.


That's already a lot for if you want to derive equations and things.


But of course, the physical world is made of many more particles.


In any case, they consider four, let's say four particles right here.


So what they're going to do, they're going to build a graph neural network that has four


vertices, one for each of the particles.


And in a graph neural network, every vertex can have properties.


So the properties of each vertex here are going to be the properties of that particle.


That means the x coordinate, for example, the y coordinate, and we're going to let's


say we're in two dimensions, right?


It's a two dimensional problem, the x coordinate, the y coordinate, the delta x, the delta y,


the mass, the I don't know, what else?


What else can we put here?


There's a lot of stuff that we can put here the charge, right?


So all of these things are properties of the vertex.


Then the other component of a graph are of course, the edges.


So the edges connect each two of all of the so each edge connects two vertices like this.


And then this particular type of graph network, we're going to consider graphs where all the


particles are connected to all the other particles like this.


So it's not like a sparse it's not a sparse graph, except I think in the cosmology example


here you can see that always there is a node that's connected to all its neighbors.


But in the Newtonian dynamics graph networks, you can see right here, everything is connected


to every thing like this.


Okay.


And why, why does that represent a physical system really well?


So the reason is going to be the following.


What we're going to try to do is we're going to try to say that in physical systems, if


I want to, for example, consider this node up here, and consider how it is pulled by


gravity by the other nodes, it's going to be pulled in this direction a little bit because


of this particle right here, it's going to be pulled in this direction a little bit because


of that one, and in this direction because of that one.


Now note that these three things are independent.


So if I want to describe the total force of gravity, I can do so as a sum over i equals


one to one, two, three of the force that the particle i pulls.


So if this is this is j right here, how i pulls on j, right, this is an independent


sum across all of the all of the neighbors of that particle.


Now you might say, wait a minute, that's it's not independent.


Because it's it's being, you know, it's not being strictly pulled in this direction is


also pulled in this direction.


Yes, but the with independent, we mean that the force, this force right here is only dependent


on this particle.


And the force diagonally is only dependent on that particle, there is no part of the


particle up here that modulates this force right here, right.


So you can calculate the total force as an independent sum across the individual forces.


And that's the simplification here.


And that's a part they claim, why so current approaches that directly try to go about finding


an equation using evolutionary methods from the data set itself, don't really work because


the space is just too high of equations.


But this right here, this is a massive constraint.


And it's lucky, first of all, that physical systems, they say, most physical systems actually


obey that constraint, most physical systems can be described as an independent sum over


contributions of interactions between just two things, right.


So we simply can sum over interactions between two things.


And that's, and that's way simpler than considering everything at once.


And second of all, it's lucky because these graph networks describe exactly this.


So each edge in the graph network is coincidentally connecting two things, right, and not more.


So the edges, they don't know about each other.


No edge knows about the other edge, they only consider whatever particles are at their respective


ends.


And that is exactly the same as this physical constraint on the physical system.


And that's why the graph networks are so adapted or are so useful in describing these systems.


So how does a graph network like this do anything basically.


So for that, you have to consider the task.


If we want to describe a system like this, a task in that if you frame it in a machine


learning way, could be, I'm going to give you these particles right here.


Okay, here, it's five, five particles, I'm going to give you for each one, I'm going


to give you all its features like the x, the y, the speed currently, and the mass.


And you're going to tell me what the acceleration is in the next frame.


Okay.


So like this, like this, like this.


Okay, considering all you know, all the interactions between the particles, just tell me, where


does it go in the very next timeframe?


That that sounds like a machine learning problem, right?


And the graph neural network can be made to predict this.


So what we want is for each vertex here, an output of a number or a vector, the acceleration.


So we want to compute an output for each vertex, how do we do this?


In a graph neural network, there are three or in this particular type, there are three


steps, we said each vertex, and we're just going to do it for one vertex, let's say the


one on the bottom right.


Let's say each vertex has these properties like these x, y, and so on.


So first, what we do is we go over the edges.


So for each edge in parallel and independent from each other, let's consider this edge


right here, what we'll do is we take the nodes that are attached to it.


And we combine their features, and we combine them.


So x, y, this also has x, y.


So we want to combine these two, we want to compute the edge right here.


Now, in a physical system, what does the edge represent?


The edge represents the force between the two particles, right?


And that's a fairly complex equation.


It's not like we can just add the features or something like this.


So the edge here already needs to compute some sort of nonlinear, complicated function.


And we know how to compute nonlinear complicated functions with neural networks.


We're in deep learning right here.


So the edge here is going to compute what's called this edge function.


And this edge function takes in two vertices, v1 and v2 right here, if this is maybe this


is v2, this is v1.


It takes in the features, these features of the two vertices, and it will compute a so


called edge message.


I think they call this ek for the edge k.


It will compute an edge message.


And this is supposed to represent the force that pulls between these two particles.


And we're going to approximate this function right here using a neural network.


Since we don't know the equation yet, right, we assume we don't know the gravitational


equation, but we can learn it right because we have data.


So we take this and we simply make it into a neural network.


So the features go in here both we can you know, concatenate them.


And then out comes this edge message.


Now this edge message here is simply going to be a vector, a numerical vector, describing


some intermediate hidden state, right, that is going to describe the force.


But for now, it's just describing intermediate hidden state.


So we do this for each edge.


So each edge is going to be maybe this is e1, this is e2, e3, e4, each edge in parallel


is going to aggregate information of its endpoints into that edge.


And then that's step one.


So step one, compute the edge messages.


Step two is going to be to compute the vertex messages or the vertex outputs.


So we said we're not actually interested in the edges, we're interested that each vertex


ends up with an acceleration with an output.


So how are we going to do this?


So consider again our graph.


If we want to compute the output for this node right here, what we'll do is we'll simply


aggregate all of the edges, all of the edge messages that connect to that vertex.


So we've computed previously the edge messages by integrating the information from all of


the attached endpoints.


Now we're going to go backwards and distribute the information from the edges back to the


vertices that are attached.


And you can see already by this two-step process, it's kind of a message passing process if


you've ever studied graphical models.


This means that in the two-step process, this vertex here aggregates information from the


other vertices via these edges.


So in this case, this vertex here is going to take in all the edge messages right here,


and it is going to aggregate all these edge messages in a function that computes the acceleration.


So that our estimate of the acceleration is going to be a function, let's call that new,


of the edges that are attached to it.


So e1, e2, and e3.


And here is where we're going to make our next physical assumption, namely the one we


said before, that the way that these edges, the way that they influence the vertex is


going to be in a form of an independent sum.


So the simplification means that this function should somehow be not of the edges, but of


the sum of the edges, right, sum of ei.


Okay, so this sum here, this is the simplification that we make to make it in accordance with


the physical system.


With this graph network, we could do any sort of complicated thing right here, right?


We could put a transformer on these things and like compute 12 layers of interaction


effects between these edges.


We're not going to do that.


We're simply going to sum them up and then come up and then, you know, run those through


a function.


So we'll sum them up.


And of course, this function right here is still going to be a complex function because


just because you sum up the forces, you don't have the acceleration yet.


So you know, as you know, that force is mass times acceleration, that means acceleration


is equal to force divided by mass.


So this here is going to be this sum over the edges, I guess, yes.


So you still need to divide it by force.


And technically, you still can do much more complicated things right here.


We still, we only say that the edges should only come in in form of a sum.


So of course, we're going to say that this function right here, since it can be any complicated


function of its input, it should also be a neural network.


So we're going to take that sum of the edge messages, and we're going to put that into


a second neural network.


And then out comes our estimate of the acceleration.


And now that we can use together from the data set, we know the true acceleration, right?


Since we have a data set, we have the observations and the labels, the labels are the true accelerations


of that system that we observed.


And we can compute a loss function right here.


And if you followed so far, everything we've done so far is differentiable.


So from this loss function that compares the output of the neural network for that vertex


to the true acceleration that we observed in the data set, we can back propagate through


this neural network that computes the vertex function, we can back prop through the sum


here to the edge messages.


And we can back prop through the edge messages to that neural network that computed the edge


messages from those features.


So everything is differentiable.


By having that loss at the end, we can train this neural network end to end to from the


observation right here, predict the numerical acceleration of the system right here.


Okay.


So this was a fairly lengthy way.


But it's important that you kind of understand what's happening.


So you build the graph network according to the physical system.


In the graph network, there are two kinds of things.


First, there are deterministic things like we're always going to aggregate in a sum.


And then there are things that you learn.


Namely, there are two neural networks, the first one computes the edge messages from


the features of the vertices, and the second one computes the output of each vertex, according


to the to the sum of the edge messages that are attached to that vertex.


Now you can say, wait a minute, there are more than just two neural networks, like each


edge here has a neural network technically, right, this edge has a neural network, this


edge has a neural network, and each vertex has a neural network.


But in this case, these neural networks are shared.


So the neural network that computes the edge message for that edge is the same as the neural


network that computes the edge message for any of the edges.


You can think of it like weight sharing, or you can think that it is actually the same


neural network, it's equivalent, right.


And the same for the vertices, there's only one neural network that in the same fashion


computes the output for each vertex.


Of course, the incoming edge messages are going to be different.


And that's why you have different outputs.


But the neural network itself is the same.


Okay, so we have a system that can describe this data set of physical observations really


well.


It's this graph neural network.


Right, so we train this end to end.


And here's a little bit of a of an analogy where they say, this, this is how you can


and now like analogize the neural network with a physical system.


So, what are the analogies here, the nodes in the graph network correspond to the particles


in Newtonian mechanics, pairs of nodes correspond to two interacting particles, the edge model


is the force between two particles, then the pooling operation, which is the summing up


of the edge messages, right that we found so important as a simplification.


This is the sum into the net force that is really given in the physical system.


So independent sum of so sorry, sum of independent forces without interaction effects, then concatenate


with node, I guess this I left this out.


But whenever you compute, whenever you compute the vertex properties, right here, I guess,


what you want to do is not only input the edge messages, but you know, each vertex has


these features that we said, and these could also be fairly important.


It's like, you technically have that information in the edge messages, because it started out


from these, but you can also just input that again into this neural network together with


the edge properties.


And that will just make its job a bit easier since for example, right here, we have to


divide by the mass in this function.


And it's just easier if you provide that mass as a as the property.


So that's a little detail I left out before.


So that you concatenate the edge mess, the aggregated edge messages with the node, then


you compute the node model, which in this case is the computation is simply the you


take this sum right here, and you divide it by the mass.


And then optionally, you can update the nodes, which is compute the next time step, which


we don't do right here, because we simply want to output the acceleration, I guess,


I mean, it should be equivalent to output the next time step and then compare, compare


with the data set what the next time step was.


In any case, you have to have some kind of loss function.


And here you can see all the black squares right here are going to be neural networks.


So now, we have learned a graph network that can describe a system, how do we make this


into an equation.


And again, here, our, our physical reality comes in that these of the like the independence


assumptions of these realities comes in, because in physics, you know, the, the acceleration


here is going to be a function of the sum and so on, what we need to do is we don't


need to develop an equation for the entire system, right?


What we need to do is simply we need to develop an equation for each vertex.


So each vertex, we need to have an equation acceleration equals something, and that something


should include some of the edges, and then the edges again should be something, right?


So we've technically, as we had two neural networks, we technically need two symbolic


equations, one that represents that first neural network that computes the edge functions,


and one that represents that second neural network that aggregates the sum of the edge


functions or that computes the output from the sum of the edge functions.


And that, you know, it's an exact correspondence.


So what we need to do is we need to take that first neural network up here, and do symbolic


regression on that, and the second neural network, do symbolic regression on that.


So what does it mean to do symbolic regression, it basically means that we want to find the


symbolic equation that describes the neural network the best, right?


And we do that in the exact same fashion as we started right here.


So we give it a bunch of these options, and then we let the system describe the neural


network as best as possible.


The way we do that, again, is we try out equations, and if they just if they get a low error,


right, we so we let the neural network run on the data set, and we let this run on the


data set, if it outputs the same thing, it describes the neural network well, and we


can iterate that until we find a good equation.


So the difference here is that we don't need to find an equation that covers the whole


system, we just need to find two equations, one governing the edge model, and one governing


the vertex model.


And that's way, way easier than the whole system.


And by finding those two equations, we and our given our physical assumptions, we can


now find the equation to the whole system by simply composing them.


Alright, so that's the entire system.


I believe I've told you the entire paper right here, without actually going into into the


paper, let's just skim the paper a bit to see that they actually tell us the same thing.


So yeah, so they graph networks, they say are ideal candidate for our approach due to


their inductive biases shared by many physics problems, a their equivalent under particle


permutations, B, they are differentiable end to end and can be trained efficiently using


gradient descent.


And C, they make use of three separate and interpretable internal functions, the edge,


the node and the global model.


Now the global model here isn't really used in the cases we're going to look at.


So it's just going to be two different neural networks, which are targets for the symbolic


regression graph networks can also be embedded with additional symmetries as in 2324.


But we don't implement these, okay, and then they say symbolic regression.


So they use this Eureka package to perform symbolic regressions and fit compact closed


form analytical expressions to these neural networks.


Eureka works by using a genetic algorithm to combine algebraic expression stochastically.


The technique is analogous to natural selection, where the fitness of each expression is defined


in terms of simplicity and accuracy.


The operations considered in the fitting process are plus minus times dot dot dot dot if as


well as real constants.


Alright, so if we look at the examples, they have three different examples.


First of all, they have Newtonian dynamics, which is, for example, this gravitational


force we looked at, they have Hamiltonian dynamics, which describes the same systems,


but in a different way in terms of the Hamiltonian.


And I don't want to go into this too much, because I think that the Newtonian dynamics


already demonstrate really well what the system can do.


And then they have dark matter halos for cosmology, which is a problem where you have universe


simulators and you try to predict where the dark matter is, depending on where other dark


matter is.


And they actually that's where they find a new unknown equation.


Okay, here is the system in a nutshell.


Now there is this is the path that you know, you have the data set.


You learn a graph network, and then you get out an equation, right.


But in between there is now you can put even more constraints to make the network really


learn a physical equation.


So we said you're going to compute these edge functions right here.


And the output of the edge functions is going to be this edge message, which is just going


to be a vector of some sort.


And that vector can be pretty large, you know, this is a hidden dimension that you can choose


as an implementer, all you need to make sure is that the output of the vertex is the same


dimension as you know what your output should be everything internal, you can choose.


Now, we know that, for example, in a 2d system, the the actual informational content of that


edge message should be two dimensional, right?


If this really describes the force in two dimensions, it, it should be two dimensional.


There's really no reason why it should have a higher dimension since all the relevant


information can be described in two dimensions.


So one thing you can do is you can simply say, all right, I will choose the hidden dimension


to be two.


And therefore, I will force my neural network to just use two dimensions.


This however, they noticed doesn't work super well, I think it works, but not that well,


they call this the bottleneck model.


And the reason why it doesn't work super well is that if you have like this constraint of


neural networks, they don't turn to, they don't tend to learn very well.


And that's what they hypothesize in the paper as well, they don't tend to really come, you


know, be good friends with the fact that they only have two floating point numbers to learn


anything.


And this is probably more a property of the optimization procedure than the problem itself.


It's property of, you know, us training graph neural networks with SGD.


So what they do instead is they put an L1 penalty on these edge messages.


So they say we apply L1 regularization.


And what that will do is that will induce sparsity in whatever you apply it to.


So L1 regularization simply means that you constrain, so the edge message, if you take


the absolute value in each entry, and the sum of that, that should be small.


So you can just add this to the loss function.


And that will induce sparsity in these edge messages.


And so now the network still has these whatever 100 latent dimensions, but it is encouraged


to use as few as possible.


That means it can use a lot during the beginning when it's really benefits from the lot of


dimensions when it learns the system.


But then as it gets better and better, it might shift a lot of the information into


very, very few dimensions.


Okay, so once we do, if we do that, we can then run a check, right?


If it is really the case that this graph network has learned the physical dynamics of the


system, then the top, we can simply look at the top two dimensions, and we start by largest


standard deviation.


So whichever two dimensions are the least sparse, right?


Have the largest standard deviation, we can look at those two and we say, well, even though


we didn't constrain the model, those two should describe our force pretty well.


And since in Newtonian dynamics, we know what the force is.


So this is, we know what the force is, we can simply check whether or not that holds,


we can check whether we can read out the force from these two components.


And here, it's made such that you, you can't guarantee that the force is, you know, this


force right here is actually.


So there are many ways to state a physical equation, because there are many symmetries


in physics.


And we cannot really make the neural network describe the equation exactly as humans would,


because there are infinite amount of equivalent formulations.


But they're, in this case, they're all covered by rotations of each other.


And that means in these graphs, if you have these message elements right here, and the


linear combination of forces right here, a linear relationship means basically that the


information is there, whereas a nonlinear relationship would mean that these numbers


don't really encode the force as is.


And here, you can pretty clearly see that the linear relationship is given.