because we only need we only know this but we want to know this so how do we calculate


iMAML: Meta-Learning with Implicit Gradients (Paper Explained)

this thing here this Jacobian that's the question right how do we derive the the algorithmic


procedure this thing rear here and the paper goes on to say well okay yeah so we need to


do this this is the entire gradient descent optimization procedure so we must compute


this thing right here and they just throw it in your face it's this boom shagga the


bomb this thing here done let's go on no so you can you can see it's it's basically putting


this just right here but we kind of want to explore where that comes from so the fact


you can see here you can derive this gradient as a close form expression of the inverse


of a matrix that contains this is the identity matrix contains somehow this lambda factor


that we saw before and it contains this Hessian matrix of the training loss right so this


is this end gradient that we can calculate easily and the second derivative of that is


the Hessian which is basically the curvature in the landscape of that loss but nowhere


in this thing is the is the SGD procedure showing up even though this thing here is


the SGD procedure and that's pretty impressive and we're going to look at how that comes


about so so where where do we start first let's take this let's take this G right here


this G function right and let's calculate the derivative with respect to the in these


parameters right here so let's go for this end gradient what's this end gradient going


to be so we'll derive the G with respect to the these parameters all right so this is


a sum this first thing is pretty easy it's going to be the gradient of this loss function


loss function is a scalar right so we can this we can count this is the something one


backward prop through the network the second thing we can also do pretty easily this is


an L2 norm all right we know how to derive a square so the two comes down and the the


this will simply result in the this vector right here so it's going to be lambda times


phi minus theta okay now this was relatively easy now imagine what happens when in this


particular thing we have one additional information namely that F the inside of F F we will


always optimize to the end we will always optimize this to its minimum right this star


denotes that the inner optimization procedure will always go to the minimum of that function


so what do we know about the minimum of a function we know that its gradient at that


particular point is zero right this is the this is an important part so now we can restructure


so if we take one to the right I might actually use black here because it's kind of burning


my eyes we can isolate this part right here so we say the phi is equal to first of all


let's um let's take this to the right side so we'll have this gradient right here and


I'm just going to write L phi let's keep the hat alive we'd have to divide this by lambda


right and then bring over the theta so we have a closed we have an expression that says


at the optimum the parameters phi the inner parameters are going to be given by this expression


now that's pretty pretty cool but we know also that these parameters aren't just you


know parameters per se they depend on these parameters right the end parameters depend


on the initial parameters because the we use the initial parameters to initialize these


end parameters so these are actually a function of the initial parameters so what we can do


is we can derive this using red again let's use blue we can derive this thing by the initial


parameters right how do the end parameters relate to the initial parameters now this


is our basic question all along but we now have an exact expression for the end parameters


which we didn't have before before we just knew they came about by SGD so important to


say this only works at the optimum right this is at the optimum that this relation counts


not anywhere and the paper is abusing this quite a bit right here so what does this do


if we derive this thing here with respect to theta it's simply giving us the identity


matrix right this is now our our Jacobian that appears here it's simply giving us this


then this one divided by lambda is going to stay and now it gets a bit tricky because


these things right here of course are also a function of theta so essentially this means


we this thing right here is a gradient of a function of another function of theta so


we can apply the chain rule again since this is already the first derivative it will give


us the second derivative with respect to the loss function right here of with whatever


goes into the loss function so that times the inner derivative now the inner derivative


is simply how to derive again the Phi by the theta okay now I'm this okay yes so you can


see first of all interesting that the expression here or the expression that we are looking


to find appears in the expression itself right since since these parameters appear over here


as a function argument as well we'll get basically this expression here twice but we can reformulate


that and find that the the this term this Jacobian is basically this here inverted so


the matrix we're looking for is sorry the inverse Jacobian the matrix we're looking


for is given by this quantity right here the identity matrix minus this Hessian term right


here okay and this is exactly what you see appearing here this is exactly that so the


derivative we're looking for sorry this is actually the Jacobian not the inverse that's


my dad what you're looking for is given by this expression now my eraser got stuck hello


cool so that's how that appears you see it's the same thing if I had done everything correctly


and so this this you do by simply shipping this to the other side which will make it


the the inverse right so you divide you basically divide both sides by this and then you get


this as an inverse now why does this work again I want to stress why did we get this


identity here why were we able to express get a closed form solution to the for the


inner thing or sorry for the end parameters in terms of the beginning parameters that


doesn't have SGD first reason because we optimized to the end to the optimum that's why we got


the equals zero right here second reason because we have this regularizer you see this directly


comes from heat from this expression right here if we wouldn't have this regularizer


then we could not make this expression we could not get Phi as a standalone quantity


here and therefore this derivation wouldn't work now why is this important because if


you look back into your drawing what you're basically doing is you are imposing a quadratic


regularizer around this initial point right here and that creates this very strong connection


between the end gradient and the initial gradient so now when you're optimizing when you have


a training loss of the inner task and maybe the training loss looks something like it


looks something like like this right here so SGD will it would go right to the very


inner point right here if you just let SGD run it would go there but now since you have


this regularizer SGD needs to find a trade-off point between the two so what it will do is


it will probably go somewhere and stop somewhere here so it will now have two forces pulling


on it the first force will be this quantity right here and the second force will be pulling


it back towards this and you can pretty much count so now SGD cannot just go to any point


right here it cannot not go to any isoline these are not equal anymore maybe mainly it


will go to the one point that points into the direction of this quadratic right here


so since it's a quadratic we have closed form formulas for relating one gradient on the


quadratic namely the one out here with the gradient back here so we can express this


Jacobian in closed form because this is a quadratic because we have this regularizer


because you have these basically two forces pulling on this point in opposite direction


one pointing towards the training loss and one pointing towards the inside of the quadratic


so that's why this method works okay I can recommend Ferran Hussar's blog post he has


some very nice animations of why this basically restricts where gradient descent can go I


can I can link to it in the description it's pretty cool to see I don't have it open right


now so what does that give us the implicit model agnostic meta learning I mammal this


is what this paper suggests while not converged do sample a batch bench of bunch of tasks


right for each task compute the meta gradient G average these gradients to get a gradient


for the outer parameters and then do gradient descent on the outer parameters pretty easy


how do you do this how do you do this in plate implicit meta gradient this is this procedure


right here so what you're going to do is met the parameters theta you initialize your parameters


with the theta by the way they don't need to be initializations they can be actually


any sort of hyper parameters that this algorithm takes any parameter ization of this algorithm


will do fine I just always said initial parameters such that it gets easier but it can be any


sort of hyper parameters of the inner task obtain task parameters using iterative optimization


solver such that the inner parameters are close to the optimum of that algorithm so


they actually extend this also in theory not so that you do not don't have to optimize


the inner objective really to the optimum but you can be like Delta close to it that's


pretty useful and that's in the part of the paper that we won't go over because this video


would be like super long but I invite you to read it if you're interested then you compute


the partial outer level gradient so this this would be your partial gradient your V would


be this gradient at the end right the gradient at the end of the optimization procedure with


respect to your validation data sets this is one back prop now we need to relate that


end gradient to the beginning and that's and we do that by multiplying it with this matrix


inverted right here now because obtaining the entire matrix this is the Hessian matrix


and inverted is very memory and computation intensive because if you have D parameters


in your neural network this is going to be a D by D matrix so if you have five million


parameters this is going to be 25 million million size matrix is just not possible and


that's why this paper extends this method to a second degree of approximation namely


you don't have to compute the exact inverse you just have to compute something that is


very close to the inverse times this inner group this final gradient and a good method


to do this is this conjugate gradient method and that method is able to to basically use


the fact that you can compute Hessian vector products without having to compute the Hessian


as a matrix this you can also do with a sort of modified back propagation algorithm also


won't go in here but see you use iterative solver for example conjugate gradient along


with reverse mode differentiation to compute Hessian vector products to compute GI so GI


is going to be the final gradient pulled back through this matrix right here to give you


the beginning gradient this meta gradient okay so two approximations here first approximation


you don't actually have to solve to the very end you can solve it Delta close and second


approximation you don't actually have to compute the inverse of that final gradient sorry compute


the multiplication of the final gradient with the inverse of this matrix right here you


can also find something that's a delta prime close to that and they have a bunch of theory


of that this still works they compare this of course to the other algorithms they observe


that their algorithm uses substantial substantially less memory and substantially less memory


and substantially less compute time once you go up to a number of inner gradient steps


and it works better than this first order mammal so this first order mammal was our


kind of initial guess of how we could do this this tends to perform very poorly as you can


see there there oh you cannot you can't actually see that here that their method is better


but their method is better and uses less time because you have this inner conjugate gradient


optimizer sorry this is the this is the outer optimizer okay so this is the error plot of


how well are these methods are able to approximate the true gradient so if you could compute


this true outer gradient you know that we did with mammal but we optimized to the end


how close are you getting of course the problem with this method right here is that you do


these approximations to you do these approximations and those could hurt you but the problem with


mammal is that you're back propagating through the optimization procedure and that means


the nonlinear errors could sort of accumulate and as you can see here even though both might


eventually you know get to the to the zero error if you give them enough inner gradient


steps especially at the low inner gradient step regime the implicit mammal is much better


than mammal now I've just said the errors accumulate but the effect probably here is


that the fact that with mammal you don't actually do good inner enough inner steps to reach


a good enough optimum of the inner tasks so these inner gradient of the tasks their gradients


when they're still very not optimized and therefore they are a very bad estimate for


your outer gradient then when you do more gradient steps so that actually hurts you


more which is also a bit surprising to me and then at the end you see this conjugate


gradient steps this is when you approximate this matrix inverse if you just do two steps


then at some point that error dominates but if you do more steps you can reach a much


lower error and ten steps isn't that much for an algorithm like this as you can see


here the ten steps your computation time will still in in the regime of many gradient steps


will still be lower than the original mammal and then they actually test this thing and


of course they're the best at pretty much everything I don't want to go into the exact


details here I invite you to check out the paper for that check out if you're interested


Reformer: The Efficient Transformer

Hi there! Today we'll look at Reformer, the efficient transformer by Nikita Kitaev,


Reformer: The Efficient Transformer

Lukas Kaiser and Anselm Levskaya. This is a paper that tries to reduce the extreme


Reformer: The Efficient Transformer

resource requirements of the transformer model. Now if you haven't seen the transformer model


Reformer: The Efficient Transformer

before, that's this thing, I suggest you go watch for example my video on it Attention is all you


Reformer: The Efficient Transformer

need it's called where the transformer is introduced. The most famous transformer is


Reformer: The Efficient Transformer

called BERT and you can also look that up. I've made a video about this. So what's the issue here?


Reformer: The Efficient Transformer

If you remember transformers, they need a lot of memory and why? That's because they compute in


Reformer: The Efficient Transformer

each layer they compute these attention things. Let's recap shortly. In a transformer you propagate


Reformer: The Efficient Transformer

information layer by layer. So you have layer here with some signal and then the next layer that you


Reformer: The Efficient Transformer

try to propagate the signal. Now what you do you assign queries to each of the next layers. So each


Reformer: The Efficient Transformer

of the next layer has queries and queries are just vectors. This is a vector, this is a vector,


Reformer: The Efficient Transformer

this is a vector and so on. So basically the next layer has the ability to ask the last layer what


Reformer: The Efficient Transformer

it wants. This is a kind of an intrinsic property of attention and as I said I explained this in


Reformer: The Efficient Transformer

detail in the video. Attention is all you need. Basically these are what's called queries, Q. And


Reformer: The Efficient Transformer

then this layer is exposing what are called keys and keys again are vectors. So vector, vector,


Reformer: The Efficient Transformer

vector, vector and so on. So keys are vectors and the way that the information is propagated to the


Reformer: The Efficient Transformer

next layer is whenever we consider for example this node here. Let's make that yellow. When we


Reformer: The Efficient Transformer

consider this node here it is going to look in the last layer which keys match my key the most.


Reformer: The Efficient Transformer

And in this case it would probably be this key and this key. They match the key the most. And here


Reformer: The Efficient Transformer

we look at the inner product so the angle between the vectors. And then information is aggregated


Reformer: The Efficient Transformer

by simply having a weighted average of the values. So information is coming in here and here.


Reformer: The Efficient Transformer

Actually information is coming into all the nodes but since only these keys match the information


Reformer: The Efficient Transformer

will be propagated like this to this unit. We could do this for another unit. For example this


Reformer: The Efficient Transformer

unit right here. What's the value of this unit? Well we have to look at the key here. Which key


Reformer: The Efficient Transformer

is it going to be matched to? It's probably going to be matched to this key right here. And probably


Reformer: The Efficient Transformer

no other key really. Maybe this key a little bit. So the information of that node in the next layer


Reformer: The Efficient Transformer

will be whatever information is coming in here routed there and a little bit of this information.


Reformer: The Efficient Transformer

So this is kind of a it's not a hard it's called soft attention. So there's a little bit of


Reformer: The Efficient Transformer

information going everywhere but the majority of the information is coming from the nodes where


Reformer: The Efficient Transformer

the keys match. So these are queries. These are keys. And technically these things coming in here


Reformer: The Efficient Transformer

are called values. But imagine the values simply as the information to be propagated and the queries


Reformer: The Efficient Transformer

and the keys are responsible for routing that information to the next layer. All of these things


Reformer: The Efficient Transformer

are learned. So the queries, the keys and the values. Now what's the problem? The problem is


Reformer: The Efficient Transformer

between the queries and the keys. As you can see what you have to do is you have to match every


Reformer: The Efficient Transformer

single query with every single key in order to find out where information goes. So this becomes


Reformer: The Efficient Transformer

order of what if you have d keys and d queries order of d squared operations that you have to


Reformer: The Efficient Transformer

do. And of course d squares values that you have to compute. And since these are all vectors,


Reformer: The Efficient Transformer

of course there is d will not only be the number of keys, but then again, this is multiplied. So


Reformer: The Efficient Transformer

there is an inner multiplication with the dimensionality, let's call that capital D of the


Reformer: The Efficient Transformer

no, sorry, that's not an inner multiplication. Let's just remain at this. So d squared inner


Reformer: The Efficient Transformer

products between vectors of capital D dimensions. So it's not an easy thing for resources to do. You


Reformer: The Efficient Transformer

need a lot of resources to hold all of this in memory at the same time and to compute all of


Reformer: The Efficient Transformer

these things. The reformer aims to solve this problem. So this giant space problem that the


Reformer: The Efficient Transformer

transformers have space memory, also computational problem to a lesser degree. Mostly it's a memory


Reformer: The Efficient Transformer

issue. All right. So what is happening here? And you see here that this product between two matrices


Reformer: The Efficient Transformer

clearly gives you this kind of squared, squared thing. So what's happening in the reformer to do


Reformer: The Efficient Transformer

this? The trick is, the trick is if we go back to this drawing, the trick is to create what's called


Reformer: The Efficient Transformer

a hashing scheme or buckets. And in creating buckets, what you want to do is you want to group


Reformer: The Efficient Transformer

similar things together. So let's say we create four buckets, bucket one, bucket two, bucket three,


Reformer: The Efficient Transformer

bucket four, right, and each bucket we label and bucket one we label with the up direction is with


Reformer: The Efficient Transformer

right direction with a down direction, the left direction as vectors. And now we simply put each


Reformer: The Efficient Transformer

of the things into the bucket where it belongs most. So let's for example, this vector here, it


Reformer: The Efficient Transformer

goes here. Sorry, that is like, absolutely not the right place. It goes probably here, right? This


Reformer: The Efficient Transformer

vector here, probably this one goes here, right? And so on. So you'll end up each of these assigning


Reformer: The Efficient Transformer

a bucket. So this, these, these all go into into that bucket. Let's continue. Actually, let's also


Reformer: The Efficient Transformer

put the keys in the same buckets. So also the keys, this key here probably goes to this bucket. This


Reformer: The Efficient Transformer

key here probably goes to this bucket. Let's say this key here probably goes to the bucket over here.


Reformer: The Efficient Transformer

You already see so before, right before we cared about this particular query, and this particular


Reformer: The Efficient Transformer

key, we just looked and we said those two will probably route information to each other because


Reformer: The Efficient Transformer

they're similar. And now you can see they both ended up in the same bucket. So the idea is to


Reformer: The Efficient Transformer

create a scheme where you throw these things into buckets such that if two vectors are similar, they


Reformer: The Efficient Transformer

will end up in the same bucket with high probability. So you'll only have to really compare


Reformer: The Efficient Transformer

things within the same bucket and not across all of these d squared elements. That's the idea. And


Reformer: The Efficient Transformer

the technique here is called locality sensitive hashing. So locality sensitive hashing. And short,


Reformer: The Efficient Transformer

this is called LSH. The idea is the following. If you have two vectors, v one and v two, and they


Reformer: The Efficient Transformer

have and you have a distance measure, distance measure d, d is a distance. What you want is if


Reformer: The Efficient Transformer

the distance between v one and v two is small, I'm getting confused with small, then you want them


Reformer: The Efficient Transformer

in the same bucket. And if the distance is large, then you want them in a different bucket. Different


Reformer: The Efficient Transformer

buckets. You know what I mean? With high probability. So all of these things where you say,


Reformer: The Efficient Transformer

you want them in the same bucket with probability p, with probability p, with high probability p,


Reformer: The Efficient Transformer

and here you want them in different buckets with high probability, or you want them in the same


Reformer: The Efficient Transformer

bucket with low probability. That's an equivalent form of stating. This is all formalized. And I can


Reformer: The Efficient Transformer

direct you to the Wikipedia page of that is pretty good. It gives a concise definition. Here you can


Reformer: The Efficient Transformer

see that. And it gives a number of examples. So one example I'd like to give here for locality


Reformer: The Efficient Transformer

sensitive hashing is, of course, the scheme of bucketing will all depend on what your distance


Reformer: The Efficient Transformer

measure is. If you consider the distance measure simply to be the jaccard distance. So let's say


Reformer: The Efficient Transformer

we have two vectors 0101. And here we have 101101. And here it's 0001. Alright, so maybe you can see


Reformer: The Efficient Transformer

the first two vectors here are much more close together than the last vector. Now, in terms of


Reformer: The Efficient Transformer

bit differences, right, one scheme to do locality sensitive hashing is to simply subsample bits. So


Reformer: The Efficient Transformer

in this case, this is a slightly constructed example, we will just subsample the first two


Reformer: The Efficient Transformer

bits, and then construct the buckets according to these bit values. So if since we sample two


Reformer: The Efficient Transformer

bits, we have four buckets, right, here's 00. Here is 01. Here is 10. And here is 11. That's the


Reformer: The Efficient Transformer

concept of locality sensitive hashing, you have these buckets, right. And then you can say, all


Reformer: The Efficient Transformer

right, this vector has 10 goes into this, this goes into this, and then that goes into sorry, the


Reformer: The Efficient Transformer

01 bucket, right. And you end up with what you have, you have the two close vectors in the same


Reformer: The Efficient Transformer

bucket, and the two far apart vectors in that bucket. Of course, that doesn't always work, you


Reformer: The Efficient Transformer

know, you can be unlucky in subsampling, but that's kind of trade off you'll have to, to go


Reformer: The Efficient Transformer

for right, if things that are close together happen with it's a low probability. But if they happen to


Reformer: The Efficient Transformer

end up in the different buckets, then basically you lose the fact that they are close to each


Reformer: The Efficient Transformer

other. And that's the trade off. The kind of locality sensitive hashing they use in the reformer


Reformer: The Efficient Transformer

now is what are called random projections. So let's say you have a bunch of vectors. And that's


Reformer: The Efficient Transformer

really what we care about, right? You have a bunch of vectors. And what you want, you want the keys


Reformer: The Efficient Transformer

and queries. So you have a bunch of vectors like this. And you want to create buckets such that


Reformer: The Efficient Transformer

vectors that are close together will end up in the same bucket and vectors that are far apart, you


Reformer: The Efficient Transformer

will end up in the in different buckets. The way a cool way to do is and this is in the cosine


Reformer: The Efficient Transformer

distance. So we care about the angle between vectors, right? A cool way to do this is to use


Reformer: The Efficient Transformer

random plane projections. And this the the cool thing about it is, it works for the cosine distance,


Reformer: The Efficient Transformer

and you can basically choose how many buckets you create, right? Let's say we want to create four


Reformer: The Efficient Transformer

buckets here again, what we need is two hyperplanes. And what we'll do is so here is the


Reformer: The Efficient Transformer

origin, we'll simply create two hyperplanes through the origin at random. So I'm going to draw a


Reformer: The Efficient Transformer

random hyperplane here, like this, and then a second random hyperplane, like this. Right, so you


Reformer: The Efficient Transformer

would agree those are pretty random hyperplanes as much as I can be a random generator. And then


Reformer: The Efficient Transformer

we'll simply label. So this will label hyperplane one, this will label hyperplane two. Right. And


Reformer: The Efficient Transformer

now we simply assign each vector bits according to the width in which on which side of the hyperplane


Reformer: The Efficient Transformer

they lie. So let's call this here the plus side. And this year, the minus side, or even Yeah, let's


Reformer: The Efficient Transformer

call this the plus and the minus. And here also, we call this the plus side, and this the minus


Reformer: The Efficient Transformer

side. So this vector here is its signs are plus, plus, right, because it's on the plus side of


Reformer: The Efficient Transformer

both of hyperplanes, this vector plus plus, this one plus plus, this one here is called it's on


Reformer: The Efficient Transformer

the negative side of plan two, but on the positive side of plane one, so it's plus minus this one


Reformer: The Efficient Transformer

here, minus minus, minus minus, minus minus. And these are your buckets. So you would group these


Reformer: The Efficient Transformer

vectors together, because they have they have the same science, you would group that vector, you


Reformer: The Efficient Transformer

would group these vectors together. The combination of this with attention, since in attention,


Reformer: The Efficient Transformer

you've seen attention uses a softmax. And the softmax is dominated usually by the largest


Reformer: The Efficient Transformer

elements. And since we compute inner products, it means that the softmax thing is dominated by the


Reformer: The Efficient Transformer

by vectors that have small inner products. So basically, you don't have to look at all of these


Reformer: The Efficient Transformer

D squared vectors, if you can find the ones that are have the closest distance, you can pretty much


Reformer: The Efficient Transformer

ignore the others. And LSH allows you to do this. So build buckets of vectors with the with similar


Reformer: The Efficient Transformer

directions, then you only have to care about these vectors comparing them to each other. So that's


Reformer: The Efficient Transformer

not a lot of vectors, generally. And that's how you save a lot of work. So you will only have to


Reformer: The Efficient Transformer

care about these three vectors, if your key vector, for example, is right here, you'll only have to


Reformer: The Efficient Transformer

care about this things in the same bucket. And you can ignore all of that rest of the space. Of


Reformer: The Efficient Transformer

course, the more hyperplanes you have, the more buckets you'll have the less vectors you'll have


Reformer: The Efficient Transformer

in the same bucket. That's the general idea, I find this explanation to be a bit easy, you can


Reformer: The Efficient Transformer

equivalently explain it by doing these kind of random rotations in the space, you can think about


Reformer: The Efficient Transformer

how that will end up actually being the exact same thing as what I just explained. I just like that


Reformer: The Efficient Transformer

my explanation better, I think. Alright, so the way they use this, they have an illustration right


Reformer: The Efficient Transformer

here is the following. So they have these keys, right sequence of queries and keys. So they do


Reformer: The Efficient Transformer

equivalent queries and keys, which is a thing you can do in transformers. Don't worry too much about


Reformer: The Efficient Transformer

it, whether they're different or not. But then they do this LSH bucketing. And here the color of


Reformer: The Efficient Transformer

the cell is just the bucket, the LSH bucket, which will end up, then they sort that, right? As


Reformer: The Efficient Transformer

you can see, and now they do an additional thing, which is called the chunk. That, as you can see,


Reformer: The Efficient Transformer

there are not the same amount of vectors in each bucket. And that is sometimes a problem. Because


Reformer: The Efficient Transformer

even though you've, you've reduced the memory, the memory is still the memory requirements are still


Reformer: The Efficient Transformer

dominated by the largest bucket, right by whatever bucket has the most number of vectors, that will


Reformer: The Efficient Transformer

pretty much be your memory requirement. Because now you don't you don't have to if this is D, right,


Reformer: The Efficient Transformer

you have to compute all the D squared things anymore. But you'll only have to compute this


Reformer: The Efficient Transformer

quantity, let's call that B. So max, the maximum bucket size, but that could still be large, right?


Reformer: The Efficient Transformer

If you look at a distribution, it's probably going to be something like this, right, where most


Reformer: The Efficient Transformer

buckets have kind of a standard number of vectors, but some buckets will have a lot of vectors. And


Reformer: The Efficient Transformer

that's sorry, some few buckets will have a lot of vectors, and your memory requirement is still


Reformer: The Efficient Transformer

dominated by this. So they do an additional thing, which is called chunking, which means they


Reformer: The Efficient Transformer

actually take fixed size chunks here fixed size, here, they always take four. And they say, All


Reformer: The Efficient Transformer

right, these are our chunks. And we will only compute attention within the chunks. Right? So it


Reformer: The Efficient Transformer

could be that they're there, there's the same bucket is actually split between chunks. And that's


Reformer: The Efficient Transformer

why they do an additional thing is that you can attend two things in a different chunk, right,


Reformer: The Efficient Transformer

right, right here, right, you can attend two things in your neighboring chunks. So you're


Reformer: The Efficient Transformer

restricted to either your own chunk or your neighboring chunk. Note that there aren't any,


Reformer: The Efficient Transformer

any arrows going over here. So you can, you can attend to have this diagram here, which things


Reformer: The Efficient Transformer

you can attend to, you can attend to yourself or attend to your neighboring thing, but not to any


Reformer: The Efficient Transformer

other thing or the other way around, right. So that's basically the the concept of saving memory.


Reformer: The Efficient Transformer

Now your memory requirements are if we call this quantity now, we call the other one B, let's call


Reformer: The Efficient Transformer

this the chunk size C, right, your memory requirements are pretty much C squared, plus


Reformer: The Efficient Transformer

whatever this unidirectional so not this isn't squared plus probably, or C, something like this.


Reformer: The Efficient Transformer

So you bring your memory requirements down quite a bit. Now, that's the general idea here. The


Reformer: The Efficient Transformer

problem they face again is so they face another problem where they say, hold on, I can find it


Reformer: The Efficient Transformer

right here. They say, hold on, we do have actually another problem. And that is that these


Reformer: The Efficient Transformer

transformers have to back propagate. So you'll have to forward propagate these things. And now


Reformer: The Efficient Transformer

we've kind of solved this D squared computation issue. But what you'll have to do is if you go


Reformer: The Efficient Transformer

from layer to layer, right, layer, layer, layer, layer, what you have to do is if you propagate


Reformer: The Efficient Transformer

information forward, you still have to back propagate. And in order to back propagate,


Reformer: The Efficient Transformer

usually, usually, you'll have to remember all of these activations, right. So these activations,


Reformer: The Efficient Transformer

these activations, in order to do backprop, it is often the case that you actually have to


Reformer: The Efficient Transformer

remember the activation of because in each forward propagation in each layer here, you might lose


Reformer: The Efficient Transformer

some information. Imagine you have a, you have to a layer that maps these two dimensional vectors,


Reformer: The Efficient Transformer

both to so here, actually, let's make this blue maps these three vectors to the following


Reformer: The Efficient Transformer

configuration. So a layer maps these vectors to this, this, and this. So it maps two things to one


Reformer: The Efficient Transformer

thing, which which you know, can be if you in a in a linear layer, it can decide to map it to a


Reformer: The Efficient Transformer

lower dimensional subspace. So you could actually try decide to map it to in fact, two points.


Reformer: The Efficient Transformer

Right, this is also a possibility could do dimension reduction. So because all of this, in


Reformer: The Efficient Transformer

order to do backprop, you actually have to remember these things in order to do proper


Reformer: The Efficient Transformer

backprop. This is a problem, again, for the transformer, because all these activations, even


Reformer: The Efficient Transformer

though we've gotten rid of the D squared computation, they will have to be remembered.


Reformer: The Efficient Transformer

And that takes a lot of memory. The way to solve this is actually to do invertible layers. What


Reformer: The Efficient Transformer

that means is that if I propagate information forward, forward, forward, forward, I can figure


Reformer: The Efficient Transformer

out what the information here was simply by looking at the backprop activations. And this is


Reformer: The Efficient Transformer

is this happens if the layer is invertible. So if this function here is invertible, so if f here


Reformer: The Efficient Transformer

technically is invertible, so I can actually write down the inverse of f and that is is defined.


Reformer: The Efficient Transformer

This, of course, is a pretty big restriction. And the way they achieve it, I like to go to the blog


Reformer: The Efficient Transformer

here. The way they achieve it is they do what's called an idea from reversible networks, where


Reformer: The Efficient Transformer

they always have two sets of activations. That's what you see here, x one and x two. And in each


Reformer: The Efficient Transformer

layer, only one of them is updated in a residual fashion. So you can see here, layer one updates x


Reformer: The Efficient Transformer

two, but x one remains the same and goes to y one, right. And then in the next layer, layer two only


Reformer: The Efficient Transformer

updates y at y one in order to construct z one, but y two remains the same to be z two. And then


Reformer: The Efficient Transformer

you can revert the layers, you can basically figure out what the activations were from the


Reformer: The Efficient Transformer

backprop signal. Now that's extremely good if you want to save memory, but of course, it restricts


Reformer: The Efficient Transformer

clearly you have to be restricted to this kind of architecture or similar. This idea actually isn't


Reformer: The Efficient Transformer

new. This has been used many times in things like normalizing flows and I want to highlight this


Reformer: The Efficient Transformer

paper actually want to highlight specific I chose this paper because they have these nice dark


Reformer: The Efficient Transformer

diagrams where they show exactly this. You see they have two sets x one and x two that in forward


Reformer: The Efficient Transformer

propagation, they only update one of them. And then in backward in what's called inverse


Reformer: The Efficient Transformer

propagation, they can figure out what those were. And they couple these in exactly the same way.


Reformer: The Efficient Transformer

Like here, this drawing might be even more similar, where they alternate between updating the two


Reformer: The Efficient Transformer

activations. So you can think of this as a way to simply make the function that you're representing


Reformer: The Efficient Transformer

with the neural network, invertible, that is a giant constraint on your architecture. But these


Reformer: The Efficient Transformer

methods here, these normalizing flow methods use that so they can actually define an invertible


Reformer: The Efficient Transformer

layer because they need the Jacobian inverse in order to compute their normalizing flow. So you


Reformer: The Efficient Transformer

see, that's why they originally did it. And I'm sure that that's not a like a new idea. Or it's


Reformer: The Efficient Transformer

particularly new again. Strangely, I haven't found the the any of the flows literature cited, they


Reformer: The Efficient Transformer

do cite the reversible residual net paper that they probably got the idea from. Alright, so with


Reformer: The Efficient Transformer

these two things, now you can save the giant computation, right. And you can also not store the


Reformer: The Efficient Transformer

forward activations. So they say they can take now giant, giant, giant input sizes, you may remember


Reformer: The Efficient Transformer

transformers like BERT. So BERT, it can use something like 512 tokens in its input sequence.


Reformer: The Efficient Transformer

That means the sequence that you can look at with BERT at a time is 512 long, and not a bit longer,


Reformer: The Efficient Transformer

right? There have been some extensions to that. For example, I believe in XL net. So XL net has


Reformer: The Efficient Transformer

pushed this to something like C times 512, where C is a smallish constant, that that where you can


Reformer: The Efficient Transformer

kind of carry over information between sequences. But this thing here, as you can see, they


Reformer: The Efficient Transformer

calculate, it could take up something like 64,000 tokens. And that would use in total 16 gigabytes


Reformer: The Efficient Transformer

of memory, which is available on a high end GPU. So this is a giant. This is a giant step forward


Reformer: The Efficient Transformer

in in producing transformers that can actually take large models. And here you see the memory and


Reformer: The Efficient Transformer

time complexity, you can look at these things yourself. But be you can see maybe here that these


Reformer: The Efficient Transformer

squares here in the from the original transformer, they now vanish from this. And all of these


Reformer: The Efficient Transformer

constants are a lot of these constants are actually smaller, for example, that chunk size is in there


Reformer: The Efficient Transformer

instead of kind of the entire sequence length. So that's basically the the paper, they show that I


Reformer: The Efficient Transformer

can actually input those long sequences, they can apply this to images, you see there's image net,


Reformer: The Efficient Transformer

pixel by pixel, which is a lot of pixels and would have been absolutely unthinkable with one of the


Reformer: The Efficient Transformer

original transformers. And with that, I invite you to check out the paper and the blog post and I'll


Reformer: The Efficient Transformer


A critical analysis of self-supervision, or what we can learn from a single image (Paper Explained)

Alright, today we'll look at a critical analysis of self-supervision or what we can learn from


a single image by Yuki M. Asano, Christian Ruprecht and Andrea Vidaldi.


This paper, I really was excited when I saw this paper because the outset is so cool and


the experiments have a very promising.


So we'll take a look.


Basically, we show that three different and representative methods by GAN, RotNet and


DeepCluster.


So this is self-supervision techniques can learn the first few layers of a convolutional


network from a single image as well as using millions of images and manual labels provided


that strong data augmentation is used.


However, for deeper layers, the gap with manual supervision cannot be closed even if millions


of unlabeled images are used for training.


We conclude that first, the weights of the early layers of deep networks contain limited


information about statistics of natural images.


That second, such low-level statistics can be learned through self-supervision just as


well as through strong supervision.


And that three, the low-level statistics can be captured via synthetic transformations


instead of using large image data set.


Okay, so I, as I said, I was kind of excited when I saw this and what they're talking about


is self-supervision.


So really quickly, for those who don't know, self-supervision is a technique where if you


have images but no labels and you would still like to learn something from them, you can


do, you can do a pre-training step for your network.


So basically you have your neural network, F, and what you would like to do is you would


like F of X to be close to Y for X and Y to in your training data set.


But you can, if you have a much larger data set of just X's, right here you have pairs


of X and Y, of just X, that are sort of similar to the X in your, in your labeled data set,


you can do, you can kind of get, get your network used to the data by doing this self-supervision.


So what you would do is you would sort of come up with your own labels for the data


points.


And one way is this, we'll take just this RotNet as an example.


So what you'll do is you'll input an image, so maybe an image of the number three in handwritten


digit recognition, but you flip it, you flip it to its side, right?


So it's the number three right here.


And then you ask the network to come up with an answer.


Is it upright?


Is it flipped to the right?


Is it flipped to the left?


Or is it turned on its head?


Which one is it?


And of course you, you who did the transformation know the correct label, right?


So this is how you come up with sort of fake labels for your data.


And this works surprisingly well, and what this paper basically says is that you do not


actually need this giant database here.


It's actually sufficient sometimes if you have one single image where you do this on.


Now the claim is a bit of a cheat, I have to say, but we'll go into that.


And further they say, okay, it's enough to have one single images to learn the features


of the lower layers of the neural network because they usually tend to extract low level


features that, you know, you can learn from a single image, but the higher level features,


you really need the supervised data set.


It's not enough to have these self supervision techniques.


And even if you have many, many, many self supervision samples, so if you actually do


have this giant data set, it still doesn't help you for the higher layers.


Just calls into question this, you have a giant data set of unlabeled things notion


that is often presented, including by me.


Okay, so what do they do?


They do, they take either single images or just very few images.


So they also have a setting where they take 10 images for the single image, they hand


select it.


So they hand select the following three images.


So this image right here, they select because it's very crowded, there's a lot going on,


right?


There's people, there's objects, there is lighting, and so on.


There's, you know, houses, these lines, you know, there's perspective.


So that's why they select image A. Image B here, they also experiment with image B is


a drawn image.


As you can see, there's also lots of stuff going on, but they basically want to, to research


how does a natural image compared to a artificial image.


And then in C, they have this is sort of a control, because there's lots of parts here,


where there's not much going on compared to here.


And most of the image, there's lots of stuff going on.


And this image on the image number C, or letter C, has large areas where there's nothing going


on.


Okay, so these are the single images.


Now why I say it's a bit of a cheat is that these images are actually super large.


They for, so for ImageNet and for C410, they, you know, this might be one of the samples


of the C410 or ImageNet classifier.


Now, of course, C410 is a lot smaller than ImageNet, but still for ImageNet, these are,


you know, there are many pictures here, not just one.


So to say this is from a single image, it's technically true.


But then if you split it into multiple images, it's technically not true.


So it would have been fun to see what actually happens with a single image when you downscale


it.


But okay, so how do they investigate this?


They have this five layer neural network right here.


So this five convolutional, I'm going to guess there's five convolutional layer.


After each convolutional layer, I think there is some batch norm and relu, and then to the


next convolutional layer, and so on.


And then at the end, maybe there is, or there's also some pooling here, max pool.


At the end, there is going to be some linear classifier that classifies it into either


a 10 or 100 or 1000 classes, whatever you want.


The way to investigate this is through linear probes, so called.


Now linear probes are somewhat of a technique to inspect how much each of the layers learn.


So if we again, draw our network right here, and this is the input x, right, so you have


the hidden representation one, hidden representation two, hidden representation three, and here


you output it to the y hat.


And that you compare with the y from your data set, right?


The x is from your data set and the y is from your data set.


Now linear probe wants to investigate how useful a given hidden representation is to


classify the output.


So what a linear probe would do is it would take the hidden representation here and learn


one single linear classifier to classify that hidden representation to come up with a y


hat given h1 or something like this.


So the important part here is that this is linear, right?


This is a linear classifier.


You do nothing more.


You take the representation and instead of this entire giant neural network on top of


it, you simply build a linear classifier and you can build these linear probes from any


layer right here.


You can build a linear classifier on top of this, on top of this, and then you basically


look how good is your linear classifier when trained on that hidden representation that


the network comes up with.


And that's how you estimate how much information about the target or let's say how, how optimal


the representation already is because at the end of the network, right, you do have a linear


classifier.


So at some point this representation must go into a form where it is now linearly well


classifiable.


And the assumption is basically that these layers of the neural network successively


make a representation that is more and more linearly classifiable.


And that is a strong assumption, right?


And this paper here uses linear probes exclusively.


And that is a bit worrisome to me because I have my troubles with this linear probe


approach because this strong assumption that more linearly classifiable is better, it just


rubs me in the wrong way, right?


We know that the information content can never increase from layer to layer about the label.


So any information about the label that is in H1, sorry, that is in H2 must also have


been present in H1.


So technically if we just built the correct classifier, we could predict from H1 just


as well as from H2, right?


Because we're actually doing it.


We're building the neural network.


But the fact that we cannot predict linearly as well using H1, so the fact that this classifier


here performs worse than this classifier here because H1 is a less optimal representation


in a linear sense, it's just, and the fact that, I mean, yes, but then to use that and


to estimate, oh, how useful is a representation, you're equating usefulness with linearly classifiable.


And that I disagree.


A representation can be extremely useful if the following layers manage to do something


useful with it.


And that can be something completely different or it can even be the opposite of the linear


classifiability, right?


So this is kind of my problem here and they don't do a good work of convincing me otherwise.


So they don't employ different techniques other than these linear probes.


In any case, when they do this linear probe, you can see right here that the percent supervised


performance, so that's how much percent of supervised performance do you get.


Oh, single image self-supervision.


We show that several self-supervision methods can be used to train the first few layers


of a deep neural networks using a single training image such as this image A, B, or even C provided


that sufficient data augmentation is used.


So what they do here is they use this self-supervision, then they take the signal from the convolutional


layer one, the hidden representation, that's H1 right here.


They train this linear probe on it and they see how well does it perform after, and this


is after the network has been self-supervised with RotNet, for example.


And then they compare that to the linear probe at layer one of the supervised network, right?


So you take the supervised network and you do the same thing.


And there they find, okay, this RotNet and all the other techniques, they perform very


well and especially if you only do a single image, they perform better as you can see


right here.


I mean, if I interpret this correctly, this one RotNet, one by again, one deep cluster,


these are these top things right here.


And the 100 is the comparison to the supervised performance, right?


So 100 means 100% of the performance of the supervised representation.


This is absolutely crazy to me.


And this, in fact, so let's just interpret it from their perspective, right?


So you also have random.


So if you, I guess if you randomly initialize a network, then with the linear, with training


a linear classifier on the hidden representation one, you could reach something like 60% accuracy,


which is impressive.


But if you do the linear probe at layer two, you reach a lower accuracy.


Now remember, this is lower accuracy compared to the supervised performance, right?


So there are two effects at play here.


The supervised performance is going to go up because the well, if you believe the assumption


that the successive layers make the representation more and more linearly classifiable.


But also, it could be that just at the same time, the self supervised performance, the


performance of the self supervised representation is going down.


So the graph here is sort of, I don't really know how to interpret it, and it really goes


down after that.


That's why they say you can learn the first layers fairly well with self supervision,


even from a single image, but you cannot learn the upper layers.


And they're basically just measuring this using this linear probe method compared to


the supervised performance.


And I would somewhat like to see is that you train, let's say you train a self supervised


network fine, but then you freeze this layer, and then you fine tune the rest of your network


on top of that representation, that would actually give you an estimate of how useful


is that representation if I had an, you know, an all powerful function approximator, which


is a neural network.


And then, of course, you're probably not going to get supervised performance.


And by the way, you'd have to compare that also to supervised with and without pre training


using self supervision.


And then you actually get a good estimate of what how well what kind of a representation


do these things learn?


In this case, all we you know, all we get out of this is this linear probe thing compared


to the supervised representation.


And it just seems a bit uninterpretable, honestly.


And the fact that here you can go beyond 100% you can actually be better than supervised


should already tell you that the linear linear probe thing might not be a good instrument


to might not be such a good instrument, especially in the lower layers, the lower layers will


be most inaccurate with these linear probe measurement.


But that's that's their finding, basically, they can learn the features of the lower layers


as well in terms of this linear probe formulation as the supervised learning.


Again, they never compare this to fine tuning on top of these representation, or compare


it to self supervision plus supervision, which I would really expect.


Alright, so they say they do a lots and lots of data augmentation since of course, they


only have a single image, they basically supercharge data augmentation, and they show that this


helps.


Now, I don't want to actually go into the into the very into the very details of what


they're doing, because they just have different methods of augmentation.


They just have different networks.


But here are the results.


So if this is on on ImageNet, if we use full supervision, we use the entire data set.


And we do these linear probe evaluation, we get a 20% accuracy after layer one, 36 after


layer two, and so on.


This goes up as we go through the layer.


So this kind of gives credence to the hypothesis that these layers sort of make the representation


more linear.


Then they have a bunch of scattering and random networks and k-means pre training, which doesn't


get you a lot like that.


But that's what they compare it to basically the self supervision to just the scattering


transforms and things like that.


But then they get into their methods.


And here we'll look at, for example, this RodNet.


So if you train on just one image, this image A, of course, if you have one image, then


you get this many this this much of the layer one.


Okay, so now that I see this here, they have this column right here, which uses the full


data set.


What I think this is, is the self supervised training using this many images.


So what if you do RodNet self supervision on this many?


It could also be the performance after supervised training, after pre training with this method.


But I think it is the performance after just after self supervision, again, with no fine


tuning on top.


And then evaluating these linear probes.


That's why this number is lower than this number right here.


But astonishingly, after you do it with just one image, you get a higher number.


And if you do it with 1000 images, you get an even higher number.


But if you do it with many more images, you do you, you somehow don't get a higher number.


This all seems a bit, it seems a bit weird, honestly.


Basically means that, okay, it is more important to augment the same thing over and over and


over in different ways than it is to incorporate different images.


I mean, there's ways I can believe that, but I'm not sure.


But you basically see that after a while, the performance compared to the first of all,


to the supervised method.


So yes, if you look, for example, here, up here, drops dramatically.


And even if you have the full, you know, now I'm convinced that this this is just self


supervision using the full data set, even if you have the full data set, but only do


self supervision, your performance still suffers compared to the supervised training.


So that's why they claim they have these two claims, you can learn the first layer representations


fairly well with self supervision, that's comparing this number to this number.


You can do so even from a single image that's comparing this number to this number.


Right.


And noticing that it's almost the same, these two numbers are almost the same.


Actually one is a bit higher.


You can learn that fairly well.


But if you go down the layers, you will basically suffer with your single image and with your


full image, self supervision.


So you need the supervised signal to learn the features of these later layers.


And that's all evaluated with these linear probe things.


Yeah.


So that is their main claims right here.


And they kind of analyze image A and image B. So they come to the conclusion that image


A works much better because it's natural and image B is not working so well.


But this depends on the self supervision used.


And image C still apparently works quite well, even though it has these large areas of nothing,


which all of this is a bit weird, but it's definitely cool to see these results.


Now again, I would like to see something like you freeze these representations and then


you actually train a neural network on top of that and look how that performs.


That would actually be an interesting thing, though.


Maybe they've done this and I'm just unaware right here.


They look at the filters that these methods have learned just from self supervision on


a single image.


And you can see these are the types of filters that we would see using even supervised learning.


If you look at the filters, they turn out to look pretty much like this.


Of course, I can't decide if these particular things are good or bad filters or not.


They do some qualitative analysis and here they have fine tuning.


Okay.


Fine tuning experiments.


The pre trained models first two convolutions are left frozen or replaced by the scattering


transform and the network is retrained using ImageNet training set.


Okay, here we go.


So if you do this fully supervised, you get to a 59.4.


Now okay, this seems very low accuracy, honestly, for even like for ImageNet.


But maybe this is their thing.


But if they do this on top of the on top of the these self supervised methods, they do


get a fairly good.


Okay, they get a fairly good accuracy right here, I would have liked to have this evaluation


right here, be applied in the table above and not these linear probes, they just seem


kind of kind of wonky.


But you can see that it is possible to learn this to learn this using just a single image


to learn the features of the lower layers.


Now how you exactly would, would put this into a training procedure, how you exactly


make use of this during training, if you already know that it's not going to help for the deeper


layers, I'm not so sure, because at least you always have your own data set, right?


So you always have at least that many images that you can self supervise train on.


But it's certainly interesting, interesting results.


And with that, I think I'm going to leave it at that.


And thanks for listening.


I hope you enjoyed this.


Python 3 Programming Tutorial - Global and Local Variables

What is going on everybody welcome to another python 3 basics video in this video. We're now going to be discussing the concept of


global and local


variables so when users start using functions


They quickly get a little confused when it comes to global and local


variables and


They get this you know dreaded. You know variable not defined


Even when they clearly see that it is defined or so they think so these terms of global and local


Corresponds to a variables reach


Within a script or program so global variable is a variable that can be accessed anywhere a local variable


the opposite


Means it can only be accessed within its you know environment or its frame


so a local variable cannot be accessed globally inherently anyways and


a


global


variable


Can be right, but you've got to turn it into a global variable and some people have a


Misunderstanding of what a global variable actually is so let's go ahead and give some examples


So if you if you haven't committed this to memory no problem. I think actually seeing


Helps under helps you to understand so x equals 6 this variable has no parent function


But it's actually not a global variable it just so happens that it's committed to memory


Before any sort of function or anything is called so for example we can say define example


And we can actually say you know z equals 5 print z


And that's totally fine, and we can call example


And as we expect it's going to print 5 so we know that works


but what


Starts giving trouble to people is instead of let's get rid of this and let's say you want to print x


Well we run this function and we see we can print x so we were able to access x


from this function and


What happens is users are able to access


What happens is users naively think that because they were able to access it now they're able to play with this variable


Let's say so print


x plus 5


Okay


So they start playing with it. They see okay?


Well, we can start modifying it and all this cool cool cool


So we have you know access to this but the problem is when they start to do something like this x plus equals 6 and


They go to run it


And you see unbound local error local variable x is referenced before assignment


Yet this makes people just pull out the hair because


First of all they never got there. They didn't get the things printed


So they're freaking out like so they didn't get the you know confirmation here because when you get rid of this it does print


It's only you only get that error whenever


You try to actually do something like add 6 to that variable and so they start pulling out their hair because up till now


They thought they were getting away with doing that but the problem is x is not a global variable


So there's a few ways that you can get around this obviously the easiest thing to do would be to convert x to a global variable


So to do that you type out global space x and what that does is it's notifying Python that hey guys x


Is actually a global variable and so now what we can do is


Print x and then we can even do x plus equals 5 print x


So now we go to run this sure enough 6 11 everything worked. We were able to use our global variable


Now that's it for global variables, but a lot of teachers you're going to find


or maybe you work for a company or something and


the CTO says nobody gets to use global functions


Or I mean global variables don't use them so some people don't like the idea of using global variables


There's actually you know a couple of things you can do to get around this


One one way that people will do it is the following they might say global globe x equals x because we already know we can access x


We just can't modify it so then you could print


Globe x and then we could do globe x plus equals 5 and print


Globe x


And we're able to do that 6 11 so we get the same output that we wanted


But maybe we're not happy with that because when we do this now we've changed variables here


And also globe x is a local variable and so we're not able to access globe x outside of this function


So say we've got multiple functions and we want them to work together


We're not going to be able to do that so what do you do instead?


Well generally what you're going to see is instead you'll get everything up to this point no problem


But instead what we'll have what users will do is they're going to return globe x with example


So when this function is run it's going to return a value


And what this means is it's almost like when you do 5 plus 5 in python


What 5 plus 5 does is it returns 10


And so by that same token what we can do is if you wanted to set a value to 5 plus 5


Or set a 5 plus 5 equal to a variable


What you would do is you would say maybe x equals the function of example


And now print x


Save and run that


We see that we were able to do that


So this is one way that you can actually end up modifying this within the main loop let's say


Without using global variables because again some people really just are very anal about global variables


So hopefully that helps some of you guys out especially with the whole accessing this local variable


But then not being able to modify it


So hopefully it will save you guys from pulling out your hair


This is something that did give me some trouble when I was starting out


So hopefully I saved you guys some of that


So hopefully you guys enjoyed


As always thanks for watching, thanks for all the support and subscriptions


Peer Review is still BROKEN! The NeurIPS 2021 Review Experiment (results are in)

Do you know how hard it is to truly generate random numbers?


I don't mean the random number generator on your phone or anything like this.


That's just algorithm that crunches something, but it's deterministic.


True random numbers are super difficult to generate.


There is even a Wikipedia article about it.


What you need to do is you need to measure some actual physical phenomenon like atmospheric


noise or thermal noise or other things that we have no idea.


They're so chaotic.


We just can't predict them and thus their results are truly, truly random.


Random.org even sells to random number generators for you.


This is big topic humanity has searched far and wide for truly random processes.


But now, ladies and gentlemen, we found it.


The NeurIPS review process is a absolutely truly random phenomenon.


So if you're not aware, a way way time ago in NeurIPS, what was that 2014 the organizers


made a little experiment where they gave certain set of papers that was submitted to the conference


not only to one committee to review, but the two separate committees in order to track


how the committees would agree or disagree.


Now the results right there were quite damning to be honest.


So not only did they not find any sort of correlation between what the reviewers scores


they gave with any sort of future citations, and that's a paper that I've covered in a


video where they look back seven years later at whether or not the reviewers could predict


anything about these papers turns out they cannot.


They also found that the reviewers mostly didn't really agree that much.


So here were these experiments.


Now of the 166 papers, most were rejected by both committees, which most papers to such


a conference are rejected.


So reject is sort of the default answer.


But here, look at that.


If committee one accepted and committee one accepted for 22 plus 21 papers, so for 33


papers, committee to only agreed on half of them.


And likewise, when committee to accepted for the 43 papers, and this is 44 papers.


So for the 44 papers that committee to accepted committee one only agreed again in half of


them.


So this means that if you were to switch committees for the papers, only half of the accepted


papers would be the same papers, half of them would be other papers that had actually been


rejected by the other committee, which is kind of crazy.


But this just shows you how noisy this process really is.


Now it's 2021.


And we've actually repeated this experiment.


So here's a Reddit post by the user, why watch young, that has scraped from open review these


scores and put together some statistics such as this one here that shows the average rating


of the papers versus how many of papers were in a particular bucket, and what ultimately


happened to them.


So we only have full data inside into the accepted papers, and the rejected papers that


have sort of voluntarily agreed to make their reviews public, which most papers that are


rejected don't.


Now the most interesting part here is this one.


This is the repetition of the NeurIPS experiment, you can see at the bottom, the total is almost


300 papers.


And again, these are not all the papers part of the experiment.


These are only the papers that were accepted because we don't know anything about the other


ones.


So the way this worked was the follows.


Papers were given to two separate committees, these two committees reached a decision independently


of each other, and then the maximum of the two decisions was taken as an acceptance criterion.


So if either of the committees accepted the paper to be published, the paper was going


to be published.


So to understand this table, the leftmost column is the final decision, which is the


max of decision one and decision two, not always, but we'll get to that.


Then the second column is the decision of the first committee.


And the third column is the decision of the second committee.


Now these things are ordered.


So it's not the same as in the last paper I've shown you.


So since there's no clear ordering, we simply always put the larger decision on the left


and the second large decision on the right.


So the most interesting part of this is how many papers were accepted by one committee, but rejected by another one.


For that we have to add together all the rows where one of the decision is a reject.


So 174 plus 16 plus nine is I think 199 papers, 199 papers out of the 298 papers that were


accepted had actually been rejected by a second committee.


So to compare, we have to do the following, we'll say that essentially the analogy would


be that 22 and 22 and 21 papers, so 65 papers would be our analogous total number from down here.


Those are the papers that ultimately ended up being accepted because they were accepted


by one of the committees, and then 22 plus 21 papers, so 43 papers would be the amount


of papers that would have been rejected by one of the two committees but ultimately ended


up being accepted because it was accepted by the other one.


So according to this, here we see 43 out of 65 papers only were accepted by one of the


committees.


And here we see that roughly 200 out of 300 papers were only accepted by one of the committees.


In both cases, it's about two thirds of the paper, which means that actually, this is


remarkably consistent.


So in the face of that, and with the explosion of the machine learning community, more papers,


more reviewers, and so on, you could actually say, it's a good thing.


It's actually surprising, this hasn't gotten much worse over the years.


Now that's one way to look at it.


And the other way to look at it is to say this is crap.


Like, come on, this is completely inconsistent.


Not only the accept reject is inconsistent, you see, of the six papers suggested to an


oral by one of the committees, this was never confirmed by another committee.


And how many were suggested for a spotlight by one of the committees?


1620 2941 4444 papers were suggested for a spotlight by one of the committees, yet only


three had actually both committees agreeing.


And again, the same results hold if you were to swap out committees, if you just differently


assign people to papers, half of the papers that are in the conference would be different


half, I don't know how people can still claim that peer review is like this esteemed thing


that is supposed to catch errors and do quality control and yada yada yada, there's something


to be said that if you have a really good paper, the probability that a different committee


also accepts it is pretty high.


And also if you have a really bad paper, the probability that two committees agree on rejecting


it I guess that's even higher.


However, most papers fall somewhere in the middle.


And that's the area of true randomness.


Essentially what you do is you throw your paper in there, and then something something


happens and then you get a random number at the end.


And remember people use this to justify archive blackouts, social media blackouts.


Oh my god, you cannot bias the reviewers, you must not bias the pristine review.


And how you cannot buy us a random number generator.


I guess you can but it makes no makes no sense.


Like honestly, this is only half joking at this point, the social media networks that


we have people surfacing interesting papers from the depths of archive and from their


social networks, all the people filtering this kind of stuff.


Yes, there's promotion going on.


Yes, there's hype.


Yes, money plays a role.


But still, this is a much better process than just like three random dudes sitting on the


toilet like scrolling through your paper a bit and then writing, not enough experiments.


Oh, reject.


I don't understand it.


It's confusing.


Look at the learning rate grafting video I did like these are the types of reviews that


reviewers have to battle with.


Yes, it hasn't gotten much worse over the years.


Yes, really good papers are consistent.


Really bad papers are consistent, but I still maintain that this situation is not really


a good one.


This is absolutely inconsistent.


It's a lottery.


Your best bet is to write as many papers as you can that are just barely barely not crap,


and then throw all of them in and through the random number process, some of them will


get accepted.


And that's a sad state because big companies do this for clout big companies do it to recruit


new people and so on.


But there are a lot of PhD students that need to get whatever their three papers published


in their four or five years that they're doing the PhD and with such randomness and with


only very, very limited amount of conferences that you can submit to over the course of


a year, there's like three or four different big conferences that you realistically can


submit to if you want a good impact factor.


This is very bad situation and a lot of people are going to be damaged just because the universe


has some random fluctuations.


The solution to this honestly starts with professors tenured professors start handing


out PhDs independent of conference submissions universities start giving professors tenure


not on the basis of the impact factor of where they publish.


Look at citations.


Look at how popular the work is in any other metric.


Stop considering impact factors of conferences.


Grant agencies stop giving out grants based on the reputations of the professors based


on the impact factors essentially disregard conference publications for anything you do.


I see some people they have to do it some professors have to get tenure and this is


a criterion PhD students have to do this because that's a requirement for their PhD.


But if you're in a position to discard all of this, do it what stops you you have tenure


tell your PhD students do three really nice really good archive publications.


If I'm happy with it PhD.


All right, that was it for me for ranting about this topic.


What do you think about it?


Let me know in the comments.


Maybe I'm completely wrong here.


But you know, I'm happy to be educated to the contrary.


AI against Censorship: Genetic Algorithms, The Geneva Project, ML in Security, and more!

Hello there, today I'm talking to Kevin Bock, who is a cybersecurity expert, and one of the main


people involved in the Geneva project. Geneva is a genetic algorithm that evades censorship by


nation states. So in real time, Geneva can evolve to the ever more present danger of censorship by


really big entities such as governments. All of this is done through an evolutionary search over


a program grammar. And in this interview, we're going to touch on a whole range of topics,


including Geneva, how it works, what it does, why people research it, and what it has done so far


in the world, but also the broader topics of security and its connections to AI, how people


can get started in this field, and what the main questions and problems are in this space. Further,


Geneva comes out of a project at the University of Maryland called breaker space, which is a sort of


lab that includes undergraduates in security research, which is a really cool project. And


I think highlighting this would be helpful to some people, maybe you're at the university,


you don't know this exists, go there, take part. Alright, without further ado,


I want to give over to the interview and have fun.


Alright, everyone, I have with me today here, Kevin Bock, who is a PhD student at the University


of Maryland, a cybersecurity researcher, and a member of breaker space, which is a pretty cool


project at the University of Maryland. He also has been in the news a little bit with a project


that's called Geneva, which uses genetic algorithms to evade censorship by nation states. And I think


that's pretty cool. So Kevin, welcome to the show. And thanks for being here.


Thank you. Thank you for having me. I'm excited to be here.


So the goal of today, it's a little bit different, because I'm a total new but security.


Most of the audience of this channel is not is into machine learning. Maybe some know about


security, some know about the the censorship apparatus that's in place around the world and


what people do about it. I think most won't. So today, I'll be asking mostly newish questions.


And we'll have you here to guide us through everything to guide us through like what's


happening in this world. So maybe you first can start off a little bit. How did you get into like,


how did you get to the place where you are? What's kind of the main things in security right now that


draw you to it? So I think security and the censorship space also is in this this really cool,


this really cool time where AI and ML techniques have been exploding and all these other fields.


And they're just over the last few years really breaking into security. And we're still figuring


out all the different applications where you can apply these techniques in security. There's new


techniques and new applications of this that people are discovering all the time from better


ways to detect spam and better ways to identify, hey, this domain is malicious or AI based scanners


for that binary you downloaded, that's probably malware, things like that. So security field is


still discovering all sorts of new ways you can apply these techniques. And that was one of my


motivations initially, actually, of bringing this censorship because this project was really


the entire field of censorship's first foray into using AI and ML like techniques.


And if you if you talk about censorship, what do you mean exactly by that?


Yes, there's so many forms of censorship in effect around the world today. I mean, everything from


political pressure to self censorship to taking down, like there's so many different types. So


I'm going to scope this discussion down a little bit, just the type of censorship that we study in


this lab. And that's this type of automated censorship that happens in the network performed


by nation states. So what do I mean by this? If you're a user in certain regimes around the world,


let's say in Iran or something, and you try and make a request, as that request, as that web


traffic crosses through the border to the country, it is scanned, parsed, and inspected by some


machines that physically reside in the network called middle boxes, called because they're in


the middle of the network. And these middle boxes examine your request and they say, is this something


we should allow or not? And if the answer is no, they either inject traffic to take down your


connection or they drop your connection or they do something else. And then they say,


drop your connection or they do something to disrupt what's going on. And you'll notice


everything I just said there, there's no human in the loop. There's no like human content review


or anything like this. It's a purely automated run by these middle boxes or firewalls deployed


by these nations that just like automatically inspect the internet traffic as they go by.


So that's really the scope of what we've been studying here.


Naive question. Why can't I just encrypt my traffic and then like every traffic looks the same


towards the outside? Yeah, that's a great question. So why can't we just encrypt everything?


People have been trying. So there's like a couple different approaches to this. You're like, well,


let's just use HTBS, right? Encrypted. We're good. Unfortunately, HTBS has a small privacy leakage.


When you first set up an HTBS connection, that very first initial is called a handshake and


that first back and forth. You as the client, as a part of the protocol, you have to announce the


domain you're talking to. And that announcement happens unencrypted. So if you're making an HTBS


handshake to Wikipedia, in the very first packet you send, it's going to include the word Wikipedia.


And that's called the server name indication field. You indicate to the server what the name


of the server you're trying to talk to. And unfortunately, sensors just read that field


and then they take down your connection if you talk to a forbidden domain. So HTBS, unfortunately,


not close, but not quite finishing the job. Now I will say there have been just a quick sidebar.


There have been some advancements in HTBS to try and fix this. There's a recent proposal to encrypt


that field. It's called encrypted SNI. And China just started censoring that last year.


So you can try and encrypt things, but these sensors are often just hostile to the idea of


just letting their citizens just encrypt all their traffic. I guess it's a little bit like


if everyone encrypts, like with HTBS nowadays, everyone does it. So you can't conceivably block


HTBS just because you don't like some traffic. But if there's a new type of encryption,


it's probably only the people that have something to hide that use that type of encryption. So is


a strategy that the rest of the world as fast as possible would use these techniques to kind of


make that approach unusable? That's exactly right. The broader topic you're actually


discovering and saying out loud here is this idea of collateral damage. Can we make a protocol or


something so popular and use so diversely that if a sensor were to try and block it, it would cause


irreparable harm to good services? There's some meaningful cost to performing that censorship.


So just like you've identified HTBS, that's everywhere. They can't just shut down all HTBS.


But rolling out a new encryption method for HTBS that's not very widely deployed,


they can nip that in the bud and prevent its rollout. So there's this kind of this interesting


race in a game between developers and these sensors that's still being played out.


Now, let's talk about more, let's say, naive approaches. What is the development of the field?


Like what has been tried before? And what has been, let's say, thwarted? Or what's the cat and


mouse game looked like in the past? I imagine different things like there's Tor, there is,


you know, all kinds of things. There is probably things that everyone installs on their end,


like VPNs and tunnels and so on. Like what's been the general development over the years?


Yeah, so the researchers and sensors have been playing this cat and mouse game for two decades


now. And it's kind of evolved and it's been playing out in multiple fronts. So you're exactly


right. Tor has been a huge front on that war, if you will, like we've developed Tor and continued


to advance it. Unfortunately, though, there are some limitations, just the Tor protocol and sensors


can enumerate the Tor entry points basically and just block you. So once you get into Tor,


you're generally great, but they try and lock you out. And there's been all sorts of techniques


people have proposed, like maybe I can disguise my traffic to look like Skype.


And then the sensors like, well, you didn't disguise it quite well enough, blocked. There's


a whole interesting field of defeating censorship or subfield, I should say, called packet


manipulation based censorship. And this is this idea where all our communication is happening


via packets. And if you just tweak those packets in just the right way, you could cause the sensor


to miss you. And historically that's also been something that's played out in this cat and mouse


game where researchers will study these sensor systems and then they'll find a loophole and


they'll deploy it and use it. And then the sensors like, oh, I'll fix that. And then we're back to


square zero. So this game has really been continuing to play. I'll call one thing out


real quickly about VPNs. Because a lot of people, particularly those who have been to China are like,


I've been able to use a VPN and it's been OK. VPNs in many places work, in many places they don't.


There's a country in the news recently, they were in the news because they rolled out a new law that


forced their citizens to swear on the Quran that they would not use a VPN in order to get Internet


access installed in their homes. It's just like a crazy sentence to say out loud. But in China,


for example, these VPNs, many of them work most of the time. But what researchers have noticed is


that around the time politically sensitive events are happening or political, such as elections,


things like this, a lot of VPNs will just mysteriously stop working. And then after


the event, they'll mysteriously start working again. And it kind of points to this broader idea


that some of these countries may be sitting on more censorship capability than they deploy on


a daily basis. And they have more power than they use. So this cat and mouse game may even be like,


the cat may even be stronger than we think it is. Yeah. Can you give us an idea of what this


packet manipulation evasions look like? Because I imagine something you mentioned before, you know,


if there's Wikipedia in the header, I don't want my population to see Wikipedia. Like, that's it,


right? What can I possibly manipulate there in order to get it to get through such censorship?


Yeah, so we think about sensors, as our computers are sending packets around, you can imagine a lot


of that communication, like you're writing mail, and your packets are envelopes that are going


that are going through the network. And in order to have a communication with a server like Wikipedia,


that's going to take a couple a couple envelopes back and forth, right? And the sensors just like


the postman in the middle reading all your letters. Unfortunately, that postman's got to process a lot


of letters, a lot of letters. And you can imagine the something that scale like China, you're dealing


with a huge, huge volume of traffic just on a constant basis. What that means is the sensor


can't just remember everything it sees. Okay, so for example, if it's trying to, if it's trying to


track that, hey, that person over there is trying to talk to that server over there, and that person


over there is talking to that server over there, that state it has to maintain, right? The amount


of state it has to maintain, it'll grow. And in the size of somewhere like China, it could grow


pretty fast. So they have to be really careful about what they remember and the state they


maintain. So you can imagine doing something like, let's say we're exchanging packets, there exists


a type of packet called the reset packet. And these are normal packets our computers send these


all the time, but they basically just exist to tell the other side stop talking to me immediately,


I'm hanging up the connection. So you can imagine doing something like, you and I are communicating,


we're sending these packets back and forth. And I just slipped one additional packet into the


connection towards the beginning and it's a reset packet. And I'll send that packet along. And when


the postman sees that packet, he's like, well, these guys have stopped communicating after this


message, he's going to ignore him forever. And then he throws away the state he's maintaining


about our connection. He forgets that we're talking because why would he need to remember


anymore? He thinks we're done. And if I craft that packet in such a way that it won't make it to you,


or you'll see it and ignore it or something like this, then we'll be able to still communicate


fine. Our communication is unimpacted, but any other packets that go by the sensors like,


I don't know who this is, and you can get through. So this is like the broad strokes,


this idea of packet manipulation based censorship, where you're tweaking the packets that go by to try


and basically trick the sensor that's in the middle into letting you continue to talk.


Now, do I do I see this correctly that there have been like, a giant amount of these schemes


proposed? And as you say, there's a cat and mouse game, one is being proposed, then they fix it,


then another one, then they fix it. So that points to the possibility of what if we could have


something dynamic, right? What if we could have something that by itself tries to invent new


things? And that's where you went with Geneva? Do I understand that correctly?


That's exactly correct. Yeah, you're spot on. Yeah, so over the years, there's been,


I want to say dozens of these that have been proposed and researchers have, it's exactly


this cat and mouse game. They studied the censorship system. I mean, the censorship


system is not public. So they're probing it, they're trying to take measurements. That's a


lot of work. And then they get an understanding, they apply their good human intuition, they


develop something cool and publish it, and the sensor fixes it. They don't tell you they fix it.


Yeah. They don't publish a paper that's like, hey, we just fixed your bug. So it just resets


the square zero. And so the idea with Geneva, which stands for genetic invasion, the idea of this was


it's an algorithm that could kind of flip this process on its head. So instead of a human having


to take the approach of let's understand how the censorship works and then defeat it, let's just


have some AI or fuzzer or automated system just attack the sensor, figure out ways through and


then give it to the human. And now after the fact, my slow human brain can go figure out why that


thing works. And now my brain is no longer the bottleneck to helping people get through the


sensor. How does this... You want to go a bit more into detail? I mean, it sounds great at the


surface, but there's a reason, right? We need security researchers probing, making sense. And


there's a reason that's the bottleneck. If I were just to be like, well, you know, fuzz a bit,


it's probably not going to work. So what does Geneva do that allows it to even be successful


where maybe humans take a long time or wouldn't be successful? Yes, there were a couple of pretty


significant challenges when we first started in applying something like a genetic algorithm or


really any AI to the space of censorship. And if you think about the way censorship works, it's not


hard to imagine like why that's the case. Because if you think about a censorship problem, right?


Like a query is either censored or it's not. It's just a binary decision. So it's not like


your traditional ML or AI where you have this nice like gradient descent. There's no error. You're


back from the sensor. The sensor doesn't tell you like, hey, if you tweak your query just a little


bit, you're getting closer. You know, there's no gradient with which you could work. So that


property alone rules out the majority of the ML field as far as approaches you can take. Is there


even a loss? Like you said, it's hard to detect if you even get through. How do you do that in the


first place? How do you notice success or failure? Yeah, so in our case, you're exactly right.


Capturing that can be difficult. What we do to make it easier in ourselves is we obtain machines


inside these censored countries and directly try to request for written content. So Geneva trains


directly against the sensor and we know we got it. When the sensor takes action is kind of obvious.


So Geneva will try and obtain some forbidden content while manipulating the packet stream.


And then if it succeeds, great. If it fails, we'll know. Yeah. Right. So this idea of how do we apply


ML AI, some fuzzing to this space, like how do we build to this? There's a couple main challenges


towards doing that. The first is this total lack of gradient that I mentioned. And really that only


leaves you with kind of a small number of approaches. And we chose to go down the route of


let's use a genetic algorithm for this. There's some nice properties. It's easily


explainable. You can understand how it works while it runs. It's a little less black boxy than


something more like a neural net or something or Markov or something like this.


But if you want to build a genetic algorithm, you need a couple of things.


You're seeing what some of these strategies look like right here.


So if you want to build a genetic algorithm, there's a couple things you need. You need some


building blocks. Something that the algorithm can compose and put together.


And you need some way for it to put those things together. I mean, us humans as examples,


as far as like genetics goes, we've got our DNA bases, right? ACTG. And we can put those together


in DNA. For the genetic algorithm for Geneva, we needed to decide what makes sense for building


blocks for the algorithm to use. And that alone is like an initial really huge challenge because


you could be creative and you can think about a million different ways an algorithm could


manipulate a packet, right? Flip a bit. You could flip this bit. There's just so many different


things you could give it to do. So one of the first challenges we had to figure out was how do


we balance what this algorithm can and cannot do to the data it has. And on one hand, we could let


it flip any bit. The downside of that is it could take forever to learn to check some,


but it's super powerful. On the other extreme there, we could just encode what previous researchers


found and let it play with those together. It would be super fast, but it'd be hard to learn


anything new, right? We'd just be building on biases directly. So the approach we ended up


taking was giving Geneva basically the same ability to change traffic as what the network


itself could do. So the network itself has just a few set primitives. It can do the packets. It


can take a packet and make multiple packets. It can duplicate them. It can change a header


to something. It's tampering a packet. You can take a packet, break it multiple times,


fragmenting. You can take a packet, drop it, which is just basically deleting the packet.


So we built out these building blocks and then allow it to compose these things together in


trees. Yeah. So like syntax, you give it a syntax and it can assemble a little program out of this


syntax, like one we see right here. That's exactly correct. Can you walk us through what this


particular thing does? Sure. Sure. This is kind of a fun strategy. So there's a few different