distribution right from which you can estimate the probability but ultimately what we want


Set Distribution Networks: a Generative Model for Sets of Images (Paper Explained)

is a generator that can sample from this okay so how do they build it they decompose this


into two parts and this is just a you know a decomposition this is standard decomposition


of probability where you'll say okay what's the probability of a set X the probability


of a set X is the probability of the latent code of that set times the conditional probability


of that set given the latent code so already we we have we ask ourselves what's the probability


of X and what we might ask is huh well X you know if I look at X it has these different


images of of this this it has these different images of that thing whatever is on the image


I can first ask what is the probability of that particular thing on the image and then


conditioned on that what is the probability that I'll get these particular images right


this is this simple decomposition already kind of builds up this model of encoding decoding


so we'll go through that step this here is going to be a deterministic function our encoder


and this here is going to be a probabilistic function the the decoder and it's probabilistic


because every time you call it basically every time you call the generator it's going to


give you a different output because you you're going to feed it different noise at the beginning


okay so so this is going to be this is going to be our encoder here and this is going to


be our decoder so they say here X is a Z is a deterministic function that maps a set X


to an element in a discrete space Z okay this is a discrete space as opposed to maybe a


regular autoencoder where you have a continuous space so here we want to discrete space and


a lot of mathematical problems are going to arise from the fact that this Z is a discrete


space and not a and not a continuous space so here P is a prior distribution with the


support given by sub C which means all the Z vectors that have some sort of set associated


with them which is a subset of all the Z so that basically means that if if we have a


given encoder there not all of these binary vectors are going to be filled even you know


with if we plug in all the world's faces into our encoder it might not fill all of the binary


capabilities that we have so this prior is only defined on this support of this and here


you already kind of see what what kind of mathematical hurdles you have to go through


if you do something like this and all the math here is going or most of the math here


is going to deal with the fact that we have this discrete thing and so on and a bit of


a bit of a little bit of a of a caveat here also is that this here they mention this here


is a prior you need a prior distribution on your Z variables and this is also not easy


so really quickly what does it mean to have a prior distribution on this kind of thing


because usually in a regular like auto encoder variational auto encoder right your latent


your latent code you'll have a prior on it and that prior can be you know some some continuous


thing like a Gaussian and even in a regular Gann as I said you you have your your noise


distribution and so on what is a prior on that thing now you can say oh a uniform prior


but again we would like to learn this prior such that it matches the data set well now


they use a prior from a paper that's called made M A D E and really quickly what it does


is it sort of kind of decomposes this thing so what you'll have is a neural network that


outputs binary vectors like this and it will sort of output them in a fashion autoregressively


so it will output one of them and then condition on that it will output the next condition


on that it will output the next and this is such that the probability of this binary vector


minus one one one minus one and so on is going to be decomposed into the probability that


there is a minus one here times the probability that there is a one here given that there


is a minus one here and so on and in different order I don't really want to go into this


but just to show you that there is a lot of consideration of mathematical consideration


if you really want to go about really want to go about this sort of thing in a formal


fashion so they define two things first of all this prior okay it's a prior distribution


that they can learn from the data set and then there is this conditional distribution


this what you might call a generator right if you're given a Z a one of these binary


codes what's the probability of a given set so you I tell you here it's a I don't know


Scarlett Johansson what's the probability of these pictures being different views of


Scarlett Johansson so that is going to be simply we're going to build this as an energy


based model you can you can do this what you'll have to do is you'll have to define an energy


and we'll just quickly discuss what that is and then you can build a construct like this


where you'll say the probability of a given set is going to be the energy that assigned


to that set divided by the energy that I'm going to assign to all of these other sets


so this is it's a form of an energy based model you can phrase very many things in terms


of these energy based models and Yann LeCun gave a talk about this at iClear I believe


where he gives a lot of different examples of energy based models so I invite you to


check this out I've also done a video on some of these energy based models and what you


can do with them here it's simply to define this probabilistic model so what we need to


do are two things we need to know what is this energy so what is this energy supposed


to do this energy is going to be a function that gives you and now I have to think so


its energy is going to be a function that gives you a high value if you are unhappy


with the input and it gives you a low value if you are happy with the input okay so you


see the negative exponential here which basically means if and also the energy is always positive


so the best if you are super duper happy with what is what your input is into the function


you'll output zero so if you output zero here you'll see that e to the exponential function


of negative something is going to be quite small and no maybe I have it wrong maybe you


output a really high number when you're really happy I'm not sure but it's one of the two


so this comes this comes from from a physics from a physics background no no no I'm right


so if you if you output if you're not happy you'll output a super high number here which


will make this negative exponential be really close to zero and therefore the probability


you say if I'm not happy the probability should be close to zero however if you're really


happy you'll output a low number right here now the energy always has to be greater or


equal to zero but the lower you go the higher this probability is going to be the bottom


thing here is simply to normalize the distribution because in a probability distribution you


always have to normalize because otherwise it's not a probability and this is what most


of these models basically are fighting over how to normalize the distribution and what


we're going to do is simply normalize it by sampling which is what most of these things


do you can build energy based models without this which GANs are a variant of that but


okay so what we need to do is we need to come up with this energy function that is going


to be a high number when we're not happy with the input now what is the input the input


is X and Z X and Z what does it mean we're not happy with the input it means that the


the image X and you see X here is one of the images of the set that particular image isn't


really congruent with the Z with the identity so you either you say what this this isn't


really a picture of Scarlett Johansson so I'm going to assign this a high value however


if you know if this is really any sort of picture of of that person then you're going


to assign it a low value and how better to do this than to build a neural network to


do that and this is going to be our discriminator so our discriminator is going to take the


role of this energy function okay cool now I said you need to normalize and I kind of


set it off the cuff and so on but the problem here is again we have this kind of sets and


and so on so our probability as you'll notice is the probability of X given Z so we are


already given the the the identity of the person so what do we need to normalize by


we can't simply normalize by all the sets of images in the world like here it's in the


integral we need to normalize by all the sets of images that are mapped to that same identity


okay so in order to normalize the distribution we'll basically ask how likely is this set


how happy are you with this particular set right here compared to all the sets that could


exist that would map to the same identity all right that's why these indicator functions


are here and as you can see the part here is simply a normalization where you say I'm


just going to produce other I'll consider all the other possible sets of images that


map to the same person and I'll simply divide by the energy of those in fact if you do it


correctly this particular X is also in that particular set but usually it's going to be


a fairly small part but to properly normalize of course you have to consider it as well


now this bottom part here as I already said is going to be the main problem of most of


these probabilistic methods and as I already said again it's usually approximated by simply


sampling a bunch of these sets and not sampling a bunch of these sets and not a by enumerating


all the possible sets of images and this sampling is going to make some further problems as


we'll see I guess down here so here is what we optimize ultimately we optimize or one


of the things we optimize is are we okay yeah so they say we apply maximum likelihood estimation


to estimate the parameters of the thing we just defined where the negative log likelihood


loss for an observed set in the training split is this so this is simply the negative log


likelihood the log decomposes into a sum so this is going to be your prior and this is


going to be your generator discriminator combo okay it's the generator producing images from


that binary code and then the discriminator assigning high or low values to that produced


images and also to images from the data set and the thing over here is simply going to


be the prior over the Z distribution that we briefly discussed now again they have to


have to do some tricks here where they say okay we can get rid of this support by using


a normalized distribution over Z which is a bound on that true prior and so on so they're


going to replace the DP with a p bar which is over the entire space of Z and that's going


to be a bound but the interesting part I feel more is here where they consider the loss


again of this conditional distribution on Z so you'll see the exact same quantity right


here but now our loss is going to be the negative log of that now since it's the negative log


you can decompose the division into a sum and this part up here you will see the indicator


function here is a bit unnecessary because the the Z here is the Z that we're considering


is going to be the Z of that particular set that we're considering and so this equality


holds on the top so disregard this and this down here as I said is simply a filter to


filter the space of all sets to the ones that correspond to the Z that we have in the energy


function so this goes here because it's simply a log of an exponential and the negative signs


cancel so you'll end up with this what does it mean you want to minimize this loss right


here and part of that is going to be you want to minimize the energy function of these inputs


okay you want you want to and this is now the case when it comes to from the data set


right so when X and Z come from the data set and E is your discriminator then you want


to make the output of the discriminator really small which means that you you want to train


the discriminator to say I'm really really happy that this particular image comes together


with this particular identity encoding now the in the in the if it comes from the generator


of course you want to do the exact opposite you want to assign it a high value remember


energy low means happy with the input okay and then the normalization down here is as


I said the problem so you'll see it stated right here because it's under the the division


it's going to get a pick up a negative sign which combines with this negative sign which


gives you a positive sign right here now this part right here is going to be intractable


because it it's not going to be feasible to enumerate all the sets of images it's not


even going to be feasible to enumerate all the sets of images that just correspond to


that particular identity and in fact it's not even going to be feasible to sample from


that because we have no clue right we can't simply generate out of the ether true other


pictures of the of a particular person what we can do of course is we can use our we can


use our model to produce more images right of that particular identity so what we'll


do is we'll replace this distribution with a variational distribution and we'll sample


from that now this isn't exactly the same this isn't this log probability anymore and


that's why first of all we have a bound here and not an equality this is this is called


a variational approximation so we bound this quantity and we can only bound this quantity


if we down here introduce the entropy of the variational distribution this is a fairly


standard trick in variational approximation methods if you want to look more into this


look into kind of a VAE explain variational autoencoders explained or anything like this


will teach you how how these methods work and in case we replace a distribution with


a distribution we can actually produce and what is that distribution look what we're


supposed to do is we're supposed to produce a set of image given so sets of images here


given particular Z and we can do that that's our generator right so we can use our generator


to produce those samples and that's what they say here here we have derived a lower bound


by introducing a variational distribution which we parameterize in the form of a generator


okay so the generator is going to produce that distribution it's going to use this noise


production so as you know the generator takes two things it takes the identity encoding


and a bit of noise and is going to produce an output set or for each noise it's going


to produce an output image very cool so that's that's the kind of math formulation behind


this model now they have a model architectures right here but this is all fairly standard


except for so for the prior they learned the prior on the Z space so you see you have Z


being this binary vectors they say we use a standard autoregressive model made with


three fully connected layers mainly for its simplicity and robustness and so again maybe


you like it took me a while to get what this prior does this prior is supposed to say it's


so it's not in again you have the Z vectors always being some sort of from the standard


noise but what you can also do is you can learn better noise distribution a better input


distribution for your again by basically making again for your input distribution so what


you'll do is you'll have a Z zero right here and then you'll use a you'll learn again to


learn better input distributions okay and this is what you do here with these with this


prior on Z this is more standard in like VA is than it is in GANs but it exists so encoder


say as a necessary option encoder for a set needs to satisfy the permutation variant property


we opt to use a simple architecture design where we let this be the average right here


so as you can see this is the average and then they they use this binarize operation


and the binarize operation here is clamping the values to one or negative one and it is


a straight through estimator which means that you will you back prop through it as if you


hadn't clamped but you forward prop through it with clamping this is kind of a trick to


get through discretization things discriminators job is to assign low energy to observed images


and high energy to generated images given a set code Z we use an autoencoder based energy


function implementation similar to twenty five and here they say we have found that


this choice is important as it enables effective learning in early stages of training so that's


why they do usually a discriminator would be the energy would be equal to this thing


right here which is a small MLP that maps a the input to a real sorry you can't see


that that maps the input to a real number either high energy I'm not happy low energy


I'm very happy here they also include this thing right here which is a decoder so it's


kind of a you can maybe think of it as a another little again another another another little


generator or the the generator part of a VAE or of an autoencoder sorry not a VAE an autoencoder


takes as input the encoding of the particular image and the identity and produces is going


to produce something that's close to the output again observe that this is now with respect


to a particular image so here we're trying to reconstruct that particular image because


we have its input thing right here and we're it's not the same as the generator that is


just asked to produce a some view some thing that corresponds to this particular identity


vector okay the generator generates a set conditioned on a set code by sampling and


random variables each of which is concatenated with Z and generates an image independently


cool so the losses they they they now introduce some margin losses on the things here but


basically you can just translate the what we have on top where we formulated negative


log likelihood into the losses right here they do have some simplifications for example


this to train the prior you what you are to train to train the encoder I think you have


to make a bit of an approximation in that the encoder is supposed to match this this


Z vector right and that's not differentiable by itself so they have this sort of L one


approximation right here they leave away the entropy from the loss and they have found


that to work well they introduced this margin losses right here I don't want to go into


that too much but basically they simply in a way with some approximations they approximate


I here it is the indicator function they approximate is this I was looking for that they they they


optimize this log likelihood from above in the way where they always optimize they keep


the generator constant and they optimize the rest of the pipeline so the encoder and the


discriminator in the prior and then they keep that rest fixed and they encode the generator


so what does that do before remember right here we had this this approximation right


here where we said you know what comes out of this we're not really optimizing this we're


optimizing we're minimizing a lower bound on it right so here's a quantity that we want


to minimize but here's a lower bound and we'll just push that lower bound down by optimizing


you know that doesn't tell us anything about this thing right here but there is actually


more to it so by optimizing the discriminator and the encoder and so on we do minimize this


lower bound so that this this loss right here you see this energy function will adjust that


whenever we adjust our whenever we adjust our that particular loss our discriminator


will adjust that energy function whenever we adjust our encoder we are going to adjust


the part that generates the z vectors right here so we'll push this down but whenever


we optimize our generator that's when we make this gap here smaller okay so we always do


two steps first we or first or second in one step we reduce this and in the other step


we'll bring these two closer together and as a result of course we hope that it's not


just the bottom one going to up down up down up down but we hope that both of them reduce


with time because the top one is the one we'll actually want to reduce that's our actual


loss or our log likelihood and that is I guess going to happen in practice so what does this


do so as I as we already saw here on top you have a set and you feed that through the encoder


feed that through the encoder that gives you a z identity and then you feed that to the


generator and the generator you can ask it you don't have to produce the same amount


of images you can produce any amount of images you like they just chose to produce the same


amount there's no correspondence but you see it's the same truck and here they manually


align these so they just produce a bunch of images on the left is the data set and on


the right I guess they just produced like a hundred images and then selected wherever


the car looked like the closest to so they ordered them by by hand and that is to show


that for example look at the the lighting on the car right here it's it's fairly similar


I guess this one has red taillights and the other one hasn't but you can see that the


the different views are pretty well captured by the generator and that just from all of


these are created from one one binary encoding of this here so this is binary encoded to


z and then all of these different views are created there's no image correspondence so


that's pretty cool and another problem when you have with sets is how do you evaluate


sets you can't you can't go and check for images or image closeness and so on so they


have to do some 3d modeling they actually take now they take these images right here


and they have to approximate their 3d shape and then compare that 3d shape with the 3d


shape of the original thing in order to just as a quant quantitatively estimate how well


they're doing in the faces the the same thing you input the top row into the encoder and


you get back the bottom row we've already looked at that but again to evaluate this


you actually have to go and use some sort of a face detector to recognize is that even


is that the same person always and is it so you can evaluate two things you can evaluate


are these right here all the same people so you can have a face detector kind of tell


you whether or not these are the same people and the second thing is are these down here


the same person as these up here right so those are the kinds of things how you can


evaluate this and they've done this and it's a fairly interesting and the results here


are not surprising when you look at the the images so these are curves curves from this


face detector and of course for real images as you can see the this is simply the performance


of the face detector so you do get some false positives if you if you want more true positives


right so this is a standard curve right here because these face detectors are not perfect


so in a given row right here in a given row even if that's from the real data set the


face detector would sometimes fail and say no that's not the same person even though


from the data set you know it is though the the to match the actual child photo from


Ali with his adult photos is even like you can forgive the face detector so that's sort


of the the gold standard we're trying to achieve and you can see within the reconstructed sets


that that is achieved fairly fairly well so compared to uniform samples this is you know


fairly fairly cool fairly close what is less close is this reckon and real and I believe


that's when you compare the identity of the real row with the identity of the reconstructed


row and that's here so that tells you already that it's the GAN or sorry the model doesn't


always preserve the actual identity as seen by a face detector and I don't know what to


say except yes that's what you see in the data right also you see that free samples


I guess so you can do two things right you can give it a set like a row and encode that


into the Z and then you can decode that again and basically reconstruct or you can just


sample since you've learned a prior on the Z variable you can simply sample you can simply


say give me some new identity maybe that I've never seen before right you have some binary


vector and now generator please give me images of that identity and these two here are actually


sampled like this and you can see again here it's remarkable that within the same row it's


pretty much the the rough identity of the person is conserved right and these are these


free samples right here I guess and they they do better than whenever you compare the reconstructed


and real but they don't do as well as when you actually input a real data and then reconstruct


this so this might be an indication that this prior isn't really working you know all too


accurately and I do have my problems with this binary encoding right here because maybe


I'm misunderstanding something but if you have these binary vectors as we said here the reason


you know the reason why you do one hot encoding in class conditional GANs is you could you


could simply say what am I doing a one hot encoding I'll simply say Z equals three for


class three and Z equals four for class four like that it should be so easy why am I doing


one hot and that's because these models see everything in a linear fashion so if you have


class three and then I have class four and then I have class nine the model doesn't see


that as three different classes the model sees this as these two are somehow closer


together than this right so the reason why we do one hot vectors is that the model cannot


do this the model has one independent dimension for each of the classes and whenever that


particular dimension is high then it knows that that particular class is activated what


this binary encoding here does is sort of it goes back to this thing right here where


it says okay there are all of these different categories here it's like you have mini classes


and the identity of whatever set you consider is now encoded in these mini classes so that


I'm going to guess the first thing here might be something like does that person have blonde


hair and the second thing might be does the image look generally bright or the images


image set as a whole look generally bright or dark and and so on so I'm going to guess


these things are encoded here and it'll sort of just end up being kind of a discrete GAN


or a discrete autoencoder rather than what they believe but maybe that was their goal


all along and I'm misunderstanding right here I just don't think this this binarization


is gives you this sort of hoped expressiveness I think there's still a lot of dependence


of whether or not a particular thing is on or off okay but enough ranting right here


I want to look at the at some more of the samples because I've only shown you the reconstructions


what I also find interesting is the free samples so here you can see uncurated shape net samples


and on the left so here you can see this effect on from the learned order regressive prior


and a uniform prior on the right and here you can see this effect of learning this prior


so if I learn the prior it's going to give me back fairly okay objects if I don't learn


the prior oh but if I learn the prior you know if I learned a prior really really well


that basically means I'm only going to ever produce sets that were in the training data


right if I learn like a perfect prior I'll see like wait this you know this particular


identity here never shows up so I'm not going to output it and the uniform prior might actually


output it and the generator is not going to be trained on that uniform prior so it's just


going to give you kind of crap and here in the in the faces you see the same thing now


again what I think I don't think that's happening what I think is happening is encoding these


kind of micro characteristics not per se identity but it's encoding probably you know a hair


color whatnot head shape and so on things like this and in each of these dimensions


and that's what is then going to produce so these each row here is in is one sample from


that prior on the left is learned which you see is working pretty well in terms of the


output and on the right you see it's from the uniform prior now you also see here first


of all that approximately identity is preserved but not as much in this uniform prior that's


first and second you see that the images are much worse which means that the generator


doesn't have as much training on that particular thing because I guess it comes from a prior


that it hasn't seen during training all right and here lastly they have reconstructions


if you give different number of views so the top row I guess is the input the this row


is when you just have four different views so I guess just the first four or something


like this input and the bottom one is when you have the full eight views and you can


I guess see or even more that this increases with number of views so the the accuracy of


this identity increases the more views you input of the set and they have a bunch of


other things right here in the appendix I do invite you to look at this and I hope you


sort of saw into a bit how you would go about something like this I I found it quite challenging


the math because I'm mainly not used to this kind of variational math but I hope this gives


you sort of an impression all right this was it from me tell me what you think and I'll


Involution: Inverting the Inherence of Convolution for Visual Recognition (Research Paper Explained)

Hello there. Today we're looking at


involution, inverting the inheritance of


convolution for visual recognition by


a number of researchers of


the Hong Kong University of Science and


Technology, ByteDance AI Lab, and Peking University.


In this paper on a high level,


the researchers tried to replace


the good old convolution operator in


CNNs by this new thing called an involution.


In its essence, involution is about halfway between


a convolution and a self-attention kind of operation.


Turns out that with some clever weight-sharing scheme,


you can achieve very good performance compared


to CNNs and self-attention networks,


while keeping the number of parameters and


the computational cost relatively low.


This, I think, is very much worth trying for


anyone who does not operate


on extremely large-scale problems.


We'll get into that a bit more when we go into the experiments.


But for now, let's go through the paper,


through what involution is,


what it does, how it's different.


If you like this, don't hesitate to share it out.


It would help a lot.


We're on the road to 100K subscribers,


and with every subscriber,


I get a subscriber.


I stole that joke.


They say here in the abstract,


convolution has been the core ingredient of


modern neural networks triggering


the search of deep learning in vision,


which correct AlexNet, ResNet, etc.


Convolution, even though transformers are


slowly taking over computer vision,


convolutions are still very, very much used.


If you're not on a super large scale problem,


a convolutional neural network is still very


probably the best way to go if you have


a computer vision problem.


They say, we rethink the inherent principles of


standard convolution for vision tasks,


specifically spatial agnostic and channel specific.


Instead, we present a novel atomic operation


for deep neural networks by inverting


the aforementioned design principles of convolution,


coined an involution.


And they say we additionally demystify


the recent popular self-attention operator


and subsume it into our involution family


as an overcomplicated instantiation.


So a lot of statements in this paper are true,


especially further down.


A lot of the experiments are really cool,


but it is a bit of an overstatement


what they say right here.


So their claim is that if you have a convolution,


what you do, you do something that's spatial agnostic


and channel specific, which means that


in a convolutional neural network,


when you have an image, let's say,


with a bunch of pixels, these are now true pixels,


not patches, and you run a convolutional layer over it,


you run a convolutional kernel over it,


you put the center of the kernel at some pixel,


then, so the kernel will be something like


a three by three kernel, you put that on the center here,


so it overlaps here, you multiply element wise,


and then you aggregate, and you can do that


in multiple channels, but essentially you do that.


And then after you've done that, you move,


you move the kernel one, let's say to the right,


you shift it, so the center is here,


you do the same thing again, and you shift it,


you do the same thing again.


So it's spatial agnostic because it repeats


the same computation over and over and over


across the image, and it doesn't care


where the computation is, right?


It does the same computation, and that is a selling point


of convolutional neural networks,


they are translation invariant.


This is, it's a form of weight sharing, right?


You share the weights across the locations,


and therefore you don't really care


where stuff is in the image, the CNN will be able


to recognize it just as well, and you don't need to learn


over and over and over the same principle


just because it's in different parts of the image.


So this is spatial agnostic, what does channel specific mean?


For that we have to go into the multiple channels realm.


So if your image has multiple channels,


let's say I'm gonna draw a new image right here


with a bunch of pixels, and it has multiple channels,


that means you can imagine it sort of as a 3D tensor here


where each pixel is a column, and every column


is a vector of a certain dimensionality.


I mean, so the original image has, of course,


three channels, which is red, green, and blue,


but if you have intermediate representations,


these channels can grow to sizes of hundreds of channels.


And the point of the channels is every entry here


is a number, and every number can sort of capture


one aspect of what's described in that particular pixel.


So maybe the first channel is, is there a corner?


The second one, is there an edge?


The third one, was it originally a blue pixel?


The fourth one, is there probably a cat here?


And so on, so these are like the different features


in the channels.


And the convolution operator is channel specific,


that means if you have the kernel,


now convolutional kernels aren't as easy as I drew them,


they're in fact four dimensional tensors.


So that is, they are four dimensional tensors,


which makes it a little bit complicated


for me to draw, honestly.


However, if you can imagine that you have one kernel,


like so, okay, that has the same amount of channels


as your image, okay.


So now you can still do the same operation, right?


You can overlay your kernel on a part of the image,


you can overlay it like so, and that's in the back.


And then you can do element wise multiplication,


and then you do an sum, you sum it all up, right?


After you do this operation, you do a big sum


over all the elements of whatever your kernel multiplied


with your image, and that gives you one number.


You do an all reduce, one number gives you one number.


And so you do this, so this is one kernel,


but you have another one right here.


Yeah, like this, and you do the same thing,


and that gives you also one number,


and you have another kernel.


I think you get the idea, you have another kernel here.


So you have many of those kernels per layer.


When you actually, if you've never looked at, you know,


how the weights look when you instantiate these layers


in a deep learning framework, I encourage you to do so.


A convolutional layer will have weights that are


of the size kernel size by kernel size,


by input channels, by output channels.


So it's a 4D tensor, and this, the orange part here,


is just one of those sub-tensors.


In fact, you have as many as you have output channels.


And that gives you, of course,


when you then go over all of these,


that gives you the next layer.


So that becomes in the next layer.


So this is the next layer representation, right?


At the point where you overlaid the kernel


in the last thing, that will become this column right here.


Okay, so you have the orange thing in the first,


the blue thing in the second channel,


green thing in the third channel, and so on.


Hope this is relatively clear.


So you have, in fact, one convolutional kernel


per output channel, okay?


So if you call the orange thing here a convolutional kernel,


then you have one kernel per output channel.


And that means it's channel-specific.


So this is a conscious choice, and it makes sense


when you think about it, because each output channel


means something different, right?


If my output channel means, is there a cat


at this particular location, then I might want to aggregate


the last layer's representation differently


than if my output channel says,


well, is this part of the sky?


Or is there a corner here or something like this?


So I want to aggregate the weights differently.


That's why I have to have a different set of weights


here, here, and here, because they mean different things.


So it's spatial agnostic, because it does


the same computation at every location.


It's channel-specific, because it does


a different computation at each channel,


even though it does it for all the locations equally.


All right, so now we're prepared to invert that.


So convolution promises we invert this.


What we want to do is something spatial-specific


and channel-agnostic, okay?


So the first thing here is the channel-agnostic.


If you've seen my last video about MLP Mixer,


this is very much the same idea, and the idea is just of,


hey, why do we have different things here?


Why do we have different computations?


Can't we just apply the same principle?


We apply to the spatial thing, where we say,


we just slide the same computation over the image,


and that is generally fine.


That's weight sharing.


It's actually good.


Why don't we just do this here?


Why don't we aggregate the information in the same way


for all the different channels?


And yeah, so you can do that.


You can just have one kernel.


So instead of having number of output channels, many kernel,


so the involution will come up with simply one kernel


that it shares across all of the,


that it shares across all of the channels.


They have a little picture down here,


and just look at the last step right here.


So here, wow, sorry, I crossed that out.


Here, this is the kernel that they have.


Sorry, it's not even by number of channels.


It's actually, you just flatten this thing, right?


So it's a K by K by one kernel,


and you simply push that,


put that over a location in the image,


and then you share the computation across.


So the image here, given that this is all in the same colors,


it means that you just multiply, you broadcast.


That's the word I was looking for.


You broadcast the operation across the channels,


and then you aggregate after that.


So you can see what involution does is broadcast,


and then not reduce, right?


You don't reduce at the end to a single number,


but you keep the channels as they are.


That's why you only need a K by K by one,


because you don't have the different computation


for each output channel,


and you don't reduce across the input channels.


So you get away with a lot less parameters.


So that's even wrong here.


Just a K by K kernel.


Now, that's one part.


The other part is,


why don't we do something that's spatial specific?


Spatial specific.


And now remember what spatial agnostic was.


Spatial agnostic was we slide the same kernel


across the image.


What they're saying in first instance,


they're saying things like, or they said something,


don't know where it was in the picture,


but they say, well, what we could do is,


if we have an image, right?


If we have an image, big image,


and we do something spatial specific,


what that means is we could have a kernel


that's just as big as the image, right?


Then no more sliding across it.


It's simply, you multiply those things together.


You broadcast it across these channels of the image,


and there you go, right?


That's it.


Also something that MLP Mixer does, right?


They just say, whatever, we don't do slidey slidey anymore.


We simply, I mean, they do weight sharing,


but essentially you're trying to get rid


of this sliding over.


You have different weight for each location.


And that means that the computation actually differs


from where stuff is in the image.


And we know that that is somewhat important


because usually the sky is up


and objects in these natural images that humans take


might be more in the middle than anywhere else.


And text goes from left to right.


And so it's not all super translation


and location invariant.


So it makes sense to have weights


that are different for each position.


But then they run into a problem.


They say, we couldn't do that very well


because now we can't just input pictures


of different resolutions, right?


That's one problem.


I think the other problem is that


this might not work too well.


So they come up with a different thing.


They say, can't we make a compromise?


And they don't call it a compromise.


They call it something different.


But they say, look, can we come up with a scheme


where we can retain a kernel


that's approximately this size, like a small kernel,


but it is different for each location.


So we still do the sort of classic convolution way


of doing things in that we do these local aggregations


across neighboring pixels.


However, the kernel that we use here


is different from the kernel that we use here.


And that's different from the kernel that we use here.


So how could you make a computation


where the kernel is always different?


You do that by coming up with the kernel in a dynamic way.


So the authors here, they say, okay,


if let's say we're at this pixel right here,


we care about this neighborhood.


How can we come up on the fly


with a kernel for this particular pixel?


And their answer is,


well, let's just generate it from the pixel.


So this is the full involution diagram.


We've now arrived at this.


So they are at this neighborhood,


which is outlined here in this black scaffolding grid thing.


The center pixel is the red pixel here, this one.


And they say, we look at that pixel and all its channels.


And we use that pixel and only that pixel.


So not the neighborhood,


we use that pixel to come up with the kernel.


So they have a computation here,


which of course is going to be a small neural network.


So this is a two layer neural network


that comes up with the kernel.


You see this, this is simply a, here is just a reshape.


So you compute the kernel across the neighborhood


from the pixel itself.


And that means that every single pixel here,


unless it's the exact same pixel,


so the exact same color in the first layer,


but already exact same representation


in the intermediate layers,


every single location gets its own kernel


for the convolution.


The computation I've already told you


is a small neural network.


Specifically, it's sort of a bottleneck neural network.


So it takes the pixel representation as a vector,


sort of bottlenecks it.


There is a non-linearity here,


and then it expands it again


to the size of the actual kernel.


And then you use that kernel and you broadcast it


instead of having one kernel per input channel.


And then you multiply,


and then you don't reduce by across the input channels.


Sorry, yeah, I said, that's it.


And that alleviates you from having to have multiple kernels,


one for each output channel, okay?


Now, this is the whole involution pipeline.


There are, I would say there are multiple


different concepts here.


So this coming up with the kernel on the fly is one concept.


And then this broadcasting scheme


is an entirely different concept.


You could do both independently of each other,


and they do them together,


which I, yeah, they do ablations further down,


but it's sort of two new things in one.


Now, the first thing here is very much,


you might think of attention mechanism


as you look at that,


because it's a form of fast weights, right?


So the weights of the computation,


they are computed on the fly from the data itself.


And that is exactly what an attention mechanism does.


However, here you do it in a slightly different way.


And they say that they have a discussion across,


about attention right here.


So they say, you know, there are a bunch of differences.


So in attention, what you'd have is,


you don't only have,


you don't only compute your weights from the actual location


where you are, even in local self-attention,


you actually compute your weights


from more than just the pixel where you are,


you compute it from the entire region you care about.


So that's the first thing.


And then the second thing is,


you don't, in self-attention,


you have the queries and the keys, right?


So you have your data, your neighborhood, let's say,


and each of those things produces a query and a key, right?


Query, and I'm gonna write the key up here.


Everyone produces a query and a key.


And then you do this sort of quadratic thing


in order to determine what,


like how you should aggregate your information.


Not in involution,


in involution you simply don't produce keys,


you only produce queries if you will, or only keys,


however you wanna look at it.


And then you don't do the quadratic thing,


rather you immediately interpret this


as sort of the weights of aggregation.


You can write this, and they say that you can write this,


you can interpret this as the positional encodings


already being present in these weights


because it's now specific to a position,


whereas in the attention literature,


you'd have to supply positional encodings.


So in order for the algorithm to know


that this is a different thing, sorry,


that this here is a different thing from this thing here,


you need to supply it with positional encodings.


Not here because the individual channels of this thing


immediately refer to different positions right here.


So this neural network is very aware


what position is where relative


to the pixel you're considering.


So they say the success of involution explains in part


why other people had lots of success


with leaving away the keys


and only using positional encodings


together with the query.


And if I'm not mistaken, this is a thing,


I think you could frame the Lambda networks


into this category where at some point,


like they never do this attention.


However, they rely heavily on positional encodings.


However, you can learn those ahead of time, right?


Or statically.


All right, that's enough of a,


so this is the connection to attention.


The connection to attention is the weights


are constructed on the fly.


However, here there's no quadratic interaction.


There is no softmax and so on.


It's just you construct the weights


from the pixel in the center.


Therefore, it's less powerful to frame attention as like,


well, it's a more complicated instantiation of our idea.


That's a bit out there.


Like the authors here, they say,


well, attention is just a more complicated thing


of our thing.


Nah.


And the second thing I worry a bit about is this is,


they say, well, this is position specific


or location specific, right?


They started out with saying convolution


is spatial agnostic.


We want to do something spatial specific.


This here is also spatial agnostic.


Like if you get the same pixel


at different locations in the image,


this thing will produce the same weights


and the computation will be the same.


In fact, you do this entire computation right here.


That is a spatially agnostic computation.


It's just, so the difference here is the same difference


that you have between slow weights and fast weights,


where you simply construct the weights


of the actual computation on the fly.


However, the way you construct these weights,


it remains position agnostic.


So that's the first thing.


And the second thing, yeah, the weight sharing,


I feel is a bit of independent thing.


Now I get it that the two work well together,


but the broadcasting and weight sharing thing


across the channels, it's almost a separate,


much simpler mention and it's a bit related to,


so if you have a depth separated convolution


and you simply share the weights across that,


that's about what it boils down to.


So, what does that give us?


In fact, it gives us a lot.


In this paper, they do experiments


and they compare against, for example,


so against ResNets and other networks


with similar number of parameters.


And I like these experiments here in that you can see


they always make sure that they have the lowest number


of parameters among the things they compare with, right?


Yet they show that they still beat these models.


They still are better than the models they compare to.


So they do that and specifically,


I guess they compare to ResNet


with the same number of layers, standalone ResNet.


This I think is self-attention.


I think here is this axial ResNet.


So that is a little bit less parameters,


interestingly enough, but yeah.


So you can see that this outperforms


on these tasks right here.


So this is ImageNet.


They also have different things


such as this segmentation task.


I think they have a picture down here.


The segmentation task where they perform better.


So here, I think this is the baseline


and you can see the involution network


that does a better job at this kind of things,


which is believable.


I think the effect that you see right here,


the fact that they are better in this number is really cool.


And it's probably a bit due to the fact


that they do this on the fly computation of weights,


which is a more powerful idea


than the static weights of a convolution.


And then the lower number of parameters, I think,


is more a result of their weight sharing scheme.


They tout here how that they is on par with ResNet-101


regarding the top one recognition accuracy


while saving 65% of storage and computation.


So I think that the saving of computation


is more due to the weight sharing mechanism.


And I think they've just here selected tasks


and they might be important tasks,


but I think it was just the case that in these tasks,


whether or not you share the weights probably doesn't matter,


doesn't hit you as hard or is even beneficial


if you don't have enough data.


And therefore, that's why they have less parameters.


All right, so this, what you can also observe here


is that differences, they get continuously smaller


as you move up the scale of network.


Now, this is all on the same data set,


but it would be interesting to see how this performs


on a really large scale, because my intuition is that,


as you go larger and larger in scale,


this approach is gonna top out and lose out


to the more general architectures like attention


and whatever, MLPs apparently, it's a clown world now.


But in the regimes, in these regimes,


and I would argue these are the regimes


where a lot of practitioners care about these


and actually smaller regimes.


So not many people are in the super high data regime.


This seems to perform reasonably well, right?


So you can see right here, the curves here,


when you compare compute to accuracy is very favorable.


As again, especially if you're in like this region here,


if you're in the low resource region,


it might be something that you wanna try out.


It remains to be seen how well this is pre-trainable


and fine tunable and so on,


but it's something you might wanna try.


Also, if you try to only use sort of parts of it,


it would be interesting to see,


if we still do convolution,


but we do this sort of weight sharing scheme,


this broadcasting scheme.


And yeah, they also have a notion of grouping in the channels.


So as I think the attention mechanism, yeah, has it.


So here they say it,


however, sharing a single kernel across all channels,


obviously underperforms in accuracy,


considering channel redundancy of involution kernels,


as long as the setting the channels shared in a group


to an acceptable range,


channel agnostic behavior will not only preserve


I guess preserve the performance,


but also reduce the parameter count and computation


or computational cost.


This will also permit the larger kernel size


under the same budget.


So it's sort of the same reasoning


as people introducing groups or different heads


in multi-head attention.


Yeah.


So try all of this stuff out.


I think it's worth it.


The code is available, code is available right here.


And I'll also put a link to that.


And that was it for me for this paper.


I wish you a very pleasant,


DeBERTa: Decoding-enhanced BERT with Disentangled Attention (Machine Learning Paper Explained)

Hi there, today we'll look at Diberta decoding enhanced BERT with disentangled attention


by Peng Cheng He, Xiaolong Liu, Zhang Fenggao and Waiju Chen of Microsoft.


This paper is an improvement on BERT the language model and the Roberta variant of it.


Specifically it suggests two improvements namely first is this disentangled attention


where they disentangle positional information and content information of the individual


tokens in the attention mechanism.


And the second improvement kind of results from the first improvement as this decoding


enhanced decoder, I guess enhanced decoder, where because they only use relative positional


information in the transformer part of the model, they have to re feed the absolute positional


information at the end, which gives them another bit of improvement.


All together with this they reach state of the art in various NLP tasks.


And this model Diberta is now available in Hugging Face for you to download for all of


your NLP needs.


So we're going to go through the paper and look at the two improvements and what they


give.


Let's and see if that's relevant.


As always, if you like content like this, don't hesitate to share it out to all of your


friends and leave a like and a comment.


I still read all the comments.


So give me your opinion.


And please also give me your opinions on the new recording setup.


There should be a title somewhere here.


Pictures somewhere here.


I absolutely want to hear feedback because I have no idea what I'm doing.


So yeah.


All right, let's dive in.


Diberta or Diberta or Diberta.


I don't know.


I think it's Diberta because it's from decoding enhanced.


Diberta is a new model architecture they say here.


We propose a new model architecture Diberta, decoding enhanced BERT with disentangled attention


that improves the BERT and Roberta models using two novel techniques.


The first is the disentangled attention mechanism, where each word is represented using two vectors


that encode its content and position respectively.


And the attention weights among the words are computed using disentangled matrices on


their contents and relative positions respectively.


Okay, we'll look at that first.


So what they mean is when you have a when you have a multi head attention layer, what


we want to do is we want to transform one sequence of tokens of token representations


into the next sequence of token representations.


Now usually, every token, let's say these are our tokens.


And this could be a sentence in a language like I am hungry.


And here is like this see this classification token that we always add when we train BERT.


Every one of these tokens is represented by a vector.


Like this is a vector.


This is a vector.


It has many entries.


Some of the vectors are thicker than others.


I mean, that's just this one just hasn't eaten enough.


So every one of these tokens is represented by a vector.


And what a multi head attention layer does is it it simply transforms this via means


of the attention mechanism into a series of vectors again, so we put in a series of vectors,


and we end up with another series of vectors.


If you want to know what a multi head attention does in detail, please go look at my video


attention is all you need, where that's explained.


Specifically, it is a attention it is sort of an information routing algorithm that sees


that sees how information needs to be routed from tokens to tokens using queries, keys,


values, and so on.


If you haven't seen the video, it's a beautiful mechanism.


But I'm not going to explain it again right here.


I'm sorry.


All right.


So in this, what usually do is you transform vectors into vectors.


And because of how the multi head attention mechanism works, the mechanism has no way


to discern where in a sentence, for example, a given token is so it cannot differentiate


between this sentence here and the sentence Am I hungry?


If it's just multi head attention is just not possible for it because it treats the


incoming sentence as like a bag of words, which is not the case in for example, a recurrent


neural network.


The recurrent neural network would go one by one over these word representations.


And it has kind of a mechanism to to see what a sequence is.


However multi head attention doesn't.


So what people usually do is they augment these representations with position encodings.


So that's at the beginning, you know where you might ask, where do these vectors come


from the very first, of course, they come from the last layer.


But the very first vectors you put in come from a table.


And these are your classic word vectors.


So at some at some point, you have a big table.


And the big table has your entire vocabulary in it.


So every word in the language that you consider so there's I and there's M and there is you


and there is Apple, and there is hungry.


And there is even the CLS token, all of them have a table entry, and all of them have a


vector associated with them.


Now these vectors are trainable.


So the neural network can decide itself what goes into these vectors.


But every word has a fixed vector in there.


And in the very first layer, because you don't have a last layer to draw from, you simply


look at what token it is, you go to the table, right here, you retrieve this vector, and


you put it here, and that's your start.


And then you transform up the layers, of course, every time from the last layer, but at the


beginning, you have embeddings.


Now the same thing you do for positions, okay.


So you also have a second table, usually, and the original transformer paper, by the


way, these were fixed vectors.


But nowadays, I think most of them are also trained.


So you label the positions.


So that's position, that's position one, that's position two, three, and four.


So for every position, two, three, four, and maybe you have also five and six, there is


a maximum length.


But right now we consider sentences of length three with the CLS token appended.


So these are length four.


So every position also has a vector.


And I'm going to actually draw these vectors in this color.


So every position has a vector, irrespective of what word there is, okay.


Right now we just have vectors for words irrespective of where they are.


And we have vectors of positions irrespective of what words there are.


And what you do is same, you look at what position is here, you go to the table, you


retrieve that embedding, and you somehow also put it here.


Now, I've made a bit of a mess here with this thing, sorry.


So how do you now you have two vectors all of a sudden per word.


So you have one, that is a position, and you have one that is the kind of the word itself


that represents the word itself.


And the neural network needs both in order to understand the sentence, right?


If every word has these two vectors at the beginning, now it can understand aha, this


is the word I that is at the beginning of the sentence.


So it's probably the subject of a sentence.


However, if the word M was at the beginning, it could be, oh, it's probably a question


because it starts with a verb like am I hungry?


Okay, and it can also evaluate the relative distances of things to each other and so on.


So given this information, the neural network has all the tools it sort of needs to understand


that sentence as a sequence.


Now what you have, you have basically two ways of combining the two things.


First of all, you can concatenate them, which means that I'm going to do it in this you


just put no, that's terrible.


You just put the I'm not too skilled yet with this new thing.


You put this on top here, imagine this is the same length and you just concatenate the


vector so now the vector is longer.


Of course, that also increases your dimensionality, computational issues, and so on.


So what a lot of people do is they simply, you know, line them up, if they're the same


size and they add them together element wise.


And you know, in the worst case, the neural network now can decide because both of these


are trained, right?


So the neural network can absolutely decide that, you know, in the top part here, it simply


learns a bunch of zeros, and then the bottom part here, it simply learns a bunch of zeros


here.


So essentially, it's a concatenation.


That's the worst case.


In the best case, the neural network can actually do some kind of information combining already


in this addition step down here.


Okay, so the you you give both encodings to the neural network as a single vector, right?


So what goes into the multi added attention mechanism is a single vector.


This paper says that is not ideal, because the positions are too much mixed with the


with the signal of the content of the words.


And we'd rather have this in a disentangled representation, such that the network can


sort of reason about the words in one line, and it can reason about the position of the


words in another line.


So their goal is to disentangle these two vectors and basically design a new attention


mechanism that always treats the content and the position as separate things.


So the new attention mechanism they propose is right here.


Of course, they're not there, they can't stay separate, right?


But they they can be disentangled through the layers.


So their new algorithm sort of is here, the way they obtain the attention matrix is due


to the following thing.


So how do you usually obtain the attention matrix, you have your input x here, this is


your sequence, and you produce two values from it q and k.


So these are matrices.


So if x is a sequence, then every single sequence element emits one key, which is a vector,


right, one key.


And then every single one also emits one query.


So like this, like this, and the key sort of the key is supposed to say, what is in


what information is this token about?


And the query is kind of supposed to say, what information does it request from other


tokens.


So now you route the information wherever the inner products line up, for example, probably


this thing would go to be routed here.


And so it's not a hard routing, it's a soft routing.


So by transforming x by linear transformations into keys and queries, you obtain your attention


matrix by multiplying together queries and keys, such that you have sort of the inner


product between each of these vectors.


And this is quadratic.


And this is the big bottleneck in transformers.


But you have the inner product between each of the two, you get a giant matrix.


And the giant matrix basically says how much does token two attend to token three, that's


the position two, three of that matrix.


And that's that seek that element is going to be the inner product of the query of token


two with the key of token three.


So that's how you do the attention matrix.


And these vectors right here, they if you do regular BERT, they always have, they're


always everything at the same time.


So you feed, you feed content and position somewhere down the layers, you feed that in


you add it together.


And the network is supposed to figure out itself how to use these two pieces of information.


This paper says no, wait, we can do better.


What we can do is, for us each sequence element, it does not only produce one key and one query,


it actually we think it should be contained, it should be made up of two vectors.


So each of these things has two different, two different components.


One is this kind of H component, which is the which is the content, content information,


and one is the P component, which is the positional information.


So here, how should how should token I attend to token J, they say, well, that is going


to be it's going to be the same thing.


It's going to be the inner product between the between the this is the query of token


I, and this is the key of token J. Okay.


However, now the queries and keys are made up of two of two different parts.


One is the content part, one is the position part in the position, as you can see, maybe


as J conditioned on either position is going to be a relative positioning.


So if you have your sequence right here, what each token would do is it would emit one vector,


oh, sorry, it would emit one vector, that is the content of the token, like before,


and then another vector would come in from the position.


So the same we did at the beginning.


But now in each layer, this positional information comes in irrespective of what word there is,


right, irrespective of what word is in the position, the position gets an encoding right


here.


And then the interesting thing is we don't add the two together, we treat them actually


separately.


So here, the keys are two vectors, and the queries are also two vectors.


So I'm just going to draw one up here.


So the query is going to be a vector.


And the query for the position is also going to be a vector.


And that also it depends only on the position and not on the incoming signal.


Okay.


So now, how do we route information?


Now, we have four different routings.


First we only consider dark blue, dark blue.


So this is kind of the classic attention, right?


This and this, they match really well, so that goes here.


That one probably doesn't go there, and so on.


But then we also so this is what they call content to content routing.


But then we also have content to position, position to content and position to position


routing.


And in all of these, so for example, in content to position, I'm sure I'm gonna there's a


5050 chance I'm going to mix this up, and I'm sure I'm going to but in content to position,


what we're going to do is we're going to look at this vector right here, which is the content


vector of the query that is produced from the token, right, the content is produced


from the token.


And we're going to attend to the position vector of the keys.


So we're going to attend to the light blue things.


So essentially, the this part is like the classic attention part, it is I am the word


am, I'm requesting all informations from all the nouns in the sentence because I'm a verb


and I would like to know who are the nouns in the sentence.


Then the content to position encodings is I am the verb am, I would like to know what


is around me.


The positions are relative positions.


So I can request the vector for, you know, the plus one position of mirror the plus two


it so the word can attend to its surroundings.


So given that it's the word am, it might be particularly interesting, maybe it has already


figured out, it's not a question, right?


From the previous layers, so it's particularly interested in what's before it.


So because, you know, am actually it probably isn't particularly interesting, because it's


always going to be I.


So actually, maybe it's exactly a counter example, where it wouldn't want information


from there.


But it can sort of attend, it can say, I want to attend to things after myself, because


I already have figured out that before me must be an I, I want to attend to things after


me like one position after me, what's right after me, what's two words after me, and so


on.


Position to content is exactly the opposite.


It is, it is saying so the token can say, well, I am in, I am in, I am in position,


plus four, to, you know, what kind of information do I want to send to things that are four


away from me, right, irrespective of what the content is.


So here, we simply consider what position is the token with respect to its neighbors,


and what kind of information doesn't want to aggregate from each of the words.


It is a bit, it's a bit weird, right?


So it says, it says, like, I, I am in, in position, a word that is two words after me,


what kind of information do I want to get from it?


And since it's attending to content that can be dependent on that can be dependent on what


word there is, but not its position.


And then position to position is simply, well, what what kind of information do I in position,


you know, three, you want to send to something in position seven, which would be useful.


But this is relative position encoding, which simply means I am always kind of in the middle.


And so this isn't really helpful, so they decide to leave this away.


So we end up with the three different attention mechanisms, so to say, we end up so there's


this one, there's this one, and there's this one, okay, corresponding to the three out


of four different ways, we can combine combine the dark blue, and the light blue keys and


queries.


Now, you can see right here, that's what they do.


And their final attention matrix is simply the addition of all of those together.


So we construct one attention from like the classic attention, we construct one attention


that is content to position, we construct one attention that is positioned to content,


and we construct one that is positioned to position, but then we leave it away because


it's we deal with relative position, so it will sort of be the same for every token.


And that's not particularly helpful.


Reason I'm going to repeat it again, the H information contains actual signal from the


last layer, while the P has no idea about the signal, it simply contains information


about the position of the tokens.


So you can decide to send information to a word that's two positions ahead of you, or


to request information from where that's three positions behind you, depending on what word


you yourself are, okay, so that's the content to position and position to content, attention,


these things are all added together.


And that makes up the final attention matrix.


So a final entry in the attention matrix could be influenced by multiple ones of them, it


could say, you know, I am the word, I'm the word am I'm in position two, I request a lot


of information from other nouns, if any noun is here, I want information, but I also want


information from things that are one or two positions ahead of me.


So that that is, and you know, since I'm the word am, and also since I'm in position number


two, I am very interested to know what the subject of the sentences now we have all of


it.


Okay.


All right.


And the rest is, is just like classic attention.


Now, you, you simply, so these P and H matrices are obtained by, sorry, the queries and the


keys for this are obtained by linear transformation.


So you see, this is the incoming signal, you send it through a linear transformation to


obtain the queries, and you also send it through a linear information transformation to obtain


the keys.


So the H is the same, but the these matrices here, these are learned weights to produce


key queries and keys.


And then you multiply them together.


That defines your attention matrix, you run that through a softmax to make a distribution


out of each row, and then you multiply it together with the values.


So this part here is kind of like the routing table, and the values are the information


to be routed, the values are obtained from these input signal.


As we said, we're going to amend that by.


So this over here is the classic key queries, keys and values.


Sorry, that's too much.


The classic queries, keys and values.


And then we augment that by two new so there is the queries and the keys for the position.


And you can see that the difference here is that again, it's learned weights.


But now there is this P thing right here.


And the P is positional encodings.


And that comes exactly out of this table we saw up here.


So the positional encodings come from this.


So and it's important to see that this here is h and this is the p values, but this is


only h zero, right?


h is actually transformed to h1 by the transformer, the first layer to h2 by the second layer


and so on.


The P always stays the same.


So you would feed the P into this layer, and you would feed it again into this layer and


you would feed it again into this layer.


So you can see, it's only positional information.


It's not content information.


And by feeding the position each time, and doing this in this disentangled way, the model


can sort of keep the content and position information separate.


I actually think it doesn't really keep the information separate because, you know, after


layer one, you certainly have position information in your h, right?


You can see that from from this path here from the actually feeding position information


into the transformer layer, h1 is already going to be a conglomerate of h0, which is


pure content plus the position somehow, this plus is not a real addition, but somehow the


information is intermingled there.


And if we weren't to feed in these things right here, it would just be like the classic


BERT, right, what they criticize.


Now by continuously feeding the positional information, that is one advantage.


You can actually do that with BERT, you can just add the position information each time.


I'm not sure if that would work super well, but you can do that.


Just gives the model a bit more side information to work with.


And then by keeping it separate.


Yeah, as I said, I'm not I'm not sure it's actually separate.


It's just that you keep feeding in position information layer after layer, therefore giving


the model sort of more information every time it makes a transformation, because otherwise


it would have to carry through the position information through all the layers just from


the very first layer.


So in this mechanism, you can see it's true that the position encoding is kept separate,


because it comes in fresh every layer, but I don't see that the content certainly has


position information in it from the last layer.


Like I hope you can see that.


So as I said, they do relative position encoding.


What does that mean?


So that means that the position encoding depends on where you look from.


So what I've drawn at the beginning, like this here, this isn't entirely correct.


You have to look at each token individually.


So for this middle token here, for example, the positions look like this, they look like


negative two, negative one, zero, one, two, and you would you'd have kind of a table not


with absolute positions, but you'd actually have a table with negative two, negative one,


zero, one plus one plus two, and so on.


And you would retrieve those vectors.


And then you when you consider the next vector, this one right here, it would look different,


it would write this would be zero, this minus one minus two, and so on.


So they do two things.


First of all, they truncate at some point, they simply say, well, our context window


is two.


So instead of going negative three here, we simply keep it at negative two.


So everything beyond negative two gets also the vector for negative two.


So that vector here is going to be just plugged into here and into here for this token, right.


And for this token for the previous token, it is only going to be plugged here.


And if and nowhere else, there are ways to efficiently implement this.


And that's this algorithm right here.


Don't want to go too much into it.


But just so you're aware, you don't have to consider each token really individually during


it attention.


That would be prohibitively expensive.


So you can do one big matrix multiply and then sort of pick and choose together from


your from the matrix that results especially with this truncation.


This is this algorithm.


So they call it efficient implementation.


Alright, so that is this position, position enhanced or disentangled information.


Why is it disentangled?


And because in every layer, they have a side input, this, this piece right here is the


side input that they sort of feed on top of this information.


And they specifically construct the attention matrix out of the three things, right?


It's almost like two contributions.


The one contribution is, hey, let's feed in position information in each layer.


And I think that has been tried before.


That's pretty simple.


The second thing is that we don't, we don't simply add the two vectors when we input it


into the attention, but we're going to construct basically three attention matrices.


And then add those together once we determine the inner products between each of those.


Okay.


So this is one of the improvements.


And that already helps a lot.


But then they run into a problem.


And this is not necessarily a problem with their method.


But this is a problem in general when you use relative position encodings.


So they say, given a sentence, a new store opened beside a new mall, right?


That's a sentence.


The words store and mall are mass.


So let's say you do this mask language model pre-training, right?


You mask out the words store and mall and you ask the model to reconstruct them.


Using only the local context, e.g. relative position and surrounding words is insufficient


for the model to distinguish store and mall in this sentence, since both follow the word


new with the same relative positions.


So from the word new, you know, relatively, it's always plus one, oopsie, it's plus one


to this word.


So the model cannot distinguish the two.


So there is a need for absolute position encodings.


Because if you had absolute position encodings, you could maybe make sense though.


You know, since I mean, you could you could figure out like store is probably kind of


a smaller thing and mall is kind of a bigger thing.


So it's more likely that the store opened beside the new mall than the mall opened beside


the new store.


So that means we need absolute position encodings or something like this, right?


And especially, we could have relative position encodings.


But if this is a very long sentence, and we truncate them somewhere, again, these two


things are not in range of one another.


And they're not going to know how far you know, they are apart and each, each one by


itself is just plus one apart.


So how do we solve the problem?


We feed in absolute position encodings.


However, that's exactly what they criticize.


They say, no, relative position encodings are much better than absolute for learning.


And that's kind of the same reasoning why a convolution is better than a fully connected


layer because you kind of slide the transformation over and it's simply data relative to each


other.


So relative positioning makes a lot of sense if when every word can do computation not


based on where exactly it is in the sentence, but how it is in relation to other words.


Otherwise, if you have absolute positioning codings, what you would have to do is you'd


have to say, well, if I'm the word m, and I'm in position two, I need to learn to attend


to position three.


However, if I'm the word m, and I'm in position three, I need to learn to attend to position


four.


And if I'm in position four, I need to learn to attend in position five.


These are all different things you need to learn.


However, if you have relative encoding, what you can do is you can simply say, I want to


attend to the word that's right after me, easy.


But we do need absolute position encoding for some things, namely disambiguate between


tasks like this.


So they feed in absolute position information.


But instead of doing it the beginning, they do it at the end.


So at the beginning, we have the word vectors, right, they go in here.


And then we have position information.


One, two, three, four, five, we have that at every single layer of the transformer,


we feed it in again, and again, and again, we feed in the same P vectors, okay, they


have different different of these, sorry, of these transformations in each layer.


So the actual transformation that makes the keys and the values, sorry, the keys and the


queries of the positional information are different, but the vectors are the same every


time.


So at the very top, so these are P relative.


So this is sorry, yeah, I mixed up this is the this is this negative two, negative one,


zero, one, two for the middle token.


And then at the end, we're going to feed in absolute position encodings.


So here we have, you know, your let's start at one, let's be good MATLAB people.


Here we have 12345, that we're going to now combine with the vectors that come out of


here.


So the reasoning is, they say there are two methods of their two methods of incorporating


absolute position, the BERT model incorporates absolute position in the input layer.


In DBERTA, we incorporate them right after all the transformer layers.


But before the softmax layer for mask token prediction, as shown in figure two, I've looked


at figure two, it's, it's not really helpful, honestly.


So that is this figure in the appendix, where they say, Okay, so in the BERT, like in the


BERT, you have the absolute position encoding somewhere down here, it goes through all the


transformer layers.


And then you have this classification layer at the top that does the language model decoding.


However, in their model, what you'd have is you have all the transformer layers here,


down here.


And then you have the absolute position encodings that come in through the side here.


And kind of the last transformer layer now has access to these absolute layers or the


last n, I think n in their case is two, or one, one or two.


So in the last layer or the last layers, now the transformer has access to the absolute


positions, and before it's just relative position at each step.


And they reason that that helps, because the transformer part learns to deal with relative


positions.


Okay, in this way, they say here, the BERT captures the relative positions in all the


transformer layers and only uses the absolute position as complimentary information when


decoding the masked words.


Thus, we call the BERT as decoding component and enhanced masked decoder.


And they compare the two, and they observe that EMD works much better.


So feeding absolute positions at the end works better than feeding them at the beginning.


We conjecture that the early incorporation of absolute positions used by BERT might undesirably


hamper the model from learning sufficient information of relative position.


In addition, EMD also enables us to introduce other useful information that is in two positions,


the yada, yada, yada, we leave it for future.


So they say you could also feed in other information.


I guess that's the case in every single neural network ever.


Yeah, but the point is they feed in the absolute position at the end, and their conjecture.


So I'm not sure, I'm not a fan of this.


I'm here, you know, this is like saying, okay, if we only feed it in at the end, right here,


this is position absolute, then we sort of limit the model.


Like right now, the model has the same information as it had before, as if we were to feed it


at the beginning, but we sort of limit it to only one layer of transformation.


So all they can do is sort of have kind of a little linear transformation in there.


And so if we don't feed that in here, whereas we do feed it in, the model can use it or


any way it wants.


And that's just not a good enough reason for me.


So I think, you know, regularization has its place, bottleneck layer has its place and


so on, restricting the capacity and so on.


But I'm not a fan of hampering the model in this way, kind of restricting it.


And I, you know, just because it makes your number better, there's not really a reason


why the same information should be worse if you give the model more steps to compute,


to compute with, you know, if you feed it in at the beginning, technically, if you train


the model correctly, it should learn to use that information in at least as good a way


as if you feed it in at the end, right, at least.


That tells me that the model that we haven't really figured out how to train these models


correctly yet with regards to positional encodings.


And again, I'm not a fan of simply saying, well, we only feed it in at the end, because


then the question immediately is, well, how many layers at the end?


How many layers at the beginning?


But when, you know, when is it too popular?


It's just, yeah, I don't think it's, it makes a lot of sense to simply give the model information,


but not let it do its best with that information, unless you have a specific kind of reasoning


why this is just not good enough for me here.


Not a criticism of the, you know, obviously, it's better, like they observe, like, you


know, all the information, sorry, all the arguments can be invalidated by, but it's


better, right?


That's deep learning.


So yeah, all respect for them for trying it out, and actually realizing it's better.


Pretty cool.


So they also do scale invariant fine tuning, where if they fine tune, which is where you


take kind of this, this model you trained with masked language modeling, and then you


fine tune it to NLP tasks, they have a bunch of tricks there like virtual adversarial training


and normalizing the embeddings before they do that.


And that apparently helps a lot, but they also say they leave the comprehensive study


of this for future work.


For now, they just want to get the good number, which is understandable, because you get published.


Alright, so here you can see, actually, we can we can skip most of the tables, they are


better, they are better, they are better, they're better in language modeling, too,


which is interesting, so you can do kind of BERT style denoising, but in classification,


you can also do actually auto regressive language model, which is pretty cool.


So here they do an ablation study of the different components, where they remove this, enhance


the decoder, and one time they remove the content to position encodings, sorry, attention


mechanism, and one time they reduce the position to content attention mechanism.


And in the table, it is sort of a wash, it depends on the task of how you look at, but


each of the components here gets you some kind of a benefit or a hit when you take it


away.


So yeah, it's not really clear that one of the components gives you all the boost, the


combination of them is obviously the best, and it's really cool when papers do these


kinds of ablations, rather than just throw a bunch of stuff at you and you it's on you


to figure out which of that stuff is important.


They compare it to Roberta in terms of training of accuracy after training.


So how much do you need pre training for a fine tuning and the Diberta as you can see


in these graphs outperforms Roberta.


So potentially, you need less pre training steps to reach the same accuracy in fine tuning


task, which is cool, also means that if you train for longer, you reach or if you train


for the same amount of time, you reach a higher accuracy.


And now for you know, their big thing they build, they scale it up, and they have a bunch


of tricks here.


And you know, pretty cool, they scale it up, I just want to highlight one trick.


We optimize the model architecture as follows.


First we share the projection matrices of relative position embeddings.


So they share the projection matrices of the relative position embeddings with each other.


So they share the position matrices with the content matrices.


So now instead of for example, so here is the query of the content, the key of the content.


Here is the query of the projection and the key of the sorry, position, position.


My battery is soon over to speed up.


So the content right here, and the position right here give rise to these matrices by


means of these help of these learned weights, right?


So here is WC, here is W, sorry, WKC, WKC, sorry, W.


That's the matrix that generates the queries from the content that generates the keys from


the content, the matrix that generates the queries from the position and the matrix that


generates the keys from the position.


So if you now share, you now want to share this and that, and also you want to share


this and that.


So if and at the end they are added, right?


So you multiply these things and then they are added.


And in my mind, honestly, what that results in, because before, let's just see.


So before you had something like if we simply multiply query times key transposed for the


context side, that would give you sort of context WQ, and now we share them.


So we don't care about C and P anymore.


WK transposed, K transposed, and oh, sorry.


Of course, context, this transposed.


And now we add them to something else.


And let's just say we have these position to position encodings that they leave away,


but we're going to consider them because it's easiest.


So it's position WQ, WK, yeah, transposed, position transposed.


If these matrices are shared, this simply ends up to be being the addition of the position


and content times these two matrices times the, again, this.


So and this is just like the old school attention mechanism.


Now I see there's these cross terms, and maybe they influence something, but it gets closer


and closer back to the old mechanism where you simply add the encodings and don't consider


them in a disentangled way, right?


If you do, if you like share the matrices of the disentangled representations, it simply


refers back to as if you were to feed the position in each layer of a traditional transformer.


So yeah, I'm not sure how much really the disentanglement is super important or whether


or not it's just more important that this positional information is actually available


at each step.


But, you know, I might be wrong here with the cross terms, I haven't actually looked


entirely at that.


Yeah, so that's the paper, they have kind of a discussion depiction of attention matrices


down here, where they show that their model, you know, does some does something kind of


different from other models in terms of where it attends, it has less of these global attention


patterns like Roberta has right here, except for the very first one, which is the CLS vector,


which makes sense, and otherwise has a rather diagonal attention matrix.


So that's, it's pretty sensible, though, you can also make the case that sometimes there


are just really important words in a sentence that everything should attend to.


I don't know, but it is state of the art, and it is a cool algorithm and is worth considering


if you build your next model.


Alright, with that, I thank you for listening.


Subscribe if you haven't.


I'll see you next time.


[ML News] AI predicts race from X-Ray | Google kills HealthStreams | Boosting Search with MuZero

Google decommissions DeepMind's health app. Jürgen Schmidhuber leads an AI initiative


in Saudi Arabia, and I have a new paper. Welcome to ML News.


Hey, hey, you. Yes, you. Do you run experiments? Machine learning experiments? Yes. How do


you track them? What? That's not a good way to track them.


Here's what you should do. You should use weights and biases. Coincidentally, this video


is sponsored by them. What is it? It's a system to track your experiments, track your artifacts,


reproduce all the things you've ever done. See metrics, data sets, models from the inception


of your idea to the final deployment and beyond. This is the ultimate tool, you can get started


with just one line of code. Yes, one line of code and be amazed at what it gives you hyperparameter


tuning, metrics tracking, resource utilization, model and data set versioning on cloud and


on premise. Get this and much more when you sign up to weights and biases. Personal accounts


are completely free. What are you waiting for? Sign up now. No, actually, watch the


video first, then sign up or sign up now and sign up later. Get your mom to sign up, get


your pet to sign up. There's absolutely no reason not to go to this URL and get your


account now. Cheers.


Hello and welcome to ml news on this beautiful glorious Monday. Let's dive into the first


story TechCrunch writes Google confirms it's pulling the plug on streams. It's UK clinician


support app. So this app has a bit of a history since 2015. DeepMind started it up originally


trying to bring more AI into the health ecosystem. Now the streams health app isn't actually


an AI focused app. It's kind of an app to track health data and assist clinicians in


making decisions. The goal was always to bring AI into the picture. But this apparently has


never succeeded. The article details the history of the app as it went through DeepMind stages,


then of course, the big scandal where it was discovered that DeepMind didn't really have


the legal basis for dealing with the data that they were dealing with. That was a weird


sentence. And finally DeepMind handing over the app to Google health, even though they


said they would never share anything about this with Google. And now finally, Google


deciding to turn off the app completely whether or not this is a result of data privacy issues


or just being a result of the business case not being strong enough. We don't exactly


know. If you're interested in this, this article on TechCrunch dives fairly deeply into the


issue. What is special is how often it is mentioned that the data is going to be deleted.


So it starts off with at least two paragraphs saying the data is going to be deleted. It


mentions it throughout and then it ends again with a paragraph on how the data is going


to be deleted. So rest assured the data is going to be deleted. I'm winking. You can't


see it. I'm winking. Now the article is also a little bit critical of Google starting up


projects and then killing them off after a short while, such as Google plus or the many,


many, many, many, many, many messaging apps that Google has released things like Google


video and so on. But honestly, I think this strategy has worked out so far. We got a couple


of very nice products out of Google that started exactly like this that we might have never


gotten if every single new product is an eternal commitment to support it. That being said,


bring back the free storage for Google photos. This was actually useful. So finally, Google


is turning off this streams app. There's apparently still one group of customers that is using


it ongoing, I guess they'll have to come to some sort of an agreement until the end of


their contract. But going further, let's just wait for the next Google inventions. There


should be like some sort of a betting market where you can bet whether or not new Google


products will make it five years past their inception could be fun.


IFLS writes AI makes strangely accurate predictions from blurry medical scans alarming researchers.


So this is an article about this paper right here reading race AI recognizes patients racial


identity and medical images that is a study into various data sets and algorithms and


whether or not they can detect a patient's race just from radiological images such as


these ones. Now there is a common pattern among articles like this one that usually


some confounding variable wasn't taken into account like source of data set or things


like this. However, this paper specifically pays a lot of attention to eliminate all such


confounding variables and really tests multiple hypotheses on how the model makes its assessment.


So there are apparently a few distinct markers of race even in these radiological images.


But even if they control for those, the models are still able to make out patients self reported


races. The really interesting thing is that even if the images are degraded, such as this


one right here and really pixelated, the models are still able to make out the patient self


reported race with a higher than random accuracy. But the pictures themselves would be completely


undiagnosable for any human and certainly humans couldn't make out the race of the patients.


So as I said, the paper is a fairly lengthy investigation into these models and data sets,


including trying to tease out race from models that have been trained not on predicting race,


which essentially means that in order to predict some health outcome, the models in some part


make predictions that correlate with race. And it is a fairly lengthy article. But if


you're interested in these things, definitely give it a read. It seems like to be a very


thorough study of these things. But the article here frames it all in terms of how terrible


this is, how biased these algorithms are. And while there's certainly truth to that,


and many of these algorithms are in fact bias when they shouldn't be and due to various


reasons, there also is the apparently rather shocking conclusions that your health outcomes


interact with your genetics, I know, new concept. So again, while we can certainly all agree


that results like this are worrisome, and there are problems with bias in AI, it seems