is going to be equal to categories dot index index lowercase whatever that


Loading in your own data - Deep Learning basics with Python, TensorFlow and Keras p.2

category is so zero for a dog one for a cat then we're gonna iterate over the


images I we don't need to show them we don't need to break anymore and all I


want to do now is resize with this new array so we'll come down here we'll


perform that resizing operation and I think that's about it so now we just


want to append this to our training data list above there so training


underscore data dot append and we want to append our new underscore array and


whatever that classification is so class underscore num okay so we want to do


that for everything the other thing we probably should do is for image we


probably should encase this in a try and accept I've been through this data set


before and some of the images are like broken so accept exception and you know


what I'm gonna do I am gonna I'm just gonna pass actually I already know that


there's some that are broken you normally you would throw the exception


so you could read it and figure out what's going wrong but I'm gonna go


ahead and just pass but there's like you'll get like an OS error and some


other warning information and all that fun stuff but I'll just pass for now


create training data and then running that and then whenever that's done let's


go ahead and print print the len of training data now a couple things we


should talk about is the balance of your training data so it's really important


that your training data especially like in the case of classification is


properly balanced so in the case of a binary choice like we have cats and


dogs you want to make sure you have 5050 right just as many cats and just as many


dogs now sometimes you can have different numbers and then when you


train the model you can inform the model and say hey these are our class weights


so they have a weights that you can pass and the way this will work is it will


weight the you know the loss a little a little differently in an attempt to


handle for your imbalance data set but if at all possible you definitely want


to balance the data set instead so but sometimes like let's say you had so let's


say you had a day so it was 75% dog 25% cat you feed that through the neural


network the neural networks gonna learn really quickly just always predict dog


and you'll be 75% right and then when it tries to learn from there it's gonna


have a really really hard time so so if you balance it so it's a perfect 5050


you'll be better the next thing you want to do is shuffle the data so we've got


the training data but as you can see the first thing we do is iterate over the


category then go from there so everything's dot all dogs and then all


cats well if you feed that to the neural network it's gonna learn okay only


predict dog and then it switches over to cats and it's like I'm wrong I'm wrong


I'm wrong okay cat cat everything's a cat and then it just keeps going back


and forth and it's probably not gonna learn very well that way either so we


definitely want to shuffle the data so we can import random and then we can do


a random not full there we go training data since training is a list mutable


there it is it's already shuffled at this point so for example we could now


for I don't know sample in training data we we can check that our labels are


correct by doing print sample once this will be the label so sample zero with


would be a how we went through them all anyways sample zero would be and let's


just do up to ten here would be the actual image array itself I don't want


to run the whole thing all over again well we'll just wait for that that's


fine so now let's do the let's take this data now and now that it's shuffled


let's pack it into the variables that we're gonna use right before we feed it


into our neural network so that's gonna be an empty list for X and an empty list


for wine general capital X is your feature set lowercase Y is your labels


sometimes you'll see train X test Y and like all these kinds of things and we


could we can split those up but we can still we can actually specify a


validation set so you don't have to split them up sometimes you may do that


anyways but you don't really have to do that so you can use built-in methods to


to properly do an out-of-sample test okay so now we're going to say for


features label in training data we're gonna X dot append features Y dot append


dot append label so we build those out into lists and we really can't pass


especially for the features we can't pass a list to the neural network I wish


we could I think we should be able to especially if we're using a high-level


API like Keras it's kind of silly that we still got to do this hopefully soon


one day this will change but we need to convert both things well actually why


can stay list but X has to be first of all a numpy array so let's go ahead and


do that so X equals NP array X but also we need to reshape it to be negative 1


by the shape of each X so negative 1 is just how many features do we have well


you can say negative 1 and that's kind of a catch-all it's anything any number


and then we can say we know the shape of the data to be image size by image size


you could also do like X 1 0 X 1 1 and so on but anyways we'll just go with


this so image size by image size and then by 1 so this one is because it is a


grayscale and just as a bit of a homework challenge for you guys I


encourage you to attempt to make a confident to work rather than with


grayscale work with color images so see if you can convert what we're going to


build here from a grayscale convolutional neural network to be color


so just as an aside that might be the hardest thing to remember to do is


change that one to a three because then it would be three values so that's what


we have to do again I think this reshape is kind of silly I wish somebody from


hopefully Keras eventually would make this not be a requirement we should not


have to deal with that shape it should just intelligently shape it because it's


always the same kind of thing like you always have to do this same step and


it's silly okay so once we have our data uh-oh X when did we convert it first of


all when do we define an X


X dot a pen when did it get converted when did we use an X I'm just tripping


out I guess because I never ran this I don't know when we had an X so why how


when was X a thing oh man someone's got to help me out with that one when did X


become a numpy array before this was run did we ever use an X variable or maybe


it's because I'm redoing this and X already did have a yeah so the first


recording got screwed up that's what happened man that was killing me I was


like wait a second so X already was if I should have restarted the kernel anyway


moving along okay the last thing that we want to do is we want to save these so


you don't want to generate your data like you don't want to like do all this


crunching and reshaping all that every single time because chances are you're


gonna be like tweaking your model so the reality of neural networks is that you


don't have the answer right away or you're not working on a problem that's


as easy as the one that we're about to be working on so it's gonna be it's not


gonna be as simple as like I'll just throw this model together use exactly


these you know features and this many nodes this many layers you're not gonna


know you're gonna be tweaking so you don't want to have to be rebuilding your


data set every time so the last thing we're gonna do is we're gonna once we've


created our training data we're just gonna import pickle and then in some way


you don't have to use pickle we could numpy dot save or whatever somehow save


your data so you don't have to redo it every time so pickle out equals open and


I'm gonna open X dot pickle X WB pickle dot dump we want to dump X where to


pickle out and then pickle pickle out dot close and then we want to do the


same thing for the Y so I'm just gonna do Y come down here Y and then later if


we want to read it back in you can do something like this like pickle


underscore in equals open X dot pickle RB and then or really no we kind of did


an extra step there but anyway X equals pickle dot load pickle in okay and then


X one for example is our image and then X so it'll be our feature basically so


that would be our image and then why one would be the label for X one anyways


that's all for now what we're gonna do is in the next tutorial we're gonna take


this data set that we've you know compiled and then we're gonna feed it


through our convolutional neural network after going over some of the basics of


confidence and all that so if you've got questions comments concerns whatever


something you think could be done better whatever feel free to leave them below


otherwise I will see you guys in the next tutorial where we are gonna feed it


through a neural network and hopefully get our correct classifications so till


Group Normalization (Paper Explained)

Hi there, today we'll look at group normalization by Yuxin Wu and Kaiming He of Facebook AI


Group Normalization (Paper Explained)

Research.


Group Normalization (Paper Explained)

So this paper is basically an engineering paper about a new normalization technique


Group Normalization (Paper Explained)

called group normalization.


Group Normalization (Paper Explained)

So what's the issue here?


Group Normalization (Paper Explained)

The issue is that pretty much throughout neural network learning we're using this technique


Group Normalization (Paper Explained)

called batch normalization.


Group Normalization (Paper Explained)

Now batch normalization is a pretty reasonable thing and it works very very well.


Group Normalization (Paper Explained)

So what's the idea behind batch normalization?


Group Normalization (Paper Explained)

The idea is if you have data points for machine learning methods and your data is in a 2D


Group Normalization (Paper Explained)

coordinate system somewhere down here, right, and you're trying to separate that from the


Group Normalization (Paper Explained)

dots which are here, it is often very beneficial to shift that distribution before you do anything,


Group Normalization (Paper Explained)

you want to shift it to the middle of the, basically you want to center it first of all


Group Normalization (Paper Explained)

such that the zero, the origin point is in the middle of the data and then sometimes


Group Normalization (Paper Explained)

you also want to do what's called normalize it and by normalizing we mean you want to


Group Normalization (Paper Explained)

kind of rescale the axis such that things are more or less sort of like Gaussians.


Group Normalization (Paper Explained)

So if you look at this distribution, first is the centering and then second is what is


Group Normalization (Paper Explained)

called a normalization.


Group Normalization (Paper Explained)

And usually we know that any sort of machine learning methods work better if you do that


Group Normalization (Paper Explained)

and that's mostly in classic machine learning methods with conditioning numbers of the data


Group Normalization (Paper Explained)

being better and so on.


Group Normalization (Paper Explained)

But if you just want to learn let's say a linear classifier you can see here you can


Group Normalization (Paper Explained)

even save one parameter because you can make it just go through the origin and that's true


Group Normalization (Paper Explained)

in general.


Group Normalization (Paper Explained)

So if we draw this in 1D you'd have a distribution that maybe is very peaky right here, you first


Group Normalization (Paper Explained)

center it to the middle of the coordinate system and sorry that's not really centered


Group Normalization (Paper Explained)

and then you would divide it by its standard deviation such that after it it is a unit


Group Normalization (Paper Explained)

standard deviation Gaussian so a normal distribution.


Group Normalization (Paper Explained)

The closer your data seems to be to a multivariate normal distribution the better these machine


Group Normalization (Paper Explained)

learning methods work, especially if you look at how signal in deep network is propagating


Group Normalization (Paper Explained)

through the layers.


Group Normalization (Paper Explained)

So the idea is if it's good for the general machine learning method that the input has


Group Normalization (Paper Explained)

a multivariate normal distribution or is normalized then it's probably good that the input to


Group Normalization (Paper Explained)

each layer is normalized.


Group Normalization (Paper Explained)

So when you look at how signal features are in between layers so this is for example the


Group Normalization (Paper Explained)

CONV53, this is a layer somewhere in the middle of a convolutional neural network and if you


Group Normalization (Paper Explained)

look at the spread of how features feature signals are throughout training you'll see


Group Normalization (Paper Explained)

that the more training progresses the larger the kind of spread of features is.


Group Normalization (Paper Explained)

So you might get really large numbers or really large negative numbers or maybe really small


Group Normalization (Paper Explained)

numbers in your neural networks and it would be better if you had a layer and the input


Group Normalization (Paper Explained)

you've normalized it right and the output then is again a distribution but it's maybe


Group Normalization (Paper Explained)

shifted that you would first transform that back into a normal unit normal distribution


Group Normalization (Paper Explained)

before you put it through the next layer.


Group Normalization (Paper Explained)

So what batch norm does is at each layer before each layer it will do a normalization procedure


Group Normalization (Paper Explained)

on the data before giving it to the next layer and you can do a basically backprop through


Group Normalization (Paper Explained)

that it's also common to learn bias and variance parameter to add after that but the important


Group Normalization (Paper Explained)

thing is that after each layer the data is normalized such that it is kind of in the


Group Normalization (Paper Explained)

most comfortable regime.


Group Normalization (Paper Explained)

What's the problem?


Group Normalization (Paper Explained)

The problem with this is that you actually need the distribution right if you want to


Group Normalization (Paper Explained)

center this data up here you need to know what the data is so you need to know the entire


Group Normalization (Paper Explained)

data if I want to figure out what is the mean of this distribution I need all of the data


Group Normalization (Paper Explained)

points to decide here's the mean I need to shift that up to here if I just have a mini


Group Normalization (Paper Explained)

batch like we usually do in machine learning so if I just have this or this and this and


Group Normalization (Paper Explained)

this point I just have four points I can't determine the mean but what I can do is I


Group Normalization (Paper Explained)

can sort of guess the mean from the four points right so my guesstimation of the mean would


Group Normalization (Paper Explained)

be somewhere here and that would be usually close enough and you can also see that the


Group Normalization (Paper Explained)

larger your batch is if you sample at random the more accurate your mean estimation is


Group Normalization (Paper Explained)

going to be.


Group Normalization (Paper Explained)

So people have been training neural networks with large batch sizes for is basically batch


Group Normalization (Paper Explained)

size have gotten larger and larger in the last year so that has not been a problem but


Group Normalization (Paper Explained)

what people are doing now is they are doing distributed machine learning where you do


Group Normalization (Paper Explained)

have your data set and you draw a batch and the batch might be large so that this might


Group Normalization (Paper Explained)

be I don't know one million images this might still be four thousand images in your batch


Group Normalization (Paper Explained)

but what they'll do especially with things like TPUs is they'll distribute that across


Group Normalization (Paper Explained)

many many many machines into batches of sometimes as small as eight samples per unit and if


Group Normalization (Paper Explained)

this is not images but maybe something longer like a sequence of text or if this is a sequence


Group Normalization (Paper Explained)

of speech or something like this you can sometimes even go down to two or one samples per unit


Group Normalization (Paper Explained)

of computation and of course you can't do batch normalization you can't calculate the


Group Normalization (Paper Explained)

mean of one sample it's just going to be that one sample itself so either you have to you


Group Normalization (Paper Explained)

have two options with your in small batch sizes let's say two either you take the hit


Group Normalization (Paper Explained)

and have your very bad estimate of the mean from just two samples or eight samples or


Group Normalization (Paper Explained)

after each layer you basically do a synchronization step such that everyone communicates to everyone


Group Normalization (Paper Explained)

else their statistics and you can basically aggregate the statistics across the batch


Group Normalization (Paper Explained)

both are not cool usually these frameworks they don't do this synchronization because


Group Normalization (Paper Explained)

it's just too slow so they'll go with the bad statistics and you can see this right


Group Normalization (Paper Explained)

here in this graph they have the image net classification error versus batch sizes so


Group Normalization (Paper Explained)

this is a resnet 50 model trained on the image net data set using eight workers so eight


Group Normalization (Paper Explained)

GPUs and if they do 32 images per now just look at the blue line here if they do 32 images


Group Normalization (Paper Explained)

per worker so it's eight workers it's eight times 32 that's the batch size that is a large


Group Normalization (Paper Explained)

number 256 maybe yeah all right so so if they do that then you can see the error is on a


Group Normalization (Paper Explained)

state of the art for a resnet 50 if they go to 16 it's still pretty good but then as they


Group Normalization (Paper Explained)

go lower and lower and lower so if they go to smaller and smaller batches and spread


Group Normalization (Paper Explained)

them out over the workers then the error starts going up and this is because the batch norm


Group Normalization (Paper Explained)

statistics get worse and worse so the goal of this paper is to find this group norm thing


Group Normalization (Paper Explained)

here the group norm they this paper claims is another normalization technique that pretty


Group Normalization (Paper Explained)

much does the same thing this centering and the normalization the scaling but it does


Group Normalization (Paper Explained)

it without relying on the batch statistics it basically does it within a data point and


Group Normalization (Paper Explained)

that means that the performance even though it's a bit smaller at the beginning for this


Group Normalization (Paper Explained)

particular example will stay constant with even in small batch size regime so this is


Group Normalization (Paper Explained)

potentially applicable as I said to things where you have to go to like two or one sample


Group Normalization (Paper Explained)

per worker because it's just the data points the single data points are just too large


Group Normalization (Paper Explained)

so if you maybe want to train something like BERT on a on a on a GPU so what is group normalization


Group Normalization (Paper Explained)

group normalization as I said works within a sample now there have been other methods


Group Normalization (Paper Explained)

that work within a sample instead of across the batch and they tend to not work as well


Group Normalization (Paper Explained)

as batch norm now this paper here claims that group norm works on par with batch norm for


Group Normalization (Paper Explained)

large batch sizes and better than on small batch sizes so here they have a schematic


Group Normalization (Paper Explained)

of what's happening in batch norm as you can see here you have this this cube now this


Group Normalization (Paper Explained)

cube here n means the batch size so these are the data points points in your mini batch


Group Normalization (Paper Explained)

this is the thing that is going to get small in the if you don't have enough memory then


Group Normalization (Paper Explained)

C would be your channels so we are talking about convolutional neural networks here but


Group Normalization (Paper Explained)

this is generalizable to other neural networks the channels are going to be the independent


Group Normalization (Paper Explained)

feature maps that you you're going to have so in a convolutional neural network usually


Group Normalization (Paper Explained)

each layer has these things called kernels and there might be three by three matrices


Group Normalization (Paper Explained)

like this and the if you have an image the kernel will be slided this thing right here


Group Normalization (Paper Explained)

will be maybe here will be slided across the image or slid is it slid okay would be slid


Group Normalization (Paper Explained)

across the image and then the numbers in here will be convolved with the pixels and that


Group Normalization (Paper Explained)

will give you the next layers representation so whatever the convolution operation is and


Group Normalization (Paper Explained)

you'll slide that over and that sliding over will give you the values in the next layer


Group Normalization (Paper Explained)

now you not only have one kernel but you actually have many kernels sorry about this let's draw


Group Normalization (Paper Explained)

that so you have more and more kernels you have a whole stack of kernels and how many


Group Normalization (Paper Explained)

kernels you have those are the different kernels are also called your different channels now


Group Normalization (Paper Explained)

the kernels refer to the weights and the channels refer to the image but the the if kernel is


Group Normalization (Paper Explained)

going to be convolving the i-th channel of the image so at the beginning the input image


Group Normalization (Paper Explained)

has three channels because red green and blue but then the intermediate images can have


Group Normalization (Paper Explained)

more channels as you have as basically as many as you have kernels in the layer right


Group Normalization (Paper Explained)

before okay and the H and the W means the height and width of the image so it combined


Group Normalization (Paper Explained)

so the image is kind of unrolled across the height or the width in this direction so what


Group Normalization (Paper Explained)

does batch norm do batch norm takes as you can see here one channel and it it takes one


Group Normalization (Paper Explained)

channel so maybe this image this is one channel let's just say this is the red channel because


Group Normalization (Paper Explained)

I drawn it in red it takes that and it calculates the mean one over and the standard deviation


Group Normalization (Paper Explained)

of that it calculates those two statistics and it uses that to do this centering and


Group Normalization (Paper Explained)

scaling operation so all of these methods are going to calculate the mean and the variance


Group Normalization (Paper Explained)

and then do the same scaling transformation the question is just how do you calculate


Group Normalization (Paper Explained)

the mean batch norm does this across the data points so it looks at a single feature at


Group Normalization (Paper Explained)

a single channel and it asks what's the mean across all the data points what are the data


Group Normalization (Paper Explained)

statistics of this channel and what was the mean and standard deviation now actually batch


Group Normalization (Paper Explained)

norm I'm not I didn't even know that in convolutional layer this works like this you can also imagine


Group Normalization (Paper Explained)

batch norm of really just taking one single feature and that means of really just taking


Group Normalization (Paper Explained)

one of these things right here so if this goes to the back and normalizing across that


Group Normalization (Paper Explained)

the important part is that it is in fact normalizing across the data points so it looks at your


Group Normalization (Paper Explained)

batch looks at the mean and the variance in that batch and it normalizes by that I think


Group Normalization (Paper Explained)

convolutional layers make sense because you have this invariance in height and width and


Group Normalization (Paper Explained)

therefore yeah so that makes sense but in a fully connected layer you'd simply go look


Group Normalization (Paper Explained)

at one feature at a time layer norm is different layer norm has been basically been proposed


Group Normalization (Paper Explained)

as an alternative to batch norm with the same reasoning that this paper has so layer norm


Group Normalization (Paper Explained)

as you can see here it does each data point individually so here we only have one data


Group Normalization (Paper Explained)

point that is normalized by itself so you do this for each data point independently


Group Normalization (Paper Explained)

and therefore it's not dependent on the batch size anymore but what you'll do is you'll


Group Normalization (Paper Explained)

look across all of the channels right here so all of the channels and all of the width


Group Normalization (Paper Explained)

and height so this entire thing here this entire thing is basically one channel right


Group Normalization (Paper Explained)

and then the next channel is here of the image and the next no that's the next image well


Group Normalization (Paper Explained)

that is a bad drawing because the image is unrolled in any case what you'll do is you'll


Group Normalization (Paper Explained)

look at so if you have a filter bank like this you have an image and the image composed


Group Normalization (Paper Explained)

of multiple channels right this is the red and then you'll have the green right this


Group Normalization (Paper Explained)

is in the green and then you'll have the blue channel and what you'll do is simply you'll


Group Normalization (Paper Explained)

calculate the mean across all of the pixels and across all of the channels you just take


Group Normalization (Paper Explained)

this whole non-py array and you just say dot mean and that gives you one number and it's


Group Normalization (Paper Explained)

just whatever that number is you subtract it and then you say standard deviation and


Group Normalization (Paper Explained)

you divide by that that's layer norm so an entire layers representation of one image


Group Normalization (Paper Explained)

is just normalized to the mean now this seems a bit drastic and that's why instance norm


Group Normalization (Paper Explained)

did the exact opposite they said wait a minute instead of normalizing across all of the features


Group Normalization (Paper Explained)

right we'll go back and do what batch norm does batch norm looks at each feature individually


Group Normalization (Paper Explained)

so basically it looks as all of these these different axes in the data distribution it


Group Normalization (Paper Explained)

looks at them differently so if one axis is scaled very widely we want to normalize that


Group Normalization (Paper Explained)

differently than if then the other axis that is just scaled very shortly and that's why


Group Normalization (Paper Explained)

we'll look at each feature individually like batch norm but also we only look at one data


Group Normalization (Paper Explained)

point at a time now as you can imagine this doesn't work anymore in in a fully connected


Group Normalization (Paper Explained)

network this basically works in a convolutional network where you have a feature map channel


Group Normalization (Paper Explained)

so you look at one individual channel and one data point so that means here you would


Group Normalization (Paper Explained)

normalize the red channel individually you would normalize the green channel individually


Group Normalization (Paper Explained)

and you'd normal normalize the blue channel individually so the image you're going to


Group Normalization (Paper Explained)

end up with is simply the red channel subtracted by its own mean and then divided by its own


Group Normalization (Paper Explained)

standard deviation and just within that data point right so maybe I should here say across


Group Normalization (Paper Explained)

the number of features or something so I hope that that's clear so the layer norm drops


Group Normalization (Paper Explained)

the dependence on the batch size but instead says we should normalize across all of the


Group Normalization (Paper Explained)

features and the instance norm says wait a minute batch norm had a good idea normalizing


Group Normalization (Paper Explained)

only across the features individually because the individual features might have different


Group Normalization (Paper Explained)

scales and we should account for that but also we don't want to be dependent on the


Group Normalization (Paper Explained)

batch size and now is where group norm comes in group norm is basically a mix between layer


Group Normalization (Paper Explained)

norm and instance norm what group norm says layer norm and instance norm have good ideas


Group Normalization (Paper Explained)

they only go across one sample they take that they say in essence instance norm has a good


Group Normalization (Paper Explained)

idea in that the features should be normalized individually but it goes sort of too far from


Group Normalization (Paper Explained)

it goes too far you might get not good enough statistics because you're now normalizing


Group Normalization (Paper Explained)

each of these things individually whereas with layer norm you're too restricted you're


Group Normalization (Paper Explained)

basically saying that the features it's fine if the features relative to each other are


Group Normalization (Paper Explained)

like this right one is maybe very high variance and one is very low variance feature norm


Group Normalization (Paper Explained)

would keep that and group norm would say maybe that's not it's not so good we should have


Group Normalization (Paper Explained)

we should normalize the individual features maybe individually but they'll their argumentation


Group Normalization (Paper Explained)

here is that maybe there are some features that by their nature already have the same


Group Normalization (Paper Explained)

sort of scaling and variance they give an example if you for example have a filter again


Group Normalization (Paper Explained)

we deal with convolutional layers here and that filter is a let's say an edge filter


Group Normalization (Paper Explained)

right so a horizontal edge filter so it's very low value here and let me mark the high


Group Normalization (Paper Explained)

value with blue so this is a horizontal edge filter if you slide this over a window and


Group Normalization (Paper Explained)

these are high numbers and these are low numbers it will respond to edges because edges have


Group Normalization (Paper Explained)

high low high right or vice versa so it will give you very positive and very negative number


Group Normalization (Paper Explained)

every time you slide across an edge now you can imagine that in natural images that filter


Group Normalization (Paper Explained)

whatever image you put in would and however you normalize would give you pretty much the


Group Normalization (Paper Explained)

same response as a vertical edge filter so the horizontal and the vertical edge filter


Group Normalization (Paper Explained)

you you'll see whatever their response is they're probably about equal in size so we


Group Normalization (Paper Explained)

could expect that in a neural network there will be groups of filters that together exhibit


Group Normalization (Paper Explained)

the same scale and therefore we can normalize across them like in layer norm so the more


Group Normalization (Paper Explained)

things we normalize across the better statistics we can gather that's why instance norm doesn't


Group Normalization (Paper Explained)

work because it only normalizes across a very small thing getting very little statistics


Group Normalization (Paper Explained)

but we should normalize if we could gather good statistics we should normalize different


Group Normalization (Paper Explained)

features differently and group norm says well since some of the features are almost guaranteed


Group Normalization (Paper Explained)

to behave the same we could normalize across those now of course you don't know at the


Group Normalization (Paper Explained)

beginning which ones those are so but you you hope that by doing group norm by basically


Group Normalization (Paper Explained)

at a priori so at the beginning of training you decide what the groups are and and naturally


Group Normalization (Paper Explained)

it's just whichever ones are next to each other those are the groups and you'll hope


Group Normalization (Paper Explained)

that through the training procedure basically those groups will learn the features that


Group Normalization (Paper Explained)

are equal of size well you basically enforce that so you kind of constrain the architecture


Group Normalization (Paper Explained)

to do that so that's the idea behind group norm you basically build these groups of channels


Group Normalization (Paper Explained)

and then you normalize across those across the groups of within the groups of channels


Group Normalization (Paper Explained)

across the entire height and width only in a single data point and therefore you gain


Group Normalization (Paper Explained)

the advantage of layer norm of normalizing within a single data point you retain the


Group Normalization (Paper Explained)

advantage of batch norm of of normalizing across single features that's what instance


Group Normalization (Paper Explained)

norm attempted but yeah so you get the best of both worlds sort of that's group norm and


Group Normalization (Paper Explained)

now we go and look what it does so they say okay basically all the normalization techniques


Group Normalization (Paper Explained)

do this they subtract a mean and divide by standard deviation that's what we saw and


Group Normalization (Paper Explained)

the difference is just across what you collect your statistics so the group norm is the following


Group Normalization (Paper Explained)

code in tensor flow as you can see you simply reshape your data and basically expand this


Group Normalization (Paper Explained)

part right here where you built you where you put the extra so this is C this entire


Group Normalization (Paper Explained)

thing used to be C and you divide it into group and index within group and then you


Group Normalization (Paper Explained)

just normalize across that and reshape to the original dimension again and the import


Group Normalization (Paper Explained)

the cool thing is in batch norm you have to keep track of these of these running means


Group Normalization (Paper Explained)

because at test time you sort of don't want the batch statistic to influence anything


Group Normalization (Paper Explained)

you don't have that here you could just back propagate through this observation through


Group Normalization (Paper Explained)

this operation and you don't need to keep these running running averages going and you


Group Normalization (Paper Explained)

always care am I in test or am I in train mode right now you just do this this operation


Group Normalization (Paper Explained)

is per data point so it's just part of your model right and they do a an experiment where


Group Normalization (Paper Explained)

they have 32 images per GPU so it's reasonably sized and they can basically show that the


Group Normalization (Paper Explained)

group norm and the batch norm they compare in their performance now I do usually don't


Group Normalization (Paper Explained)

believe the experiments that you see in single papers but I think this has been replicated


Group Normalization (Paper Explained)

a couple of times now you see this is the train error where group norm even behaves


Group Normalization (Paper Explained)

a bit better and then in the validation error it behaves a bit worse but one could say it


Group Normalization (Paper Explained)

is it is kind of more closely together than the other methods are to the group norm or


Group Normalization (Paper Explained)

to each other these instance dorm and layer norm so it at least it's better than instance


Group Normalization (Paper Explained)

norm and layer norm and then once you go into the smaller batch size regime of course that's


Group Normalization (Paper Explained)

where the group norm starts to shine so if you go from the 32 images per GPU which is


Group Normalization (Paper Explained)

this low black curve here all the way to two images per GPU and I believe they could even


Group Normalization (Paper Explained)

do one image per GPU with group norm but of course you can't do that with batch norm because


Group Normalization (Paper Explained)

you need batch statistics you can see that the the performance of batch norm degrades


Group Normalization (Paper Explained)

drastically whereas with group norm this experiment is just funny they just have to do this even


Group Normalization (Paper Explained)

though you know exactly what turns out so you could the lines are all exactly in the


Group Normalization (Paper Explained)

same place I mean come on like you know you're just having time to probably one of the reviewers


Group Normalization (Paper Explained)

was like but did you really do the experiment to put it in so yeah so you can see that the


Group Normalization (Paper Explained)

batch norm beats the group norm in this setting with the when you have the larger batch sizes


Group Normalization (Paper Explained)

but the group norm pulls ahead quite drastically when you have the smaller batch sizes and


Group Normalization (Paper Explained)

that is the main advantage so now you can turn to models that require small batch sizes


Group Normalization (Paper Explained)

or small batch per worker and generally it's a pain in the ass to just keep track of those


Group Normalization (Paper Explained)

statistics for test time they do verify which I find pretty cool that this phenomenon of


Group Normalization (Paper Explained)

the responses going apart during training in the internal feature maps batch norm counteracts


Group Normalization (Paper Explained)

that so with batch norm you'll get actually a convergence of responses during training


Group Normalization (Paper Explained)

so the more you train the more normalized basically your internal features will be and


Group Normalization (Paper Explained)

they show that this is exactly the same with group norm so group norm is as it seems it


Group Normalization (Paper Explained)

is a replacement it's not an addition it doesn't the gains don't come from different place


Group Normalization (Paper Explained)

it seems to be a substitute for batch norm though they don't have an experiment where


Group Normalization (Paper Explained)

they do both I believe maybe I'm wrong actually maybe they do but yeah it seems like you just


Group Normalization (Paper Explained)

kind of have to bring some calmness on some standardization into your signal and how exactly


Group Normalization (Paper Explained)

you do that doesn't seem that important as long as you do it with some precision and


Group Normalization (Paper Explained)

some some real overall statistics yeah what I don't like about this is now you have of


Group Normalization (Paper Explained)

course a new hyper parameter which is this number of groups right so that that seems


Group Normalization (Paper Explained)

rather annoying and the gains like this usually come from the introductions of new hyper parameters


Group Normalization (Paper Explained)

and and that just it's not so it's not that ideal for a method to introduce a new hyper


Group Normalization (Paper Explained)

parameter at least layer norm and instance norm didn't and now as you can see the number


Group Normalization (Paper Explained)

of groups is is not super influential but does have a bit of an influence on the on


Group Normalization (Paper Explained)

the performance so if you go a number of groups or here number of channels per group of course


Group Normalization (Paper Explained)

these two numbers are inversely related the more groups you have the less number of channels


Group Normalization (Paper Explained)

per group you have if you go to one extreme you will get to the layer norm basically so


Group Normalization (Paper Explained)

the layer norm is an extreme case of group norm where you just have one group all the


Group Normalization (Paper Explained)

channels are in the same group then the performance as you can see here is quite a bit worse if


Group Normalization (Paper Explained)

you go to the other extreme where every channel is its own group that's equivalent to instance


Group Normalization (Paper Explained)

norm again the performance is quite bad and somewhere in the middle here with 32 groups


Group Normalization (Paper Explained)

is seems to be a good sweet spot so I don't again I don't like the hyper parameter seems


Group Normalization (Paper Explained)

to be some somewhat of a thing where you really have to hit a good value and well I guess


Group Normalization (Paper Explained)

we'll see over time if that value is always going to be about the same you know like the


Group Normalization (Paper Explained)

beta 2 of Adam it's it's always like people never change it from point nine nine nine


Group Normalization (Paper Explained)

because it just tends to work or whether that's really going to be another hyper parameter


Group Normalization (Paper Explained)

to fit that seems to be annoying they do a bunch of ablation studies and tests on as


Group Normalization (Paper Explained)

we said the for example object detection segmentation so so models where you must go almost to small


Group Normalization (Paper Explained)

batch sizes just because so video classification so if you want to classify an entire video


Group Normalization (Paper Explained)

that's a lot of data right and you almost have to go small batch sizes for that they


Group Normalization (Paper Explained)

do a lot of experiments and generally as I said I believe these results for group norm


Group Normalization (Paper Explained)

have been replicated and across the community a bunch of times now and I would definitely


Group Normalization (Paper Explained)

consider group norm if you are thinking of a especially a distributed machine learning


Group Normalization (Paper Explained)

project all right with that I hope you enjoyed this paper I've been talking for way too long


Group Normalization (Paper Explained)

now I wish you a nice day if you haven't already please subscribe like share comment or whatever


Group Normalization (Paper Explained)


Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments (Review)

Hello, this is a comprehensive paper review on a paper called avoiding catastrophe, active


dendrites enable multitask learning in dynamic environments. This is a very cool paper because


it combines ideas that come from biology, which are active dendrites and ideas that come from


deep learning, namely the problems that we face in multitask learning and in continuous learning.


Catastrophic forgetting is one of the main problems of these areas and the method of active


dendrites directly inspired by biology can really help with that. So this video is a comprehensive


review on the method of active dendrites in deep learning as the paper describes it. By the end of


the video, you'll have a good understanding of what is in the paper. In the next video that I'll


publish tomorrow, there will be an interview with the authors, which was also super interesting. And


I definitely invite you to check out both. As always, if you have any comments, please leave


them in the comments on YouTube, leave a like if you do like the video, and I'll see you around.


Bye bye. Hello there. Today we're going to look at avoiding catastrophe, active dendrites enable


multitask learning in dynamic environments. This is by researchers of Nemento, Cornell and Stanford.


So this paper proposes to bring some of what has been lost in translation from real biological


neurons to deep learning neurons to bring some of that back into the deep learning neurons,


specifically the concept of what they call active dendrites, and also a bit of sparsity that is to


be found in biological neurons. So they bring these back into deep learning neural networks.


And it turns out that that is pretty useful to combat something known as catastrophic forgetting,


thus the title of the paper avoiding catastrophe. So catastrophic forgetting is a phenomenon where


in multitask learning or continual learning, a network has to learn many things at once. And


then these things interfere with one another. And it turns out that our methods of training neural


networks using backpropagation aren't really good at that. So either they don't learn any of the


tasks because they conflict with each other, or in continual learning, they do this catastrophic


forgetting where as soon as a new task comes in, they've completely forget about the old task. So


many solutions obviously have been proposed. And this right here isn't like is not entirely ultra


novel, but it is interesting. It ties together biology and sort of practical applied deep


learning. And it does have some connections to, for example, modern transformer architectures and


so on. So I'd also be interested to hear what you think how this stuff is all connected. So they


start out saying that a artificial neural networks, they call these ANNs. So whenever you in this


paper, ANNs means sort of the deep learning neural networks, we have to be a bit careful when we talk


about things that involve biology, because neural networks is an ambiguous term there, like the


neural networks is an ambiguous term because it appears in both domains. So they they claim they


fail dramatically when learning multiple tasks, a phenomenon known as catastrophic forgetting. And I


already said catastrophic forgetting, it essentially means that you can't learn many things at once.


So it says learning multiple sequential tasks can lead to significant interference between tasks.


They look at two different they look at two different tasks right here. One is multi task


reinforcement learning, and the other one is continual learning. So in multitask reinforcement


learning, it's essentially reinforcement learning with multiple tasks. So you're some sort of an


agent, and you're in some sort of environment, and you have this basic loop of sending an action and


getting back some kind of observation and reward. However, however, there are multi, there are many


tasks in this environment. So maybe you see it, and maybe you don't. That's part of the definition


of the problem. I think in this particular environment, you also get back kind of an


indicator of which let's call that T the task indicator. So which task you currently supposed to


fulfill. So the same environment has many tasks. And then obviously, your reward is going to be


dependent on which task is currently active. So you're going to give the agent a mixture. So every


new episode, the agent tackles the task is different. And therefore, if the agent just does


the same thing, as in the last episode, it might get a completely different reward because the task


is different, right. So that is multitask reinforcement learning. And it turns out that and


this papers have established this before, and I think we've even made a video on some of them, that


if you look at the gradients, they often conflict with one another. So learning one task would pull


a weight in some direction and learning another task would pull it sort of in a different direction.


And there are papers that try to make these gradients as like orthogonal as possible or


project them somehow into a task specific subspace. But as it stands, conflicting gradients can arise


in these multitask settings. And therefore, the classic way of training neural networks with back


propagation to update all the weights at the same time, just isn't very conducive, even worse in


continual learning. So here, we're not necessarily in reinforcement learning anymore, but although we


could be. So this is this is simply continual learning, where you present a neural network. So


you have a neural network, the neural network is able to, you know, take whatever picture, let's say


it's a picture classification and give you some sort of a class label for that picture. And now you


have different tasks. So you have task one, task one, might be classify, you know, classify cats


from dogs, then task two might be classify, I don't know, cows from beavers, task and so on. So


there is also a bit of a specification gap. Some of these continual learning benchmarks, they will


always have the same classes, but different data sets, some will have different classes, some will


have new classes, and so on. In this particular case, we're looking at permuted MNIST, which is


sort of the MNIST data set. So you know, there is whatever picture, and there is some sort of hand


written digit in here. And the the permuted MNIST data set is simply that every task that you


consider, so task one, would have a permutation applied to all the pixels in in this picture, but


always the same permutation. And then task two would apply sort of a different permutation,


permutation one, permutation two. So it's kind of a different task. It's the same classes, you're


still classifying digits into zero to nine, but the permutation is different. Therefore, it's like


you have to learn a new task if you don't have some sort of built in symmetry prior in your


neural network. Obviously, this, we're not going to use convnets right here, because convnets would


make no sense if your pixels are permuted. We're simply going to use feed forward networks. The goal


isn't to get state of the art. The goal is to show the difference between what if we use regular


neural networks, and you can imagine right here, if I train on task one right here, task one has


some kind of a permutation in the pixels, I'm able, you know, these neural networks, they're able to


learn that because if they're feed forward networks, they don't care about neighborhood


anyway. So they they are able to, you know, we train, we train these weights right here to to


completion. And then I activate task two, right, right after task one, I stop giving the network


data from task one, and I start giving it data from task two. So also different permutation. I


also label my images, give it to task two. Now, I'm going to train these weights, I continue


training these weights. And there is some effect when we talk about large language model pre


training in that whatever you pre train on that kind of stays around. So any fine tuning in large


language models isn't going to completely erase the pre training. So it actually matters what you


pre train. Although, this is not the same right here. First of all, we're dealing with way smaller


networks. And these way smaller networks, they're able to be kind of overwritten mostly. And also,


we're dealing with classification tasks right here and not some sort of language modeling task. So


yeah, these these weights, they will just be overridden to the point where task one is forgotten.


It's nowhere. So we've again, if we draw up some sort of a weight, task one would pull it in this


direction, that would be the gradient. So the weight would slowly update by update going this


direction. And then all of a sudden, we activate tasks to which will pull it in this direction. So


the weight would then travel into this direction, and essentially forget about task one. So it is


nowhere near where it should be for task one. As I said, there are some methods of solving this with


orthogonal projections and so on. But as a basic rule, our deep networks aren't very good at that.


So what what do we do about it? This paper's idea is that since our deep networks use a model of the


neuron that looks very much like the thing on the left, so you have your your input weights, which


are commonly known as the weight matrix or the weights of the layer. This is just one row or


column, I guess. Well, it depends on how you specify the layer. But these are just all the


input weights going into one neuron, they're summed up. So this is the matrix multiplication. And then


there is some sort of a non linearity right here, which could be a sigmoid, which could be a tanh,


which could be a relu. And that's essentially still the model that we have. This is like an over,


like it's it's decades old, this this model, and it served us pretty well, but it has forgotten some


very important aspect of biology. Here on the right, you see a pyramidal neuron, a pyramidal,


I'm just going to call it pyramidal because of pyramid. So this is obviously way different. So,


well, first of all, it's not a schematic, it's kind of like an actual drawing, you see the axon


right here, and the axon splits up into different parts, which is, you know, is like our regular


neurons, they connect to all the neurons in the next layer. Although one difference is you can


already see that there are way less connections from here than you would have in a fully connected


layer. So there is a degree of sparsity in biological neural networks that is not represented


in the deep neural networks that we build. And then the inputs right here, we just consider all


the inputs to be the same. However, there is a difference between what they call proximal inputs


and distal inputs. So proximal inputs would be inputs that are very close to the cell's body,


and those behave very much like the linear influence that we see in our model. However,


there are also these distal, by the way, these things are called dendrites, there's a difference


between the axon, which is this thing here, and the dendrites, which is this thing here. Every


neuron has one axon, but can have many, many dendrites. And dendrites are sort of like,


they're just kind of elongations of the cell body. So any other axon could dock either directly on


the cell body, or close to it, or could dock on any of the dendrites. So you can make connections


from axon to body or from axon to dendrites. And dendrites are kind of like harbors, like


ports or docks for incoming traffic. Yeah, that's how I can explain it. However, these distal


dendrites, they're not acting like as much as like linear things. What they are doing is, and this


paper describes that, is they act like their own little subunit that computes its own function. So


it's almost like a mini neuron inside a neuron, and that mini neuron can then influence or modulate


the cell body. So whenever that mini neuron is, for example, very high, is very activated,


it will raise or lower the activation threshold for the main cell body. So it can sort of influence


the main cell body in a multiplicative way. And that's exactly what we're going to see


in this architecture. So, yeah, I've sort of skipped a lot of the text right here. Yeah,


if you're a Patreon, you get these notes, I hope they help. I've never considered my scribbles to


be super duper helpful, but I've started pre annotating and I hope it helps someone. But yeah,


these are mostly for me to see what I have to what I have to look at. So what does that have to do


with continual learning? Well, they describe right here, they hypothesize that biological


properties of pyramidal neurons in the neocortex can enable targeted context specific representations


that avoid interference. So pyramidal neurons, which comprise most cells in the neocortex are


significantly more sophisticated, demonstrate a wide range of complex nonlinear dendrite specific


integrative properties. And they are hypothesizing that this modulation property that we've just


discussed, this modulation property could battle this catastrophic forgetting. Specifically,


what they say is that, well, we have many of these dendritic distal sub modules, and these could


learn and there are some biological evidence for that to recognize different contexts in which you


are in. And depending on which of these is active, that means which context is recognized, it can


modulate the body of the cell. So the cell could react differently depending on the context. And


that is one of the ingredients exactly that we need to avoid this catastrophic forgetting or do


multiple tasks at the same time is to say, hey, I'm only going to activate my cell body if I'm in


the correct context, meaning, for example, a particular task is active. So the cell body can


learn its weights to do to specialize on a given task and rely on the subunits to recognize when


it needs to fire. And obviously, if there's some structure to the tasks, we can also think of these


being sub tasks. So sub tasks are sort of being activated that can then generalize and be integrated


into multiple tasks and so on. So there's a bit bit of related work. The active dendrites that is


pretty much pretty much what I just described. You can see each distal dendritic segment act as a


separate active subunit performing its own local computation. When input to an active dendritic


segment reaches a threshold, the segment initiates a dendritic spike. So this is not a neural like


axon spike, it's a dendritic spike that travels to the cell body. Okay, I've apparently memorized


this passage, and can depolarize the neuron for an extended period of time, sometimes as long as half


a second, they don't model time dependency right here, by the way, that's something they don't


integrate right here. During this time, yet the cell is significantly closer to its firing threshold


and any new input is more likely to make the cell fire. This suggests that active dendrites have a


modulatory long lasting impact on the cell's response with very different role than proximal


or feed forward inputs. So they say they typically receive contextual input. That is a different


input than received in proximal segments, proximal are the near ones. These context signals can


arrive from other neurons in the same layer, neurons in other layers, or from the top down


feedback. Another thing they don't model right here is any sort of top down feedback or same


layer or anything like this, just, I'm just taking this away. What they do model is these dendritic


subunits. The second thing they're very interested in is sparsity. So sparse representations are


ubiquitous in biological neural networks, not so much in deep neural networks. They claim that


studies show that relatively few neurons spike in response to a sensory stimulus across multiple


sensory modalities. Sparsity is also present in the connectivity. And they claim that one advantage


of sparsity and representations is that vectors for two separate entities have low overlap. So


they're now talking about deep networks because biological networks don't have vectors. So they're


talking about how if you impose sparsity in a deep neural network, and you are in high dimensions,


then your representations likely will not collide, because a lot of the entries are zero. Low


representation overlap among unrelated inputs may be particularly useful when an artificial neural


network is learning multiple unrelated tasks. And that's why they are interested in the sparse


representations, because if different things don't, aren't likely to overlap, they're not likely to


interfere with each other. And therefore, they might be useful to combat catastrophic forgetting.


So two things, we're going to implement these active dendrites into our models. And also,


we're going to implement a degree of sparsity. And we're going to observe how these two things


work together to combat the catastrophic forgetting phenomenon. That is essentially what this paper


suggests. So let's look at exactly how they do this. I think it's best to jump to the model


right here. So this is one of the models or one of the architectures they use. This is the actual


arc, they use two layer neural networks. So yeah, these are not huge networks that they use


right here. It is for reinforcement learning. So it is kind of a soft actor critic, they use this


benchmark right here, where a robotic arm needs to perform multiple tasks in the same world. And in


this particular task, the agent always gets the information which task is active. So which task


is active goes into this context vector on the left, this is a one hot vector that is fed as a


context signal. What's special about this network is that first of all, you can see that there is a


linear layer. And that is not some classic linear layer that is a special linear layer, namely,


the active dendrite linear layer. So the active dendrite linear layer has a feed forward signal,


and that feed forward signal is treated just as a classic deep neural network feed forward signal.


So that would be the feed forward signal would essentially be whatever the input here is,


in this case, probably the robots state or something, and its position and it's maybe the


position of the whatever object it needs to grab, if that's not always at the same place and so on.


So that's the state input. And if it if we're only one task, the network could just learn from this


input. However, this is multiple tasks, so it gets the context vector. The alternative, the baseline,


what the baseline would do is it would append the context vector right here, and just sort of


extend this feed forward layer. And it would say, well, the network essentially has access to this


information right here in its input. So it should technically be able to handle that. However,


they're going to show that, you know, they're going to implement this in a baseline going to


show that that's not as helpful as what they're doing. So we have a feed forward signal, and that


computes some output, you can see that's independent of this context vector. So the feed


forward layer, the weights of the feed forward layer, which sit approximately here, they're going


to be, you know, multiplied by the weight matrix summed up. And then there's some output signal


right here, just in a classic feed forward layer, the context vector comes in here. And what it's


what it's going to do, remember, this is a one hot vector. For now, they make it more complicated


later, it is going to be matched with each of what these things are, these things are called dendritic


segments. So it is going to be matched with each of them. And the matching is simply done via an


inner product. That's what this little sum symbol does right here. So there's an inner product


between the context vector and the dendritic segment. And then they're going to select


whatever dendritic segment matched the highest, and that is going into here, and then here is a


modulation function. So the signal that is the highest, the highest inner product with whatever


dendritic segment is going out here and modulates that signal. And that's going to be the output.


Now let's look at how these dendritic segments work, because that's really sort of the meat


right here. Here you can see the forward signal, the forward signal is your classic signal right


here. There's a weight matrix or vector in this case, there's the input, there's a bias. Okay,


the dendritic segments are, they're just vectors. These are trained, okay, every single one of these


dendritic segments is a set of weights that is trained. And it's different as far as I can


understand, each neuron has its own dendritic segments and for each dendritic segments, it has


it has its own weights. So there's no weight sharing going on among the dendritic segments,


which would I think break the whole the whole thing, although I guess one could come up with


some sort of smart like meta weight sharing right here. But the idea is that, as you can see from


the formula, we're simply going to take the context vector, calculate the inner product with all of


these dendritic segments, take the max dendritic segment, that's going to be some kind of a number,


right? This is an inner product. So this is the strength of whichever dendritic segment matched


the most. And then we're going to take a non linearity, in this case, a sigmoid function,


and we're going to multiply the at the feed forward signal that we have with this sigmoid


function of the of this inner product. So this can, you know, the sigmoid is between zero and one,


I think, yeah, I think they retain the sign. So they take the max absolute value in the end. But


let's leave that out for now. So whichever segment matches the most, that's some number that goes


through a sigmoid. So let's think about this, when is this thing one? It's one whenever one of these


dendritic segments activated, right? So we take since we take the max, one of them needs to activate,


and then this thing is one. So the dendritic segments, they're sort of like, like receptors


for contexts that where this neuron could be relevant. So they are sort of like, you know,


feature detectors. And if they they expose some kind of some kind of vector, they are obviously


vectors. So in the space, there's like here, like, you know, I have maybe I have three of these


dendritic segments. And I say, well, I'm interested, if, if my representation, if my


context representation is any of those three in that direction, then I'm interested. So if the


context comes in like this, they they're just like not no one is interested, therefore, the sigmoid


maximum is going to be zero, and it's going to block the signal right here. However, if the


context comes in is very close to what one of these segments is, then it's like, oh, wow, this actually


might be relevant for this neuron. Therefore, the sigmoid, so the inner product is high, the sigmoid


of the inner product is high, and the signal is going to be propagated through. Interestingly,


in the experiments, they always expose like as many dendritic segments per neuron, as they have


tasks, which I thought to criticize that because I was like, well, that's kind of cheating. But now


I don't even know if that is that is necessarily like, wouldn't one dendritic segment suffice,


like if it could perfectly recognize if every neuron was only relevant for one task, and if


that could be perfectly recognized by the context vector, I guess that would, that would work. But


this is more powerful, right? You can present a number of situations where you would be interested


in. Ah, I guess, okay, if you have as many dendritic segments as you have tasks, then every


neuron could be relevant for every task. So a neuron could be relevant for all tasks, or for


just two of the tasks and so on. So yeah, I still maintain it's a bit of it's a bit of cheating to


make as many dendritic segments as you have, as you have tasks, because that's implicitly telling


the network how many tasks you have. But you do get you do get the task as the context. So you


already know anyway, right? In any case, that's that's what this network does. It exposes these


things, it's able to take this context signal and modulate that signal. The second thing it does is


this k winner takes all. And this is this is very much like maybe the sort of sparse mixture of


experts that you might know from from transformers or the concept. So what it does is it simply


calculates a maximum maximum activation over the entire layer. And it only lets through the highest,


the highest k many things. So it's k winner takes all k could be three or five or something like


this. But in any case, it is not as many as you have neurons and all the other neurons, they're


just set to zero, therefore, they also don't receive any gradient. So here you can see how these


two things play together. First of all, we're going to modulate. So we're going to block a lot


of the signals right here. Blocking means we're just going to multiply them by a very small number,


if they're not relevant. And then it's not just that they're very small. Actually, we're just


going to pick like the top five. So all the numbers that are small, we're just going to


eliminate completely. I don't know if this you know, this method of achieving sparsity is


necessarily the best one to pick the k best, or if it'd be better to just threshold somewhere,


because k then is some sort of other hyper parameter that you might, you know, set via


cheating, or that you might have to try out and some sort of a threshold might be more robust,


especially since the sigmoid is fairly, fairly steep function. Yeah, that's, that's the


architecture, essentially. So I hope you can see how this sort of connects to other things,


especially, I'm interested in this modulation property. And I'm also interested in the


sparsity approach. Obviously, if you have sparse representations, there's not going to be any


gradient flowing back through the neurons that weren't activated. And therefore, there's not


going to be any gradient into these neurons. That means these weights here aren't trained for that


particular neuron, it means these dendritic segments, which are, again, these are parameters,


trainable parameters. So these blue arrows are back propagate trainable, they will only update


if the neuron has actually been selected in its forward pass. So they're random at the beginning,


and then with time, they will fine tune for specific contexts. So they will sort of move.


And yeah, there's a bit of a danger that some of these are just become ghost parameters. But I


guess as stuff moves around, and as initializations are diverse and random enough, almost everything


will become sort of selected at some point, if your inputs are diverse enough. Yeah, so that's


that I've skipped a lot of these, a lot of the text right here, you can see the K, the KWTA,


the K winner takes all representation, we're simply going to let the signal through,


if it's in the top K activations, and it's zero otherwise. Yeah.


Exactly. So here they say only the neurons that were selected by the WTA function will have non


zero activations and thus non zero gradients, only the weights corresponding to those neurons will be


updated. And that's how the two things work together to battle catastrophic forgetting, in


that if the context, if the dendritic segments successfully learn to recognize different tasks,


that means that only the neurons that are involved in a particular tasks will be updated


by that task. And therefore, the network will not forget the other tasks or not forget them as


easily, because the sparsity also the sparsity kind of forces not all parameters to be updated,


and the dendritic segments forces these sparse updates to be in a very structured,


very consistent fashion. And yeah, they also say that only the dendritic segment


J that was chosen by the max operator is updated, all other segments remain untouched. So even if a


neuron is part of this K top K activations, only one dendritic segment is updated, namely the one


that matched the most with the context. And this again ensures that maybe if a neuron is relevant


to different tasks, the other dendritic segments they can, they can keep their place, even if we


train in a new task where this neuron is also relevant, if it was relevant to an old task,


that might be stored in a different dendritic segment than the one that is activated right now.


And that dendritic segment due to the max operator will not receive a gradient and will just


remain as it is. Of course, this doesn't scale, you know, forever, and to all degrees of noise.


And there is a, there is a way in which tasks can be too related. So I would guess that in a model


like this, if tasks are very related, they will activate the same dendritic segments and therefore


override each other. But then also, if tasks are very related, you would expect that there is some


form of generalization or crossover among them. But the difficulty has never been that much with


generalization, it has always been with the fact that if you think of, for example, large language


models, I also think of large language models as continual training, they often they don't even run


an it was single epoch over some of the data, and they still learn from it. So they see a data point


once right and, and then, you know, that's that's that, and they still are able to incorporate that


somehow. So how are they not subject to catastrophic forgetting, they also in a way


implement different tasks, because I can query GPT-3 with so much stuff, like it can do so much


different diverse things, it is all it is like a bit of, you know, sure, it's always the same loss,


and the gradients don't necessarily conflict of that loss, but it's kind of a multitask learning.


And one key difference is that GPT-3 is presented with sort of an IID shuffled sample of the training


data. However, here, the all the data of task one comes first, and then all the data of tasks two


comes later. So even if there's some generalization aspect, I would expect if tasks are close


together, task two will override task one, because the same dendritic segments might activate. And


just from the model here, they don't have a way to, I feel they don't have a way to battle that


maybe they are there of a different opinion. But maybe some sort of, how should I say this,


some sort of a contrastive method, like a contrastive addition to these dendritic segments,


like pushing them apart from each other for different tasks, you know, if they have the task


information, or just plain pushing them apart from each other, maybe hallucinating pseudo tasks for


that may be a way to to automatically adjust to how close together or far apart the different tasks


are. Yeah, that's just my, what I would guess might help. But maybe I'm completely wrong. Tell


me what you think. They say we hypothesize that a functional specialization will emerge where


different dendritic segments will each learn to identify specific context vectors. So that's the


model. Now they go into the experiments. As we already said, they do two things, multitask


reinforcement learning, this is this robot thing. So it's all at the same time. In this particular


case, it's not one after another, it's all at the same time, I think each batch is always from the


same task, but like the next batch will be of a different tasks, I think, yeah, but it's different


tasks, right? So the same actions don't lead to the same reward. And that is means conflicting


gradients. They use a very basic RL algorithm right here, which is not necessarily important


for our discussion. Just to say that the networks are quite small, right? They have two hidden


layers, each with 2800 neurons, which okay, that's sizable. So they're quite fat hidden layers,


but it's just two of them. And then each one is followed by a K winner takes all activation


function. And then there's a final output layer. They say the first layer has standard neurons,


whereas the second layer hidden, the second hidden layer contains active dendrite neurons,


which are modulated by the context vector. In this case, the context vector just encodes the task ID


as a one hot vector. And yeah, each active dendrite neuron in our network has exactly 10


dendritic segments, the same as the number of tasks to learn, they do ablations where they


increase that number of dendritic segments. But yeah, I do think they're giving their model the


absolute best chance to learn right here, by setting some of these parameters with essentially,


okay, it's not hidden information in this particular case, but it is in the next case,


where we're not getting the task ID, as you will see. So this is how the model looks. There's the


state vector, there's feed forward, we have some sparsity enforced by these. Notice that it's


really interesting that sparsity is even enforced here without any without any modulation. And they


do also some ablations on that. But I'd be interested why they didn't choose to also have


dendritic segments in the first layer. It seems quite odd, honestly, to set up an experiment like


this. Yeah. And the other thing is they say, although we control the hidden sizes to yield


approximately the same number of total non-zero parameters, we note that MLP baseline contains


nearly 500k more non-zero parameters than our active dendrite networks. They speak a lot of


these non-zero parameters, and they count the network sizes in non-zero parameters. So I would


be interested what are, what's the difference between parameters and non-zero parameters,


and what was it a non-zero? I don't, I've not seen this exactly explained in the paper. Is that


like at the end of training, if a parameter is zero, you don't count it? Or is it somehow different?


I don't know. But safe to say they do try to make the networks as you know, with the same number


of parameters, which means that if they have these dendritic segments, which are quite a number of


parameters, they have to, I mean, not that many compared, but they have to turn down


the other parameters. So here you can see the results at the beginning, the active dendrites


network in blue is sort of underperforming, but then it overtakes the baseline, the MLP baseline.


And yeah, the errors here, the variances are quite large, as you can see. They do run another


analysis where they just select the top five for each. And you can see that it separates a bit more


cleanly. Although I'm not sure if that is like, is that a thing? Like, can you say I'm just going to


select like the top five of each to reduce the variance? I'm not sure if the max distribution is


the same as the mean distribution. Like, could I do that in practice? Maybe not. If I just have


one run, which is essentially what I'd want to do in practice, I couldn't necessarily do that.


I don't know. In any case, they beat the MLP baseline in both cases. You can see that sometimes


there are pretty significant differences, especially in what they claim are the harder task,


like the pick place tasks. And these are also the tasks that have very little overlap with the other


tasks. So you would expect greater interference. And that's where they have a lot of gains against


the baselines. In continual learning, they use this permuted MNIST as we've discussed. And so


yeah, here's sort of the comparison. Yeah, you can see, also you can see here the variants are huge


for some of these tasks. Yeah, in the permuted MNIST dataset, they, okay, they don't have a


graph, I believe, but in the permuted MNIST dataset, they also are beating or advancing against


the baseline significantly. So we have somewhere, there are the results.


So you can see right here, there isn't a baseline in this particular diagram, but you can see that


the drop off is not very steep. And usually if you do this with regular MLPs, they just fail,


like they fail, which means that, so this test accuracy is on all the tasks you've seen. So


far, so you get presented with whatever, 20 tasks in sequence, and then you evaluate on all of them.


And regular MLPs, they just suck at this, like they forget the previous tasks. And yeah, that's


that. So the fact that these networks are able to sort of hold up across, and here you can see up


to like a hundred tasks is already pretty remarkable. They have two different variants. One


where the prototype is given while training, which essentially means they have information about which


task they're in. And one is where the prototype is inferred and they describe these up here.


So what they do, they now switch over from not providing the task ID as a context signal, because


that's kind of cheating. And they provide now this prototype. So what is a prototype? A prototype is


essentially a data point, or it can be a latent vector, but here I think it's just a data point


that is kind of the mean data point. So this will be the prototype of task A, the mean data point


of all the data points in a particular task. So they provide that as the context, as the


context signal. Now, what they can do now is, here you can see how that works. It's just a mean


I told you what they can do is if they don't have a task annotation, if they don't know what task


goes with a particular data point, they can simply collect data points during training. They can say,


well, here's a data point, here's one, here's one, and here's one, right? And it helps that they have


the guarantee that each batch has the same task. And then they say, well, okay, we're going to make


a prototype right here, and that's going to be our context vector. And then the next batch comes in,


and it's kind of like over here, and they say, well, this is not very close, so we're going to


make a new prototype right here. And then the next batch comes in, and it's like here, and they say,


ah, that's probably of the same thing again, so we're going to use that prototype to provide to


the system. So it's kind of this heuristic thing, averaging the data points, which I find to be


quite weak, like averaging the pure data points is like, it might work in permuted MNIST, but there's


definitely room for improvement right there, because that that is not going to be informative


at all in in many or most tasks. And obviously, there's also like a hyper parameter to set like


you know, what's what's the appropriate distance measure right here. And also, this just going into


this as the context signal. And the context signal is essentially just worked out by inner product,


as we saw up, sorry, up here. So the signal is just, it's just an inner product with some of


these u vectors. If this gets any more complicated, there's going to need to be a lot of machinery


in front of the context vector, like, I would expect, we need to pass it at least through some


hidden layers to compute something of value. But for permuted MNIST, it's going to be enough, right?


So they recognize which tasks they're in. Now, I am interested, why exactly they switched from


providing the task ID, like, at least in first in a first instance, and why they switched over to


providing these prototypes right here, as the context signal, right, just experimentally,


they have this one experiment in this one setting, where they, they just provide the


task ID. And then they have the other setting where they do something different. I would,


I would get it if they did both things in the same setting. But having two different settings and just


doing two different things is a bit suspicious, I guess. And also here you can see they provided


actually to both layers, and not just to one layer. I would like to know the story of this


and not just to one layer. I would like to know the story behind this. They also compare to a


baseline, which is called SI. So SI, as they describe here, it is a thing that operates


solely at the level of synapses, it maintains an additional parameter per weight that controls the


speed of weights adapting to specific tasks. The two approaches are complementary, that's why they


can be combined. You can see on the right, so on the left hand side, you can see what happens if


you infer these prototypes during training. And you can see it's just a little bit worse, which I


think is like 100%. So I don't know how much better or worse they would be if they actually gave the


task ID. But I think this distance right here, that is only going to be possible on non-permuted


MNIST. Maybe I'm wrong, maybe I'm wrong. So here you can see, interestingly, right, here is the


active dendrites. This is kind of the curve from the left. And then these SI method just by itself


actually beats the active dendrites. However, you can combine both, as you can see, and both


together are stronger and give you an even better, better boost. So that is, I mean, it's


good if you can combine all the tricks that you had so far. I would have liked to have here like a


like, okay, the MLPs, they just suck, because right now it's not exactly clear how much they suck.


Although I'm sure that there's some appendix table and I haven't looked, I haven't found it. The


paper is quite long. So here they compare to a different method, which is called


XDG, which is context dependent gating, sorry. They say this is the implementation closest to


theirs. This is another idea. However, that one uses hard coded distinct sub network for each task.


So this is pre allocated. It pre allocates as you sub network, you're for task one, you're for task


two, you're for task three, they engineer this in a way where they expect some overlap between the


tasks and some separate neurons. And then they only train the sub network. So they need the task


ID to be provided. The implementation invokes task specific subset of the hidden layer. Other neurons


are forced to have an activation value of zero. This requires a task ID that determines exactly


which neurons to turn on or off. It turns out, so the way they emphasize all of this is that it


turns out that they do beat the baseline, as you can see right here, when you just do them by


themselves. But as soon as you combine them with this SI technique, the XDG outperforms the active


dendrites. So obviously, they need to highlight the differences right here, which is a good tactic,


right? And it's valid. They do do more. So here they say task information is inferred, it's not


provided via this prototyping, where this provides a system with a task ID during training and


testing, right? And it's important to see that even if they do the prototyping with the information


of the task ID, they claim that during inference time, there is no task ID provided. And they


simply, you know, they see whatever if a data point is whatever prototype the data point is closest


to, that's the prototype they take. The second thing, sub networks automatically emerge via the


use of dendritic segments in their model, whereas the baseline, it pre allocates different sub


networks for each tasks. And that's, that's legitimate. However, I don't, I can't shake the


feeling that they've like evaluated it. And then this thing was better. And they were like, ah,


rats, now what can we what can we do? Okay, we can't beat it. How can we make it? How can we make


it different enough? And maybe that's when they decided, okay, let's try to like not provide the


task ID, but let's try to come up with like, a dynamic way of figuring out the task or something


like this. Maybe that's the story behind why this prototyping exists. Or maybe that has like, that


just turned out like it is. I don't know. But, you know, it's interesting. It's interesting to see


sort of, there might, there might be a research process behind this. And which is cool, because


the research process sort of leads to more innovation, which is neat. There is an important


question, one that which I also had during reading of this paper. And no, that's not it. This is,


we're gonna get to that. First, they check their hypotheses. So they say the hypotheses of our work


are twofold. First, active dendrite networks modulate an individual neurons activations for


each task. Second, the winner takes all activations use this modulation to activate sub networks that


correspond to each task. They provide some evidence for this. So here, on the left and the right, you


see the two tasks they tackle. And they give you an impression of which hidden units are active for


for which particular task. And they you can see that it's fairly sparse. So if you look at any given


column or at any given row, then not many light up in dark green, which means that not many things


are activated per tasks and a given unit is kind of specialized to particular tasks or a particular


set of tasks. Now, without a comparison to a sort of regular neural network, or without a comparison


to one of the two features of the network ablated, it's kind of hard to see whether this is a lot or


not a lot, especially on the on the right, you can also see like, is this sparse? Or is this not


sparse? I don't know. I'm gonna guess it is. Yeah, so I don't know, I'm gonna believe them that this


is especially sparse. And I think they also measured it at some point, actually the sparsity,


but just the graphic alone isn't necessarily enough for me. They look at single neurons. So


in the single neuron, they wonder which dendritic segment is responding to which task, right? There's


a neuron A and neuron B. And you can see at initialization, a lot of the segments are responding


to a lot of the tasks. However, after learning, it becomes much more quiet, and only very few segments


are responding to any or each of the tasks. However, also here, first of all, it's not super


clear what we are to compare this with, because this could just be a phenomenon of kind of like


the scale of stuff being wrong, like at initialization, just the scaling of things being


kind of out of whack, because you can see right here, there are entire regions that are just kind


of dimming down, right? So yeah, obviously, a given neuron isn't going to respond to all the


tasks, right? With all the segments, it's not going to be involved in all of the tasks that would


actually, you know, this is a valid prediction of their hypotheses. And you can also see that,


especially neuron B here, if you look at segment eight, multiple dendritic segments are reacting


to signal eight, which might be an indication that there is some, you know, they have learned to


recognize different features that all indicate that four, no, segment eight responds to multiple


tasks. Ah, okay, that's different. Okay, negate my argument, forget what I said. I thought


it was a smart recognition. But, you know, it is definitely evidence for the fact that there's


specialization going on. But without a comparison to anything, it's hard to tell if that is that, or


just some sort of a scaling issue that just after training, things are scaled differently. But just,


you know, from all the other evidence, they make a convincing case that there is this


sparsity and specialization going on. So here is the last thing I want to discuss. And this is a


question that I had when reading this paper, which is, aren't like, isn't this, isn't there an


equivalence to larger networks? Like, aren't you just sort of, sort of, you know, designing this


network in this special way? And can't I achieve the same thing with sort of a regular neural


network if I just make it a bit larger? They say multiple studies have suggested that dendritic


computations performed by pyramidal neurons can be approximated by artificial neural networks that


have one or more hidden layers. From a computational and deep learning perspective, this is equivalent


to claiming that ANNs with dendrites can be substituted by larger ANNs without dendrites,


without dendrites, as proposed. And I have tried, so they are going to make the case right here


that that is not the case that they are outperforming, for example, three layer MLPs,


which are about the same size, and MLPs that are much larger, so much deeper. So they're going to


outperform them at you can see right here, number of tasks 100. Oh, this is, this is probably the


graph I was looking for before. No. Yeah. So here you can see how much, how much the the MLPs suck.


So yeah, they show that even if you scale them up, in fact, the 10 layer MLP is even worse,


which is interesting, which might be might be interesting in itself. Like, why is it? Why is


it worse? And is there like a crossover point here? But in any case, these MLPs, they get the


context vector as an input, right? So technically, technically, they have all the information to do


the same thing. However, the paper argues that it's the training procedure, back propagation,


updating all the weights for the given data that is presented to us. This is particular to an IID


setting of data, which we don't have right here. So no matter how big you make your neural network,


supposedly, if they're correct, this, it would always result in the same problems due to the way


that you train them. On the left, you see an ablation of the two ingredients. So the active


dendrites only the sparse representations only, and the combination. One second.


So they do certainly give empirical evidence. And by the way, here is also an ablation on having


more dendritic segments. On the top, they're trying to learn 10 tasks. On the bottom, they're trying


to learn 150 tasks. And it's interesting to see that the gains here are kind of negligible,


although maybe that's just a property that they're very close to 100% already. And here you can kind


of see gains until 50. And then, well, okay, I might be imagining things that there's stronger


gains here than here, after you pass sort of the number of tasks barrier. But safe to say that,


you know, more more dendritic segments might also be useful. And maybe my skepticism of them setting


parameters exactly, exactly as many as sort of exactly to the number of tasks they have is not


super warranted. Also interesting is the the fixed number of dendritic segments and varying activation


density level. So here is this K, so how many things they let through each layer, you can see


a increases to the right, this would be 100%, which would regress to a classic MLP. See if you activate


100%, it's really bad. And there are two things right here. Again, they're trying to learn 10 tasks


or 50 tasks. Interestingly, interestingly, if at the beginning, obviously, that nothing through,


it kind of sucks, then you let some things through, it's already really good. And then it gets better.


So there's some kind of an optimum around 10% ish or so. Interestingly, that's the case for both


the things, even though one is trying to learn significantly more tasks, which is interesting,


right? Then there is a drop off for both things, which you would expect. But then there is kind of


like a flat flattening, followed by another drop off. And it's also interesting to, to think about


why that's the case. So here, it might be that this is the situation where very few things are


overlapping. And therefore, the network is able to use specialized subnetworks for all the things


that it needs to do. And in this entire region up until here, it might be the case, you see,


it kind of drops off at the end after like 80%. It might be the case that most of the things are


shared. However, the network can kind of encode stuff in the non shared part. And that can itself


within the network kind of modulate whatever the shared stuff is doing. It's kind of like a shared


feature extractor, followed by some modulation of the non shared parts. I would Yeah, it's interesting


to think and then that crashes together once there is no more non shared parts. And there's no way


of doing anything different in the different task settings. I was thinking myself, you know,


getting back, sorry, getting back to, can I just achieve the same thing with a larger network?


I was thinking myself of how to do that. So they claim, no, you cannot. And I guess it's true.


Let's think of, okay, this, let's leave the sparsity away. Let's just think of this dendritic


activation, right? I have my x, that's multiplied by by w. And let's also leave the biases away. So


I have my x vector down here, I have some w, which is a weight matrix. So everything's connected to


everything till here. Now, can I also and I have my context vector, can I somehow build a feed


forward network that would also, you know, have the appropriate weight connections that I could


build myself the function w x times sigmoid, you see, let's also leave away the max right,


right here, I guess, okay, we can't, that's an integral part. And yeah, it's not clear to me


how that would work necessarily with a single layer. And it's also not entirely clear to me


how that would work with multiple layers, like, you would have to build some very, like various


contraptions of additions, maybe, you know, once you get a ReLU out on all of that, it might be


more possible. But it's not easy to get this multiplicative interactions between signals


working in a feed forward network. However, however, in transformers, that might be different,


right? So, you know, this here, this, you know, we can do this in transformers, I guess,


in feed forward networks too. And then the max, we have we have soft maxes in transformers, right?


So what we could do is we could have these things here as let's call them queries, right? And these


things here are the keys. And we apply the softmax in a transformer. And the values might just be a


constant vector of ones. So the values might just be constant vector of ones, which would mean that


if we multiply the softmax by this thing, we would simply select sort of the maximum out of that,


and that's going to be one and everything else might be zero. Maybe I might, maybe I have this


wrong, but maybe not. Yeah, I guess that that would work, right? So and then in the next layer,


so that could be our output signal for layer one. And that could be our output signal for layer one


in a different attention head. And then the multiplicative interaction, again, we can get by