you what, show you what the benchmark looks like. So the benchmark has this particular


Neural Architecture Search without Training (Paper Explained)

form, this particular form, there's the skeleton. And in this skeleton, there is this block,


and it's always repeated. And you're basically your task is to determine what this block


should be. So this block has an input node a and an output node D, and two intermediate


nodes. And what you have to do is basically you have to determine these connections right


here. So there are six connections. And for each one, you have the option of putting different


things there. Like you can see you put can put a convolution, you can put the identity


function, which is a skip connection, zero wise, I'm I don't, maybe that's the zero function.


So it basically means nothing. I'm not so sure, honestly. But you could technically


put a convolution here and here, right? Or, and or different convolutions or things like


this. So there are these 15,625 possible cells, okay. So the NAS benchmark contains 15,625


possible architectures that you'll have to search. And they take these architectures


and they plot now they plot for each architecture, the validation accuracy after training and


the training protocol is standardized, you don't have to care about that, right. And


the score that they measure at the beginning of training. And what you can see is that


there is a linear relationship sort of like a sort of from from these experiments, what


you'll get is like this sort of feeling. What they're going to propose is that you should


take that score as a as a measure. And here again, also, sort of sort sort of there is


a there is a clear trend, as you can see, right here, though. Yeah. Though this, as


you can see, this sort of spreads out and the most right one is ImageNet, which is the


most difficult one, of course. So, and this is CIFAR 100, which is more difficult than


CIFAR 10. So we can see that this sort of relationship at the top, it doesn't really


hold anymore if the task gets difficult. And this is so what I think is happening, this


is kind of an interjection of my own opinion. What's happening here is that this score that


they discover, allows them pretty efficiently to see which networks are just degenerate,


and cannot be trained. Like if you try to train them, they just perform really poorly.


Okay. That it's probably a very good score for weeding those out. And that would mean


if you kind of barrier here somewhere, right, you could just discard a whole lot of this


crap, or even even here, right, you could just discard a whole lot of this crap. And


also, now here, just, you know, all of this crap. Yeah, whereas here, as you can see,


some of this score, sometimes it's higher than these ones, even though they perform


better. And again, you could probably discard a lot of the crap. But it's not as distinctive


for the well performing networks, because these here are all not the degenerate version,


right? They're not degenerate in the sense that they're, they have some fundamental flaw


where the function lacks now expressivity from the very start, so you can't train it.


And then probably other factors come into play other factors, then you can simply determine


with this particular score. But, you know, there is this relationship that's, that's,


you know, you can see that. And they do some ablations on this here. For example, are your


scores a proxy for a number of parameters? And they say, no, the number of parameters


works way worse than this particular score, which you know, is a is a cool thing, then


how important is the specific mini batch and initialization? And they say, look, right


here, we for some architectures, we do different mini batch sizes. And you can see each of


those groups, they don't vary too much in how they're it influences their score. This


is, I believe this is the same architecture. So it's always an architecture that achieves


in this case, for example, wow, that's not a straight line. 77% or so. And you can see


if you go for different mini batches, the score varies only minimally. initialization


is a bigger variance inducing thing. But also here, the scores don't vary too much. But


it is interesting that the different initialization to get you to different score, because it


would directly support kind of my hypothesis that what's going on here is that you sort


of measure initial degeneracies. And you can sort of make up for these initial degeneracies


in the architecture sometimes with sort of a different initialization. So the different


initializations give you differently performing networks. We already know this from things


like you know, lottery ticket hypothesis and so on, that the initialization can matter


to some degree in these types of things. Now, that being said, they always train to the


same, it seems, but their their score varies. So I might be backwards correct here or not


correct. But in any case, the initialization here matters more, but also you can still


see this linear relationship. And this is particularly interesting. This is even the


case when you just input white noise. So instead of the data, you measure that score by just


inputting noise that I guess has some sort of the same magnitude as the data would have,


but it's just noise. And you can still sort of see this linear relationship, which is


very interesting. And that I think also shows some that you what you find is a property


of the network itself. And the fact that it is initialized and built in such a way that


it allows you to train it in a very, in a sort of a benign manner, it has no degeneracies.


Okay. So in last experiment, they go here and they say, we evaluated the score on initialized


networks in the PyTorch CV library. So they go to this library that has a lot of these


networks. But these networks are not the same as this benchmark. This benchmark is specifically


designed to do architecture search. Now the networks in this library, they are all designed


to perform really well. Some are designed to be quite small, some are designed to be


quite fast and so on. But in general, they are all of their goal is to perform well.


And they have been sort of found by humans to perform well. So they take now these networks


on CIFAR-10 and they test them. So as you can see here, here is the test accuracy again,


and here is their score that they give it. And they say, I can't move this anymore. Hello.


Well, okay. They say that this linear relationship still sort of holds. It doesn't hold super,


super well. But you can still sort of if you squint, if you squint hard, you can see that


it sort of goes upward, though, you really have to squint hard, like what are these things


right here. And what, again, what's the case is that if the score is low, you will sort


of be able to cut off the cut off the worst performing ones. But really at the top here,


it doesn't seem like there is a particular relation between between these networks and


this initial score, which sort of strengthens my hypothesis that what this does is just


kind of weed out the bad ones. But it's pretty cool because you can weed out the bad ones


without any training, right? You'd simply forward prop, backward prop, there you have


it. So cool. Now they come they, here is the experiment where they now really do this NAS


benchmark and they compare with other methods. So some of these other methods are designed


to do the call weight sharing, which basically is a technique where you can sort of speed


up the speed up the algorithm as compared to non weight sharing and the non weight sharing.


That's one of these we have discussed initially. That was my initial example with the controller


and so on, where it takes super long. So here you see the method and how long each method


takes. Now the best ones, as you can see already, the best ones here, or these these methods


right here are the best ones, they score somewhat like a 93.9 or so on CIFAR 10. Whereas these


weight sharing ones, they don't perform too well, except this one seems to perform quite


well. And in this hours case, they perform worse than that, but they still perform better


than a lot of the weight sharing ones. So what their point is basically is that they


get a pretty good score, which is a 91.5 on CIFAR 10, which is, you know, it's at least


not degenerate. It's a good accuracy. They score that with simply evaluating 10 architectures,


right? And as n goes up as they evaluate more and more architectures, they do they do get


better, but not much. So they have a discussion here. I'm having trouble moving this. All


right, so we'll sort of go through the discussion, we report results, yada, yada, yada, yada.


As the setup the non weight sharing methods are given a time budget of 12,000 seconds


for our method and the non weight sharing methods are averaged. Accuracies are averaged


over 500 runs. For weight sharing methods, accuracies are reported over three runs. With


the exception of GDAS, our method is able to outperform all the weight sharing methods


while requiring a fraction of the search time. And that you may see at the table, this is


the real, I mean, this is the real deal here, they only use here 1.7 seconds compared to


the 12,000 seconds of the other methods, and you reach almost the same accuracy. Now to


be said, 2% in this particular regime on CIFAR 10 is still a sizable difference. And that's


the same benchmark, right? With the same sort of the same training schedule, and so on.


So there's not too much room to tune here, you simply have to find a better architecture.


So these things are still sizably ahead of this. And what it appears to me that these


methods here that don't perform well, they're they're simply crap. It seems they're simply,


I don't I don't know, but they might be trying out something or, you know, doing something


researchy or whatnot. But it seems like if you're well able to weed out the bad architectures,


you might be getting to a score like this. And then if you are actually performing a


search to find the best one, then you might be getting to somewhere like this. And you


can see this here throughout. So in CIFAR 100, they achieve a better score than these


things, but a worse score than the non weight sharing method. And in ImageNet, it gets even


the difference is even larger. So, again, what I can see here is that there's is a good


method to maybe get you like, let's say 90% of the way of the way you want to go. And


what's interesting is that here they say, we also show the effect of sample size, we


showed accuracy of the networks chosen by our method for each end. So that's the sample


size, we list the optimal accuracy for sample sizes 10 and 100 and random selection over


the whole benchmark. So in this case, they have the the optimal one, which I guess they


just draw 10 samples and then take the best one. So they train all of them and then take


the best one, you can see that already gets you to the 93. And whereas in their case,


sometimes when they add more, they get worse. So here they get better, but then they get


worse again. So they comment on this right here. We observe that the sample size does


not have a large effect on the accuracy of our method. But note that as sample size increases,


our method suffers from a small amount of noise, increasing the gap between our score


and the optimal result. And of course, the key practical benefit is execution time. So


again, they are massively faster than the other methods. But to me, it seems you could


just think of combining these methods, right, you combine this with this, in that what you


want to do is actually actively search for the best ones. But by doing so you could if


you could pretty quickly weed out the bad ones using this method down here, you might


already have like a big speed up. Because, again, with comparison to this random ones,


what appears to happen is that they get good at finding, you know, your 90% architecture,


but then they fail to differentiate the top performance performers from each other, where


you'd really have to train the network to find out what's you know, which one's better.


So yeah, here they say they visualize the trade off between search time and accuracy


for c410 for different NES algorithms on the NES benchmark are moving the need for training


our method is able to find accurate networks in seconds instead of hours. And here you


can see the accuracy and here you can see the time and all the good ones are either


way over here or here. And there's is almost at zero while being quite close to the accuracy


of the other ones. All right, yeah, that was that was this paper. Again, I think this is


pretty valuable if you are especially if you're in a new domain, where you might not know


what kind of network to build, you might just be able to write a little script that generates


networks, run it through this algorithm. And at least you get an idea of which ones are


certainly not worth considering. And then you can simply select one of the other ones


it doesn't you know, often it doesn't need to be the best ones. And you can then tweak


it a little bit manually the ones you found, maybe you see some regularity. And yeah, that


was my two cents on this paper. I hope you liked it. If you did consider sharing it out


and telling your friends about it and subscribing liking and leave a comment if you agree or


Neural Networks from Scratch - P.4 Batches, Layers, and Objects

What's going on everybody welcome to part 4 of the neural networks from scratch videos.


In this video what we are going to be doing is taking our currently a single sample of


inputs and converting that to be a batch of inputs.


We'll explain why and how momentarily.


Also we are currently modeling a single layer of neurons and what we would like to do is


model two layers of neurons which then we could extrapolate out to however many layers


of neurons we want at which point it will become pretty apparent that we would benefit


pretty greatly from converting this to from this code now to an object instead.


So we'll be getting into doing object oriented programming creating our first layer object.


With that in mind as a reminder this series is based on the neural networks from scratch


book which you can get at nnfs.io at the time of my recording this this book is in draft


form but if you pre-order any version of the book you get within hours access to the ebook


draft version which is currently in a google doc so you can go in there highlight ask questions


comment in line with the actual text also the draft at some point it won't be a draft


anymore obviously the book is currently scheduled to ship out September 2020 and but right now


the draft version of the book is quite a bit further ahead than we are right now covering


everything from obviously creating the the neurons in the neural network creating a full


neural network doing a full forward pass obviously with also activation functions calculating


loss doing the partial derivatives back propagation with various optimizers also regularization


techniques like dropout and l1 l2 regularization so quite a bit further ahead so if you want


to get a jump on the material or you just want to review the material before or after


the videos that's a great way to do it and again access to the ebook version is given


within hours I'm still doing that manually so it's just if I'm around I'll give access


so as far as I know no one has waited more than a day for the access so anyway let's


get into the code now so the first thing that we're going to cover is batches first of all


why batch so there's kind of two major reasons for batches the first one's pretty obvious


and simple to understand we can can we can calculate these things in parallel so the


first thing that allow that batches allow us to do it's and it's really a cherry on


top I'm not sure I think this is just kind of a nice thing that happens I don't really


think this is the primary reason I would say batch but because we're calculating things


in parallel the bigger the batch the more parallel operations that we can run now this


is also why we tend to do neural network training on GPUs rather than doing them on CPUs so


I don't know what the average CPU core account is these days some are probably between four


and eight that's not very many cores compared to your typical GPU which is going to have


hundreds or thousands of these cores that we can actually run calculations on so that's


why it's so much faster on a GPU so and it's and really the CPU cores right those are meant


for much harder calculations than what we're doing here which is typically like matrix


multiplications and pretty simple calculations so so that's why we use GPUs rather than CPUs


now that's a benefit of doing things with batches but the other reason why we do things


in batches is because it helps with generalization so if you just think for a moment so as a


reminder what are what are inputs right now it's a single sample and that sample currently


contains four features so given our let's let's continue the example of hey we're trying


to predict whether or not a server is going to fail obviously we've made up all of these


values but just pretend for a moment in that example maybe each of these features is a


reading from a sensor so this could be your temperature sensor this is your humidity sensor


value and so on so these are these four values are values from unique sensors at a specific


point in time so that's a single sample now we could convert that to be a batch of samples


and have many readings of of of sensors at any specific point in time and then we're


going to attempt to fit those to something now right now we don't have the intended output


or anything like that we're not calculating loss yet or anything like that so for now


just make sure you have a good idea of what what are these inputs right now they are features


from a single sample and so this this describes the current status of this server at a single


point in time right now and what we want to be able to do is pass a batch of these samples


instead so so the second reason why this is helpful is it helps us to generalize because


rather than showing the machine the algorithm one example at a time and then it's basically


going to try to fit to that sample and then the next sample and the next sample we can


instead show it multiple at a time that way it's much more likely to generalize and I


think the best way to understand that concept is to just visualize it so let's check that


out so to visualize the concept of batches and generalization what we're going to do


is we're going to take a theoretical sample set of 512 samples and then we're going to


ask a single neuron to attempt to fit to that sample now what's going on here is we're just


fitting one sample at a time to this neuron and you can just see that this neuron is just


bouncing around with that fitment line so that line there is the fitment and the dots


that you're seeing these are individual samples so we're just asking it to fit to this one


then this one then this one this one it's very confusing and just imagine yourself if


you were shown just one sample at a time and you were asked draw the fitment line to represent


the entire set of samples you would have a probably a pretty hard time doing it right


whereas when we show multiple samples or you can see all of the samples kind of fading


into the background here you can see okay I could probably what like if you just had


a better representation of all the samples you could probably do a better job of fitting


to those samples so if we increase the batch size to 4 we can see that it does a better


job there's a lot less wiggling around now we expect that when we're just showing a small


number of samples at a time that the fitment line is going to move a little bit we expect


that but you can already see how much less movement we're seeing so it's already become


much easier for this neuron to fit this exact same sample set when we increase the batch


size to 4 if we continue to increase to 16 we see even less movement for the most part


right this line is pretty much in the same kind of standard deviation range once it catches


up via the learning rate you don't need to understand what the learning rate is necessarily


doing here just know the learning rate is kind of the way of kind of moving forward


so how much of the previous knowledge do we want to keep versus the new knowledge or something


like that we'll get to learning rate soon but I think if you just imagine again yourself


like if you were trying to create a fitment line yourself imagine you are the neuron if


you're just showing one sample at a time that's very difficult whereas if you're showing 32


at a time suddenly it becomes a little easier now you might wonder why not all the samples


at a time this can generally the problem with this is it can cause over fitment so if you


show all the samples at once now you're telling the neural network hey just fit to all of


these and the neural network is going to try to fit as good as possible to all the samples


at once which is actually likely to hurt your generalization because now it has knowledge


of all of the samples and it's going to actually fit your in sample data it will fit your in


sample data likely pretty good and then do pretty bad on your out of sample data so you


don't really want to pass all of your data at once and you don't necessarily want a huge


batch size either generally something between 32 64 maybe as high as 128 that's pretty rare


a batch size of 32 is actually a pretty common batch size you know from what I've seen different


problems will vary with batch size for sure but that is the concept of batch size alright


so now that we're all convinced of the wonders of batches let's convert our single sample


of inputs to a batch of inputs so right now it's a list that means we're going to convert


it to be a list of lists and then we're going to match the values from the book so the second


sample here is 2.0 5.0 and then it's negative 1.0 2.0 the third sample is negative 1.5 2.7


3.3 and negative 0.8 the next question you should ask yourself is there anything that


we need to change about weights and biases do we need to add weights or any biases or


anything like that just as a general review of how things are working so the answer to


that is of course no because weights and biases are associated with individual kind of unique


neurons are we adding any neurons to like this in theory is our layer are we adding


any neurons to our layer by adding more of a batch of inputs has anything changed there


no so we don't need to change anything in our weights and our biases getting to the


output is there anything that we need to change in this dot product or adding the biases or


anything like that so now that both weights and inputs are going to be a matrix let's


talk briefly about the matrix product and how that works essentially it's going to be


very similar except for the fact that now that the inputs are going to be a matrix let's


talk about how that how we do a matrix dot another matrix so to do this what we do is


we take that first row vector from the first input matrix so in this case it was matrix


a and then we're going to do the dot product of every single column vector from matrix


b this results in the scalar which becomes the first value in that first row vector for


the output matrix so again still the first row vector in that a matrix we're going to


do the dot product with the second column then the third column fourth column and fifth


column of matrix b and these individual scalar values as the result of that dot product become


the first row then we continue on now we're at that second row that second row vector


from the first matrix and so on until we've populated the entire output of this matrix


product so remember in the previous tutorial that we hit an error because because we were


passing inputs by weights right and recall that the size at index at the index of one


of the first element in the dot product needs to match the size at index zero of the second


thing that we passed so in this case at index zero so weights is what it's a three by four


so that means inputs needed to be the four by none really or four by one that it was


so now is that true does that still is that still the case well no right because actually


inputs and weights are currently the same shape so if we go to run this we're going


to see uh-oh right we get the shape error right and again just like I said before this


is like a perfect shape error that you probably would see fairly frequently in other deep


learning frameworks and then you would probably you know google or duck duck go search to


figure out what is the answer to this problem because it doesn't make sense to you but it's


pretty clear if you now that you understand how this works that it's telling you right


now you know you've got a dim one here that's of size four right so that's weights it's


telling you right now at this at the first of index so the zero is this stuff so you


have three sets of weights at the first of index well how many weights do we have in


each weight set four so it's telling you right now you got four here but unfortunately at


dim zero of inputs right you have uh three right you got three in this batch so you've


got three samples this is a problem and it doesn't get any better by changing things


so if we refer back to inputs by weights for example we get another uh shape error because


it still hasn't changed we have to uh we have to do something else so what do we do because


so let's go back to what are we trying to do what we are trying to do is inputs times


the weights so what's happening right now is the dot product is attempting to take this


value multiplied by this value then this value multiplied by this value this value multiplied


by this value and then we get to 2.5 and the dot product is like what what do i do it doesn't


have a value instead what we're trying to do is this value times this value this value


times this value and so on all the way through four and then suddenly uh the shapes are correct


so how do we do this so what we really would want to do is take this weights matrix and


we want to switch rows and columns right that's what we that's how we would solve this problem


so luckily for us that's called transpose so again the concept of a transpose is just


to swap rows and columns so all you're going to do is take the first row that becomes a


column you're going to take the second row that becomes a column and so on throughout


this entire matrix and that's all there is to a transpose of a matrix all right so let's


go ahead and transpose the weights now to do that we're actually going to convert weights


to a numpy array now why do we need to do that so right now inputs and weights are lists


of lists in python when we pass those to numpy's dot product function here what's happening


is in the back end these inputs in this weight these are actually getting converted to numpy


arrays but since we want to use numpy to transpose the weights we actually are going to convert


weights now to numpy array weights and then to transpose that it's actually just dot capital


T boom transpose let's go ahead and run that and what we get is an actual batch of outputs


so we should be able to recognize this one is this is the same one that we've been seeing


uh so far which is the outputs from the three neurons from uh this as your input and then


these are some new outputs from those same three neurons based on the the other inputs


that we just just started adding so okay pretty cool now the next thing that we're going to


talk about is adding one more layer here but before we do that let's make sure everyone's


with us as far as what happened why did we have to transpose here so before we move along


to adding another layer let's let's review what's happened so far so initially we were


able to do the inputs and the weights without a transpose because the weights was a matrix


but inputs was just a vector so in this case the shapes lined up we could do a dot product


as long as the weights came first and everything was fine but when inputs became a batch of


inputs the shapes did not work anymore because what we need is index one of that first element


needs to match index zero of that second element so once we transpose weights suddenly it was


able to work again so let's review the actual calculations that have taken place in our


example the first thing we do is transpose our weights so that we could even do the matrix


product then we take the matrix of our inputs dot the weights which means we're going to


take each individual row vector from the inputs matrix and do the dot product with each column


from the transposed weight matrix doing this for each of the row vectors in the inputs


matrix which gives us our final output for this matrix product okay so our final calculation


is to add this biases row vector to our matrix product output from the inputs in the transposed


weights to add a row vector to a matrix all we need to do is we are going to just add


this biases row vector to each of the rows from that matrix product output to do that


all you're doing is adding each element together so at the same index so it's 2.8 plus 2 6.9


plus 3 and negative point 59 plus point 5 okay so now what we're going to do is add


another layer and to do that we just need another set of weights and biases and then


basically the size of weights that's how many neurons we're going to have the length or


size of biases again that's how many neurons we're going to have so for now we'll just


go with the exact same sized layer just for simplicity's sake and just so you understand


how this is kind of working and then what we'll do is convert this to an object so again


like before I am going to copy the values from the book so we're going to say 0.1 negative


0.14 0.5 for that first set of weights the second set of weights will be negative 0.5


0.12 negative 0.33 and that final set of weights is going to be negative 0.44 0.73 and that


negative 0.13 and then finally the biases that will go with negative 1 2 and negative


0.5 so that is for our second layer now what we're going to do is rather than just calling


this straight up output we're going to call this layer layer 1 underscore outputs so this


is now the outputs of layer 1 which then become the inputs for layer 2 so all we're going


to say is what we're going to do is we're going to take this line copy we paste this


line and then we're going to rather than inputs here this becomes layer 1 inputs we'll call


this layer 2 outputs and then so the inputs change now to be the inputs from layer 1 now


what we want to do is take weights 2 biases 2 and then what we can do is we can print


you know layer 2 outputs at this point go ahead and run that and just to confirm that


is the same from the book yes it is cool so okay so that's as easy as adding another layer


now of course you know you might want to be able to really quickly change the size of


that layer like how many neurons are you going to have and adding more layers and of course


at this point we should hopefully recognize that this is very quickly going to be quite


quite unruly so instead of doing things this way what we're going to do is convert this


to be an object right so to convert the concept of layers up to this point what we're going


to do is we'll keep inputs and these will be this will serve as actually our sample


data and instead what I'm going to do is I'm going to delete basically all of this and


we will start coding the other thing I will do is I'm going to I'm going to change the


name of inputs to X as a capital X it's kind of a standard with machine learning that your


your input feature sets is going to be denoted by a capital X so you'll have X and maybe


you'll have X train and X test and so on but for now we'll just make it a capital X so


that signals that this is our actual training data set now so we'll assume this is the input


data to the neural network and then now what we're going to define is the two hidden layers


and again they're called hidden only because we as the programmer don't really specify


how that layer we're not in charge of how that layer changes I suppose like we kind


of define you know how we define the optimizer we define various hyper parameters for the


optimizer and stuff like that but but the neural network is responsible for tweaking


that whereas the input data you know a lot of times what we're going to do as a programmer


is we're going to scale that maybe between 0 and 1 or negative 1 and positive 1 and so


on for now we'll just leave it as is but that is going to be our input data so in this case


we'll say we've got three samples so so that's that the next thing that we're going to do


is let's just go ahead and class and we're going to call this layer underscore dense


no performs they're required and then we're going to define our init method in this case


we'll just pass self for now and then I'm going to hit pass and then I'm also going


to define the forward method and again we'll pass self and then we'll also just say pass


for now and so let's talk about how we initialize a layer so in the case of neural networks


there's kind of two ways you're going to initialize a layer the first is you've got a trained


model that you saved and you actually want to load in that model so the what happens


when you save a model what all you're doing is you're actually just saving the weights


and the biases so when you load the model that's all you're doing is you're setting


the weights and biases as whatever they were in that saved model in this case though let's


say we're making a new neural network what how do we initialize well you've got weights


and biases that need to be initialized so first we're going to initialize weights the


way that we tend to initialize weights is as random values between in the range of negative


one and positive one but the smaller that range or the more tight that range is probably


the better we want generally with neural networks we want small values so the reason


why we want small values is we hope that things continue to tend in the range of negative


one to positive one now just as a single example of why we do that is if you have input data


and the weight is say 1.2 or 1.5 or 5 or some larger value what you're doing is data passes


through this network is you're making it bigger and then for each of those connections if


each of those weights is above one you're making it bigger than bigger than bigger than


bigger and then you get this like explosion so one of the things that we try to do is


keep including with our input data so in this case we have a few values that are that exceed


the range so one of the things you might do is first normalize this data data set and


then scale this data set and by scale we generally mean take it and change the values so that


like kind of the meaning stays the same but the scale has changed in such a way that these


values are all between negative one and positive one well we will talk about that later but


in this case when we initialize our layers we want to kind of adhere to this principle


of keeping values small rather than letting them even at the outset just explode so for


our weights you know we you know maybe something between the values of negative 0.1 and positive


0.1 that's maybe a good starting point for weights then for biases we tend to initialize


those as just zero now there's times where you might not want to do that because if for


whatever reason your neurons aren't firing initially so if they they go through the neural


network inputs times the weights it's not big enough to actually produce an output your


bias being zero means it's that neuron is going to output a zero so then what happens


well that output zero gets multiplied by whatever the weight is for the next you know the next


connection zero times any weight is still going to be zero if that neurons bias is also


zero it's going to output zero and then boom you've just propagated through this entire


network all zeros and the network is dead so to speak so if that if you're having a


problem like that just know okay maybe we need to start the biases off as a nonzero


number like I said in general that won't be an issue but if that is an issue that is that's


one solution and again that's why we learn about neural networks from scratch so that


when we hit issues like this we know ah it's probably a dead network okay how do I solve


that I maybe want to initialize biases of something other than zero so with that in


mind let's do it so for the weights we want we're going to use random and numpy and just


so everybody following along hopefully we'll get the same values we're going to set np.random.seed


we're going to set this to zero so for initialization we need weights and biases so we're going to


say self dot weights and this is going to be equal to np.random.randm and then we need


the shape that we want to pass so in this case we need to know kind of two things from


the programmer when this layer is created which is what's the size of the input coming


in and then what is the size how many neurons do we want to have so in this case what is


the size of the input coming into this layer it will be the size of a single sample so


in this case you're going to pass a batch but in this case it's four right so you would


say you would say four but instead what we're going to say is n inputs to the network and


so what we want from numpy is to create a weights that will be the size of n inputs


multiplied by how many neurons do we want to have so we'll say n neurons and then what


we're going to do is in the parameters that we're going to add are going to be n inputs


and neurons so when we create this dense layer we'll specify these values so that is np.random.randm


n inputs and neurons now just to show what this does because we are actually going to


we're going to make this even smaller let's go ahead and print and we'll paste this in


and let's just say the n inputs is four and we're going to have three neurons we can save


and run that and you can see here is our weights so what randm does is it's just a Gaussian


distribution bounded around zero but as you can see some of these values are actually


bigger than one so what we want to do is instead let's just say 0.10 times and then run it


again I don't know why why is that not running there we go okay and then this looks a little


bit more like what we hope our weights to look like so that's what we're going to do


here so we're just going to say 0.10 times and that will be our weights and this and


again this is just the shape that it makes for us and then we're going to say self.biases


and we're going to say that is equal to np.zeros and in this case our biases the shape is one


by however many neurons that we have so actually two things to point out before we move on


to this forward method first of all the input to np.zeros to pass the shape actually the


first parameter is the shape so we need to actually pass the actual shape as a parameter


so it'll be just a tuple of the shape versus np.random.randn where the parameters are the


shape so just kind of note those differences but actually both of these functions return


a matrix of the shape of you know this and this and or the shape of this and this so


kind of confusing I almost messed that up so make sure you don't so the other thing


I want to point out is the shape of weights so how did we shape weights we are saying


this matrix is n inputs by the number of neurons so we're viewing part three what was the shape


of weights the shape of weights in this case was actually the number of neurons by the


number of inputs so it's the opposite so why did we do that so the reason why we're doing


that is when we go to do our forward pass if we have shaped our weights and so in this


way already when we do the forward pass we don't have to do a transpose every time we


do a forward pass it's just a silly operation to be doing every time when now we've got


full control right here in the initialization so what we're going to do is actually just


not need the transpose anymore it just makes a lot of sense so take note of that as well


so for the forward method the forward method is going to just take inputs so whatever inputs


are this could be really one of two things it's either going to be if this is the first


hidden layer the inputs are going to be your actual your actual training data or the inputs


will be the self dot output from the previous layer now the self dot output this is going


to be mp dot dot and in this case it's going to be your whatever the inputs are multiplied


by the self dot weights and then of course plus self dot biases and that is your forward


method so let's make use of this and then we'll be done with this tutorial so the first


thing we're going to do is we're going to make our two layers so we're going to say


layer one equals a layer dense object and we specify basically here the size of the


inputs and then the how many neurons do we want to have so the size of the inputs we


already know that it is so the inputs it's basically how many features in each sample


so in this case we have one two three four so we know the input size is going to be four


and then for number of neurons it's anything you want you can make this any number you


want I'm going to for now just say five so we can look at it on the output but you could


make this five twelve it does it it's whatever you want so and that's why we want to make


this an object because it's going to it just makes so much more sense it's so much more


dynamic so so that's our layer one and then we can go ahead and just specify our layer


two now the only requirement for layer two is that the output from layer one is going


to be the input to layer two therefore if this is five on the output layer two has to


be of shape five for the input but again the output can be any shape or any size that you


want so we'll say two okay again it could be anything you want I'm going to choose two


so now we have these two layer objects now we can pass data through these layer objects


and when we do that we get a self dot output for that layer so first let's start with layer


one so we're going to say layer one dot and we'll do a forward pass or the forward method


and we'll pass in the x data when we do that that means we will now have a layer one dot


output value so let's go ahead and print that so print layer one dot output and we can see


here that sure enough because or how big was the data that we passed we had three samples


so we get three outputs basically of this batch and each one has how many values well


five because that's how many neurons we said this layer has and again you could have picked


any number here so now layer one output becomes the input to layer two so we will say layer


two and then we do dot forward and we will say layer one dot output and then what we


can do is print layer two dot output and let me just silence this one so it's super clear


go ahead and run that and what we get is again of size three because that was the batch of


the input data that we passed we now get three examples of these two neuron outputs based


on the data that we passed in so pretty cool this has been a pretty long time to get to


such a basic concept but as you hopefully can see there's so much that even in this


super basic concept of inputs times the weights plus bias so much has gone into that just


alone that hopefully at this point I think it's pretty cool to take your input data and


pass it through this network and get you know whatever shape you could possibly desire I


just think that concept alone is actually pretty cool so anyway that is our dense layer


object at least to the point of doing a forward pass of some data now at this point the next


kind of major thing that we're missing is in our activation functions so right now we're


just kind of passing data we do have these weights and these biases but the next kind


of major element that we tend to tack on here is going to be an activation function so that's


going to be the subject of the next video and then of course as we continue to build


after that we get to finally trying to figure out okay well what's the loss of this of this


network and then we start to determine okay well how can we optimize these weights and


these biases to decrease that loss and get the values that we want so pretty cool stuff


to look forward to beginning to get into the more sexy parts of deep learning hopefully


you guys are enjoying again you can check out the book at nnfs.io where we are already


at the sexy parts of deep learning also if you have questions comments concerns whatever


VOS: Learning What You Don't Know by Virtual Outlier Synthesis (Paper Explained)

Outliers, we all know them, we all hate them.


How can these data points just be out of distribution, not in the training data, things that we haven't


seen before, things that we don't even expect.


Well they suck.


So today we're going to look at what you can do about it.


Specifically we're going to look at the paper learning what you don't know by virtual outlier


synthesis.


This paper presents a technique to generate what it calls virtual outliers, which are


synthetic data points that are out of distribution.


The core idea is that rather than trying to come up with data space out of distribution


samples, this paper comes up with latent space out of distribution samples, which is much


easier and much more useful.


They're then designing a loss that pushes down the energy of the model wherever the


outliers are and pushes up the energy wherever the data is.


This paper is really interesting because it presented very successful results on a multitude


of benchmarks.


So definitely this technique looks like it works.


However when I read the paper I was quite critical.


I had a lot of criticisms, I had a lot of open questions and that's why I've invited


the authors for an interview to the channel.


So this video right here is a comprehensive paper review.


I'll explain in detail what is in the paper, what the method does, what its contributions


are, what its experimental results look like, what is good about it and what I think is


bad about it.


Then in the next video released tomorrow I'll interview the authors of the paper.


The authors will have seen my review and therefore are able to respond to any criticism and any


questions that I had.


So be sure to check out the interview part as well because it was really really cool


to get all my questions answered.


As always let me know how I can improve these videos by leaving a comment, leave a like


if you do like and I'll see you around.


Bye bye.


Do you have audio of someone talking?


Do you want that transcribed?


Boy do I have the product for you.


Assembly AI builds accurate speech-to-text APIs which means that developers can use these


APIs to automatically transcribe and understand audio and video data in just a few lines of


code.


This works in the traditional way where you upload audio and you get back the transcription


but they can also do this real time so you get a web socket to their neural network powered


back end and in real time it gives you back text for your speech.


That's insane.


But this is not all they have a ton of features on top of that.


For example they can do summarization, they can do topic detection, they can do bad word


detection, content moderation in your audio.


And I have to say this is really good.


In fact I have uploaded this video right here to their APIs and the text you see on screen


is the raw output of that model.


So judge yourself how good it is.


We'll actually try some Swiss German words on it.


It is an English model but we'll just give it a shot.


Hohe Haschle, Gelretle, Half a Khas.


Oh well isn't that great.


So give them a try they even have a basic free tier their documentation is super extensive.


They give you walkthroughs and examples of all the parameters that you can send.


They have a great blog where they describe different feature sets and different ways


of applying their technology.


And yeah it's a really cool thing.


Now I've only scratched the surface right here they do much more.


They have features upon features on this but it's best you check them out yourself.


So thank you very much to Assembly AI for sponsoring this video is really great.


Please check them out.


A link is in the description and I wish you a lot of fun.


Hello there today we'll look at VOS learning what you don't know by virtual outlier synthesis


by Xuefeng Du, Zhao Ning Wang, Mu Cai and Yixuan Li.


This paper presents a model that can do out of distribution detection in object detection


networks.


But not only in object detection they show it on object detection but it is a general


framework for detecting out of distribution data at inference time.


If this really works this could mean a lot for especially for safety critical applications


networks that are deployed as a classifier or a detector somewhere and they would be


able to recognize accurately when they are presented with something they didn't learn


at training time like some out of distribution class.


In this particular case on the left here you see an image which is an object detection


network at inference time.


It has correctly recognized the car on the right hand side.


However it thinks that the moose here is a pedestrian.


It doesn't even classify all of the moose but it recognizes there is an object and the


class is pedestrian probably because it hasn't seen moose's meese.


What's the plural of moose?


In any case it hasn't seen a moose or multiple meese at training time and therefore it cannot


classify it and very often these networks make very very high confidence predictions


for classes that they haven't seen.


This paper tackles this and proposes this technique called virtual outlier synthesis


to which we'll get to in a second.


As I said it's a general framework they demonstrated on object detection which is a particularly


hard task but this could also be applied to image classification.


They do make the point that if you have an image like this and you haven't seen the moose


class during training most of the image will still be in distribution like this will not


be a particularly out of distribution image except for that small part with the moose.


However if you do object detection then the object itself here is out of distribution


and maybe that makes actually their tasks as researchers a bit more easy because they


are less often in these ambiguous cases where like half the data point is out of distribution.


In any case they mention here that the networks that we currently have they often struggle


to handle the unknowns and they assign high posterior probability for out of distribution


test inputs.


Now why might that be?


If you train a typical classifier the classifier will just attempt to separate classes from


each other you see this here in the middle.


This is a projection of the last layer of a neural network right before the classifier


layer so right before the softmax.


So the last the classification layer all it can do is it can lay linear decision boundaries


essentially through the distribution of data points.


So what the model does is it sees three classes right here.


So this is class one this is class two this is class three and what it needs to do is


linearly separate them.


So it says well okay I'm gonna this is not an ideal color for this.


I'm going to just put my decision boundaries like this and now I've essentially separated


the classes because all that is important to a classification loss is that you know


points in class three are away from points in class one and away from points in class


two.


So that also means that the more away from classes one and two I go the better like the


more likely it is to be class three because all I've ever seen at training is samples


from class three and my entire objective was just to make it to push it away or distinguish


it discriminated from class one and class two.


So obviously if I go more into the direction of class three the network will become will


output a more and more confident number about this being class three even though as you


can see the data is all in this region right here and out there there is no data yet the


network is still very very confident.


Red here means quite confident.


An ideal situation would be if the network was very confident where the training data


is right here.


However again we have the decision boundaries like this.


However if you go further out it will say something like wait a minute even though this


is not class one for sure and not class two for sure it's most likely class three but


still I haven't seen any training data around that area.


So I'm also going to be to just output low a low probability or a low confidence score


I'm going to say it's class three but I'm going to assign it a low confidence because


I haven't seen actual training data in that vicinity.


Now this all seems intuitive and makes sense and so on.


Mostly that is because low dimensionality and high dimensionality data is very different


and can deceive if you look at it in this in a kind of a very simple projection like


this.


If you're a human you see this data and you go like of course that makes total sense.


However this becomes very different if you look at high dimensional data.


Note that there is a reason why our classifiers do the thing on the left because the thing


on the right essentially amounts to like a probabilistic model of the data distribution


right.


The thing on the right it has an idea where all the data is right.


The thing on the left it just needs to separate data from each other.


Three lines are enough for that.


The thing on the right actually needs to model the data in the latent space which can become


pretty complicated in high dimensions and it needs some very very distinct assumptions


to make it tractable.


So the right thing is essentially a generative model of the data like a distributional model


of the data which needs a lot more resources and power and could pull away resources from


the classification task to be solved.


So what does this model do?


First of all they have some notation right here which I found to be well let's just first


look at the diagram right here.


So this is the whole model architecture.


They have an input over here.


So there's input input X right.


I'm going to use the green highlighter I guess for this stuff.


There's input X.


You can see this is the input image.


In general first you have this proposal generator and that proposal generator will generate


bounding boxes.


So some of these detection networks they have two stages first proposal generation and then


a sort of a post processing stage where they assign labels to the proposals.


So the proposal generator would simply ask where are objects you know any sort of object.


The objectness property it sort of generalizes between objects so it makes sense to train


the object detector to just predict where are bounding boxes.


In this case it would predict well there is one here there is an object and there is an


object here and then it would pass on those two to the classifier to determine what's


in the bounding boxes and you can already see the object detector has done a good job.


It detected that this thing right here is an object.


However the classifier it what can it do other it has to assign a label.


There is no option for it to say no actually this isn't an object and previous methods


have tried this they've just added like an extra class for outlier.


It usually doesn't work too well because the reason is pretty simple.


In order to do that here on the left you'd have to introduce like another line and say


okay so I'm going to introduce another line I'm running out of colors here introduce another


line you know like right here so this would now be outlier sorry outlier space.


Well that doesn't cover that doesn't cover this region or this region or the region back


here right so having a single class for outliers is sort of useless because there are just


so many places where outliers could be and not just like a single a single slice of the


space.


So you'd have to have many you'd actually have to have like a lot and ultimately that


amounts to exactly the situation on the right where you know ultimately you're going to


train a classifier that is a threshold between low and high density areas and that's exactly


a generative model of the data.


All right first stage is the bounding box proposal this thing right here and you pass


on the bounding box to multiple things.


First of all there is a loss that's simply concerned with did you detect the objects


correctly so during training the proposal generator would simply be trained with that


loss right here.


Now everything here is back propagated obviously but that would be the main loss to localize


the bounding boxes.


The second stage here would be the assignment of a label this would be the so-called classification


head.


So that would take the latent representation that is generated including the bounding box


right so we're going to feed this through a neural network and that will give us a latent


representation this H thing mean that they call that the latent representation right


before the classification layer and the classification layer would assign a label to it and that


would be the normal way of doing things and now we augment that by a bit.


Just to say they formulate this here as saying we have a data set the data set here contains


X is data B is bounding box and Y is labels so B and Y would be the labels right that


those would be the things to predict and then they say they split it up into two things


so first of all the P of the bounding box and then the one of the label and I don't


think that's correct I think that's a typo right here I think this should be the probability


of the bounding box given X not the label and this should probably be the probability


of the label given X as well as the predicted bounding box let's call this B hat right here


the predicted bounding box of B hat would be sampled from this but this is minor because


the rest of the paper essentially treats it as I think I write it down in any case what


they do in addition to that is they also have this classifier right here the classifier


that takes into a sample and the bounding box and it tries to predict this number G


and G is 1 if the object is in distribution and G should be 0 if it's out of distribution


so this is a binary classifier that classifies any sample into in or out of distribution


independent of what the classifier head says what class it is so that would amount to the


situation on the right where if you're anywhere in this region right here the classifier would


still say well that's clearly class 3 because that's the region of class 3 but your other


classifier would say yes but the outlier probability is very high the inlier probability is very


low for that region so you can do outlier detection at inference time now how do we


do this we do this by generating these virtual outliers during training virtual outliers


are essentially outlier data points that you synthesize now you what you could do and they


mentioned that what you could do is you could train like again you can simply train a generative


model of the data and then use that to sample out of distribution data however they mentioned


that synthesizing images in the high-dimensional pixel space can be difficult to optimize instead


our key idea is to synthesize virtual outliers in the feature space so the feature space


is if you have your have your image right let's just talk about classifier you feed


it through a bunch of neural networks and then here is the last layer and all you do


at the end is you have a classification head that classifies it into multiple classes and


this right here is just described by a matrix W this is just a linear layer that goes from


the amount of features I guess D or something like this to the amount of classes C that's


the dimensionality so in this space at the end you would do in this space right here


that's the space we've seen in in these diagrams up there here is where we would sample the


virtual outliers so what we would do is we would look at our training data where does


our training data fall and we say aha okay there is class 1, 2 and 3 as we had it then


we build a Gaussian mixture model of the training data essentially we'd assume that each class


has is described well by a high-dimensional by a multivariate Gaussian they all share


the covariance matrix by the way and then we would say well okay given that that is


the case which ends up at the situation on in the right we would sample we'd sample data


points from outside of those Gaussians so that have a sufficiently low probability so


these would be these virtual outliers we would just sample them anywhere where we where our


Gaussian mixture model says that there is no data but still we sample according to the


Gaussians so we're not going to be like way out here in undefined space just because this


is in our support set we're still going to sample from these Gaussians but we're going


to sample until we get a sample that has a very low likelihood so we're deliberately


going to sample outliers from these Gaussians and that those are going to serve as samples


for our outlier classifier so then the outlier classifier what it needs to do is it needs


to find a decision boundary between these virtual outliers and and the data you can


see him draw this right here so there's going to be a decision boundary now you can see


this decision boundary gets quite a bit more complicated than the decision boundary of


the of between the classes especially you know given that we do it in the last layer


so we'll go on in in the paper a little bit what we just said is going to come up in a


second here so they say we assume the feature representation of object instances forms a


class conditional multivariate Gaussian distribution and they state this right here so every class


has a mean all the classes share a covariance matrix and they do calculate they don't learn


these things they do just calculate them from the training data in an online fashion so


this is in the penultimate layer of the neural network as I just said yeah they compute empirical


class mean and covariance of training samples and they do this in an online sorry about


that in an online estimation fashion which means that as they train the network they


collect the training data and then in an online fashion they compute these metrics to always


be up to date they do say here we assume the feature representation is this Gaussian they


say see figure three and figure three is a UMAP projection of UMAP visualization of feature


embeddings of the Pascal VOC data set and I'm not sure what they mean by look at figure


three this is a UMAP this is like a a projection a nonlinear projection into low dimensional


space if I'm I'm not exactly remembering what UMAP does but for sure this is a projection


like this doesn't convince me that the data is Gaussian it convinces me that the data


is kind of in one place ish right like or like it convinces me that all the blue points


are closer or most of the blue points are closer to each other than they are close to


for example the green points here like that that is what is convincing to me from this


graphic it is not at all convincing that in the original high dimensional space where


they come from they are somehow a cluster or a Gaussian even or even that all of these


classes would have the same covariance matrix even if they were Gaussians so that that is


it is a wild assumption but you know it seems to work so the result of the paper or that


they are very very good at this outlier detection they reduce false positive rates by a lot


so you know it seems to work I'm just saying this does not convince me or maybe I don't


understand UMAP maybe there is something so here is where they say they sample the virtual


outliers from in this feature representation space using the multivariate distributions


so they would simply sample the virtual outliers from the Gaussians but the Gauss but then


evaluate them and only take them if they are their likelihood is smaller than some epsilon


they say it's efficiently small so that samples sample outliers are near the class boundary


yeah these outliers would then be converted to to the output so this would be the output


the classifier head by the classifier matrix now that is that is how they sample the outliers


and you know all all good so far I have a few concerns right here for example what you're


going to teach the model is you know successfully if in the last layer before the classifier


there is a data point and that data point does not is not where the training data is


then if this model works it will in fact it will recognize it as an outlier right what


will not happen and this seems yeah okay what will not be the case if if that that moose


right here for some reason right an earlier layer already confuses it with something right


an earlier layer thinks oh this you know it's four legs it's probably like it looks like


a dog right then the moose will come will come to lie really inside of the dog class


because it would have the features of a dog which the lower layers would have confused


it so you'd have to have done this technique in one of the lower layers and there you could


see that that this isn't an outlier but the lower the layers you go you know the less


your data the even less your data looks like a Gaussian I mean ultimately you'd have to


do it in the input layer right and there it becomes clear that this is just like a distribution


of the data that you're trying to approximate and in the input layer certainly this is not


Gaussian at all so I think this only works for specific outliers if there is an outlier


that as I say has like the same features as some in distribution data resulting that in


the last layer they are in like inside of this cluster then this method will not be


able to detect it yeah that is that is kind of my one concern the other concern I've already


said is that this separating these outliers is naturally a harder task because yeah as


well it essentially amounts to a generative or a distributional model of the data rather


than just a discriminative classifier so how are they incorporating this into training


during training we still we still don't know right we have so up here right we have our


loss right here for the localization we have a classification loss which is is fine is


good the sort of classification loss tells us if we have the class correctly but we still


need a third thing which is this uncertainty loss we are going to estimate the uncertainty


which is going to be our measure of how much out of this how much the model thinks that


this is an out of distribution data point or not and how are they doing it they are


using the log partition function for that so the log partition function is a it's this


thing right here it's essentially essentially what is at the bottom of the softmax if you


use a softmax for classification so if the F here is the logit of class K so if this


is the output of your classifier and then you do a softmax in the last layer across


your logits the softmax would look like this right so you'd have the class Y at the top


and then you'd have that log sum X of the of all the classes at the bottom so the bottom


right here is kind of like a measure of how peaky your distribution is right if if you're


if your logits are you know one is just standing out heavily then that is kind of a measure


for low uncertainty like you're quite sure about what you're you're doing and if the


all the logits are kind of the same then this they they are they're all more even so this


measure is a little bit of an indicator of certainty right so this was already this was


already shown to be an effective uncertainty measurement for out of distribution detection


so what we're going to do is we're going to use this as a uncertainty loss right here


so what we're going to do is we're going to to train or not to train we're going to have


a log logit based loss so we're going to say we are going to use a sigmoid and what we


want is we want this measure right here we want the we want this right here which is


one is the one is the logit and one is one minus the logit I can't remember which one


is which in any case we want this measure to be high for in distribution data and low


for out of distribution data or the other way around I want the uncertainty to be high


for out of distribution data and low for in distribution data so if we get a data point


we'll plug it in to this free energy well the by the way this the negative of the log


partition function is called the free energy sorry I forgot to mention that that would


make some connections to other even other fields of science so we're going to take our


data point and we're going to not plug it into the classifier but just this bottom part


of the classifier right to measure is the distribution that we're getting very certain


or very uncertain and then what we want is that if we have a true data point then we


want the we want the uncertainty to be very low if we have a fake data point we want the


uncertainty to be very high so by adding this loss right here by adding this loss what this


does is this trains our classifier to be more certain if the data point is real and less


certain if the data point is fake which ultimately right will result in decision boundaries like


this or or certainty estimates like this on the right here so the certainty estimate on


the left would just be if we just train the classifier objective the thing will get more


and more certain as we go away from the classification boundaries if we look at this certainty measure


and now we explicitly train the model to only be certain around the data and to be again


very uncertain around all the virtual all the virtual outliers so that's why you see


blue anywhere away from the data we explicitly train the model to to do that so our uncertainty


classifier that we talked about where was it this thing right here our uncertainty classifier


is not in fact an additionally trained model it is simply us plugging a data point into


this uncertainty measure and during training we make sure that this measure is low for


fake data and high for clean data now this loss if I see this correctly this uncertainty


loss initially it will only it will directly affect this parameter set right here since


we only generate the fake data in the last layer the only parameters that are really affected


by this loss in that case is the classification weights right here however implicitly obviously


by saying that the true data here must have a high certainty or a low uncertainty and


by contrasting this with the fake data in the last layer it may also be that through


back propagation the entire network is shaped such that the latent space will be more optimal


for doing this classification however right I cannot conceive super well how this you


know how all the effects and counter effects and so on are gonna work out but it would


be interesting to think a bit more clearly through that yeah so what we're gonna end


up with is a probabilistic score for out of distribution detection our loss is going to


be a mixture of these classification and localization losses and the uncertainty loss added with


a given hyper parameter so this is going to be our detector for in distribution we simply


take predicted or we take an inference sample we take the predicted bounding box we'll plug


it into this uncertainty estimate right here so this here is this free energy we plug it


into the sigmoid formula here and that will give us one if the classifier is very certain


and the zero if it's very uncertain that this is in distribution data we can define a threshold


and that's going to be our out of distribution classifier so that's it for the method they


go through a bunch of results now I'll shorten the results by saying they're just very good


at everything like it at the data sets they try against the baseline baselines they do


ablations and particularly noteworthy for example here is the false positive rate where


lower is better you can see if they were just to add an outlier class this would hurt the


performance quite a bit like more than other modifications right here which I found interesting


to see yeah they detect they compare against other outlier detection methods and they they


do have I believe some samples right here needless to say I have my concerns but it


does work pretty well so and I'm just a person that looks at this paper for the first time


and hasn't worked in this field at all and hasn't tried anything so I'm going to give


the the right away to the authors right here but let me know what you think and I'll see


How I Read a Paper: Facebook's DETR (Video Tutorial)

Hi there, people. So a lot of you have asked me how I read papers. And honestly, I don't think


there is any super special method to it. But you know, I thought because people have asked me to


make a video on it, so I'll make a video on it. And I'll try to share my method of reading papers.


And hopefully, this is going to be somewhat of a mini series or a series where I every now and


then discuss how I read one of the papers that I make videos about. And I'll try to select them


such that different things are highlighted. Now, I've selected this one right here for really for


no particular reason, other than I sort of remembered it. And I'm going to try to go with


you through how I read this, and how I encountered this and kind of try to honestly share what I


thought at the first time when I read it. And I hope this helps some of you. If it does help you,


and if you like content like this, of course, feel free to share this out and subscribe. If you


haven't seen my original video on this paper, it might be worth to go watch it, I'll link it. And


with that, let's dive in. So again, this might be not really something new, but I'll just go through


it. Okay. So first thing I do is, of course, read the title. So the title has three parts,


end to end object detection with transformers. So what I noticed that I do myself is I like,


through reading a paper, it's like read the paper with an open mind, I don't do that. I almost


immediately form an opinion and a hypothesis of what's going on. Like, so I see transformers. So


I know what transformers are, if you don't, I've made a video of my lots of videos on transformers,


attention is all you need is the base paper for that. So I know what a transformer is okay. And


I know that transformers are usually in NLP are usually used in NLP, though there are things like,


you know, other things with transformers, but it's usually an NLP model. Then I read object


detection. And I know object detection is a computer vision task. So immediately, this here


is sort of a, a difference. And I immediately try to assess what's the new idea in this paper. And


in this case, it might be okay, it might be applying transformers to object detection. But


then I also see end to end. And the only reason to put that in a title is because that's the novelty,


because usually in deep learning, we're sort of used to systems being end to end. And even if


there aren't, if most systems aren't end to end, a lot of people don't is like end to end image


classification on ImageNet, like thanks. So I was guessing that the reason they put in end to end


into the title was because that's actually something that's special about the model. So now


I have like two competing hypotheses of why this paper matters, first of all, because it does it


with transformers. And second, because it does it end to end. And the, of course, the truth here is


that the combination of end to end transformers, all of that is what makes this model. And I


already form like a hypothesis of whether I like this or not, like I have to be I have to be honest,


I have very quick judgment of papers of whether I like them or not. And then I sort of catch myself


each time. And I still try to, so there are for most papers actually that I have sort of a negative


opinion at the beginning where I will negative. There are papers where I think like there is no


way this is going to, you know, work or something like this. I'm actually positively convinced


throughout the paper. So for most for most papers where I that I read, I'm trying to find the


positive things in there. But I do form an opinion pretty quickly, usually. Alright, so the second


thing, this part right here, I like I don't even see this is like advertisements on on like Twitter,


I like you just, I have always had issues with author names, like people will come to me and be


like, Oh, have you seen the new vinyls paper? And I have no clue. And then when they say like, oh,


that's where they use this character level model to do that. And I'm like, oh, that paper. So I


like, do not care who the authors are of a paper. Like I don't, I can't remember the papers by their


author names. I've gotten better at it, I have to say, but I've always had trouble with this. Now,


that's not to say that a name doesn't pop out to me. Like, if this would be like, like Yoshua


Benjo or some someone, like really famous, then of course, that would catch my eye. And, but I also


know that, you know, Yoshua Benjo's paper, Yoshua Benjo's lab is huge. So just because a big name is


on the paper doesn't mean that the paper is going to be of any good or bad quality. Sometimes the


authors give you an indication of what kind of research is there. Like if you see Jeff Klune, or


Ken Kenneth O. Stanley, you know that there's going to be this certain type of, you know, learning to


explore and kind of a bit more out of the box thinking in their papers, which I really like,


but it doesn't immediately give you a clue. Maybe if you go by first authors, it's much more


indicative if you have already read some of their papers, but most often I just ignore authors and


go on. The affiliation sometimes matters in in that it's a bit of a vicious cycle. If there's a


big name affiliation, like Facebook AI, Google AI, and so on. These papers, also they get more


exposure in like the press and so on. So whenever Google AI published the paper, all of these all


these pop site magazines like The Verge and this and Lifehacker and Hacker News and whatnot, they


like like write a blurb about it. So often they get much more scrutinized for these papers, they


get much more, they get much more the public attention, but they also get much more scrutiny,


which in turn means that there is a bit more pressure on them to do good experiments. So that


biases me like a little bit into the direction of believing their experimental evidence more. Now,


usually, this is also backed up by the fact that I am actually convinced by their experiments,


usually. So these, these big name papers, often I find myself even without or disregarding the


affiliation to be convinced more than of like regular papers, my most often issue with papers is


that I don't believe the experiments. And I make no difference. Like even if it's Facebook, I still


my prior is the experiments are crap, and I don't believe them. And they have to convince me of


the opposite. But some like, I can't say that it doesn't affect me that it's like a big name


affiliation. Okay, so then the second thing is I sometimes I see the paper on archive, and I skim


the abstract. Sometimes the abstract is informative, and sometimes not. So here, it's like blah, blah,


blah, a new method that views object detection as a direct set prediction problem. And I'm like, Oh,


yeah, okay. So streamlines the detection effectively removing the need for many hand design


components like non maximum suppression, yada, yada, yada. The main ingredients called detection


transformer, a set based global loss that forces unique prediction via bipartite matching, and the


transformer encoder decoder architecture. So they make it clear here why it matters. And that's


that's what I what I want to get at is sort of what's the new thing in this paper, most papers


are even though they're all very long and have lots of math and so on. They often have like


one or maybe two new core things that they really tell you sometimes zero. But a lot of times it's


like one thing that they really do. And you you sort of have to, but they're trying to cloak it


often because they need to make their research as impactful as possible, right? But you need to


sort of figure out what it is they're doing here, they make it fairly easy for us in that they say,


okay, they remove the need for many hand design components like non maximum suppression, which


tells me that they're building something that's easier than what came before them. And that


already tells me it's not necessarily going to be better. Their argument is more that it's going to


be easier, right? There are sort of two kinds of experimental results, the ones where you try to


beat what came before you. And the ones where you're trying to say, look, our thing works just


as well as this other thing while being more advantageous in some other metric. So I would


place this already in the sort of second category. And then they say, what are the actual


ingredients? It's a set based global loss that forces unique predictions via bipartite matching.


Now I, at this point, I know what these terms mean. But at this point, I actually don't have to


know what the terms mean. What I need to recognize is that I simply have to go later in the


model later and figure out what that is, and a transformer based encoder decoder architecture.


Okay. So there are two things right here that I remember I need to pay attention to later. There's


this loss, which seems to be special. And there is the transformer architecture, which seem which


they say, okay, that that's the model basically consists of those two things. And then they have


a description of what it does, given a fixed small set of learned object queries, that your


reasons about the relations of the objects and the global image context to directly output the


final set of predicted in parallel, that almost tells me nothing of Yeah, okay, the model reasons,


maybe this in parallel is something but the model is conceptually simple and does not required


a specialized library. Unlike many other modern detectors, this sort of repeats this enforces my


hypothesis that they're going with the Hey, this is a much easier way of doing things approach.


Deter demonstrates accuracy and runtime performance on par with well established that


further confirms my hypothesis that this is on par, right, the runtime performance on par with


the current state of the art. And at the end, they say, moreover, Deter can easily be generalized


proceed to produce panoptic segmentation in a unified manner. We show that it significantly


outperforms competitive baselines, training code and preterm models are available. Okay.


Now this last part, when I first read it, it's like, okay, can easily be generalized to produce


this panoptic segmentation. This is I didn't know yet whether this is like a central claim of their


paper that it can do this segmentation or whether this is like an added benefit to their paper,


because you can read it in both ways. And I'm just ready to find this out in the paper. Now,


after I've read the abstract and sort of already formed the hypothesis of what's going on. So


here, I already in my mind, I already sort of have a model of how would I do that? Right? How would I


how would I do that? And then what would I do? So right now, what I might be thinking is if I have


a transformer over images that directly outputs the predictions in parallel, I'm imagining like


an image, and the image somehow needs to go into a transformer. So maybe there's like an encoder,


like a CNN encoder that gives me image features. And then it's it's so maybe you sample this down


this image. This is just me hypothesizing what could be going on, right. And then I might be


unrolling that right this image into a vector of these lower pixels. And then so in my mind,


what I would do right here, without knowing anything more would be to do something like


BERT span prediction. So I would have BERT right here. And I so for I would input the sequence


right here. And then to detect an object, I would sort of think that maybe the BERT, you know,


BERT has an output that is the same length as the input, right? So it's it's very good at


sequence tagging and things like this. So maybe how it detects an object is going to be that it


sort of like tags the tags the center location in the pixel of an object right here, or it tags


somehow the corners of the of the bounding box. But then I don't know how this is going to be in


parallel, maybe BERT outputs like a score for each location, and then you do some kind of matching


right here. So this is my initial hypothesis of what's going on. And then I scroll through.


And honestly, the first thing I do is I go and find the pictures. And no, no different in all,


like since since your first book you read, that's what you do, I go and find the pictures,


because usually if someone proposes anything new that they're going to try to make a picture of it.


Luckily, I don't do like super theoretical, what not your Bayesian generalization bounds,


and I don't know. So most often papers I read have some sort of picture. And that's very helpful to


me. I know I know I know. But yeah, so I find this picture. And here I see okay, you have image,


you have CNN, okay, gives you set of image features are so far so good, then transformer,


encoder, decoder, then set of box predictions. So all of them come out here and already read


there in parallel, and then bipartite matching loss. So here, they I can see they color these


in different ways. And these color appear to match with these colors right here, right in the green


here. And these they they also this is a very good graphic, right from this, I can already read that


these here go to the no object. A lot of times, the graphics aren't very good. So this this is


what I'm not saying in every paper, you can learn by looking at the graphics, like sometimes the


graphics are terrible. And you're like, what's going on here? I like I don't, this this makes no


sense. This happens a lot in this paper right here. This happens to be very, very good explanatory


graphics. So I'll take advantage of that. And I do the same thing in the other papers, right. But


then later, when it doesn't match what I read in the text, I'll have to, you know, update my belief


and so on. But here, I see that these go to no object. And this goes to no object. So I don't


know yet that this is the test set. At the point where I read this, I was sort of confused by this.


But I recognized that each of these boxes right here is going to be either resulting in a bounding


box or in the no object prediction. So from that, I could conclude that these things here are maybe


some sort of a fixed set, right. But I still thought that, you know, these that this would


actually be the output of these image features. So that, in this case, you'd have like six set of


image features. And then you'd have like Bert here. Even though that's not an encoder decoder,


I still this was still my running hypothesis that somehow you'd map these image features


to these boxes right here. So and I didn't know what to what to make of this this thing right


here. So then I went through some more and look for more pictures. And they're not sometimes I


also kind of glance at the formulas. But okay, when I ever see this, this is just, I mean, this


is kind of useless, like, okay, cool, you minimize the loss, thanks. This, okay, didn't really pay


attention to that. Ah, new picture. Cool. So this picture is much more informative than the other


picture. I believe with the other picture, they were trying to showcase this loss, how they do


the matching. And even though I could read a lot from that picture, I did not get that part. And


therefore, I felt when I saw this, and I just glanced at it, I'm like, wait, what's what's


different than up here? It seems like the same. But okay, let's look at this. So again, we see,


okay, you have set of image features that comes out of the CNN. So that conforms with my belief.


But then this here goes into a transformer encoder. And this comes out. So immediately, I see, oh,


this is not the same as these boxes here, right? That was my hypothesis that these things here


would be the colored boxes. So I I say, okay, obviously, that's not what happens.


This thing here seems to be sort of the encoded image information, then that's somehow fed into


here. And that then there are these object query things, and they seem to correspond to this.


So I'm a bit more confused right now, what I can see is that these then will result in these in


these boxes. Okay. So being confused by that, I look for more pictures. So I go look for more


pictures. This here seems to be like a visualization, a lot of these papers have some


sort of ablation experiments or so and so on. This I just find really cool picture for now. I don't


know yet what it means this I don't know yet what it means. And I go down, skip all of this. And then


back here in the appendix, I find this here, which I immediately map to the previous where this is


the end. And this is a decoder. And I've already read the attention is all you need paper. And at


that point, it clicked in me is like, ah, this is not a BERT transformer. This is one of these


transformers that has an encoder in the decoder, even though they told me like 50 billion times.


Already, I was too stupid until this point. So now I know, okay, okay, I see what's going on. So the


image goes through here. And then this goes as a side input, like as an attention from the decoder


to the encoder, like I know in NLP, right. So in NLP, this here would be a source sequence, like


maybe if you do translation, and this here would be a target sequence. So now, whenever I see a


now, whenever I see a transformer like this, and it outputs something at this, I, I look at it as


okay, this here is sort of the input that goes as like a side input over here. And usually, here,


you have the target sequence, but that's not the case right here, right? You have these,


these object queries. So this is how far I get from the pictures. Now, I go up, so I have a sort


of I have questions now. I have questions. And that's when I start reading the paper. Only now


do I start reading the paper after I've looked through all the images form the hypothesis,


and sort of have questions on how this works. And we'll go a bit faster from now on, to just not


bore you with all the things. So the introduction is often very important, even though it's called


introduction. And maybe, you know, if you read a book, like, if there's like introduction, or


prologue, or something like this, it's often kind of pointless. Introduction in these research papers


is one of the most important points, because all of these papers, they try, basically, all of them


try to convince a reviewer to accept them. And in order to do that, they will set up their main


points and their main story immediately in the introduction. So what you'll usually have


is a problem statement, which is here, like why what's what's wrong right now. And then you have


like a story of how their paper addresses the issue. Okay, and that's, that's here.


We streamline the training pipeline by viewing object prediction, yada, yada, yada. This is often


formulates in words, what the paper is about, and what contribution the paper makes, right? This is


like a, this is like a longer abstract, the abstract is often very, very cryptic, very dense.


This here is often much more informative of what the paper does. So for understanding the paper and


a high level, the introduction is the best place. But given that I've already looked at the images


and so on, I don't actually draw many new much new information from this thing. Then there's


related work. And honestly, I skip it. Like, unless I'm the actual reviewer of a paper, like when I'm


the reviewer of a paper, I read the related work. But often related work is a very, very complex


work. But often related work is just like, you, first of all, you cite a bunch of your friends,


and then you cite the mandatory papers, and then you cite every single person that you think could


be a reviewer because, or you've actually been rejected from a conference with a reviewer


claiming that you're you haven't compared or you haven't cited that or that paper, you can pretty


much be sure that that's the if if it's not a glaring omission, if it's like a niche paper,


and you haven't cited it, then you're like, okay, I'm gonna cite it just because the next


conference you could be my reviewer again. So I'm not I'm not sure that these related work sections,


they're necessary. Like if someone wants to write their thesis, and they go and read this paper,


and they want references. Oftentimes, this is a good place, but a lot of it is just blah, blah,


blah. Okay, I know, I know, disagree with me if you want. Oh, yeah, to maybe to reading quality.


So I tend to at this point, I tend to not skim. So at first I skim. But at this point, I tend to


read every sentence and read it closely and understand it. And when I realized like I'm


tired or something, I don't just skim the paper, I've tried to skim papers and it doesn't doesn't


work. Try to read every sentence, understand every sentence. And okay, if you don't understand it,


don't stop reading because of that. But try to not skim and be like, Oh, yeah, yeah, okay,


I gotta go to go to go to that's not helpful. Except related work skip completely cool. Then


a lot of times in this paper now is the model. And this is the section I'm actually interested


in, right. So I read very, very closely here. And then I find out what their, their loss is all


about. And again, I stress read these things and understand them, right. Sometimes it's hard. But


if you're if you're confused, that means you either they've done a bad job, or they made a


mistake, or that you haven't understood something. If you can't understand the sentence, try to read


on maybe it's clarified later, and then you know, go back. But again, do not do not, like just start


a lot of times, when I read paper previously, like I wouldn't understand something quite well


yet. And then I would be like, Oh, yeah, yeah, yeah. And then I noticed that I start skipping


and skimming more and more, because that would, you know, pop up again and again, and I wouldn't


understand it again and again. And then at the end, I would just be kind of glancing at the paper.


And I don't want to do that right here. So I want to read every sentence and understand it. Okay,


so here, then I find out about the loss. And then I if I don't know something here, then I'll go


and look it up on maybe on Wikipedia or something like this. Now, I don't need to actually, I don't


need to understand every single part of it, right? That's maybe I should correct myself. So for


example, this bounding box loss here, they talk about the second part of the max and quats and


Hungarian puzzle is this box loss that scores bounding boxes. Unlike many detectors that do box


prediction with some initial the yada yada yada, they say the most commonly used l one loss will


have different scales for a small. So here they basically talk about how they mix the losses,


they say overall our box loss is that defined as this and this. Now, I haven't, I don't know what


these losses are. I just assume there's some bounding box losses. So when I it's not true when


I say understand everything. Understand the things that are integral to the story of the paper,


right? How exactly they compute bounding box losses at this point, I don't care. I just assume


that there's some loss that I can back propagate, right? I what is important is that they do this


Hungarian matching thing, right? As soon as I get that I'm like, oh, that was this, you know, this,


this thing, no, this thing up here. This thing, this with the matching thing. Now I get it. Now I


know there are always the same amount of boxes here. And there are always the same amount of


labels here. And all we need to do is somehow match them. And I immediately think why is that


relevant? Oh, because when something is already matched to an object, some other thing cannot be


matched to the same object. And that's how we you know, prevent the fact that all the things predict


the same thing, right? And so that immediately becomes clear. And as I said, there is usually


like one or two ideas in a paper. I don't assume or I don't care what their exact loss function is


because I've sort of gotten the idea up here of what the loss is about. All right, so I hope that's


clear on the very closely read the things and understand the things that are necessary for the


story. If you find if you think something's not necessary for the story and then later end up not


understanding that maybe come back and you know, read it again. In any case, I would I would rather


I would rather skip something and assume it's not necessary if I think so, and then come back,