have any questions especially during the software labs they can help you answer any questions that you might have and finally we really want to give a huge thank to all of our sponsors who without their help and support this class would have not been possible ok so now with all of that administrative stuff out of the way let's start with the the fun stuff that we're all here for let's


start actually by asking ourselves a question why do we care about deep learning well why do you all care about deep learning and all of you came to this classroom today and why specifically do care about deep learning now well to answer that question we actually have to go back and understand traditional machine learning at its core first now traditional machine learning

MIT 6.S191 (2020): Introduction to Deep Learning


algorithms typically try to define as set of rules or features in the data and these are usually hand engineered and because their hand engineered they often tend to be brittle in practice so let's take a concrete example if you want to perform facial detection how might you go about doing that well first you might say to classify a face the first thing I'm gonna do is I'm gonna try and


classify or recognize if I see a mouth in the image the eyes ears and nose if I see all of those things then maybe I can say that there's a face in that image but then the question is okay but how do I recognize each of those sub things like how do I recognize an eye how do I recognize a mouth and then you have to decompose that into okay to recognize a mouth I maybe have to recognize these


pairs of lines oriented lines in a certain direction certain orientation and then it keeps getting more complicated and each of these steps you kind of have to define a set of features that you're looking for in the image now the key idea of deep learning is that you will need to learn these features just from raw data so what you're going to do is you're going to just take a


bunch of images of faces and then the deep learning algorithm is going to develop some hierarchical representation of first detecting lines and edges in the image using these lines and edges to detect corners and eyes and mid-level features like eyes noses mouths ears then composing these together to detect higher-level features like maybe jaw lines side of the face etc which then


can be used to detect the final face structure and actually the fundamental building blocks of deep learning have existed for decades and they're under underlying algorithms for training these models have also existed for many years so why are we studying this now well for one data has become much more pervasive we're living in a the age of big data and these these algorithms are hungry


for a huge amounts of data to succeed secondly these algorithms are massively parallel izybelle which means that they can benefit tremendously from modern GPU architectures and hardware acceleration that simply did not exist when these algorithms were developed and finally due to open-source tool boxes like tensor flow which are which you'll get experience with in this class


building and deploying these models has become extremely streamlined so much so that we can condense all this material down into one week so let's start with the fundamental building block of a neural network which is a single neuron or what's also called a perceptron the idea of a perceptron or a single neuron is very basic and I'll try and keep it as simple as possible and then we'll try


and work our way up from there let's start by talking about the forward propagation of information through a neuron we define a set of inputs to that neuron as x1 through XM and each of these inputs have a corresponding weight w1 through WN now what we can do is with each of these inputs and each of these ways we can multiply them correspondingly together and take a sum of all of them then we take this single


number that's summation and we pass it through what's called a nonlinear activation function and that produces our final output Y now this is actually not entirely correct we also have what's called a bias term in this neuron which you can see here in green so the bias term the purpose of the bias term is really to allow you to shift your activation function to the left and to


the right regardless of your inputs right so you can notice that the bias term doesn't is not affected by the X's it's just a bias associate to that input now on the right side you can see this diagram illustrated mathematically as a single equation and we can actually rewrite this as a linear using linear algebra in terms of vectors and dot products so instead of having a


summation over all of the X's I'm going to collapse my X into a vector capital X which is now just a list or a vector of numbers a vector of inputs I should say and you also have a vector of weights capital W to compute the output of a single perceptron all you have to do is take the dot product of X and W which represents that element wise multiplication and summation and then


apply that non-linearity which here is denoted as G so now you might be wondering what is this nonlinear activation function I've mentioned it a couple times but I haven't really told you precisely what it is now one common example of this activation function is what's called a sigmoid function and you can see an example of a sigmoid function here on the bottom right one thing to note is that this function takes any real number


as input on the x-axis and it transforms that real number into a scalar output between 0 & 1 it's a bounded output between 0 & 1 so one very common use case of the sigmoid function is to when you're dealing with probabilities because probabilities have to also be bounded between 0 & 1 so sigmoids are really useful when you want to output a single number and represent


that number as a probability distribution in fact there are many common types of nonlinear activation functions not just the sigmoid but many others that you can use in neural networks and here are some common ones and throughout this presentation you'll find these tensorflow icons like you can see on the bottom right or sorry all across the bottom here and these are


just to illustrate how one could use each of these topics in a practical setting you'll see these kind of scattered in throughout the slides no need to really take furious notes at these codeblocks like I said all of the slides are published online so especially during your labs if you want to refer back to any of the slides you can you can always do that from the


online lecture notes now why do we care about activation functions the point of an activation function is to introduce nonlinearities into the data and this is actually really important in real life because in real life almost all of our data is nonlinear and here's a concrete example if I told you to separate the green points from the red points using a linear function could you do that I


don't think so right so you'd get something like this oh you could do it you wouldn't do very good job at it and no matter how deep or how large your network is if you're using a linear activation function you're just composing lines on top of lines and you're going to get another line right so this is the best you'll be able to do with the linear activation function on


the other hand nonlinearities allow you to approximate arbitrarily complex functions by kind of introducing these nonlinearities into your decision boundary and this is what makes neural networks extremely powerful let's understand this with a simple example and let's go back to this picture that we had before imagine I give you a train network with weights W on the top right so W here is 3 and minus 2 and the


network only has 2 inputs x1 and x2 if we want to get the output it's simply the same story as we had before we multiply our inputs by those weights we take the sum and pass it through a non-linearity but let's take a look at what's inside of that non-linearity before we apply it so we get is when we take this dot product of x1 times 3 X 2 times minus 2 we mul - 1 that's simply a


2d line so we can plot that if we set that equal to 0 for example that's a 2d line and it looks like this so on the x axis is X 1 on the y axis is X 2 and we're setting that we're just illustrating when this line equals 0 so anywhere on this line is where X 1 and X 2 correspond to a value of 0 now if I feed in a new input either a test example a training example or whatever


and that input is with this coordinates it's has these coordinates minus 1 and 2 so it has the value of x1 of minus 1 value of x2 of 2 I can see visually where this lies with respect to that line and in fact this this idea can be generalized a little bit more if we compute that line we get minus 6 right so inside that before we apply the non-linearity we get minus 6 when we


apply a sigmoid non-linearity because sigmoid collapses everything between 0 and 1 anything greater than 0 is going to be above 0.5 anything below zero is going to be less than 0.5 so in is because minus 6 is less than zero we're going to have a very low output this point Oh 200 to we can actually generalize this idea for the entire feature space let's call it


for any point on this plot I can tell you if it lies on the left side of the line that means that before we apply the non-linearity the Z or the state of that neuron will be negative less than zero after applying that non-linearity the sigmoid will give it a probability of less than 0.5 and on the right side if it falls on the right side of the line it's the opposite story if it falls


right on the line it means that Z equals zero exactly and the probability equals 0.5 now actually before I move on this is a great example of actually visualizing and understanding what's going on inside of a neural network the reason why it's hard to do this with deep neural networks is because you usually don't have only two inputs and usually don't have only two weights as


well so as you scale up your problem this is a simple two dimensional problem but as you scale up the size of your network you could be dealing with hundreds or thousands or millions of parameters and million dimensional spaces and then visualizing these type of plots becomes extremely difficult and it's not practical and pause in practice so this is one of the challenges that we


face when we're training with neural networks and really understanding their internals but we'll talk about how we can actually tackle some of those challenges in later lectures as well okay so now that we have that idea of a perceptron a single neuron let's start by building up neural networks now how we can use that perceptron to create full neural networks and seeing how all


of this story comes together let's revisit this previous diagram of the perceptron if there are only a few things you remember from this class try to take away this so how a perceptron works just keep remembering this I'm going to keep drilling it in you take your inputs you apply a dot product with your weights and you apply a non-linearity it's that simple


oh sorry I missed the step you have dot product with your weights add a bias and apply your non-linearity so three steps now let's simplify this type of diagram a little bit I'm gonna remove the bias just for simplicity I'm gonna remove all of the weight labels so now you can assume that every line the weight associated to it and let's say so I'm going to note Z that Z is the


output of that dot product so that's the element wise multiplication of our inputs with our weights and that's what gets fed into our activation function so our final output Y is just there our activation function applied on Z if we want to define a multi output neural network we simply can just add another one of these perceptrons to this picture now we have two outputs one is a normal


perceptron which is y1 and y2 is just another normal perceptron the same ideas before they all connect to the previous layer with a different set of weights and because all inputs are densely connected to all of the outputs these type of layers are often called dense layers and let's take an example of how one might actually go from this nice illustration which is very conceptual


and and nice and simple to how you could actually implement one of these dense layers from scratch by yourselves using tensor flow so what we can do is start off by first defining our two weights so we have our actual weight vector which is W and we also have our bias vector right both of both of these parameters are governed by the output space so depending on how many neurons you have


in that output layer that will govern the size of each of those weight and bias vectors what we can do then is simply define that forward propagation of information so here I'm showing you this to the call function in tensor flow don't get too caught up on the details of the code again you'll get really a walk through of this code inside of the labs today but I want to just show you


some some high level understanding of how you could actually take what you're learning and apply the tensor flow implementations to it inside the call function it's the same idea again you can compute Z which is the state it's that multiplication of your inputs with the weights you add the bias right so that's right there and once you have Z you just pass it


through your sigmoid and that's your output for that now tension flow is great because it's already implemented a lot of these layers for us so we don't have to do what I just showed you from scratch in fact to implement a layer like this with two two outputs or a percept a multi layer a multi output perceptron layer with two outputs we can simply call this TF Harris layers dense with units equal to two to indicate that we have two


outputs on this layer and there is a whole bunch of other parameters that you could input here such as the activation function as well as many other things to customize how this layer behaves in practice so now let's take a look at a single layered neural network so this is taking it one step beyond what we've just seen this is where we have now a single hidden layer that feeds into a


single output layer and I'm calling this a hidden layer because unlike our inputs and our outputs these states of the hidden layer are not directly enforced or they're not directly observable we can probe inside the network and see them but we don't actually enforce what they are these are learned as opposed to the inputs which are provided by us now since we have a transformation between


the inputs and the hidden layer and the hidden layer and the output layer each of those two transformations will have their own weight matrices which here I call W 1 and W 2 so its corresponds to the first layer and the second layer if we look at a single unit inside of that hidden layer take for example Z 2 I'm showing here that's just a single perceptron like we


talked about before it's taking a weighted sum of all of those inputs that feed into it and it applies the non-linearity and feeds it on to the next layer same story as before this picture actually looks a little bit messy so what I want to do is actually clean things up a little bit for you and I'm gonna replace all of those lines with just this symbolic representation


and we'll just use this from now on in the future to denote dense layers or fully connected layers between two between an input and an output or between an input and hidden layer and again if we wanted to implement this intensive flow the idea is pretty simple we can just define two of these dense layers the first one our hidden layer with n outputs and the second one our output layer with two outputs we can cut


week and like join them together aggregate them together into this wrapper which is called a TF sequential model and sequential models are just this idea of composing neural networks using a sequence of layers so whenever you have a sequential message passing system or sequentially processing information throughout the network you can use sequential models and just


define your layers as a sequence and it's very nice to allow information to propagate through that model now if we want to create a deep neural network the idea is basically the same thing except you just keep stacking on more of these layers and to create more of an more of a hierarchical model ones where the final output is computed by going deeper and deeper into this representation and


the code looks pretty similar again so again we have this TF sequential model and inside that model we just have a list of all of the layers that we want to use and they're just stacked on top of each other okay so this is awesome so hopefully now you have an understanding of not only what a single neuron is but how you can compose neurons together and actually build complex hierarchical


models with deep with neural networks now let's take a look at how you can apply these neural networks into a very real and applied setting to solve some problem and actually train them to accomplish some task here's a problem that I believe any AI system should be able to solve for all of you and probably one that you care a lot about will I pass this class to do this let's


start with a very simple two input model one feature or one input we're gonna define is how many let's see how many lectures you attend during this class and the second one is the number of hours that you spend on your final projects I should say that the minimum number of hours you can spend your final project is 50 hours now I'm just joking okay so let's take all of the data from


previous years and plot it on this feature space like we looked at before green points are students that have passed the class in the past and red points are people that have failed we can plot all of this data onto this two-dimensional grid like this and we can also plot you so here you are you have attended four lectures and you've only spent five hours on your final exam


you're on you're on your final project and the question is are you going to pass the class given everyone around you and how they've done in the past how are you going to do so let's do it we have two inputs we have a single layered set single hidden layer neural network we have three hidden units in that hidden layer and we'll see that the final output probability when we feed in those


two inputs of four and five is predicted to be 0.1 or 10% the probability of you passing this class is 10% that's not great news the actual prediction was one so you did pass the class now does anyone have an idea of why the network was so wrong in this case exactly so we never told this network anything the weights are wrong we've just initialized the weights in fact it has no idea what


it means to pass a class it has no idea of what each of these inputs mean how many lectures you've attended and the hours you've spent on your final project it's just seeing some random numbers it has no concept of how other people in the class have done so far so what we have to do to this network first is train it and we have to teach it how to perform this task until we teach it it's


just like a baby that doesn't know anything so it just entered the world it has no concepts or no idea of how to solve this task and we have to teach at that now how do we do that the idea here is that first we have to tell the network when it's wrong so we have to quantify what's called its loss or its error and to do that we actually just take our prediction or what the network


predicts and we compare it to what the true answer was if there's a big discrepancy between the prediction and the true answer we can tell the network hey you made a big mistake right so this is a big error it's a big loss and you should try and fix your answer to move closer towards the true answer which it should be okay now you can imagine if you don't have


just one student but now you have many students the total loss let's call it here the empirical risk or the objective function it has many different names it's just the the average of all of those individual losses so the individual loss is a loss that takes as input your prediction and your actual that's telling you how wrong that single example is and then the final the total


loss is just the average of all of those individual student losses so if we look at the problem of binary classification which is the case that we're actually caring about in this example so we're asking a question will I pass the class yes or no binary classification we can use what is called as the softmax cross-entropy loss and for those of you who aren't familiar with cross-entropy


this was actually a a formulation introduced by Claude Shannon here at MIT during his master's thesis as well and this was about 50 years ago it's still being used very prevalently today and the idea is it just again compares how different these two distributions are so you have a distribution of how how likely you think the student is going to pass and you have the true distribution


of if the student passed or not you can compare the difference between those two distributions and that tells you the loss that the network incurs on that example now let's assume that instead of a classification problem we have a regression problem where instead of predicting if you're going to pass or fail to class you want to predict the final grade that you're going to get so


now it's not a yes/no answer problem anymore but instead it's a what's the grade I'm going to get what's the number what so it's it's a full range of numbers that are possible now and now we might want to use a different type of loss for this different type of problem and in this case we can do what's called a mean squared error loss so we take the actual prediction we take the the sorry excuse me we take the


prediction of the network we take the actual true final grade that the student got we subtract them we take their squared error and we say that that's the mean squared error that's the loss that the network should should try to optimize and try to minimize so ok so now that we have all this information with the loss function and how to actually quantify the error of the


neural network let's take this and understand how to train train our model to actually find those weights that it needs to to use for its prediction so W is what we want to find out W is the set of weights and we want to find the optimal set of weights that tries to minimize this total loss over our entire test set so our test set is this example data set that we want to evaluate our


model on so in the class example the test set is you so you want to understand how likely you are to pass this class you're the test set now what this means is that we want to find the W's that minimize that total loss function which we call as the objective function J of W now remember that W is just a aggregation or a collection of all of the individual w's from all of


your weights so here this is just a way for me to express this in a clean notation but W is a whole set of numbers it's not just a single number and you want to find this all of the W's you want to find the value of each of those weights such that you can minimize this entire loss function it's a very complicated problem and remember that our loss function is just a simple


function in terms of those weights so if we plot in the case again of a two-dimensional weight problem so one of the weights is on the x-axis one of the weights is on this axis and on the z axis we have the loss so for any value of w we can see what the loss would be at that point now what do we want to do we want to find the place on this landscape what are the values of W


that we get the minimum loss okay so what we can do is we can just pick a random W pick a random place on this this landscape to start with and from this random place let's try to understand how the landscape is changing what's the slope of the landscape we can take the gradient of the loss with respect to each of these weights to understand the direction of maximum


ascent okay that's what the gradient tells us now that we know which way is up we can take a step in the direction that's down so we know which way is up we reverse the sign so now we start heading downhill and we can move towards that lowest point now we just keep repeating this process over and over again until we've converged to a local minimum now we can summarize this


algorithm which is known as gradient descent because you're taking a gradient and you're descending down down that landscape by starting to initialize our rates wait randomly we compute the gradient DJ with respect to all of our weights then we update our weights in the opposite direction of that gradient and take a small step which we call here ADA of that gradient and this is


referred to as the learning rate and we'll talk a little bit more about that later but ADA is just a scalar number that determines how much of a step you want to take at each iteration how strongly or aggressively do you want to step towards that gradient in code the picture looks very similar so to implement gradient descent is just a few lines of code just like the pseudocode


you can initialize your weights randomly in the first line you can compute your loss with respect to those gradients and with respect to those predictions and your data given that gradient you just update your weights in the opposite direction of that event of that vector right now the magic line here is actually how do you compute that gradient and that's something I haven't told you and that's something it's not easy at all so the


question is given a loss and given all of our weights in our network how do we know which way is good which way is a good place to move given all of this information and I never told you about that but that's a process called back propagation and let's talk about a very simple example of how we can actually derive back propagation using elementary calculus so we'll start with a very


simple network with only one hidden neuron and one output this is probably the simplest neural network that you can create you can't really get smaller than this computing the gradient of our loss with respect to W to here which is that second way between the hidden state and our output can tell us how much a small change in W 2 will impact our loss so that's what the gradient tells us right


if we change W 2 in the differential different like a very minor manner how does our loss change does it go up or down how does it change and by how much really so that's the gradient that we care about the gradient of our loss with respect to W 2 now to evaluate this we can just apply the chain rule in calculus so we can split this up into the gradient of our loss with respect to


our output Y multiplied by the gradient of our walk or output Y with respect to W 2 now if we want to repeat this process for a different way in the neural network let's say now W 1 not W 2 now we replace W 1 on both sides we also apply the chain rule but now you're going to notice that the gradient of Y with respect to W 1 is also not directly computable we have to apply the chain


rule again to evaluate this so let's apply the chain rule again we can break that second term up into with respect to now the the state Z ok and using that we can kind of back propagate all of these gradients from the output all the way back to the input that allows our error signal to really propagate from output to input and allows these gradients to be computed in


practice now a lot of this is not really important or excuse me it's not as crucial that you understand the nitty-gritty math here because in a lot of popular deep learning frameworks we have what's called automatic differentiation which does all of this back propagation for you under the hood and you never even see it which is incredible it made training neural


networks so much easier you don't have to implement back propagation anymore but it's still important to understand how these work at the foundation which is why we're going through it now ok obviously then you repeat this for every single way in the network here we showed it for just W 1 and W 2 which is every single way in this network but if you have more you can just repeat it again


keep applying the chain rule from output to input to compute this ok and that's the back prop algorithm in theory very simple it's just an application of the chain rule in essence but now let's touch on some of the insights from training and how you can use the back prop algorithm to train these networks in practice optimization of neural networks is incredibly tough in practice


so it's not as simple as the picture I showed you on the colorful one on the previous slide here's an illustration from a paper that came out about two or three years ago now where the authors tried to visualize the landscape of a of a neural network with millions of parameters but they collapsed that down onto just two-dimensional space so that we can visualize it and you can see that


the landscape is incredibly complex it's not easy there are many local minima where the gradient descent algorithm could get stuck into and applying gradient descent in practice in these type of environments which is very standard in neural networks can be a huge challenge now we're called the update equation that we defined previously with gradient descent this is that same equation we're going to update our weights in the


direction in the opposite direction of our gradient I didn't talk too much about this parameter ADA I pointed it out this is the learning rate it determines how much of a step we should take in the direction of that gradient and in practice setting this learning rate can have a huge impact in performance so if you set that learning rate to small that means that you're not really trusting your gradient on each step so if ADA is


super tiny that means on each time each step you're only going to move a little bit towards in the opposite direction of your gradient just in little small increments and what can happen then is you can get stuck in these local minima because you're not being as aggressive as you should be to escape them now if you set the learning rate to large you can actually overshoot completely and diverge which is even more undesirable


so setting the learning rate can be very challenging in practice you want to pick a learning rate that's large enough such that you avoid the local minima but small offs such that you still converge in practice now the question that you're all probably asking is how do we set the learning rate then well one option is that you can just try a bunch of learning rates and see what works best


another option is to do something a little bit more clever and see if we can try to have an adaptive learning rate that changes with respect to our lost landscape maybe it changes with respect to how fast the learning is happening or a range of other ideas within the network optimization scheme itself this means that the learning rate is no longer fixed but it can now increase or


decrease throughout training so as training progressive your learning rate may speed up you may take more aggressive steps you may take smaller steps as you get closer to the local minima so that you really converge on that point and there are many options here of how you might want to design this adaptive algorithm and this has been a huge or a widely studied field in


optimization theory for machine learning and deep learning and there have been many published papers and implementations within tensor flow on these different types of adaptive learning rate algorithms so SGD is just that vanilla gradient descent that I showed you before that's the first one all of the others are all adaptive learning rates which means that they change their learning rate during training itself so they can increase or


decrease depending on how the optimization is going and during your labs we really encourage you again to try out some of these different optimization schemes see what works what doesn't work a lot of it is problem dependent there are some heuristics that you can you can get but we want you to really gain those heuristics yourselves through the course of the labs it's part


of building character okay so let's put this all together from the beginning we can define our model which is defined as this sequential wrapper inside of this sequential wrapper we have all of our layers all of these layers are composed of perceptrons or single neurons which we saw earlier the second line defines our optimizer which we saw in the previous slide


this can be SGD it can also be any of those adaptive learning rates that we saw before now what we want to do is during our training loop it's very it's the same stories again as before nothing's changing here we forward pass all of our inputs through that model we get our predictions using those predictions we can evaluate them and compute our loss our loss tells us how


wrong our network was on that iteration it also tells us how we can compute the gradients and how we can change all of the weights in the network to improve in the future and then the final line there takes those gradients and actually allows our optimizer to update the weights and the trainable variables such that on the next iteration they do a little bit better and over time if you


keep looping this will converge and hopefully you should fit your data no now I want to continue to talk about some tips for training these networks in practice and focus on a very powerful idea of batching your data into mini batches so to do this let's revisit the gradient descent algorithm this gradient is actually very computationally expensive to compute in practice so


using the backprop algorithm is a very expensive idea and practice so what we want to do is actually not compute this over all of the data points but actually computed over just a single data point in the data set and most real-life applications it's not actually feasible to compute on your entire data set at every iteration it's just too much data so instead we pick a single


point randomly we compute our gradient with respect to that point and then on the next iteration we pick a different point and we can get a rough estimate of our gradient at each step right so instead of using all of our data now we just pick a single point I we compute our gradient with respect to that single point I and what's a middle ground here so the downside of using a single point


is that it's going to be very noisy the downside of using all of the points is that it's too computationally expensive if there's some middle ground that we can have in between so that middle ground is actually just very simple you instead of taking one point and instead taking all of the points let take a mini batch of points so maybe something on the order of 10 20 30 100 maybe


depending on how rough or accurate you want that approximation of your gradient to be and how much you want to trade off speed and computational efficiency now the true gradient is just obtained by averaging the gradient from each of those B points so B is the size of your batch in this case now since B is normally not that large like I said maybe on the order of tens to a hundreds


this is much faster to compute than full gradient descent and much more accurate than stochastic gradient descent because it's using more than one point more than one estimate now this increase in gradient accuracy estimation actually allows us to converge to our target much quicker because it means that our gradients are more accurate in practice it also means that we can increase our


learning rate and trust each update more so if we're very noisy in our gradient estimation we probably want to lower our learning rate a little more so we don't fully step in the wrong direction if we're not totally confident with that gradient if we have a larger batch of gradient of data to they are gradients with we can trust that learning great a little more


increase it so that it steps it more aggressively in that direction what this means also is that we can now massively paralyze this computation because we can split up batches on multiple GPUs or multiple computers even to achieve even more significant speed ups with this training process now the last topic I want to address is that of overfitting and this is also known as the problem of


generalization in machine learning and it's actually not unique to just deep learning but it's a fundamental problem of all of machine learning now ideally in machine learning we want a model that will approximate or estimate our data or accurately describes our data let's say like that said differently we want to build models that can learn representations from our training data


that's still generalize to unseen test data now assume that you want to build a line that best describes these points you can see on the on the screen under fitting describes if we if our model does not describe the state of complexity of this problem or if we can't really capture the true complexity of this problem while overfitting on the right starts to memorize certain aspects


of our training data and this is also not desirable we want the middle ground which ideally we end up with a model in the middle that is not too complex to memorize all of our training data but also one that will continue to generalize when it sees new data so to address this problem of regularization in neural network specifically let's talk about a technique of regularization


which is another way that we can deal with this and what this is doing is it's trying to discourage complex information from being learned so we want to eliminate the model from actually learning to memorize the training data we don't want to learn like very specific pinpoints of the training data that don't generalize well to test data now as we've seen before this is


actually crucial for our models to be able to generalize to our test data so this is very important the most popular regularization technique deep learning is this very basic idea of drop out now the idea of drop out is well actually let's start with by revisiting this picture of a neural network that we had introduced previously and drop out during training


we randomly set some of these activations of the hidden neurons to zero with some probability so I'd say our probability is 0.5 we're randomly going to set the activations to 0.5 with probability of 0.5 to some of our hidden neurons to 0 the idea is extremely powerful because it allows the network to lower its capacity it also makes it such that the network can't build these memorization channels


through the network where it tries to just remember the data because on every iteration 50% of that data is going to be or 50% of that memorization or memory is going to be wiped out so it's going to be forced to to not only generalize better but it's going to be forced to have multiple channels through the network and build a more robust representation of its prediction now we


just repeat this on every iteration so on the first iteration we dropped out one 50% of the nodes on the next iteration we can drop out a different randomly sampled 50% which may include some of the previously sampled nodes as well and this will allow the network to generalize better to new test data the second regularization technique that we'll talk about is the notion of early


stopping so what I want to do here is just talk about two lines so during training which is the x-axis here we have two lines the y-axis is our loss curve the first line is our training loss so that's the green line the green line tells us how our training data how well our model is fitting to our training data we expect this to be lower than the second line which is our


testing data so usually we expect to be doing better on our training data than our testing data as we train and as this line moves forward into the future both of these lines should kind of decrease go down because we're optimizing the network we're improving its performance eventually though there becomes a point where the training data starts to diverge from the testing data now what happens is that the training day


should always continue to fit or the model should always continue to fit the training data because it's still seeing all of the training data it's not being penalized from that except for maybe if you drop out or other means but the testing data it's not seeing so at some point the network is going to start to do better on its training data than its testing data and what this means is


basically that the network is starting to memorize some of the training data and that's what you don't want so what we can do is well we can perform early stopping or we can identify this point this inflection point where the test data starts to increase and diverge from the training data so we can stop the network early and make sure that our test accuracy is as minimum as possible


and of course if we actually look at on the side of this line if we look at on the left side that's where a model is under fit so we haven't reached the true capacity of our model yet so we'd want to keep training if we didn't stop yet if we did stop already and on the right side is where we've over fit where we've passed that early stopping point and we need to like basically we've started to


memorize some of our training did and that's when we've gone too far I'll conclude this lecture by just summarizing three main points that we've covered so far first we've learned about the fundamentals of neural networks which is a single neuron or a perceptron we've learned about stacking and composing these perceptrons together to form complex hierarchical


representations and how we can mathematically optimize these networks using a technique called back propagation using their loss and finally we address the practical side of training these models such as mini batching regularization and adaptive learning rates as well with that I'll finish up I can take a couple questions and then we'll move on to office lecture


It's a scientific fact that the hormones of stress downregulate genes and create disease. Long-term effects. Human beings because of the size of the neocortex, we can turn on the stress response just by thought alone as I think about our problems and turn on those chemicals That means then our thoughts Could make us sick So if it's possible, that our thoughts could make us sick then it is possible then our thoughts could make us well, the answer is absolutely yes

How To BRAINWASH Yourself For Success & Destroy NEGATIVE THOUGHTS! | Dr. Joe Dispenza


Everybody welcome to Impact Theory our goal with this show and company is to introduce you to the people and ideas that will help you Actually execute on your dreams Alright today's guest is a New York Times bestselling author and one of the most sought-after speakers in the world He's lectured and given advanced workshops in more than 30 countries Across five continents all with the aim of helping people better understand and unlock the power of their mind


His expertise is the intersection of the fields of neuroscience Epigenetics and quantum physics and he's partnered with other scientists across multiple disciplines to perform extensive research on the effects of meditation Using advanced technologies such as epigenetic testing brain mapping with EEG s and gas-discharge visualization technology. Through his work


He is endeavouring to help advance both the scientific community and the public at large as understanding of mind derived health optimization, a topic he covered extensively in his groundbreaking book, You are the placebo. His teaching has had such a profound impact on the way that people perceive a wide range of brain related topics around Mindfulness and well-being that he's a faculty member at the quantum University in Hawaii the Omega Institute for holistic studies in New York


And the Kerr Paulo Centre for yoga and health in Stockbridge, Massachusetts He's also an invited chair of the research committee at life University in Atlanta As well as a corporate consultant where he delivers his lectures and workshops for businesses So, please help me in welcoming the man who has appeared in such films as Heal, People versus the state of illusion and Unleashing creativity


The author of the recent book Becoming supernatural. Dr. Joe Dispenza Thanks for being here So, diving into your world and how you perceive the sense of self and the way that you marry science to - the way that we form memories the way that we live in a perpetual state of Reliving our past and things like that It's really, really incredible and I want to dive into the whole notion of you sort of being a habitual


Construct like what? What is that? What is the habit of you? Well a habit is a redundant set of Automatic unconscious thoughts, behaviors and emotions that's acquired through repetition The habit is when you've done done something so many times that your body now knows how to do it better than your mind So if you think about it people wake up in the morning they


Begin to think about their problems Those problems are circuits, memories in the brain, each One of those memories are connected to people and things at certain times and places and if the brain is a record of the past The moment they start their day, they're already thinking in the past. Each one of those memories has an emotion Emotions are the end product of past experiences


So the moment they recall those memories of their problems, they all of a sudden feel unhappy, they feel sad, they feel pain Now how you think and how you feel creates your state of being. So the person's entire State of being when they start their day is in the past. So what does that mean? The familiar past will sooner or later be predictable future so if you believe that your thoughts have something to do with your destiny and


You can't think greater than how you feel Or feelings have become the means of thinking by very definition of emotions you're thinking in the past And for the most part you're going to keep creating the same life, so then people grab their cell phone They check their WhatsApp. They check their texts. They check their emails. They check Facebook They take a picture of their feet. They post it on Facebook. They tweet something, they do Instagram


they check the news and now they feel really connected to everything that's known in their life And then they go through a series of routine behaviors They get out of bed on the same side. They go to the toilet. They get a cup of coffee They take a shower, they get dressed, they drive to work the same way. They do the same things They see the same people that pushed the same emotional buttons and that becomes the routine and it becomes like a program


So now they've lost their free will To a program and there's no unseen hand doing it to them. So when it comes time to change the Redundancy of that cycle becomes a subconscious program. So now 95% of who we are by the time we're 35 years old is a Memorized set of behaviors, emotional reactions, unconscious habits, hardwired attitudes, beliefs and perceptions that function like a computer program


So then person can say with their five percent of their conscious mind. I want to be healthy I want to be happy. I want to be free but the body's on a whole different program So then how do you begin to make those changes? Well? you have to get beyond the analytical mind because what separates the conscious mind from the Subconscious mind is the analytical mind and that's where meditation comes in because you can teach people


through practice how to change their brainwaves, slow them down and when they do that Properly they do enter the operating system where they can begin to make some really important changes. So Most people then wait for crisis or trauma or disease or diagnosis, you know, they wait for loss some tragedy to make up their mind to change and my message is why wait and and


You can learn and change in a state of pain and suffering or you can learn and change in a state of joy and inspiration I think right now the cool thing is that people are waking up that's really interesting and where I found the the deepest hooks into how powerful this can be for somebody is when you talk about trauma and you've talked about how People experience a traumatic event, but they then basically rehearse it and how that then has this knock-on effect. So, what is that?


Why do people find it so hard to get past trauma? Well? the the stronger the emotional reaction You have to some experience in your life the higher the emotional quotient The more you pay attention to the cause and the moment the brain puts all of its attention on the cause It takes a snapshot and that's called a memory. So long-term memories are created from very highly


Emotional experiences. So what happens then is that people think neurologically within the circuitry of that experience and they feel chemically within the boundaries of those emotions and So when you have an emotional reaction to someone or something most people think that they can't control their emotional reaction Well, it turns out if you allow that emotional reaction, it's called a refractory period to last for hours or days


That's called the mood. I say to someone. Hey, what's up think so I'm gonna move well, why are you in a mood? well I had this thing happen to me five days ago and I'm having one long emotional reaction if you keep that same emotional reaction going on for weeks or months That's called temperament. Why is he so bitter? I don't know. Let's ask him. Why is he so bitter? Why are you bitter?


Well, I had this thing happened to me nine months ago And if you keep that same emotional reaction going on for years on end that's called a personality trait And so learning how to shorten your refractory period of emotional reactions is really where that work starts So then people when they have an event what they do is they keep recalling the event because the


Emotions of stress hormones the survival emotions are saying pay attention to what happened Because you want to be prepared if it happens again Turns out most people spend 70% of their life living in survival and living in stress. So they're they're always Anticipating the worst-case scenario based on a past experience and they're literally out of the infinite


potentials in the quantum field they're selecting the worst possible outcome and they're beginning to emotionally embrace it with fear and their Conditioning their body into a state of fear do that enough times Body has a panic attack without you you you can't even predict it because it's programmed subconsciously So then you say to the person why are you this way?


And they'll say I am this way because of this event that happened to me 15 or 20 years ago and what that means from biological standpoint is that they haven't been able to change since that event So then the emotions from the experience tend to give the body and the brain a rush of energy So people become addicted To the rush of those emotions and they use the problems and conditions in their life to reaffirm their limitation


So at least they can feel something. So now when it comes time to change you say the person why are you this way? Well, every time they recall the event they're producing the same chemistry in their brain and body as if the event is occurring firing and wiring the same circuits and Settings the same emotional signature to the body. Well, what's the revelant behind that? Well your body is the unconscious mind


It doesn't know the difference between the experience that's creating the emotion and the emotion that you're creating by thought alone So the body's believing it's living in the same past experience 24 hours a day seven days a week 365 days a year and so then when those emotions influence certain thoughts and they do and Then those thoughts create the same emotions and those same emotions influence the same thoughts


Now the entire person's state of being is in the past. So then the hardest part about change is not making the same choice as you did the day before a period and The moment you decide to make a different choice get ready because it's going to feel uncomfortable It's going to feel unfamiliar. It's there's gonna be something so why does it feel so uncomfortable? Is it because of the the


neurons that fire together wire together so I've there's like an Easiness to that loop just because literally and you've talked very eloquently about this the way that the neurons connect in the brain how rapidly I've seen you show footage of how Rapidly those connections happen, which is pretty incredible Is is that what makes it so? discomforting for people I think that I think that the bigger thing is that we we keep


Firing and wiring those circuits they become more hardwired. So there you have a thought and then the program runs but it's the emotion that follows the thought if you have a if you have a Fearful thought you're gonna feel anxiety the moment you feel anxiety your brains checking in with your body and saying yeah, you're pretty anxious so then you start thinking more corresponding thoughts equaled how you


While the redundancy of that cycle conditions the body to become the minds. So now when it comes time to change Person's steps into that river of change and they make a different choice in all of a sudden They don't they don't feel the same way So the body says well you've been doing this for 35 years Well, you're gonna just stop feel suffering and stop feeling guilty and stop feeling shameful and you're not gonna complain or blame or make excuses


Or feel sorry for yourself long The body's in the unknown so the body says I want to return back to familiar territory so the body starts influencing the mind then it says Start tomorrow, you're too much like your mother. You'll never change. This isn't gonna work for you. This doesn't feel right And so if you respond to that thought as if it's true that same thought will lead to the same choice


Which will lead to the same behavior, which will create the same experience which produce the same emotion I want to talk about that notion of Give me a little more detail. We mean by the body becomes the mind or the unconscious mind. What do you mean by that exactly? Well, those are two different things your body is your unconscious mind in a sense if you're sitting down and you start thinking about


Some future worst-case scenario that you're conjuring up in your mind and you begin to feel the emotion of that event your body doesn't know the difference between The event that's taking place in your world outer world and what you're creating by emotion or thought alone. So most people then They're they're constantly reaffirming their emotional states So when it comes time to give up that emotion


they can say I really want to do it but really the body is stronger than the mind because it's been conditioned that way so The servant now has become the master and the person all of a sudden once they step into that unknown They'd rather feel guilt and suffering because at least they can predict it being in the unknown Is a scary place for most people because the unknown is uncertain people say to me. Well, I can't predict my future


I'm in the unknown and I always say the best way to predict your futures have created Not from the known but from the unknown what thoughts? Do you want to fire and wire in your brain? what behaviors do you want to demonstrate in one day the act of rehearsing the mentally closing your eyes and rehearsing the action the rehearsing the reaction of what you want or the action of what you want by closing your eyes and mentally rehearsing some action if


You're truly present. The brain does not know the difference between what you're imaging and what you're experiencing in 3d world so then you begin to install the Neurological Hardware in your brain to look like the event has already occurred Now your brain is no longer a record of the past now It's a map to the future and if you keep doing it priming it that way


the hardware becomes a software program and who knows you just may start acting like a happy person and then I think the hardest part is To teach our body emotionally what the future will feel like ahead of the actual experience. So, what does that mean? You can't wait for your success to feel empowered. You can't wait for your wealth to feel abundant you can't wait for your your new relationship to feel love or


Your healing to feel whole I mean that's the old model of reality of cause and effect, you know Waiting for something outside of us to change how we feel inside of us and when we feel better inside of us We pay attention to ever or whatever caused it But what that means then is that from the Newtonian world that most people spend their whole life living in lack


Will hitting through something to change out their what do you mean the Newtonian world? Newtonian world is all about the predictable It's all about predicting the future But the quantum model of reality isn't is about causing an effect the moment you start feeling Abundant and worthy you are generating wealth the moment you're empowered and feel it You're beginning to step towards your success the moment. You start feeling whole


Your healing begins and when you love yourself and you love all of life You'll create an equal and now you're causing an effect and I think that's that the difference between living as a victim in Your world saying I am this way because of this person or that thing or this experience They made me think and feel this way when you switch that around you become a creator of your world and you start


My thinking and my feeling is changing an outcome in my life And now that's a whole different game and we start believing more that were creators of reality. So, how do we go from? Okay, I have this negative emotion. It's controlling my life. It's got me in this cycle of I think about this emotion which triggers a chemical reaction which trains my body to feel that way which makes it easier more likely I will do it again and


so now I'm in this vicious line unconscious and it's unconscious right and you You said does your thinking create your environment orders your environment create your thinking which I thought was really really interesting. So how do we then go from that like mechanistically To begin this visualization process of something that's empowering its me in a different state. It's my future self


Is it meditation is it what does that look like if you're not being defined by a vision of the future? Then you're left with the old memories of the past and you will be predictable in your life and If you wake up in the morning and you're not being defined by a vision in the future as you see the same people and you go to the same places and You do the exact same thing at the exact same time


It's no longer that your personality is creating your personal reality Now your personal reality is affecting or creating your personality Your environment is really controlling how you think and feel? unconsciously because every person every thing every place every experience has a neurological network in your brain every Experience that you have with every person produces an emotion. So some people will use their boss to reaffirm their addiction to judgment


They'll use their enemy to reaffirm their addiction to hatred to use their friends that we affirm their addiction to suffering So now they need the outer world to feel something. So To change them is to be greater than your environment to be greater than the conditions in your world and the environment Is that seductive so then why is meditation the tool well?


Let's sit down. Let's close our eyes. Let's disconnect from your outer environment So if you're seeing less things is less stimulation going to your brain if you're playing soft music or you have earplugs in Less sensory information coming to your brain. So you're disconnecting from environment if you can sit your body down and Tell it to stay like an animal stay right here. I'm gonna feed you when we're done


You can get up and check your emails You can do all your texts, but right now you're gonna sit there and obey me So then when you do that properly and the you're not eating anything or smelling anything or tasting anything? You're not up experiencing and feeling anything. You would have to agree with me that you're being defined by a thought, right? So when the body wants to go back to its emotional past


And you become aware that your attention is on that emotion And where you place your attention is where you place your energy? you're siphoning your energy out of the present moment into the past and you become aware of that and You settle your body back down in the present moment because it's saying well, it's eight o'clock You normally get upset because you're in traffic around this time and here you are sitting and we're used to feeling anger and you're off


Schedule. Oh, it's 11 o'clock and usually check your emails and judge everybody. Well, the body is looking for that that predictable chemical state every time you become aware that you're doing that and your body is craving those emotions and You settle it back down into the present moment. You're telling the body it's no longer the mind that you're the mind and now your will is


Getting greater than the program and if you keep doing this over and over again over and over again over and over again Just like training a stallion or a dog. It's just gonna say I'm gonna sit and the moment that happens when the body's no longer the mind when it finally surrenders There's a liberation of energy We go from particle to wave from matter to energy and we free ourselves from the chains


Of those emotions that keep us in the in the familiar past and we've seen this Thousands of times. In fact, we can actually predict it now on a brain scan. That's so interesting Let's go a little bit harder on Metacognition the notion that you don't have to believe everything you think I love the way that you talk about that. Hmm Yeah, and we have a huge frontal lobe. It's 40% of our entire brain and


most people when they have a thought they just think that that's the truth and I think one of my greatest Realizations in my own journey was just because you have a thought it doesn't necessarily mean it's true so if you think 60 to 70 thousand thoughts in one day and we do and 90% of those thoughts are the same thoughts as The day before and you believe that your thoughts have something to do with your destiny


Your life's not gonna change very much Because the same thought leads to the same choice the same choice leads to the same behavior the same behavior creates the same experience and the same experience produces the same motion and so then the act of becoming conscious of this process to to begin to become more aware of How you think how you act in how you feel? It's called metacognition


and so then why is that important because the more conscious you become of those unconscious states of mind and body the Less likely you're gonna go unconscious during the day and that thought is not gonna slip by your awareness unchecked because your It means to know thyself and the word meditation means to become familiar with So as you become familiar with the thoughts the behaviors and the emotions of the old self


You're retiring that old self as you fire and wire new thoughts and condition the body into a new emotional state if you do that Enough times it'll begin to become familiar to you. So it's so important Just like a garden if you're planting a garden, you've got to get rid of the weeds You got to take the plants from the past year and you got to pull them out the rocks that sift to the top that


Are like our emotional blocks they have to be removed that soil has to be tenderized and broken down We have to we have to make room to plant the new garden So primarily we learn the most about ourselves and others when we're uncomfortable because the moment you move into that uncomfortable state normally a program jumps in When that program jumps ins because the person doesn't want to be in the present moment and engage it consciously


So when you teach people how to do that with a meditative process Turns out that when they're in their life They're less likely to emotionally react they're less likely to be so rigid and believe the thoughts they were thinking they're more aware of when they go unconscious back into a habit and that is what starts the process of change and So we have to unlearn


Before we relearn we have to break the habit of the old self before we reinvent a new self We have to pre synaptic connections and sprout new connections. We have to unfired unwire and refire and rewire. We have to unmemorable Body to a new mind into a new emotion like the program and reprogram that's the act and it's a two-step process Yeah, I like the way that you call that out as an action


There was another thing that you said that I thought was really powerful about how insights themselves are essentially inert. They don't do anything What what then do we do with an insight? How do we take a breakthrough moment and make sure that it's not just a breakthrough moment Like I guarantee people watching right now are having like a hundred aha moments for sure


That was definitely the case for me as I was researching you and when you said that I was like and that's the danger that You have the AHA and then nothing. Yeah Yeah and that's it's a it is a danger because then people will will shrink back into mediocracy and they'll use the insight to Excuse them from taking a leap. They'll say yeah, you know, I have a chemical imbalance in my brain. Yeah, my father was


Really overbearing he was a perfectionist. That's why I am the way I am you know people they come up with stuff to to excuse themselves. The insight is Actually giving them permission to stay limited and it's an amazing idea because they'll say to you And they really want to get over their anxiety. But let's ok. Let's take your ex-husband. Let's put him in a straitjacket


Let's duct tape them and shoot them to the moon know what I mean. What are you gonna do now? You still have to make those changes. And so then the person's enemy dies or they're something shifts in their life And that person's gone. They'll find another person to hate. This is just how we function as human beings. We just slide another Reason to feel those emotions. So I think I think when people start to understand this, you know,


I think knowledge is power but knowledge about yourself is self empowerment. So how much of this is really learning to? just bifurcate the world into there's negative emotions that have negative neuro chemistry associated with and you said that in those states if you're living in a perpetual state of stress hormones and things like that illness is like a step away and


Then just the other side of that is understanding but there's this whole other side of positive energy which happiness joy Empowerment whatever that you know neurochemical cocktail is but that when you're on that side Your immune system is more likely to function. Well, like is that Just sort of bringing it down to like a really basal. Yeah, that's sort of one of the biggies


Well, let's talk about it in terms of survival or creation As I said 70% of the time people live in stress and living in stress is living in survival now All organisms in nature can tolerate short-term stress, you know a deer gets chased by a pack of coyotes when it out runs the Coyotes it goes back to grazing and the event is over and The definition of stress is when your brain and body are knocked out of balance out of homeostasis


The stress response is what the body and Nate Lee does to return itself back to order. So you're driving down the road Someone cuts you off you jam on the brakes You may give them the finger and then you settle back down and the event is over and boom now. Everything's back back to normal But what if it's not a predator that's waiting for you outside the cave, but what if it's your coworker?


Sitting right next to you and all day long you're turning on those chemicals because they're pushing all your emotional buttons When you turn on the stress response, and you can't turn it off Now you're headed for a disease because no organism in nature can live an emergency mode for that extended period of time It's a scientific fact that the hormones of stress down regulate genes and create disease long term affects


Human beings because of the size of the neocortex we can turn on the stress response just by thought alone Which I think about our problems and turn on those chemicals That means then our thoughts Could make us sick So if it's possible that our thoughts could make us sick. Is it possible that our thoughts could make us? Well, the answer is absolutely Yes So then what are the emotions that are connected to survival? Let's name them


anger aggression hostility hatred competition fear Anxiety worry pain suffering guilt shame unworthiness the envy jealousy. Those are all Created by the hormones of stressin and psychology calls them normal human states of consciousness I call those altered states of consciousness So then we tend to remember those traumatic events more because in survival, you better be ready if it happens again


that's an and in one's survival gene is switched on you could have ten really great things that happen to you in your day and you just have one bad thing that happens and you cannot take your attention off that bad that that unhappy thing because The survival gene is switched on it's really interesting How does epigenetics come into play and all this like what's actually happening? You've talked pretty profoundly about?


Proteins and like really at a deep level how we're signalling to our genetics to create these kinds of changes What does that actually look like? Well epigenetics epi means above the gene and Many years ago after the DNA helix was discovered by Watson and Crick They said the blueprints of life, you know, all diseases are created from genes it turns out less than 5% more like 1% of people on the planet are born with a genetic condition like type 1 diabetes or


Tay-sachs disease or sickle cell anemia the other 95 to 99 percent Are created by lifestyle and by choices you can take to identical twins Exact same genome one dies at 51. The other one dies at 85 same gene different environment, so All of a sudden they said we lied That was wrong. It's not genes that create disease. It's the environment that signals the gene that creates disease. Well, ok, but


That's not the whole truth too because you could have two people working side by side in the same factory One gets cancer after being exposed to a carcinogenic for 25 years both working for 25 years The other one has no cancer at all. So there must be some internal order That would cause one person to not get it while another one does So is it possible then if?


The environment signals the gene and it does and the end product of an experience in the environment is called an emotion Can you signal the gene ahead of the environment by embracing an elevated emotion? We've done the research on this where we measured 7,500 different gene expressions in a group of people it came to an advanced event for four days and we Had them doing a seated meditation a walking meditation a laying down meditation a standing meditation and at the end of four days


Just four days The common eight genes that were upregulated two genes to suppress cancer cells and tumor growth Two genes for neurogenesis the growth of new neurons in response to novel experiences and learning the gene that signals stem cells To go to damaged areas and repair them the gene for oxidative stress was upregulated We started seeing all these genes that are very very healthy to cause the body to flourish


Imagine if people were doing that for three months. We also measured telomeres the little Shoestrings on the end of DNA that tell us our biological age. We asked people to Do the work meditation five out of seven days for 60 days Measure their telomeres that determine their biological age sixty days later seventy four percent of the people lengthen their telomeres 40 percent


significant change twenty percent a very remarkable change That means that they got a little bit of their life back if it lengthened by ten percent They got 10% of their life back. That's incredible Before I ask my last question tell these guys where they can find you online Sure. My website is just dr. Joe Dispenza dot-com. You can follow us on Facebook Twitter Instagram


We're all over and then my final question. What's the impact that you want to have on the world? I? think that the end game for me is to empower people to Such a degree that they realize that they need less things outside of them to make them happy less things outside of them to regulate their moods and their behaviors and that they begin to use the kind of the power that we


All have access to and into really and to change the world to make a difference so that there's more peace There's more homeless. There's more connection that we support and love each other and we serve better and and I think that we have to start for the most part if everybody's working on themselves and and Trying doing their best to present the greatest ideal of themselves to the world. I think the world would be a better place. And so


That's my passion and I'm witnessing it happening now The more than I ever thought I would was incredible Joe. Thank you so much for being here and amazing having you Guys Go watch this man's videos They are some of the best explanations of what's going on inside the mind that I've ever come across There were of several that I literally have people in my life that I'm going to force to sit down and watch these things


It's just incredible explanations of how you create yourself out of the things you do Habitually the way that you think creates a feeling the way that you feel creates thinking that matches that and then you get in this cycle and that coming down to that personality ultimately being a finite set of patterns in your brain I think is really really illuminating in terms of how we actually experience the world and


I think when people understand that that it's within your control that you don't have to believe every thought that you think that you can step outside of that that you can leverage metacognition to think about your thinking and Deconstruct and decide what you want to think about and start focusing on that and create an entirely different version of yourself that has new


Elevated feelings that's over on the side of the positivity empowering yourself I think it's really incredible and he gets deep into the mechanistic stuff Which I love you guys will not regret diving deep into this man's world. I think you will get some incredible revelations All right, if you haven't already be sure to subscribe and until next time my friends be legendary. Take care


hi there today we'll look at self training with noisy student improves image net classification by chidze sie mintan luang eduard hovi and kwok v li so this paper takes an imagenet classifier that's been trained on the imagenet dataset and uses that classifier as a teacher model to label a whole bunch of unlabeled images and then it trains a student model that

Self-training with Noisy Student improves ImageNet classification (Paper Explained)


is larger than the original teacher model on those teacher labeled images and that turns out to improve the classification on the imagenet validation set now that there is a couple of things that make this all work and today we're going to explore how this paper does it and what they say is important if you enjoy content like this as always don't hesitate to share it out or tell


your friends about it and if you're not subscribed yet then do so um i would appreciate that and you'll get more content so win-win so this this paper is about semi-supervised learning in um in effect so it's at the intersection actually of semi-supervised learning knowledge distillation and transfer learning so what do we mean by semi-supervised learning usually in


supervised learning you'll have some sort of data set and the data set will contain let's say it's an image net it's image data set so the data set will contain images this is an image with like some sort of cat on it and it will contain the labels according to that so cat now in semi-supervised learning you you assume that so this is supervised learning in semi-supervised


learning you assume that only part of your data set has the labels so like only this part down here has the labels and the upper part does not have the labels so that's semi-supervised learning it's often the case when it's very expensive to get labels so you can only get labels for a couple of images in your data set but very often in semi-supervised learning you still


assume it's the same data set there is a slightly different setup here that's called transfer learning so in transfer learning what you'll have is you'll have your data set that has the labels but it's very small so you'll notice i've drawn it smaller that means you have very little that is also the case when it's very expensive to get labels but also it's expensive to get the data


itself this is often the case like say in medical data where not only is it expensive to get labels for like a ct scan it's actually expensive to get the ct scan so what the goal in transfer learning is is to say well i do i do have only this small data set but i do have this giant other data set over here now can't i it's not the same it's maybe they're not ct so these are ct


scans maybe these are x-rays right they're fairly similar similar technology um if you slice the ct it will give you sort of an x-ray can i you know train my model pre-train my model on x-ray data and then fine-tune it on the ct data so that's called uh transfer learning usually now this can be done with or without labels so it can be that for the x-ray data set you do have the


labels or you don't have the labels there are techniques for all of those now what we're going to look at today is kind of this situation right here it's the transfer learning situation where you do not have the labels for this x-ray data set but other than in this x-ray example what we're going to look at is the small data set is going to be our imagenet database so our original


picture with label database so you'll see immediately the difference here is that in the transfer learning setting we usually assume that the data set we want to train on is fairly small here you know imagenet is already sizeable but what we have is we have a much larger database of unlabeled images that we can just get from the internet so we can scrape the internet for any


kind of pictures and that will be our unlabeled data set and what we'll try to do is somehow incorporate this unlabeled data set here into the training process to get better on the imagenet data set okay so this is the the problem statement is you have the imagenet dataset and you have a second much larger data set of unlabeled images and you somehow want to make use of them


so i hope you see how this is sort of connected to the others it's essentially sort of a transfer semi-supervised learning setting but with the exception that usually in transfer learning you assume that the the labeled data set is like super small which is not the case here and that's going to result in us being able to apply a different technique so this different technique is


called the noisy student now usually what you might do in a transfer learning setting is you might want to start with that big data set right because that's the data set that's sizeable enough to allow you to train a really big model on it and then you fine tune and you you sort of hope that the information transfers over here on the other hand what we want to


do is we start with the imagenet data set so first we train this in a supervised learning fashion into our model now this model is going to be called the teacher model we know how to do this we know to train imagenet models right so we can train this into a teacher model that has a reasonable accuracy on the imagenet data set step two we're going to take that big


data set over here and use the teacher model to label the unlabeled images so for each image for each image coming in here the teacher so maybe this is again another cat the teacher will say that's a cat okay so that gives you the big data set where now you have images along with labels just the labels aren't true labels they're generated by the teacher and then in the third step you train


this big data set you train on this big data set and that's what you call your student model and then the student model in this paper will see how can we make it such that the student is then better at the original imagenet task than the teacher ever was which seems counterintuitive at first because all of the information that the student is trained from is basically what the teacher already


knows right all the labels here come from the teacher therefore the student shouldn't be able to outperform the teacher but in this case the student will be able to outperform the teacher and their argument here is that this is mainly due to the fact that you use noise in this training procedure so when you train the student what you'll do is you'll use noise and one of the types of noise is


that you severely augment this data right here in order to train the student now we've known for a long time that data augmentation for example in the frameworks of self-supervised learning and so on can have a very large benefit to training and here the fact that we incorporate this at extra data and we use noise and augmentations on it is going to result in a student that can


sort of learn more about the data than than the teacher did know okay this this is basically it and as you can see this is kind of their main final result where they say on imagenet our top one accuracy sort of increases right here and uh even on these kind of subsets of imagenet or these are sort of corrupted sets of imagenet they make even more substantial improvements as you can see


here now we'll go into what these corrupted subsets are but you know just for now these here are very difficult variants of imagenet they can be severely corrupted or or distorted and so on and you can see that the model improves severely over the previous state of the art which basically means that this model is more robust and that's a direct consequence of the noise


now one last thing i should say is that the student here is also larger than the teacher so that's also one thing that makes the student better so what you will make is the student model is larger than the teacher model as a model as the architecture so in combination with the noise right here with the noise in combination that means the student model is probably able to


capture more of the variance of the data it's larger it has more parameters it can learn more about the data together with the noise it can probably be a more robust and that's what makes it generalized better and we'll also see as we see here it's more robust to these transformations and it's also going to be more robust to adversarial perturbations so the the technique again is uh


illustrated here as as we said it's pretty simple first so step one step one train the teacher model with label data as you would step two you infer the pseudo labels on unlabeled data step three you make a student you make sorry with step three over here train an equal or larger student model with combined data and noise injected so they don't they use the original label data here


and the pseudo-labeled data right here in order to train the student but still this the student doesn't have more information more label information than the teacher had it simply has this teacher labeled teacher labeled unlabeled data also to train on now the crucial part here is well first of all that the student can be larger and second of all that there can be


noise and the noise comes in three different forms so first of all you use data augmentation which we've already seen this is sort of like random cropping or mild rotations color jitter whatever they use a rand augment here which is a specific technique to apply these augmentations um they use dropout which is a fairly old technique where you in the student model that you train


you randomly drop out connections which makes it more robust and more generalizing and then you also use stochastic depth now stochastic depth is a technique when you train a model what you'll do during training instead of always passing your data forward through the layers like this you use some sort of a drop out but with entire layers so what you'll do is you'll pass your data forward and


then randomly you'll skip a layer and then pass it forward again now these these might seem weird first because uh yeah it might seem weird but in if you know that most models especially computer vision models nowadays are residual networks which means that their layers look like so you have the input you have some computation and then you have the output and then there is


already a residual connection that basically adds the original signal together to the result of the computation so all you do in this stochastic layer dropout or this stochastic depth right here is you basically disable use you disable this connection right here and all the signal has to flow through here if you read the residual the resnet original resnet paper they make it


pretty clear why the residual connection is a good idea basically they say these computations here they if you have a very deep network each layer only has to basically um do very a little bit of computation that that can be bypassed uh fairly efficiently for a lot of data points so it's not that hurtful to bypass a layer and in this case they actually use it to


just bypass some of these small computations and inject some more robustness into the student model so with these three strategies to bring noise into the training process one is on the data and two is on the student model itself they train the student model and then fourth and this is what we didn't have before fourth or maybe we put four here make the student a new teacher so now


you can iterate you can use the student model that you just trained to again label the unlabeled data and then you can use another student model again under the influence of noise to train from that student model and so on and you can go on and they do up to like three iterations of this where they always take the new the student as the new teacher and then use a new student model to train from


that teacher and they get better and better as they do this of course there's like a diminishing returns but it's pretty impressive that this even works right the new students in fact aren't even larger than the old students it's just that the students are larger than the original teacher model in most of these cases so here's the algorithm written down you'll require labeled


images right here and unlabeled images which are the ones with the tilde so first you learn the teacher model which minimizes the cross entropy on labeled images this we already know this right this is the label this is the image according to the label and you train the teacher model which is this thing here and you can see here noised so already in the teacher training process


you want to introduce this noise you want to introduce these data augmentations these are as i said these are standard techniques to make models more robust and therefore more generalizable yeah we know from these from these self-supervised papers that these augmentations are very powerful and the way you design them basically if you one of these augmentations is a random crop which


means if you have an image you randomly crop out like part of that image and then that's your training sample and not the entire thing so by doing this you basically teaching the model to ignore the exact location and scale of things on an image and you can do this because you as a human know that you know i can zoom in i can zoom out into something and it won't


change what's on the picture and so that's you use these augmentations to kind of heuristically tell the model what it should be invariant to and that is that is a very powerful technique uh to regularize basically to to robustify these deep deep methods and this is used the same here so already in the teacher model we train with these noise and then step two use a normal i.e not noise


teacher model to generate soft or hard pseudo labels for the clean i.e not distorted unlabeled images and this is important they stress this here that when you when you label the unlabeled images you want to use the model that is without the noise and you do it on the not distorted unlabeled images so when you infer the labels it's very important that you have


clean accurate labels without any sort of noise in them so label noise is not something that they have found to help in this case so not label noise on the teacher that is so you can see right here on the unlabeled images we'll use that teacher model without the noise um to infer the labels now they say these can be hard model hard labels or soft labels so what does that mean if we generate


hard pseudo labels that means that the y here is simply going to be either 0 or 1 or 2 or 3 and so on so just the index of the class whichever class is most likely that's going to be our label this is exactly how the supervised data sets come right so this is what you'll think first when you see that however soft pseudo labels means that the y will be a distribution


so instead of being of class 0 it will be sort of let's say 90 percent of class zero but also five percent class one and five percent class two right so you'll output the distribution um instead of the just the label and they have found that the soft pseudo labels work slightly slightly better than the hard pseudo labels okay thanks so they use the soft pseudo labels here


because they work slightly better but you can do it with hard or soft labels the important thing is that you use the teacher to generate as accurate as possible labels for your unlabeled data then third we've already seen this learn an equal or larger student model which minimizes the cross entropy loss on labeled images and unlabeled images with noise added to the student model so


as you can see labeled images and unlabeled images so we're in this semi semi supervised learning setting right now you take in both together with noise and noise here is in bold which means they stress it again this is important so you can see that the loss is composed of two different things these are the true images of your original model and you use that


and this means you noise the student model and that noise can be on the data or in the model itself and here also the unlabeled images that you have labeled with the teacher you do the exact same thing so you train on both of these data sets and step four is if you want to do iterative training use the student as a teacher and go back to step two now they have uh some more tricks when


they do this iterative training they are also up the batch size during the iterative training and so on so they do a lot of things to make the student learn something more something better than the teacher and i think this the whole paper it doesn't it doesn't state it explicitly but i think the whole paper everything they do here is to kind of force or allow the student


to become better than the teacher by by giving more noise by making the student larger by making the batch size for the student larger and so on so you you want to sort of inject as much invariance as you can and that will make the student learn more so they say here noising student when the student is deliberately noised in its it is trained to be consistent


to the teacher that is not noised when it generates the pseudo labels in our experiments we use two types of noise input noise and model noise all right first data augmentation is an important noising method in noisy student training because it forces the student to ensure prediction consistency across augmented versions of an image specifically in our method the teacher


produces high quality pseudo labels by reading in clean images while the student is required to produce to reproduce those labels with augmented images as an input second when dropout and stochastic depth function are used as noise the teacher behaves like an ensemble at inference time when it generates pseudo labels whereas the student behaves like a single model


in other words the student is forced to mimic a more powerful ensemble model we present an ablation study so this it's a bit weird what they say here um don't be confused you use the dropout and the stochastic depth on the student model and they they say here if you do this the teacher behaves like an ensemble at inference time whereas the student behaves like a


single model and yeah it's it's a bit of a weird formulation but it's it's true like the teacher the teacher will produce these same uh the label for different pathways through the student if you use dropout and kind of stochastic depth and therefore the student is kind of required to approximate each time each forward pass has a different forward pass through the layers through


the connections with dropout and it's forced to approximate that teacher label with all of these um different things so you see that you you put in a lot a lot of techniques so they have even other techniques um there is one additional trick and it's not and it's not one actually they have so many tricks and if you look at their experimental setup that it's crazy


like they describe exactly we reduce the learning rate like this and the batch size like this and so on so to get state of the art on imagenet it's not enough to just have a good idea of a new thing to do what you you have to have the good idea and then execute it almost like really well um because you have to regard all of these additional tricks that people have figured out over the


years in any case they say it works better with an additional trick data filtering and balancing specifically we filter images that the teacher model has low confidence on since they are usually out of domain images so that goes to a point where if you see we have this imagenet label data set right and we have the larger data set now the larger dataset simply contains


images and there is no guarantee that the images are actually of the classes that we have in the imagenet data set right here we have a thousand classes here there's no guarantee that these images fit into any of those classes yet we still ask the teacher model to put them in some of these classes now you can filter out part of those images um if you can look at the teacher model and you look


at its confidence so when it outputs a distribution if if there's just two labels let's say if it outputs a distribution like this that's wildly different than if it outputs a distribution like this both are class 1 labels but one is much more confident than the other so what you want to do is you want to filter out these low confidence labels because you know the model isn't


really sure but it has to assign a class but that's usually an indication that it is an out of domain image so if they filter this it works better and then also to ensure that the distribution of the unlabeled images match that of the training set we also need to balance the number of unlabeled images for each class as all classes in imagenet have a similar number of labeled images


for this purpose we duplicate images in classes where there are not enough images for classes where we have too many images we take the images with the highest confidence okay so this is just another technique this has basically nothing to do with their core idea but this is just another thing uh where they say okay we can treat this big uh thing that we scrape from


the internet you know we can somehow filter and balance it smartly and that will work even better all right so let's go into the experiments of course there um so what they do i think where is the graphic what they do is they take an image net sorry they take an efficient net right here and they trade they first train an efficient net um a smaller efficient net as we said


for to be the teacher and then they train a larger efficient net for the student the best model in our experiments is a result of three iterations of putting back the student as a new teacher we first train an efficient at b7 on imagenet as the teacher model so you can see in the table right here what the b7 achieves the efficient net b7 here you can see it has 66 million parameters which is


fairly small compared to these other kind of previous state-of-the-art methods on imagenet right so they first train this and that will achieve something like an 85 percent accuracy now if you just train a larger model this efficient net l2 right here that has you can see 480 million parameters so a lot of more mainly parameters but you just train it on the same data set on imagenet you


will get a 0.5 improvement and you can see that here with noisy student training with the exact same model so it has the same amount of parameters you'll actually get an 88.4 so i like a more than a three percent improvement and that's with the same model just with this different training procedure and inputting these 300 million unlabeled images that you have laying around but the


all the information about all the label information comes from the imagenet dataset and comes from this efficientnetb7 teacher model so that's basically you can it's a testament that out of this out of this 85 you can make this 88 uh just by smartly using the information that the model that this model has learned about the data and transferring it to new data so they train


an efficient net b7 that's the small model as a teach model then by using the b7 model as the teacher we trained an efficient net l2 model with the unlabeled batch size set to 14 times the labeled batch size and they stress that it's important that you up the batch size that's another thing that makes the student learn more than the teacher then we trained a new efficient net so