that person who's totally happy that was abused every day of her life and they don't have the genetic disorder any longer you're gonna say wow that person doesn't look like a movie star that person doesn't look like they're vegetarian now that person doesn't look young and after whatever it is they're gonna look like a normal person and you're gonna say I identify with her identify her she can do it I'm into it now here's the cool part yeah we've seen then when people do this with the same stand on the stage and tell the story


that there's that the person in the audience with the same genetic condition does it in a shorter amount of time because the evidence becomes a loudest voice why because it's in testimony yes there's truth right in front of you that's it's it's it's right in front of you there's you know we had a guy in Dubai that had he was in a wheelchair with this was this tumor in his spine and the doctor sent him home to die he came across my book like a week before they got the book becoming supernatural mm-hmm and and then he read the book and

DO THIS Everyday To Unlock The FULL POWER Of Your Mind! | Joe Dispenza


he somehow got in there someone gave him a spot he was in a wheelchair stage for cancer or go home and died severe paralysis a limitation crowding the spinal cord the whole bit nothing we can do for you all these pain meds excruciating pain one week after the week long his tumor reduced by 30% I just saw him in Munich he's he can walk he's walking without his wheelchair I mean he isn't it he's in a new experience he he's believing now in himself and when you believe in yourself you believe in possibility you can't


have one without the other do you believe in yourself when you've never believed in yourself and you've got doubt inside of you all day long knowledge knowledge knowledge keep learning keep studying keep listening to it sooner or later that'll become a louder voice in your head than I can it's too hard I'll never change yeah and you me knowledge of knowledge of new philosophies knowledge of new skills knowledge of new anybody study if you don't want to read get on YouTube and one of course talk about how you can


change your cheese just start looking starts looking at the testimonies and websites I mean we have over 450 testimonies now people heal themselves not small burritos amazing start looking to see what did that person do and when they tell their story that's worse than yours you're gonna start going wow that person really had a tough one and a day overcame it well jeez why like well I could just forgive my father right now I give you right now I want to let them go and I want to be free but I don't want I don't want to give them my attention


because I give him my energy I wanna but I put my attention in my future soon later you're gonna come to it the way you do but if you don't have the knowledge then you believe in it last see you see there's people that we I saw this just recently and I was looking closely and there are people that do this work do do this transformation work in the meditations that we teach and they're so impatient and they're so entitled and they want an instantaneous change from the lack of feeling right that they never overcome themselves in


their meditation they never overcome themselves in the meditation and when they finish their meditation they believe in this work less hmm then there are people who say I can tell you the moment I made up my mind to change because I had reached the end and I made a decision and that decision to change carried an amplitude of energy that was greater than the hardwired programs in my brain and the emotional conditioning in my body and my body literally responded to my mind in that moment that the choice that I made became a moment


in time I would never forget and they'll tell you and that's the moment I remembered when I was going to change now those people then when they sit down to do the work there that the chemotherapy has worked the injections didn't work the radiation didn't work the surgery didn't work the diet didn't work the yoga didn't work this is this is now their end they have nothing else to believe in but themselves hmm and they go all in not 50% not 60% they're going all in they have nothing else to believe in now listen when they


go a little bit outside the known they've got they went a little further than where they normally would stop they push themselves to that next limit they started believing in themselves that they could do it a little bit more they they finish the meditation and they get up and believing it's it's it's working more than working less they're the person that's believing in themselves that's why because it's not the work it's your belief in yourself right it's and when you believe in yourself you believe in possibilities when you


believe in possibilities you got to believe in yourself who wasn't gonna believe it so people make these great strides in in the and and their own personal growth is a testament to the living organist and the living organism the species of human beings that's starting to believe well maybe we're not so a slit limited as we've been programmed to believe maybe we are more unlimited in and I'd rather throw in with that and if you don't think you you're your immune system isn't aware of viruses that could it could handle any


virus if I got the right signal and if it was in a state of wholeness and your thymus was activated and your blood flow to that Center was turned on because you decided to turn it on and and you decided to release those chemicals that suppress the survival centers in your brain because oxytocin does and you wanted to stay there for a period of time and memorize that feeling I guarantee you that thymosin would begin to signal those t-cells and those t-cells would activate their t-cell receptors and those t-cell receptors


good morning everyone the first talk of today is by Jason Lee Corporation University of audacious of departing star customer meetings and over characterization and transition yeah so of course obviously deep learning super successful at blah blah blah it solves a lot of stuff it's successful in a lot of things some people I've even gone as far as claiming it's the new electricity of course when we have electricity we should look there's a field call electrical engineering the studies it I guess it's kind of a purpose of us being

On the Foundations of Deep Learning: SGD, Overparametrization, and Generalization


here this summer so deep learning is not quite a science yet it's fairly crazy that things regardless of whether these things where people will try crazy things like these cyclic learning rates learning rate to nice games are pretty crazy they're fragile even things like changing seeds and it's sort of like we're trying to build a working system by trial and error instead of understanding something like civil engineering and then sort of you know been our deep learner has built this thing but you didn't take a structural


engineering class so that's what happened so this talk is trying to about unkind understand deep learning so motivate of course by this well-known paper I added an optimization at the end I think they forgot that they left out something pretty important there okay so understanding deep learning and we so there's obviously two aspects our goal is to minimize test error we want to do say multi-class classification as usual you can think of as two components an optimization error and the generalization error something


controlled by the algorithm and something simply statistical so there's of course this problem is challenging in both aspects the optimizations non-convex non-smooth the global landscape right nothing is very good about it lots of local minima and just in general it's difficult the reason about the only thing we can see are really gradients at Hessians and so forth local information and you want to draw global conclusions on the statistical side similarly bad there's more parameters than samples that generalization error literally


depends on everything your algorithm you're learning race schedule your architecture initialization scheme algorithm parameters such as momentum so forth okay so how to deal with optimization luckily this is probably where it's most well understood is simply people have realized that you can always change the model that if one model is difficult to fit we don't have ever believed any models correct or anything like that you just fit a bigger model and bigger models are easier for SPD to fit which is somewhat of an


empirical fact I know so you can have this model with many bad local minima do something and then SGD works so over privatization okay here's an experiment showing this this experiment is essentially replicated from this paper of living Isha Mirror and she'll have shorts so on the left hand side the data comes from a teacher network that's fairly small compact as 50 neurons on the learner side the network has doubled the number new ons there's a hundred neurons but still the data comes from a network with only 50 neurons so if you


try to run SGD on the well-specified architecture which is very sensible assuming you saw through a statistics class if the model comes from the family you want to be well specified Emily's asymptotically efficient da da da whatever by the end of the day actually fails so statistical efficiency doesn't matter right if you can't get training our low why do you care about statistical efficiency because you cannot find the mo e you run SGD five times it gets five different answers none of which are the global minima I


could have done this five thousand times I don't think you would have found the global minimum okay now in stark contrast simply the same data comes from the same source and I simply change slightly change the architecture the only thing is doubling the number of neurons and I run STD again five times and every single time it's lost value zero and this is also tester this is STD on fresh cause of data this is not training here so SGD is finding the global minima of the population loss empirically so seemingly over


privatization does in fact make optimization easier and in this case even made because it's tests are you could have even extracted some statistical statement here okay so recently in many a series of papers by myself yen's it also briefly talked about this yesterday Trenton has also worked on this I'm summarizing all of these results because they're all kind of similar enough I don't want to spend time distinguishing they basically say if you have very wide networks and you initialize randomly with appropriate


variances that are not crazy gradient descent converges to an epsilon global minimizer as long as the loss is convex your learning rate is small roughly speaking or works for any model so when you're sufficiently over privatized you get an epsilon approximate global minimizer of the training loss let me sketch a proof of why this could be true of course when you're over parametrized looking at the parameters is not a great idea they're not identifiable there's permutations and all sorts of other invariances that come out that the


parameters don't really respect it's a better idea to look at how the prediction changes so if you compute the change in the loss you'll find that it's related to the outer product of a certain Jacobian matrix or though or the gram matrix as long as this gram matrix is strictly positive definite you get a contraction so then if I could ensure that this thing stayed positive definite then you convert to a global minimizer so why does this the first thing is to establish that it started positive definite that's pretty clear random


matrices are positive definite so this is some sort of just a concentration and perturbation analysis to show that initialization the gram matrix state is positive-definite and then simply when you're very over Prime Christ M is the width of the networks then it stays positive definite there's always hiding a bunch of stuff it's whatever it's something and then you need am big enough to make this smaller than lambda zero okay so over parameterization essentially what it does is if it's forcing in this initialization scheme it's forcing the


grab matrix to stay very stable close to this initialization and thus stay well conditioned and finally you get the great from this you can extract statements of this type foursquare loss you can say exactly converges to global minima at a log 1 over epsilon rate if this will say logistic loss you could say that it converges an epsilon global minimizer where Absalon is determined by the width if you want in smaller you would need wider because logistic loss is not global we strongly come back to so many locally well regardless for wide


enough networks as long as your loss is any convex function your you can get a statement like this ok so after we so here's a simpler way to interpret these statements if you don't want to read 15 pages of calculating algebra one way to interpret these statements is a statement about the local geometry of the loss function you take your random initialization draw a ball around it of some norms see for some appropriate seat that depends on the width you set every critical point in B is a global minimizer and there's at least one


global minimizer here and Gd store stays in this set so it's a statement about local geometry is saying that locally things essentially look convex as long as you find a critical point in this local set it will be a global minimizer and in fact you do stay in this set so roughly the global geometry of deep networks is it's probably pretty bad I think there's definitely it should have exponentially many bad local minima local minima with even high values of crane loss however locally there is a global minima locally there exists


global minima and you can find a region in which there are global minima but no local minima no local minima of higher loss yeah one way is you take in under parametrize network and start introducing neurons and then set them to be like to sort of cancel each other configure the ways you add them in a way that cannot help but it's still a local minima I can make at least I can make it a higher order saddle up certain but I'm not saying you'll find them yeah I'm saying there are ones and certainly not near random initializing


oh I think I can always do this I take your architecture I just make a tiny one if you believe there's one in a very small architecture I start augmenting neurons in some useless way so it will look like a standard architecture but maybe the parameters will be in a weird configuration at least I can make a satellite high order satellite okay so global geometry probably not very good local geometry very good there is a global minima and gradient descent converges to it so we're kind of happy and then oh you think about it a little


bit more and then some paper start coming out and then you realize of course locally it's very good locally you can write down any function so f theta is my neural network function it's essentially equal to some F 0 it's not important think of f 0 as 0 or very close to 0 some order one quantity a gradient term and a second order term so of course since I'm local the second order term should be thought of as smaller than the delayed the linear term so we kind of just throw it away and that's what these papers are doing and


then so what's happening here is that if you're able to change the predictions f of X by an order one quantity while the Jacobian the gradient stay constant in a relative sense then the second order term is essentially vanishing the second order term is how fast this changes so if I can move f of X by order one which is all I need to move it because I need to move it to match Y well this stays constant then I'm done I've found that I've constructed a global minimizer that is very close to my initialization close


in the sense that there's no neg there's no potential negative curvature and that's essentially what that proof was saying it was saying this H of the H matrix the gram matrix is simply outer products of these gradients and it was not moving a lot under these initialization schemes on widths and so forth ok so she's out and Bach came up with a nice sufficient condition of when you should expect this they call it this kernel regime to happen essentially the sufficient condition is that think of Y minus F 0 as order one


we initialize in network so it outputs an order one quantity that's random so this is like 1 because Y is order 1 and I basically pointed out that if you're Hessian divided by the square norm of the gradient is smaller than one then the second order term doesn't matter and your gradient dynamics track very closely for some constant amount of time the dynamics of a of gradient descent on a kernel machine this is very intuitive because it's essentially saying exactly that how much the Hessian changes is very small to the gradient and the


gradient when the gradient is big then you only need to move the parameters a little bit and you move your predictions by a lot so roughly in all of the ways we initialize the Hessian is something like going to zero with the width and the gradient is of one and so we have this sort of linear behavior in some region around the initialisation okay so any questions here before I move on to you saying the final state is an interpretive approximation an initial and that this linear approximation is good is this sort of an idealized model


motivated by a team that works and how you think about this analysis appropriate does this change so it was all like this so I would this result I would think as if this sufficient condition holds and you use a very small learning rate then the gradient on F theta F theta sub peak and you can construct a new function that is linear call it f bar theta T these are very close for a constant amount of time so a linear that gradient dynamics track those of a gradient dynamics on a linear model for some amount of time right but


that's like a statement of the result yes the motivating slide you had a prompt was an appearance a set of empirical selfish genre that you know sort of empirical results instead in our models you know with these sort of things no this is my state oh yeah there was the ICL I think John was the little thing for two years ago I know that was that was on a state of the art right this paper was about model this was about generalization I'm only talking about optimization right now this paper is about generalization this statement


has nothing to do it these I mean you can make extract a statement about statistical error here but it's not a strong state the this is really an optimization thing I would say it's saying that I mean that's what I'm asking well in your mind what does this say about the state of the art motivation like that this would you be able to give me you continue could ask whether the initial configuration the final configuration of pulse like this they're not close really not close like this so I mean that in your mind and


with God is does this I think it tells you initial that you're you can get training over small very quickly initially in the first few steps so it seems that one could come up with empirical protocols and somehow see how relevant this is to what's going on in practical also I've said a word that they're not quotes does that mean that we run experiments and you know this does not really capture what's going on in in standard models yes there's there's some careful papers not buying me circle papers that say that this is


not yes and is there any evidence that indicates otherwise or is it depends on like your parameter settings if you set the learning race small it agrees well if you set the learning rate big then no it disagrees if you set a small aircrews yeah but there are certain values that we usually use in practice so and the ones like a good test error disagree Cilicia pretty simple question I hope but how should we think about Hessians for models that don't actually have Hessians you should think about it as okay one quantitative


way to think about it is the change of activation pattern which is a second sigma prime is the activation pattern and sigma prime prime is how much those change in some measure the Hessian is a border one over the has seen itself without the Hessian you don't care about the Hessians movement while you carry we are worried about is a gradient moving a lot because that means your feature scheme is moving a lot you want to ensure the feature scheme does not move so you need about the size of the movement of the feature


scheme which is the Hessian relative to the size of the gradient because the gradient multiplied by your movement is your change in F that's why it's a relative measure yeah so which Darby are measured that has some potential because still the size of the everything is just awesome Park is the only parameter moving is f so then the norm doesn't matter but the the size of the parameter depends on that sorry you should be L to a spectrum but everything in that paper pretend so the only thing changing us out sure but


I mean just even the site also yeah if you're careful you it's polynomial this paper doesn't talk about I'm not sure they talk okay so somewhat this at least tells us that at least locally we optimization is not a big deal so perhaps you can believe that if you your learning rate initially is not very big you can get your optimization error kind of small if you try to use this kind of method to get an analysis of generalization error unsurprisingly since your feature scheme is not changing what you'll find is


simply it has the same prediction exactly as that of a kernel method the kernel is called the neural tangent kernel and your generalization bound will look exactly like rate regression will be something like like label Y transpose kernel inverse y / n whole thing under square root so that's some generalization bound you can extract from this style of analysis so I have no I I don't think it's very tight in fact I'll talk about later okay so let's look a little bit more at generalization error what's happening a very standard


found in generalization or some complexity over N for example in kernel methods it would be Y transpose K inverse Y that's sort of the RK just norm in Ridge regression it could be VC dimension VC dimensions roughly number of parameters times the depth in feed-forward networks the required list all kind of big they're bigger the sample size which is exactly what the I Claire paper was pointing out it was pointing L essentially that the number of parameters is roughly 30 20 30 times the number of samples in many


models and of course of course so this slide is slow and from naki or I think from telling you actually who stole it from naughty and I copied it from time so we've all seen this many times I'm sure by now so what they did in there my clear paper is the number so you the black line is a training curve and if you took statistical learning you were to kind of expect the red line kind of goes well it doesn't it goes down so an over parameterize ation does not seem to hurt generalization in this case in fact in this one example it kind of helps a


little you can improve your generalization even after interpolation and there's even more evidence that number of parameters are hurting if you plot the imagenet top one error over time then you see that in fact for these very very big networks the top one error is quite small it means be creasing here I mean this is cherry pick from this paper but there's clearly networks with 600 million parameters and it has passed they are very low so throwing in parameters in a non stupid way will not hurt your generalization I'm here okay


so what's going on here let's turn to margin theory and see what it tells us so this margin theory is very classical it basically says that if you're very far from the decision boundary you should be very good why is that Pete Bartlett and Mendelssohn coalesced this almost 20 years ago now basically your generalization cap is upper bounded by some complexity measure divided by the margin so if you can ensure some sort of Rademacher complexity that's sort of size free independent of the explicit independent of the width then


you can get good generalization error so here's some papers that talk about how to opera BAM the numerator here I'm only going to talk about how to lower bound the denominator okay so how do you get solutions with good market let's look at the simplest loss function with the simplest regularization scheme is you have logistic loss with a sum norm regularizer for example weight DK l2 norm you would kind of hope so this is the Globo max margin this is the best you can do if you search over all models in your parametric family and you thought


you kept the one with largest margin gamma star so you would hope you can get gamma star and in fact if you do a very good job of minimizing this regularized functional for small lambda then you do get gamma star so in other words if week l2 regularization leads to assuming optimization works week our two regularization will get you very high margin or the best possible March okay this this proof is a simple unlike a lot of you into this proofs I think this is genuinely simple okay the proof is essentially you write down


the logistic loss realize that logistic loss when this argument is very small can be approximated by Taylor's theorem and I could see an exponential exponential of a lot of terms added together you only need to keep the smallest one because the rest are exponentially smaller okay so that this line is Taylor's theorem and then this one says if you add a bunch of like e to the minus ten plus e to the minus hundred you only need to care about either minus ten so only the smallest of these terms matter that's why there's a min here and


then this from this you can kind of read off the results if you lambda very small it's saying let me use as much norm as possible but among those I need a fit I need to make the worst case margin good so among solutions of the same norm we prefer the largest margin okay so how does so how does over parametrization improve the margin this is completely obvious if I have a network and this site if I have a sub network of this network the margin can only improve so if you have a Rademacher bound that's a independent of the explicit number of


parameters the margin is only improving this denominator can only get bigger okay so let's talk a little bit about how to optimize when you have a regularizer the first thing that realize is that none of these results using n key K and locally convex or whatever can ever handle this the regularizer induces a sort of tie breaking among the global minimizer's and the global minimizer's are not equivalent like you can have two global minimizer's the unregular eyestrain loss and they will have drastically different regularization


effects so the regularization induces a tiebreaking and now you really do need to find like a global minimizer of this regularize objective so how to do this the answer is not fully satisfying but we can say something so take a very very very very wide network think of this as like wider than anything you've seen exponential in D or X Panetti whatever infinite okay then run gradient descent with a particular noise scheme then you convert your global minimizer in polynomial time polynomial in the dimension and in one over epsilon


something like Pauly D over epsilon four or something okay so gradient descent I'm very very very over parameterize networks converges to global minimizer's so over parameterization does help even when you have regularization the mechanism in which it helps is very different from this local convex intuition the intuition here is that when you're very very over primate rising there's a descent direction in function space but if you have a finite a small width network you might miss the function space because you might miss


the direction of this sign in the function space that's essentially what Frank wolf algorithm will do but computing as Frank will sup is a exponential time so but instead if you had a ton of neurons and you add noise then there might be an exponentially small fraction of your neurons I see this Frank wolf direction but then great relu is homogeneous and then although it's exponentially small signal it goes up exponentially too and these things carefully balanced and at the end you do get a polynomial time result


so because the number is very large are you what's relationship for the ndk regime this no more I'm Kiki and you guys caught yeah you change anything NDK Scott it's very fragile and so I know I remember some of these results at that on laminate extremely small you set lambda and whatever way let's say you want gamma Magnus point one and you set lambda depending on that it's polynomial time to get within constant a polynomial iteration it's not pollen on with one iteration is exponential time because you have exponentially many parameters


some polynomial iterations or time in the pde sense and one other question so in the generation ban you're talking about a size free Rademacher bound right obtaining the margin with a very light regularization parameter president Ron a MicroBot might depend on say norms which is n have to be balanced against this resulting mark yes you should think of whatever norms is this let's concretely take the goal of which one which is just a Frobenius norm of everything then that would be weight decay okay all right so so you might ask um


okay so why add this regularizer I can already get the global min of the logistic loss without this regularization term what am I getting here is my stat sample complexity much better and it turns out it is let's look at a very simple data set on the first two coordinates the data looks like this it's like two eggs or essentially then every other coordinate is just standard normal so there's two coordinates that have signal and that's the ones you want to pay attention to the rest of the coordinates are completely uncorrelated


with the label so they're noise coordinates are kind of for you and they exactly do full a kernel method a kernel method is unable to localize onto the first two dimensions so it has to look over all dimensions and it pays sample complexity at least d squared to get some air last if you want R less than some absolute constant you need to pay at least d square samples because a cardinal method to solve this problem needs at least form pairwise degree to polynomials that has complexity d squared however for that sort of to XOR


construction there's a neural net with four neurons and then so then the sample complexity of learning a neural net with four neurons is something like four D over N so D over and roughly so there's a clear sample complexity separation with regularizer you can learn with D samples without regularizer uni at least d square samples so regularization explicit regularization or math you yes yes yes good you might put it it's not but we don't yeah it's not clear yeah yeah they might take exponential time we don't know no we know it takes


exponential time how you scale things but I can scale this so it takes you expert but then you can increase your learning rate anyway to analysis in the specific city yeah I guess I should be more precise to be more precise this one is better than the NDK one let's play it that way so then it covers the unregular eyes and the regular ice case okay so so do you need regularization it turns out you can do pretty well if you look at these numbers without regularization so like the gain here is not from regularizing you gain 5% but if you just


change the architecture you can gain around 10% so SGD without regularization does already have very good generalization perhaps it's not state-of-art but certainly the bulk is not from these regularization methods okay so let's turn to a simple example of logistic regression so even if you have separable logistic regression the problems convex but not strongly or strictly convex so there's many many global minimizer's and you may be wondering which one does it converge to this is exactly what soldiery Hoffer and


cerebro asked a year or two ago and they showed that this converges to the SVM solution in a very precise sense if you run gradient descent for a long long long time then you normalize because the norm blows up you get exactly the auto SVM solution after doing some normalization this is quite amazing if you haven't seen it before there's all these directions why the heck should gradient descent get the 1 of minimum how to norm that maximizes the l2 margin right I mean this seems maybe I should depend a loading rate how


you initialize it could depend on anything and this result is quite stable any learning rate that's stable it converges to the whole to SVM solution ok so then so what is then you might be thinking what is special about gradient descent let me write down gradient descent in a very suggestive way I'm just write it down so it's trying to maximize the correlation you know yeah because the logistic lost gradient is never zero so it's impossible to have a zero vanishing gradient even if you this is in contrast to least-squares I


think which is where you're thinking yes okay what was I saying yes this norm okay the important thing is that you write gradient descent in this suggestive way so what it's saying is that gradient descent is trying to maximally decrease the lost value of the function while paying infinitely small amount of l2 norm the key thing is is trying to use the least amount of Al to norm to achieve this goal so now you might ask why do I change a norm and this gives you a family of steepest descent algorithms and in fact you can


prove that for this entire family of steepest descent algorithms you converge the SVM given in that normal okay so for example that one norm this is exactly a form of boosting a to prove this a long long time ago in s PhD that in fact does maximize the l1 margin the same proof works for all norms okay so some examples coordinate descent which is steepest ascent with respect to l1 norm you're going to maximize l1 margin it's not quite adaboost it's adaboost with some damping step size so add a boost with a dampened step size that maximizes


our margin adaboost itself does not sign gradient method commonly used now the save on communication this gets you some L infinity bias and of course gradient descent is steepest descent with respect to l2 okay so that's kind of that's great we understand logistic regression very well that's sort of always the starting point so now the question is what does it do on deep networks if you're if you're very very very optimistic you may hope that it solves max margin even if you do know regularization because then you will


have some generalization bound that only depends on this up here and that seems good because now you can get the optimal margin of course notice that this is an SVM problem but it's a non convex SVM problem now this SVM problem unlike the linear case has many many stationary points as first order stationary points so I can order station points on a global max so of course you cannot sort prove that it converges to the global Maximizer well you're able to prove is that it converges the first order optimal point on the following SVM this


is some nonlinear program it has first order KKT conditions and assuming the only assumption here is homogeneity then if you do gradient descent exponential loss logistic loss you get a first order optimal point the first order stationary point of this okay so you wanted to prove it gets max margin that's probably not possible because you're running a local search algorithm but at least you can characterize that the whatever you converge to is very special in the sense that it's precisely a critical point of


a fairly intuitive optimization problem okay so I've talked about I didn't really talk about the implicit regularization of n key K but if you think about what kernel methods is you can write down the sort of implicit bias of NPK or the inductive bias of it as sort of you're trying to find the thing that best that maximizes the margin the worst-case margin but you stay infinitely close to your initialization so this top one theta hat K is exactly what would happen if you did n TK on logistic loss and terminated after some amount of time a


very long time you had approximately get maximum margin maximum with respect to a different sense in the sense that you have to stay infinitely close to this ball what the this previous result by us and Haiphong Liu and Jen we chose that actually you get a stationary point of this following program and of this program you're letting the prompter move infinitely far from its initialization so it's completely forgetting where it's initializing it's running forever and ever which you'll always do when you have a


logistic loss and when you run forever and ever you try to maximize the margin okay these are of course two endpoints of an extreme case on one of these you're trying to say infinitely close to your initialization the other you're moving infinitely far so they're very different with endpoints probably in my opinion what's interesting is probably when this is not quite zero you're going slightly further than the linear regime but you're clearly not going to this super asymptotic regime you've deviated slightly from the N key


K and what is happy there we don't really know but I think that's probably likely to correspond to closer to practice than the other end point because things like large learning rate finite width will cause you to sort of stay infinitely close yeah okay so the final thing I want to talk about is how does architecture matter so I've told you kind of that gradient descent asymptotically the bias it gets you is an l2 regularization on all of the parameters but that's sort of uninteresting why should you ever care


about the parameters again parameters have no meaning in your own nets the only thing that matters at the end is your function f this is all you see a test time the parameter is just a way to encode F so what you should really ask is how does this bias translate over here so what is the bias on the prediction function and this is clearly where architecture matter so for example here 1 3 & 4 there was no regularization the algorithm was just SGD the only thing that changes is the architecture yet you get a huge improvement in the test error


luckily I don't know the answer for neural nets but in linear networks there's a very crisp answer you can take a linear function write it as a product of matrices then if you have outer regularization on w's which comes from gradient descent it translates a shot in quasi norm on theta on the linear function okay so regardless of the depth and the width the prediction function class stays linear you know you can write a linear map a matrix as a product as one matrix as a product of five matrices is still matrix the function


class number changes or simply varying the parameterization combined with algorithm changes the inductive bias here and here you can characterize in a very precise sense but whatever gradient can descend converges to is a stationary point of some problem with this shot and quasi norm okay a similar phenomenon in convolutional networks same thing you can take a linear convolutional net and write it as of comp of predictors you'll also get a quasi norm except now it's sparsely instead of on the singular values okay


no it stops its learning like it's always changing zone on versatility the function is infinitely flat so in fact you can even grow the life some solar league rate of those convergence yeah you need sufficient design there's nothing okay so some random thoughts let me finish so of course we've seen now that by training Colonel training neural nets now stay close to initialization using a kernel classes so how'd it go beyond kernels UN's I mentioned this yesterday and of course one thing missing here is how will distributional assumptions help


most of these results have not used distributional assumptions in any meaningful way I would say I mean there's a lot of results about learning single relu single convolutional filters they use Gaussian assumptions of course and those argue and even then you can see the Gaussian are helping you that much there were other ways to learn it just maybe not quite gradient descent so what are some reasonable distributional assumptions that can help us learn things via kernels don't really depend on the distribution their optimization


at least so what happens right after the ndk regime we figure out the end point in a sense we figured out what happens locally we've figured out what happens if you move infinitely far but the whole middle I don't know it's kind of interesting in architecture design an inductive bias so you know people come out with new architectures every week and how does this change the inductive bias of SGD if I do SGD on ResNet versus amoeba net block then you know what is changing here let's say I just use STD on both but the


functions I learn are different how are they different and why does amoeba net whatever generalize better [Applause] nothing GD is just easier to analyze most of the results hold for a skewed yeah that was for single oh no that's a matrix this one's a matrix this one's a matrix you minimize there's two results one is you minimize cross-entropy with explicit weight DK you get the quasi norm second one is you minimize cross-entropy with TD no there's no optimization these are statements about global minimum okay


the first statement is a statement about global minimum the second statement is invoking this following theorem this says that you convert a force or the optimal point of a problem an SVM where this is the Shatan clause anymore so the inductive bias of having deep that's in a linear network using logistic loss is getting closer to is a crank this is about first-order okay so gradient descent the algorithm is involved here it's about gradient descent you run gradient descent something happens and what happens is a


[Music] in this video we'll be discussing the concept of semi-supervised learning semi-supervised learning kind of takes a middle ground between supervised learning and unsupervised learning as a quick refresher recall from previous videos that supervised learning is the learning that occurs during training of an artificial neural network when the data in our training set is labeled unsupervised learning on the other hand is the learning that occurs when the data in our training set is not labeled so now onto semi-supervised learning

Semi-supervised Learning explained


semi-supervised learning uses a combination of supervised and unsupervised learning techniques and that's because in a scenario where we'd make use of semi-supervised learning we'd have a combination of both labeled and unlabeled data let's expand on this idea with an example say we have access to a large unlabeled data set that we'd like to train a model on and that manually labeling all of the state ourselves is just not practical well we could go through and manually label some portion of this large data set ourselves

Semi-supervised Learning explained


and use that portion to train our model and this is fine in fact this is how a lot of data use for neural networks becomes labeled but you know if we have access to large amounts of data and we've only labeled some small portion of the data then what a waste it would be to just leave all the other unlabeled data on the table I mean after all we know the more data we have to train a model on the better and more robust our model will be so what can we do to make use of the remaining unlabeled data in our data set well one thing we can do is

Semi-supervised Learning explained


implement a technique that falls under the category of semi-supervised learning called pseudo labeling this is how pseudo labeling works so as just mentioned we've already labeled some portion of our data set now we're going to use this label data as the training set for our model we're then going to train our model just as we would with any other labelled data set okay and then just through the regular training process we get our model performing pretty well so everything we've done up to this point has been just regular old

Semi-supervised Learning explained


supervised learning in practice now here's where the unsupervised learning piece comes into play after we've trained our model on the labeled portion of the data set we then use our model to predict on the remaining unlabeled portion of data we then take these predictions and label each piece of unlabeled data with the individual outputs that were predicted for them this process of labeling the unlabeled data with the output that was predicted by our neural network is the very essence of pseudo labeling now after labeling the unlabeled data

Semi-supervised Learning explained


through this pseudo labeling process we then train our model on the full data set which is now comprised of both the data that was actually truly labeled along with the data that was pseudo labeled through the use of pseudo labeling were able to train on a vastly larger data set we're also able to train on data that otherwise may have potentially taken many tedious hours of human labor to manually label the data as you can imagine sometimes the cost of acquiring or generating a fully label data set is just too high or the pure

Semi-supervised Learning explained


act of generating all the labels itself is just not feasible so through this process we can see how this approach makes use of both supervised learning with the labeled data and unsupervised learning with the unlabeled data which together give us the practice of semi-supervised learning so hopefully now you have an understanding of what semi-supervised learning is and how you may apply it and practice through the use of pseudo labeling and I hope you found this video helpful if you did please like the video subscribe suggest

Semi-supervised Learning explained


I want to talk to you about the future of medicine. But before I do that, I want to talk a little bit about the past. Now, throughout much of the recent history of medicine, we've thought about illness and treatment in terms of a profoundly simple model. In fact, the model is so simple that you could summarize it in six words: have disease, take pill, kill something. Now, the reason for the dominance of this model is of course the antibiotic revolution. Many of you might not know this, but we happen to be celebrating

Soon We'll Cure Diseases With a Cell, Not a Pill | Siddhartha Mukherjee | TED Talks


the hundredth year of the introduction of antibiotics into the United States. But what you do know is that that introduction was nothing short of transformative. Here you had a chemical, either from the natural world or artificially synthesized in the laboratory, and it would course through your body, it would find its target, lock into its target -- a microbe or some part of a microbe -- and then turn off a lock and a key with exquisite deftness, exquisite specificity. And you would end up taking a previously fatal, lethal disease --


a pneumonia, syphilis, tuberculosis -- and transforming that into a curable, or treatable illness. You have a pneumonia, you take penicillin, you kill the microbe and you cure the disease. So seductive was this idea, so potent the metaphor of lock and key and killing something, that it really swept through biology. It was a transformation like no other. And we've really spent the last 100 years trying to replicate that model over and over again in noninfectious diseases, in chronic diseases like diabetes and hypertension and heart disease.


And it's worked, but it's only worked partly. Let me show you. You know, if you take the entire universe of all chemical reactions in the human body, every chemical reaction that your body is capable of, most people think that that number is on the order of a million. Let's call it a million. And now you ask the question, what number or fraction of reactions can actually be targeted by the entire pharmacopoeia, all of medicinal chemistry? That number is 250. The rest is chemical darkness. In other words, 0.025 percent of all chemical reactions in your body


are actually targetable by this lock and key mechanism. You know, if you think about human physiology as a vast global telephone network with interacting nodes and interacting pieces, then all of our medicinal chemistry is operating on one tiny corner at the edge, the outer edge, of that network. It's like all of our pharmaceutical chemistry is a pole operator in Wichita, Kansas who is tinkering with about 10 or 15 telephone lines. So what do we do about this idea? What if we reorganized this approach? In fact, it turns out that the natural world


gives us a sense of how one might think about illness in a radically different way, rather than disease, medicine, target. In fact, the natural world is organized hierarchically upwards, not downwards, but upwards, and we begin with a self-regulating, semi-autonomous unit called a cell. These self-regulating, semi-autonomous units give rise to self-regulating, semi-autonomous units called organs, and these organs coalesce to form things called humans, and these organisms ultimately live in environments, which are partly self-regulating and partly semi-autonomous.


What's nice about this scheme, this hierarchical scheme building upwards rather than downwards, is that it allows us to think about illness as well in a somewhat different way. Take a disease like cancer. Since the 1950s, we've tried rather desperately to apply this lock and key model to cancer. We've tried to kill cells using a variety of chemotherapies or targeted therapies, and as most of us know, that's worked. It's worked for diseases like leukemia. It's worked for some forms of breast cancer, but eventually you run to the ceiling of that approach.


And it's only in the last 10 years or so that we've begun to think about using the immune system, remembering that in fact the cancer cell doesn't grow in a vacuum. It actually grows in a human organism. And could you use the organismal capacity, the fact that human beings have an immune system, to attack cancer? In fact, it's led to the some of the most spectacular new medicines in cancer. And finally there's the level of the environment, isn't there? You know, we don't think of cancer as altering the environment.


But let me give you an example of a profoundly carcinogenic environment. It's called a prison. You take loneliness, you take depression, you take confinement, and you add to that, rolled up in a little white sheet of paper, one of the most potent neurostimulants that we know, called nicotine, and you add to that one of the most potent addictive substances that you know, and you have a pro-carcinogenic environment. But you can have anti-carcinogenic environments too. There are attempts to create milieus, change the hormonal milieu for breast cancer, for instance.


We're trying to change the metabolic milieu for other forms of cancer. Or take another disease, like depression. Again, working upwards, since the 1960s and 1970s, we've tried, again, desperately to turn off molecules that operate between nerve cells -- serotonin, dopamine -- and tried to cure depression that way, and that's worked, but then that reached the limit. And we now know that what you really probably need to do is to change the physiology of the organ, the brain, rewire it, remodel it, and that, of course, we know study upon study has shown


that talk therapy does exactly that, and study upon study has shown that talk therapy combined with medicines, pills, really is much more effective than either one alone. Can we imagine a more immersive environment that will change depression? Can you lock out the signals that elicit depression? Again, moving upwards along this hierarchical chain of organization. What's really at stake perhaps here is not the medicine itself but a metaphor. Rather than killing something, in the case of the great chronic degenerative diseases --


kidney failure, diabetes, hypertension, osteoarthritis -- maybe what we really need to do is change the metaphor to growing something. And that's the key, perhaps, to reframing our thinking about medicine. Now, this idea of changing, of creating a perceptual shift, as it were, came home to me to roost in a very personal manner about 10 years ago. About 10 years ago -- I've been a runner most of my life -- I went for a run, a Saturday morning run, I came back and woke up and I basically couldn't move. My right knee was swollen up,


and you could hear that ominous crunch of bone against bone. And one of the perks of being a physician is that you get to order your own MRIs. And I had an MRI the next week, and it looked like that. Essentially, the meniscus of cartilage that is between bone had been completely torn and the bone itself had been shattered. Now, if you're looking at me and feeling sorry, let me tell you a few facts. If I was to take an MRI of every person in this audience, 60 percent of you would show signs of bone degeneration and cartilage degeneration like this.


85 percent of all women by the age of 70 would show moderate to severe cartilage degeneration. 50 to 60 percent of the men in this audience would also have such signs. So this is a very common disease. Well, the second perk of being a physician is that you can get to experiment on your own ailments. So about 10 years ago we began, we brought this process into the laboratory, and we began to do simple experiments, mechanically trying to fix this degeneration. We tried to inject chemicals into the knee spaces of animals


to try to reverse cartilage degeneration, and to put a short summary on a very long and painful process, essentially it came to naught. Nothing happened. And then about seven years ago, we had a research student from Australia. The nice thing about Australians is that they're habitually used to looking at the world upside down. (Laughter) And so Dan suggested to me, "You know, maybe it isn't a mechanical problem. Maybe it isn't a chemical problem. Maybe it's a stem cell problem." In other words, he had two hypotheses.


Number one, there is such a thing as a skeletal stem cell -- a skeletal stem cell that builds up the entire vertebrate skeleton, bone, cartilage and the fibrous elements of skeleton, just like there's a stem cell in blood, just like there's a stem cell in the nervous system. And two, that maybe that, the degeneration or dysfunction of this stem cell is what's causing osteochondral arthritis, a very common ailment. So really the question was, were we looking for a pill when we should have really been looking for a cell.


So we switched our models, and now we began to look for skeletal stem cells. And to cut again a long story short, about five years ago, we found these cells. They live inside the skeleton. Here's a schematic and then a real photograph of one of them. The white stuff is bone, and these red columns that you see and the yellow cells are cells that have arisen from one single skeletal stem cell -- columns of cartilage, columns of bone coming out of a single cell. These cells are fascinating. They have four properties.


Number one is that they live where they're expected to live. They live just underneath the surface of the bone, underneath cartilage. You know, in biology, it's location, location, location. And they move into the appropriate areas and form bone and cartilage. That's one. Here's an interesting property. You can take them out of the vertebrate skeleton, you can culture them in petri dishes in the laboratory, and they are dying to form cartilage. Remember how we couldn't form cartilage for love or money? These cells are dying to form cartilage.


They form their own furls of cartilage around themselves. They're also, number three, the most efficient repairers of fractures that we've ever encountered. This is a little bone, a mouse bone that we fractured and then let it heal by itself. These stem cells have come in and repaired, in yellow, the bone, in white, the cartilage, almost completely. So much so that if you label them with a fluorescent dye you can see them like some kind of peculiar cellular glue coming into the area of a fracture, fixing it locally and then stopping their work.


Now, the fourth one is the most ominous, and that is that their numbers decline precipitously, precipitously, tenfold, fiftyfold, as you age. And so what had happened, really, is that we found ourselves in a perceptual shift. We had gone hunting for pills but we ended up finding theories. And in some ways we had hooked ourselves back onto this idea: cells, organisms, environments, because we were now thinking about bone stem cells, we were thinking about arthritis in terms of a cellular disease. And then the next question was, are there organs?


Can you build this as an organ outside the body? Can you implant cartilage into areas of trauma? And perhaps most interestingly, can you ascend right up and create environments? You know, we know that exercise remodels bone, but come on, none of us is going to exercise. So could you imagine ways of passively loading and unloading bone so that you can recreate or regenerate degenerating cartilage? And perhaps more interesting, and more importantly, the question is, can you apply this model more globally outside medicine?


What's at stake, as I said before, is not killing something, but growing something. And it raises a series of, I think, some of the most interesting questions about how we think about medicine in the future. Could your medicine be a cell and not a pill? How would we grow these cells? What we would we do to stop the malignant growth of these cells? We heard about the problems of unleashing growth. Could we implant suicide genes into these cells to stop them from growing? Could your medicine be an organ that's created outside the body


and then implanted into the body? Could that stop some of the degeneration? What if the organ needed to have memory? In cases of diseases of the nervous system some of those organs had memory. How could we implant those memories back in? Could we store these organs? Would each organ have to be developed for an individual human being and put back? And perhaps most puzzlingly, could your medicine be an environment? Could you patent an environment? You know, in every culture, shamans have been using environments as medicines.


Could we imagine that for our future? I've talked a lot about models. I began this talk with models. So let me end with some thoughts about model building. That's what we do as scientists. You know, when an architect builds a model, he or she is trying to show you a world in miniature. But when a scientist is building a model, he or she is trying to show you the world in metaphor. He or she is trying to create a new way of seeing. The former is a scale shift. The latter is a perceptual shift. Now, antibiotics created such a perceptual shift


in our way of thinking about medicine that it really colored, distorted, very successfully, the way we've thought about medicine for the last hundred years. But we need new models to think about medicine in the future. That's what's at stake. You know, there's a popular trope out there that the reason we haven't had the transformative impact on the treatment of illness is because we don't have powerful-enough drugs, and that's partly true. But perhaps the real reason is that we don't have powerful-enough ways of thinking about medicines.


It's certainly true that it would be lovely to have new medicines. But perhaps what's really at stake are three more intangible M's: mechanisms, models, metaphors. Thank you. (Applause) Chris Anderson: I really like this metaphor. How does it link in? There's a lot of talk in technologyland about the personalization of medicine, that we have all this data and that medical treatments of the future will be for you specifically, your genome, your current context. Does that apply to this model you've got here? Siddhartha Mukherjee: It's a very interesting question.


We've thought about personalization of medicine very much in terms of genomics. That's because the gene is such a dominant metaphor, again, to use that same word, in medicine today, that we think the genome will drive the personalization of medicine. But of course the genome is just the bottom of a long chain of being, as it were. That chain of being, really the first organized unit of that, is the cell. So, if we are really going to deliver in medicine in this way, we have to think of personalizing cellular therapies,


and then personalizing organ or organismal therapies, and ultimately personalizing immersion therapies for the environment. So I think at every stage, you know -- there's that metaphor, there's turtles all the way. Well, in this, there's personalization all the way. CA: So when you say medicine could be a cell and not a pill, you're talking about potentially your own cells. SM: Absolutely. CA: So converted to stem cells, perhaps tested against all kinds of drugs or something, and prepared. SM: And there's no perhaps. This is what we're doing.


Let's imagine for a few moments what our life would be like if we could access let's say 20% of our brain's capacity If you want to have something show up in your life The kind of person you would like to become manifest something new into your life something powerful, whatever it might be You obviously must first be able to imagine it Your imagination is able to do all that you ask in proportion to the degree of your attention So what kind of attention do you place on your desires? Einstein's most famous quote one of his most famous observations. He said imagination is more

YOU ARE THE CREATOR | Warning: This might shake up your belief system! Morgan Freeman and Wayne Dyer


important than knowledge Knowledge is limited Imagination encircles the world Logic will get you from A to B, but imagination will take you everywhere Make your future dream a present fact by assuming the feeling of the wish fulfilled That which you feel yourself to be you are and You are given that which you are. So assume the feeling that would be yours were you already in possession of your wish and your wish must be realized so live in the feeling of being the to be and that you shall be if this assumption about what you would like to become is


persisted in until it becomes your dominant feeling the attainment of your ideal is Absolutely inevitable You must first assume the feeling of a wish fulfilled in all aspects of your life Don't allow anybody elses opinions Don't allow what it says on the internet. Don't allow the research. Don't allow what anybody out there tells you is possible or not possible for you if you advance confidently in the direction of your own dreams and Endeavor to live the life, which you have imagined You will meet with a success unexpected


in common hours it will chase after you if you can place into your Imagination what it is that you would like to attract and begin to feel it Start retraining your subconscious mind and your subconscious mind it responds to what it is that you suggest to it The subconscious mind moves your life 96 to 97 percent of everything that you do is done as a result of your subconscious mind And when your subconscious mind gets programmed it goes ahead and Respond to whatever it is. Your conscious mind has placed into it


You are the creator this is the mystery This is the great secret known by the seers and prophets and mystics throughout the ages. This is the truth that you can never know intellectually Many of you as I have been as I am are where you are in your life Based upon what you believe and it's not just what you think you believe on the surface It's also your shadow beliefs that are holding you back from moving into the life that you believe You deserve What I know is if you're not looking at the shadows if you're not looking at what is


Subconsciously running through the tape in your mind telling yourself. You're not good enough. You're not worthy enough. You're not smart enough You're not enough which is a tape that's playing for a lot of people If you're not conscious of that then you end up acting out of that Belief system and not out of what you know to be the truest or want to be the choice for yourself You are where you are today in part because of what you've been saying about yourself Words are like seeds when you speak something out. You give life to what you're saying if you continue to say it


Eventually that can become a reality you Are planting seeds when you talk at some point you're going to eat that fruit. My challenge is make sure You're planning the right kind of seeds if you want apples You have to sow apple seeds if you want oranges You can't plant cactus seeds poison ivy seeds mushroom seeds You're going to reap fruit from the exact seeds that you've been sowing in other words You can't talk negative and expect to live a positive life You can't talk defeat and expect to have victory you can't talk lack


You're not enough can't afford it Never get ahead and expect to have abundance if you have a poor mouth. You're going to have a poor life And this is great when we're saying things like I'm blessed I'm strong I will accomplish my dreams I'm coming out of there That's not just being positive. You are prophesying victory Prophesy and success prophesy new levels and your life will move in the direction of your words But too many people go around prophesy on just the opposite. I never get any good brights. I'll never get back in shape


Business is slow. I'll probably get laid off Flu season is here. I always get it. They don't realize they are prophesy and defeat It's just like they're calling in bad breaks mediocrity lack You don't become what you want because so much of wanting is about Living in the space of what you don't have that's why Jim Carrey's story is so powerful Because he started to act as though he already had it. He would go up to Mulholland Drive He would drive away sing thinking. I already have those things I just haven't accessed them as yet


I believe Those things are going to come to me and I'm going to act like they are so I'm gonna move forward in my life in order to Draw that to myself in such a way that my actions are in alignment with what I say, I believe So if you start to think about that really why are you where you are in your life the choices that you have made? Have been because of what you believe to be true for yourself The time is now the time is now to express and for people to believe in themselves The time is now for it to be okay to be great


People in this world shun people for being great for being a bright color for standing out but the time is now to be okay to be the greatest you You can talk yourself out of your destiny Negative words can keep you from becoming who you were created to be don't fall into that trap Quit calling in defeat quit talking about how it's not going to happen You should wipe down your Write down what you want to see happen in life Any areas that you're struggling in where you need to improve write it down like it's already done and then every day


hi everyone, let's get started. Good afternoon and welcome to MIT 6.S191! TThis is really incredible to see the turnout this year. This is the fourth year now we're teaching this course and every single year it just seems to be getting bigger and bigger. 6.S191 is a one-week intensive boot camp on everything deep learning. In the past, at this point I usually try to give you a synopsis about the course and tell you all of the amazing things that you're going to be learning. You'll be gaining fundamentals into deep learning and

MIT 6.S191 (2020): Introduction to Deep Learning


learning some practical knowledge about how you can implement some of the algorithms of deep learning in your own research and on some cool lab related software projects. But this year I figured we could do something a little bit different and instead of me telling you how great this class is I figured we could invite someone else from outside the class to do that instead. So let's check this out first. Hi everybody and welcome MIT 6.S191 the official introductory course on deep learning to taught here at MIT. Deep


learning is revolutionising so many fields from robotics to medicine and everything in between. You'll the learn the fundamentals of this field and how you can build some of these incredible algorithms. In fact, this entire speech and video are not real and were created using deep learning and artificial intelligence. And in this class you'll learn how. It has been an honor to speak with you today and I hope you enjoy the course! Alright. so as you can tell deep learning is an incredibly powerful tool. This was


just an example of how we use deep learning to perform voice synthesis and actually emulate someone else's voice, in this case Barack Obama, and also using video dialogue replacement to actually create that video with the help of Canny AI. And of course you might as you're watching this video you might raise some ethical concerns which we're also very concerned about and we'll actually talk about some of those later on in the class as well. But let's start by taking a step back and actually introducing some of these terms that


we've been we've talked about so far now. Let's start with the word intelligence. I like to define intelligence as the ability to process information to inform future decisions. Now the field of artificial intelligence is simply the the field which focuses on building algorithms, in this case artificial algorithms that can do this as well: process information to inform future decisions. Now machine learning is just a subset of artificial intelligence specifically that focuses on actually teaching an algorithm how to do this without being explicitly programmed to


do the task at hand. Now deep learning is just a subset of machine learning which takes this idea even a step further and says how can we automatically extract the useful pieces of information needed to inform those future predictions or make a decision And that's what this class is all about teaching algorithms how to learn a task directly from raw data. We want to provide you with a solid foundation of how you can understand or how to understand these algorithms under the hood but also provide you with the


practical knowledge and practical skills to implement state-of-the-art deep learning algorithms in Tensorflow which is a very popular deep learning toolbox. Now we have an amazing set of lectures lined up for you this year including Today which will cover neural networks and deep sequential modeling. Tomorrow we'll talk about computer vision and also a little bit about generative modeling which is how we can generate new data and finally I will talk about deep reinforcement learning and touch on some of the limitations and new


frontiers of where this field might be going and how research might be heading in the next couple of years. We'll spend the final two days hearing about some of the guest lectures from top industry researchers on some really cool and exciting projects. Every year these happen to be really really exciting talks so we really encourage you to come especially for those talks. The class will conclude with some final project presentations which we'll talk about in a little a little bit and also some awards and a quick award ceremony to


celebrate all of your hard work. Also I should mention that after each day of lectures so after today we have two lectures and after each day of lectures we'll have a software lab which tries to focus and build upon all of the things that you've learned in that day so you'll get the foundation's during the lectures and you'll get the practical knowledge during the software lab so the two are kind of jointly coupled in that sense. For those of you taking this class for credit you have a couple different options to fulfill your credit


requirement first is a project proposal I'm sorry first yeah first you can propose a project in optionally groups of two three or four people and in these groups you'll work to develop a cool new deep learning idea and we realized that one week which is the span of this course is an extremely short amount of time to really not only think of an idea but move that idea past the planning stage and try to implement something so we're not going to be judging you on your results towards this idea but rather just the novelty of the idea


itself on Friday each of these three teams will give a three-minute presentation on that idea and the awards will be announced for the top winners judged by a panel of judges the second option in my opinion is a bit more boring but we like to give this option for people that don't like to give presentations so in this option if you don't want to work in a group or you don't want to give a presentation you can write a one-page paper review of the deep learning of a recent deepening of paper or any paper of your choice and this will be due on the last day of


class as well also I should mention that and for the project presentations we give out all of these cool prizes especially these three nvidia gpus which are really crucial for doing any sort of deep learning on your own so we definitely encourage everyone to enter this competition and have a chance to win these GPUs and these other cool prizes like Google home and SSD cards as well also for each of the labs the three labs will have corresponding prizes so it instructions to actually enter those respective competitions will be within


the labs themselves and you can enter to enter to win these different prices depending on the different lab please post a Piazza if you have questions check out the course website for slides today's slides are already up there is a bug in the website we fixed that now so today's slides are up now digital recordings of each of these lectures will be up a few days after each class this course has an incredible team of TAS that you can reach out to if you have any questions especially during the software labs they can help you answer


any questions that you might have and finally we really want to give a huge thank to all of our sponsors who without their help and support this class would have not been possible ok so now with all of that administrative stuff out of the way let's start with the the fun stuff that we're all here for let's start actually by asking ourselves a question why do we care about deep learning well why do you all care about deep learning and all of you came to this classroom today and why specifically do care about deep learning


now well to answer that question we actually have to go back and understand traditional machine learning at its core first now traditional machine learning algorithms typically try to define as set of rules or features in the data and these are usually hand engineered and because their hand engineered they often tend to be brittle in practice so let's take a concrete example if you want to perform facial detection how might you go about doing that well first you might say to classify a face the first thing I'm gonna do is I'm gonna try and


classify or recognize if I see a mouth in the image the eyes ears and nose if I see all of those things then maybe I can say that there's a face in that image but then the question is okay but how do I recognize each of those sub things like how do I recognize an eye how do I recognize a mouth and then you have to decompose that into okay to recognize a mouth I maybe have to recognize these pairs of lines oriented lines in a certain direction certain orientation and then it keeps getting more complicated and each of these steps you


kind of have to define a set of features that you're looking for in the image now the key idea of deep learning is that you will need to learn these features just from raw data so what you're going to do is you're going to just take a bunch of images of faces and then the deep learning algorithm is going to develop some hierarchical representation of first detecting lines and edges in the image using these lines and edges to detect corners and eyes and mid-level features like eyes noses mouths ears then composing these together to detect


higher-level features like maybe jaw lines side of the face etc which then can be used to detect the final face structure and actually the fundamental building blocks of deep learning have existed for decades and they're under underlying algorithms for training these models have also existed for many years so why are we studying this now well for one data has become much more pervasive we're living in a the age of big data and these these algorithms are hungry for a huge amounts of data to succeed secondly these algorithms are massively


parallel izybelle which means that they can benefit tremendously from modern GPU architectures and hardware acceleration that simply did not exist when these algorithms were developed and finally due to open-source tool boxes like tensor flow which are which you'll get experience with in this class building and deploying these models has become extremely streamlined so much so that we can condense all this material down into one week so let's start with the fundamental building block of a neural network which is a single neuron


or what's also called a perceptron the idea of a perceptron or a single neuron is very basic and I'll try and keep it as simple as possible and then we'll try and work our way up from there let's start by talking about the forward propagation of information through a neuron we define a set of inputs to that neuron as x1 through XM and each of these inputs have a corresponding weight w1 through WN now what we can do is with each of these inputs and each of these ways we can multiply them correspondingly together and take a sum of all of them then we take this single


number that's summation and we pass it through what's called a nonlinear activation function and that produces our final output Y now this is actually not entirely correct we also have what's called a bias term in this neuron which you can see here in green so the bias term the purpose of the bias term is really to allow you to shift your activation function to the left and to the right regardless of your inputs right so you can notice that the bias term doesn't is not affected by the X's it's just a bias associate to that input


now on the right side you can see this diagram illustrated mathematically as a single equation and we can actually rewrite this as a linear using linear algebra in terms of vectors and dot products so instead of having a summation over all of the X's I'm going to collapse my X into a vector capital X which is now just a list or a vector of numbers a vector of inputs I should say and you also have a vector of weights capital W to compute the output of a single perceptron all you have to do is take the dot product of X and W which


represents that element wise multiplication and summation and then apply that non-linearity which here is denoted as G so now you might be wondering what is this nonlinear activation function I've mentioned it a couple times but I haven't really told you precisely what it is now one common example of this activation function is what's called a sigmoid function and you can see an example of a sigmoid function here on the bottom right one thing to note is that this function takes any real number as input on the x-axis and it transforms that real number into a scalar output


between 0 & 1 it's a bounded output between 0 & 1 so one very common use case of the sigmoid function is to when you're dealing with probabilities because probabilities have to also be bounded between 0 & 1 so sigmoids are really useful when you want to output a single number and represent that number as a probability distribution in fact there are many common types of nonlinear activation functions not just the sigmoid but many others that you can use in neural networks and here are some common ones and throughout this presentation you'll find these tensorflow icons like you can


see on the bottom right or sorry all across the bottom here and these are just to illustrate how one could use each of these topics in a practical setting you'll see these kind of scattered in throughout the slides no need to really take furious notes at these codeblocks like I said all of the slides are published online so especially during your labs if you want to refer back to any of the slides you can you can always do that from the online lecture notes now why do we care about activation functions the point of


an activation function is to introduce nonlinearities into the data and this is actually really important in real life because in real life almost all of our data is nonlinear and here's a concrete example if I told you to separate the green points from the red points using a linear function could you do that I don't think so right so you'd get something like this oh you could do it you wouldn't do very good job at it and no matter how deep or how large your network is if you're using a linear activation function you're just


composing lines on top of lines and you're going to get another line right so this is the best you'll be able to do with the linear activation function on the other hand nonlinearities allow you to approximate arbitrarily complex functions by kind of introducing these nonlinearities into your decision boundary and this is what makes neural networks extremely powerful let's understand this with a simple example and let's go back to this picture that we had before imagine I give you a train network with weights W on the top right so W here is 3 and minus 2 and the


network only has 2 inputs x1 and x2 if we want to get the output it's simply the same story as we had before we multiply our inputs by those weights we take the sum and pass it through a non-linearity but let's take a look at what's inside of that non-linearity before we apply it so we get is when we take this dot product of x1 times 3 X 2 times minus 2 we mul - 1 that's simply a 2d line so we can plot that if we set that equal to 0 for example that's a 2d line and it looks like this so on the x axis is X 1 on the y axis is X 2 and


we're setting that we're just illustrating when this line equals 0 so anywhere on this line is where X 1 and X 2 correspond to a value of 0 now if I feed in a new input either a test example a training example or whatever and that input is with this coordinates it's has these coordinates minus 1 and 2 so it has the value of x1 of minus 1 value of x2 of 2 I can see visually where this lies with respect to that line and in fact this this idea can be generalized a little bit more if we compute that line we get minus 6 right


so inside that before we apply the non-linearity we get minus 6 when we apply a sigmoid non-linearity because sigmoid collapses everything between 0 and 1 anything greater than 0 is going to be above 0.5 anything below zero is going to be less than 0.5 so in is because minus 6 is less than zero we're going to have a very low output this point Oh 200 to we can actually generalize this idea for the entire feature space let's call it for any point on this plot I can tell you if it lies on the left side of the line that means that before we apply the non-linearity the Z or the state of that


neuron will be negative less than zero after applying that non-linearity the sigmoid will give it a probability of less than 0.5 and on the right side if it falls on the right side of the line it's the opposite story if it falls right on the line it means that Z equals zero exactly and the probability equals 0.5 now actually before I move on this is a great example of actually visualizing and understanding what's going on inside of a neural network the reason why it's hard to do this with deep neural networks is because you


usually don't have only two inputs and usually don't have only two weights as well so as you scale up your problem this is a simple two dimensional problem but as you scale up the size of your network you could be dealing with hundreds or thousands or millions of parameters and million dimensional spaces and then visualizing these type of plots becomes extremely difficult and it's not practical and pause in practice so this is one of the challenges that we face when we're training with neural networks and really understanding their


internals but we'll talk about how we can actually tackle some of those challenges in later lectures as well okay so now that we have that idea of a perceptron a single neuron let's start by building up neural networks now how we can use that perceptron to create full neural networks and seeing how all of this story comes together let's revisit this previous diagram of the perceptron if there are only a few things you remember from this class try to take away this so how a perceptron works just keep remembering this I'm


going to keep drilling it in you take your inputs you apply a dot product with your weights and you apply a non-linearity it's that simple oh sorry I missed the step you have dot product with your weights add a bias and apply your non-linearity so three steps now let's simplify this type of diagram a little bit I'm gonna remove the bias just for simplicity I'm gonna remove all of the weight labels so now you can assume that every line the weight associated to it and let's say so I'm going to note Z that Z is the


output of that dot product so that's the element wise multiplication of our inputs with our weights and that's what gets fed into our activation function so our final output Y is just there our activation function applied on Z if we want to define a multi output neural network we simply can just add another one of these perceptrons to this picture now we have two outputs one is a normal perceptron which is y1 and y2 is just another normal perceptron the same ideas before they all connect to the previous layer with a different set of weights


and because all inputs are densely connected to all of the outputs these type of layers are often called dense layers and let's take an example of how one might actually go from this nice illustration which is very conceptual and and nice and simple to how you could actually implement one of these dense layers from scratch by yourselves using tensor flow so what we can do is start off by first defining our two weights so we have our actual weight vector which is W and we also have our bias vector right both of both of these parameters


are governed by the output space so depending on how many neurons you have in that output layer that will govern the size of each of those weight and bias vectors what we can do then is simply define that forward propagation of information so here I'm showing you this to the call function in tensor flow don't get too caught up on the details of the code again you'll get really a walk through of this code inside of the labs today but I want to just show you some some high level understanding of how you could actually take what you're


learning and apply the tensor flow implementations to it inside the call function it's the same idea again you can compute Z which is the state it's that multiplication of your inputs with the weights you add the bias right so that's right there and once you have Z you just pass it through your sigmoid and that's your output for that now tension flow is great because it's already implemented a lot of these layers for us so we don't have to do what I just showed you from scratch in fact to implement a layer like this with two two outputs or a percept a multi


layer a multi output perceptron layer with two outputs we can simply call this TF Harris layers dense with units equal to two to indicate that we have two outputs on this layer and there is a whole bunch of other parameters that you could input here such as the activation function as well as many other things to customize how this layer behaves in practice so now let's take a look at a single layered neural network so this is taking it one step beyond what we've just seen this is where we have now a single hidden layer that feeds into a


single output layer and I'm calling this a hidden layer because unlike our inputs and our outputs these states of the hidden layer are not directly enforced or they're not directly observable we can probe inside the network and see them but we don't actually enforce what they are these are learned as opposed to the inputs which are provided by us now since we have a transformation between the inputs and the hidden layer and the hidden layer and the output layer each of those two transformations will have their own weight matrices which here I


call W 1 and W 2 so its corresponds to the first layer and the second layer if we look at a single unit inside of that hidden layer take for example Z 2 I'm showing here that's just a single perceptron like we talked about before it's taking a weighted sum of all of those inputs that feed into it and it applies the non-linearity and feeds it on to the next layer same story as before this picture actually looks a little bit messy so what I want to do is actually clean things up a little bit for you and I'm gonna replace all of those lines with just this symbolic representation


and we'll just use this from now on in the future to denote dense layers or fully connected layers between two between an input and an output or between an input and hidden layer and again if we wanted to implement this intensive flow the idea is pretty simple we can just define two of these dense layers the first one our hidden layer with n outputs and the second one our output layer with two outputs we can cut week and like join them together aggregate them together into this wrapper which is called a TF sequential model and sequential models are just


this idea of composing neural networks using a sequence of layers so whenever you have a sequential message passing system or sequentially processing information throughout the network you can use sequential models and just define your layers as a sequence and it's very nice to allow information to propagate through that model now if we want to create a deep neural network the idea is basically the same thing except you just keep stacking on more of these layers and to create more of an more of a hierarchical model ones where the


final output is computed by going deeper and deeper into this representation and the code looks pretty similar again so again we have this TF sequential model and inside that model we just have a list of all of the layers that we want to use and they're just stacked on top of each other okay so this is awesome so hopefully now you have an understanding of not only what a single neuron is but how you can compose neurons together and actually build complex hierarchical models with deep with neural networks now let's take a look at how you can


apply these neural networks into a very real and applied setting to solve some problem and actually train them to accomplish some task here's a problem that I believe any AI system should be able to solve for all of you and probably one that you care a lot about will I pass this class to do this let's start with a very simple two input model one feature or one input we're gonna define is how many let's see how many lectures you attend during this class and the second one is the number of hours that you spend on your final


projects I should say that the minimum number of hours you can spend your final project is 50 hours now I'm just joking okay so let's take all of the data from previous years and plot it on this feature space like we looked at before green points are students that have passed the class in the past and red points are people that have failed we can plot all of this data onto this two-dimensional grid like this and we can also plot you so here you are you have attended four lectures and you've only spent five hours on your final exam


you're on you're on your final project and the question is are you going to pass the class given everyone around you and how they've done in the past how are you going to do so let's do it we have two inputs we have a single layered set single hidden layer neural network we have three hidden units in that hidden layer and we'll see that the final output probability when we feed in those two inputs of four and five is predicted to be 0.1 or 10% the probability of you passing this class is 10% that's not great news the actual prediction was one


so you did pass the class now does anyone have an idea of why the network was so wrong in this case exactly so we never told this network anything the weights are wrong we've just initialized the weights in fact it has no idea what it means to pass a class it has no idea of what each of these inputs mean how many lectures you've attended and the hours you've spent on your final project it's just seeing some random numbers it has no concept of how other people in the class have done so far so what we have to do to this network first is


train it and we have to teach it how to perform this task until we teach it it's just like a baby that doesn't know anything so it just entered the world it has no concepts or no idea of how to solve this task and we have to teach at that now how do we do that the idea here is that first we have to tell the network when it's wrong so we have to quantify what's called its loss or its error and to do that we actually just take our prediction or what the network predicts and we compare it to what the true answer was


if there's a big discrepancy between the prediction and the true answer we can tell the network hey you made a big mistake right so this is a big error it's a big loss and you should try and fix your answer to move closer towards the true answer which it should be okay now you can imagine if you don't have just one student but now you have many students the total loss let's call it here the empirical risk or the objective function it has many different names it's just the the average of all of those individual losses so the


individual loss is a loss that takes as input your prediction and your actual that's telling you how wrong that single example is and then the final the total loss is just the average of all of those individual student losses so if we look at the problem of binary classification which is the case that we're actually caring about in this example so we're asking a question will I pass the class yes or no binary classification we can use what is called as the softmax cross-entropy loss and for those of you who aren't familiar with cross-entropy


this was actually a a formulation introduced by Claude Shannon here at MIT during his master's thesis as well and this was about 50 years ago it's still being used very prevalently today and the idea is it just again compares how different these two distributions are so you have a distribution of how how likely you think the student is going to pass and you have the true distribution of if the student passed or not you can compare the difference between those two distributions and that tells you the loss that the network incurs on that


example now let's assume that instead of a classification problem we have a regression problem where instead of predicting if you're going to pass or fail to class you want to predict the final grade that you're going to get so now it's not a yes/no answer problem anymore but instead it's a what's the grade I'm going to get what's the number what so it's it's a full range of numbers that are possible now and now we might want to use a different type of loss for this different type of problem and in this case we can do what's called a mean squared error loss


so we take the actual prediction we take the the sorry excuse me we take the prediction of the network we take the actual true final grade that the student got we subtract them we take their squared error and we say that that's the mean squared error that's the loss that the network should should try to optimize and try to minimize so ok so now that we have all this information with the loss function and how to actually quantify the error of the neural network let's take this and understand how to train train our model


to actually find those weights that it needs to to use for its prediction so W is what we want to find out W is the set of weights and we want to find the optimal set of weights that tries to minimize this total loss over our entire test set so our test set is this example data set that we want to evaluate our model on so in the class example the test set is you so you want to understand how likely you are to pass this class you're the test set now what this means is that we want to find the W's that minimize that total loss


function which we call as the objective function J of W now remember that W is just a aggregation or a collection of all of the individual w's from all of your weights so here this is just a way for me to express this in a clean notation but W is a whole set of numbers it's not just a single number and you want to find this all of the W's you want to find the value of each of those weights such that you can minimize this entire loss function it's a very complicated problem and remember that our loss function is just a simple


function in terms of those weights so if we plot in the case again of a two-dimensional weight problem so one of the weights is on the x-axis one of the weights is on this axis and on the z axis we have the loss so for any value of w we can see what the loss would be at that point now what do we want to do we want to find the place on this landscape what are the values of W that we get the minimum loss okay so what we can do is we can just pick a random W pick a random place on this this landscape to start with and from


this random place let's try to understand how the landscape is changing what's the slope of the landscape we can take the gradient of the loss with respect to each of these weights to understand the direction of maximum ascent okay that's what the gradient tells us now that we know which way is up we can take a step in the direction that's down so we know which way is up we reverse the sign so now we start heading downhill and we can move towards that lowest point now we just keep repeating this process over and over