so it's exactly you know very very much the same idea I showed before except now the the layers of the kind of feed-forward part of the network I kind of drawn on this side and then the the sort of resolution increasing half is drawn on that side with skipping connections going directly so it looks like a u and these sort of variations of this so this is work from my colleagues that at NYU who are working on with medical image analysis these are 3d MRI scans and so the cognate here is three-dimensional the completions take

Self-Supervised Learning


place in three dimensions over the three spatial dimensions and and every voxel now is labeled as you know one of a number of categories so you can do things like segment hip bones and things like this for you know preparing for hip replacement surgery and stuff like that and and it works really it works much better if you use 3d rather than 2d because you get the consistency of all the slices so as you see at the top here is there are artifacts of the recognition if you use kind of 2d segmentation perhaps with a little bit

Self-Supervised Learning


of cleanup and if you really you you sort of get with your of those artifacts the same team with a different subset of people has applied this to things like mammograms so this is 2d data but you have multiple images from sort of multiple with use angles of view and here's a surprising thing so this is the kind of application that you some of you may not have heard of which is the application of comp nets in physics this is an example in astrophysics so this is a paper from I think was in PNAS published in PNAS a

Self-Supervised Learning


few months ago from the Flatiron Institute which is a private research institute in in New York and what they did here was use a accomplished on a to accelerate the solution of partial differential equations solvers so what they were interested in these are cosmologists and they're interested in you know what are the initial conditions of you know baby universe that will cause the kind of universe verbs or in today what you have to do for that is basically simulate the entire universe at its birth you know the expansion the

Self-Supervised Learning


first expansion phase of the universe you can do this because you could do this in principle because you know you have the density of matter and dark matter ordinary matter dark matter photons whatever at every location and you can solve a partial differential equation which basically is just you know physics at every location and I'm gonna compute the evolution of the universe this way the problem of with this is that she only do this at the scale of the universe given the size of the the grid the grid that you have to

Self-Supervised Learning


use to solve this equation it will take too long and so what they did here was they used one of those solvers known PDE solvers to kind of solve those equations on kind of small domains small four-dimensional domains right because it's three dimensions of space and one dimensional time and they train a convolutional net to produce the same result but that convolutional net has a bigger grid right so a PDE solver basically takes one value when vauxhall okay four dimensional voxel if you want or 3-dimensional Vauxhall and then looks

Self-Supervised Learning


at the neighbors and then passes it to some function that computes the new value for the next time step let's say of the the center or the central grid cell so it's a completion like operation except that you know it's maybe nonlinear so so what they did was train accomplish on that with a few layers and they it you use this it uses this unit architecture so we can take a fairly large context into account not just the neighboring cells but sort of a bigger neighborhood rather big grid cells and it's trained to produce the result that

Self-Supervised Learning


the PD solver would would produce and they can easily generate data by running the PD solver but they run it on kind of small 3d domains and then once they have this commercial net they can run it on the the big scale of you know universai universe size scale if you want and and what they get is those kind of match here which are the displacement maps of densities and there's also different methods and the colors indicate errors blue is low error and you know red is high error and those are also various ways of doing this and this is kind of

Self-Supervised Learning


their proposed method and compared with what the PDE solver we do for a relatively small domain so that's kind of an interesting thing which is to use neural nets or D planning in general as a phenomenological model of something that we might possibly know the underlying physics but it's computationally too expensive people are doing this also for predicting the properties of materials for solving problems in molecular dynamics so for example confirmation of protein where the two proteins are going to stick to

Self-Supervised Learning


each other you know things like that which is of course super important for ticks like drug design I was at Harvard a couple weeks ago and I talked to people who are trying to use neural nets to predict the property of certain solids so if you take graphene which is a two-dimensional mesh of carbon atoms and you take two and it's cool you know just a single atom thick layer you take two layers of graphene and the one on top you twist it just a little bit compared to the the one at the bottom there's a particular

Self-Supervised Learning


angle at which this material becomes superconductor and nobody has any idea why and so there's some idea of using neural Nets to kind of build phenomenological models of all those properties so that perhaps we could predict other properties there's interesting work along those lines also by Pascale flora who is actually originally a vision guy at a pea ferry and what he's doing what he's been doing was to predict the aerodynamic or hydrodynamic properties of a solid using a by training accomplish on that feel

Self-Supervised Learning


accomplished on that basically so you feed the shape of the the solid to the to the system and again using fluid dynamics computation to generate data you train it to produce the properties of that shape for example its drag or lift if you are interested in designing air foils for for you know blades of propellers or airplanes or hydrofoils or whatever and then what you have now is on your net that predicts those properties and so because it's on your own that is differentiable so now you can optimize the shape by doing gradient

Self-Supervised Learning


descent in input space you can optimize the shape so as to get the properties you want on the output which you you know you can't really do it with a regular computational fluid dynamics piece of code so it's really interesting he actually has a startup that works on this yes huh well I guess you have to really know the underlying physics to be able to make that generalization so I mean you can test on relatively small spatial domain because you can run the PD solver so you know how accurate your new ComNet is the

Self-Supervised Learning


question is when you extend the size is it still accurate okay and there is a leap of faith there's no question now to your comment that this has nothing to do with Commerce on that no it does it has very very much to do with Congress on it because all of those piggies are local operations that basically look like convolutions that are essentially the same you know press some nonlinear thing because you know if you do navier-stokes equation for free dynamics you have to do some projection afterwards that's nonlinear but you know but it's a local

Self-Supervised Learning


operation and it's the same operation you do everywhere in the image so or if we're in the in the volume so it is accomplish on that it's it's directly it's probably one of the most appropriate use of kinetically imagine so of course it's been quite a bit of progress in in things like you know start driving cars it's working progress these are actually videos that are quite a few years old I think about five years five years old from this one is from mobile I which is now Intel and NVIDIA and there's a huge amount of work on

Self-Supervised Learning


surviving cars as you know a lot of engineering goes into this but all the perception system use you know some sort of cognitive process either images from cameras or from various other types of sensors like lidar and other things like this okay so all this is great it's all supervised and reinforcement and one big question that we can ask ourselves is is this going to take us to the possibility of building you know truly intelligent machines machines were that you know I'm not talking about human level intelligence but maybe intelligence of

Self-Supervised Learning


the house cat or something like that so a house cat has more common sense than any AI systems that we build that we can build today and the answer is no we need we need you know significant can a conceptual progress if you really if you really want to make machines that are more intelligent that we have today so we can do all the stuff we have on the on the left assuming that we put enough efforts in them engineering efforts like you know separate cars you know semi autonomous cars better medical image analysis systems you know

Self-Supervised Learning


all kinds of stuff stupid chat BOTS you know that are entertaining but we can't have things that you know we the technology we have is not enough to get machines that have common sense to get to build things like intelligent personal assistants that really help us it can help us in the daily lives answer any question we have and you know kind of be a bit more like like human assistance we can have really smart chat BOTS we can have household robots that you know take care of all the chores in their house we don't really have a jar

Self-Supervised Learning


in dexterous robots they are a giant dexterous and very kind of specific situations but it's sort of very brittle and we can't have artificial intelligence so in general we can't have artificial general intelligence because that concept does not exist there is no such thing as general intelligence and I hate this term AVI there's a lot of people who claim that you know they are going to get to AGI by scaling up reinforcements running just having more computation this is completely false okay those people are after investment so

Self-Supervised Learning


they're ready to either you know self be so deluded or or kind of stretch the truth a little bit but in my opinion we're not gonna get there with the current type of running that we are that we're using and so why is there no such thing as artificial general intelligence and that's because there is no general intelligence human intelligence is incredibly specialized I'm sorry to say that okay that applies to everyone in this room but our intelligence is super specialized you know we're built by hit by evolution

Self-Supervised Learning


to kind of survive in our environment and we have this sort of impression that our intelligence in general but we just suck at a lot of tasks okay and in fact a lot of the tasks that computers can do quite well we totally suck at it so there was this idea that you know before alphago if I go 0 etc that humans were the best go player in the world were very very close to the ideal player okay good god alright that you could you could get just a few stones of handicap with idea player and basically beat the idea player two or three stones and you

Self-Supervised Learning


care for something like this turns that no turns out the best human players are horrible you know current machines are much much better than they are like by a huge margin so we just suck at it we're really bad which means you know that's not part of the stuff that evolution kind of built into our our brain to be able to do well now the thing is the reason why people were thinking that you know they were very close to the ideal player was because they could not imagine you know smarter considerably smaller entities and so we cannot

Self-Supervised Learning


imagine all the stuff that we're not able to do and therefore we think of ourselves as having general intelligence it's just that or imagination for what you know what functions we need to be able to do is very limited let me give you another more specific example anymore I wouldn't say mathematical but it's you more quantitative your your optical nerve has 1 million fibers so imagine we just take the 1 million fibers coming out of your optical nerve that goes to your brain and imagine that they're just binary so what you see is just a binary

Self-Supervised Learning


image ok 1 million bits how many so a particular recognition function if you want ok recognizing your grandmother or whatever is a boolean function 1 million bits in the input and 1 bit in the output and the question is how many search functions are there anyone has any idea how many search function how many boolean functions of 1 million bits are there any suggestion any idea to to the 1 million yet you are off by a huge factor but it's a good start 25 yes 2 to the 2 to the 1 million that's a correct answer ok so you have 2

Self-Supervised Learning


to the 1 million input configuration of 1 million bits right and for each of those 2 to the 1 million configuration you have one output bit that's the truth table of a particular boolean function ok so the number of configurations of 2 to the 1 million bits is 2 to the 2 to the 1 million it's an adorably large number I mean it's just a ridiculously large number now among all of those functions how many do you think what proportion do you think your brain can actually compute your visual cortex visual cortex has you know order between

Self-Supervised Learning


10 and 100 billion neurons order 10 to the 14 synapses okay so it has 10 to the 14 synapses let's say to be generous each synapse scanning can store 10 bits ok so there's 10 to the 15 bits in your entire visual cortex that's your that's what determines the function of you know of your your visual cortex that means the number of functions your visual cortex can possibly implement is 2 to the 10 to the 15 that's a lot less than 2 to the 2 to the 1 million not a lot less it's just you know this is like no comparison right one so the number of

Self-Supervised Learning


functions that your visual cortex can implement compared to all possible functions it's just this tiny tiny tiny sliver we're super specialized in particular if I play a trick on you I cut your article nerve I'm gonna do it ok and I put a device between your your retina and your brain that permutes all the pixels in your head in your optical nerve with a random permutation but a fixed one ok so now there is no spatial consistency in the signal you get to your visual cortex I don't think you can envision because your cortex has local

Self-Supervised Learning


connections and those local connections are there to exploit local correlation and now you break the circle correlation but doing this permutation you can you can see anyone you might see at very low resolution because the higher layers have big context but what is it true what is true yes okay so it is retin-a topic so the connection between the the optical nerve and the visual cortex is within a topic which means the topology is preserved the connections are largely local there are long range collections but this is only

Self-Supervised Learning


a small number of them and so you don't you don't have a huge amount of you know communication bandwidth for the long range you you have you know big bundles of connections from sort of low layers to high riders if you want from V 1 to V 2 and V 2 to V 4 and once you get to the higher layers the the spatial distribution is not represented anymore it's like a cornet where you are pulling and so in the high high layers you don't need that organization but by the time you get there the spatial resolution is lost so we can do the experiment

Self-Supervised Learning


actually it would be fun ok so next question is how do humans and animals learn so I don't know how many of you were were here yesterday at the inauguration probably not many but there is this idea that humans learn in a very different way from either reinforcement or supervised running and I'll call this later so supervised running but this is just a hypothesis but if you but you know babies learn concepts they learn sort of basic facts about basic knowledge about the world basically just by observation in the first few weeks

Self-Supervised Learning


and months of life and Emanuel depuis who is a colleague of Jean and me Vaness and on Facebook put together this chart that shows at what age babies learn different concepts so things like being able to make the difference between animate and inanimate object that pops up around three months and the notion of object permanence the fact that an object that is hidden behind another one is still there still exists the notions of solidity rigidity stability and then intuitive physics like gravity inertia are things like

Self-Supervised Learning


this pop up around eight months so if you show a six-month-old baby the scenario on the top left where you put a little car on a on a platform and you push up you push the car off the platform and it doesn't fall it's hidden in the back put the baby you can see that it's a trick six months they they're not surprised to just you know that's how the world works it was one more thing I need to learn after nine months they've learned that objects are not supposed to float in the air that you're supposed to fall and they go like

Self-Supervised Learning


this okay and you can measure how long they stare at it and with how much attention and so that's how you know that a concept has been has been known on that we know if a concept is is violated by a particular scene that you show the baby the baby is going to be really surprised and you can measure the degree of surprise if you want so how is it that babies learned is just basically by observation you know young babies you know before a few months old or completely helpless I just observed and don't really have any way of affecting

Self-Supervised Learning


the physical world around them so how does that happen so it's a different type of learning that either refers mental supervised ready and it's not just babies it's you know almost animals learn learn this kind of stuff this is a baby or wrong.we Tong is being shown a magic trick whether it's a an object in the cup and the object is removed but he doesn't see that and now the cup is empty and he's running on the floor laughing so obviously you know his model of the world includes object permanence and objects are not supposed to

Self-Supervised Learning


disappear like this and you know and when we see something that surprises us we are you know we laugh or we get scared because here is something we didn't predict and it can kill us so it's all kinds of concepts like this the reason for this animation here at the top is is that these very basic concepts like the fact that the world is filled emotional that perhaps we can learn but there's training of cells to predict very simple things so if I if I train myself at train my brain or if I train a running machine to predict what the

Self-Supervised Learning


world is going to look like when I'm when I move my my head or the camera a few centimeters to the left the view of the world changes depending on you know objects move with parallax depending on the depth the distance to my eyes and so if I train myself to predict what the world's going to look like when I move the camera perhaps I can automatically infer that every object in the world has a depth because that's the best explanation for the simplest explanation for how things change okay so the notion of depth the

Self-Supervised Learning


fact that the world is feel emotional might just simply emerge from training ourselves to predict what the world looks like when we move our head once you have that you have occlusion edges you know objects that are nearby don't move the same way that objects that are far away and so you see them as as objects okay there's a bunch of you know so weekly supervised vision systems that exploit this this kind of this kind of property once you have objects you know you have objects they can they can move independently of others your background

Self-Supervised Learning


objects you have you know the notion of obstacle you know things like that localization so you could think of you know concepts like this being sort of built hierarchically by just training yourself to predict and then representing you know coming up with good representations of that allow you to do a good job at predicting okay so this is not supervised it would be an unsupervised way form of running and that led some of us to so this is the joke from earlier shell force it's a play on some poem from the 1960s or 70s

Self-Supervised Learning


the variation will not be supervised so the future is in a new form of learning that will allow a machine to accumulate all those background knowledge about how the world works mostly by observation a little bit by interaction but mostly without supervision mostly without reinforcement so maybe that's the salvation so supervised learning what is it the the basic concepts the basic concept is I'll give you a piece of data let's say a piece of video for the sake of being concrete and I'm going to mask a piece of that video

Self-Supervised Learning


perhaps the second half of the video which is in blue at the top I'm going to train a machine to predict the future of the video from the past and the present okay but the general concepts of supervised learning is you have a piece of data you mask a piece of it and you ask the machine to predict the piece that is masked from the piece that is not mask if the piece that is masked is always the same so is the future for example you know you can use a some sort of prediction architecture for that but more often than not you don't actually

Self-Supervised Learning


know which piece is gonna be masked for example in this scene here right now you don't see my back but you can have you might have some good idea that you know what it looks like and you know maybe your brain sort of unconsciously tries you know it's predicting what I look like from the back and once I turn around you know your belief about this is updated it you can you can train yourself you can train your model you know same for all kinds of parts of the scene here which are currently occluded from your from your

Self-Supervised Learning


view so this principle of sort of learning to predict things that you will eventually see I think is a good one now this is so again you could train yourself to predict the past from the present to predict the top of the image from the bottom you know whatever doesn't doesn't matter exactly what it is the advantage of this is that this is something that you know Jeff Fenton has claimed for a long time is that the amount of information you're giving to the machine at every time step but every trial at every sample is enormous you're

Self-Supervised Learning


asking to predict every pixel you know in a bunch of frames in a video which is a lot of information much more than the label of a of an image for example which means you're putting a lot more constraints on the parameters of the machine which means you can train the machine to learn a lot of knowledge with a relatively small number of samples and furthermore those samples are free because you know we have more video data than then we can deal with so essentially if you think about sort of a hierarchy of the type of loading paradigms that we've

Self-Supervised Learning


been talking about here in South supervisors running there's a huge amount of feedback you're giving to the machine you know you're giving it a piece of video and then you were asking it you're telling it you know predict all those pixels it's an enormous amount of information this is a technical issue with it which I'll come back I'll come to in a minute supervise running you give a relatively small amount of feedback you tell the Machine this is class number three out of a thousand it's not a huge amount of information

Self-Supervised Learning


and I should say right now the the reason why neural nets work so well on image net is not just that it has 1 million training samples it's that it has 1,000 categories having a problem with lots of categories helps a lot to kind of construct good representations and then reinforcement learning is a very very weak feedback you're only telling the Machine once in a while you got it right or you get it wrong you're giving just a scalar value it's absolutely no way that a machine can learn anything complex without lots

Self-Supervised Learning


and lots of interactions using basic reinforcement running it's just no way you're just not giving it a lot of information yeah you know that's what you know a learning theory the only theory is called this sample complexity and it's just completely obvious that there is no way you can run complex stuff without tons and tons of interactions when you're giving just one scalar value once in a while as a feedback so the path to human intelligence you know my go through reinforcement running but it's not gonna be necessary

Self-Supervised Learning


it's not gonna be sufficient that's for sure that led me to this obnoxious analogy of intelligence as a cake and you know if so supervised running is the bulk of the cake machine learning is in the same embarrassing situation as physics in the sense that physicists have no idea what 95% of the mass in the universe is you know it's dark matter and dark energy they have no idea what it is we only know the 5% that is actually real matter and but the rest we don't know what it is so here's the same thing we can do we can

Self-Supervised Learning


make the chair you can make the icing on the cake but we can't actually bake the cake okay so I give you a preview earlier of what's missing and Wes missing is the ability to learn models predictive models of the world you know in the example of the car you if you want to train your system to drive a car it has to have some sort of politi model of what's gonna happen so as not to try stupid things like running into a tree or off a cliff so sorry years ago a few of my colleagues at Facebook ran this experiment which they you know they did

Self-Supervised Learning


so cooked up a bunch of simple physical situations where you stack a bunch of cubes and then you either run a game engine to can assimilate the cubes following or not following or you actually have real data where you take videos of stack of cubes and you train a video version of cornet basically to predict what's going to happen to the cubes so what you see here is what actually happens those are so the segmentation maps of the various cubes and it says what the continent is producing and the predictions are kind

Self-Supervised Learning


of blurry the reason why they vary is because there is no there's no way of exactly predicting what's going to happen there is a little bit of uncertainty about where other cubes actually are and everything and so what the system produces is a prediction which is sort of an average of multiple futures that can happen and that's a blurry prediction so how to deal with uncertainty is going to become the the main the main problem here right we're gonna make we're going to predict what's going to happen in the world by doing video prediction or we're

Self-Supervised Learning


going to take an image or a piece of text mask some parts of it and then ask a system to reconstruct and there is no way to make those predictions exactly and so what we had to have our systems that can deal with the uncertainty in the prediction they can represent the uncertainty of the prediction and it's for that reason to introduce the notion of energy based learning so you can think of it as kind of a weaker form of learning that people are used to which is sort of learning probabilistic models or learning the cities running you know

Self-Supervised Learning


distributions and the reason that we have to weaken it is because in high dimensional continuous spaces like images we don't have good ways of representing distributions that mean anything useful so let's say our entire world consists of two scalar variables y1 and y2 we're not going to do prediction here I mean we could consider that one variables observe the other one is not but we don't know in advance which one so let's say you observe y2 okay and this is this is our training set each each each point here is a

Self-Supervised Learning


training sample so this is obviously some structure in our world here you know all the values of y1 y2 seem to rely on some sort of surface line I mean your curve and if I give you a value of y2 you can predict that the value of y1 will be sort of around here or around there this is this is 30 okay I couldn't tell it was like a comma or not ok 30 minute that's perfect so right so there are multiple possible predictions so if you train a neural net or whatever primary trace function to make one prediction of y2 of y1 as a

Self-Supervised Learning


function of Y - it's not gonna work because you can only predict one output if you train the system willie square when it sees half of the sample on this side the other half on that side what is going to produce is the mean of the two okay that's the best way to minimize the square error that's not a good prediction for y1 for this value of y2 right that's those blurry predictions I was showing you about that's the brewery prediction here right in the middle so how do we kind of turn a prediction with sort of multiple possible outputs into

Self-Supervised Learning


kind of an architecture if you want and the proposal of energy based model so if you of course if you are probabilities you say well you know this is just a joint distribution I'm just going to learn the density of the Joint Distribution between those two variables and I'm done yeah you can do this in two dimensions you can't do this in 1 million dimensions when the those those things represent natural images for example so what I'm proposing is we're going to learn a energy function as its energy function so think of it as the

Self-Supervised Learning


negative log of a probability but it's not going to be normalized we're not gonna care about normalization okay so that in that way it's a little more general then then then probably stick approaches so give if if our data are those blue beads here an appropriate energy function that captures the dependency between the two variables is look something like this it takes low energies on the on the samples and higher energies outside okay and if we have a system like this if we have a function that has two inputs in this

Self-Supervised Learning


case when I put that gives us the basically the compatibility between the two inputs the two values that we give it we can use it to predict you know I'll give you a value of y2 and then buy gradient descent you can find or buy some search our method you can you can find the two values that produce a low energy on the output and they correspond to the two values of by one that are compatible way to okay so that's that's how inference works in those in those systems the system doesn't produce an output it only has

Self-Supervised Learning


inputs if I could strain the value of some of the inputs I can compute the value of the other inputs that will minimize the energy using some some scheme so now the second question is how do you train this box that produces the energy function and the training will do things like shaping the energy function so that it will take low energy on the on the blue beads and higher energies outside so if you have a primary Trice function say in the form of neural net that produces a scalar output it's very easy to show it a sample and then tune

Self-Supervised Learning


the parameter so the output goes down right so show it a sample one of the blue beads and then tweak the parameters on your net so that the output goes down so you get low energy four data points another second question is how do you make sure the energy is higher outside because if it's if the energy function is flat it just gives you low value for everything it doesn't play any interesting role so you have to make sure the energy is higher outside of the region of data and this is something that probably stick model give you know

Self-Supervised Learning


like maximum likelihood and normalize probabilistic model do automatically and the way you transform an energy based model into into a public one is it's really gives distribution I mean you have several ways but that's kind of a very natural when to do so take your energy function and take the exponential minus multiplied by some arbitrary positive constant and then normalize so you get a bunch of numbers between zero and one that's sum to one okay in the discrete case that's that's called soft max in the continuous case

Self-Supervised Learning


it's it's called a gives distribution in the general case where the energy function is complicated this normalizing turn normalization term is intractable you can't compute this integral so you can't actually turn the energy into into a probability and that's why I'm arguing for just manipulating the energy function now going to a density because you can't normalize it you can only normalize when the energy is kind of has a trivial form and it's not that interesting so right so that's the Department of Energy

Self-Supervised Learning


baserunning it's easy to make the energy low on the samples it's higher to make it it's harder to make sure it's higher outside it's possible to interpret classic running algorithms in terms of sort of energy type models you know classic and supervisor algorithms like PCA or k-means or gets a mixture model or square ICA things like this those are two examples here for PCA and k-means so this is in two dimensions the variable has is just a vector with two dimensions and the energy function for 4k means is for PCA I'm sorry it's just the square

Self-Supervised Learning


root construction error so you take a vector Y but you know project it on the principle subspace which is done by this matrix W and then multiply by W transpose and assuming things are appropriately normalized you get a reconstruction of the original point which is just in the original space the location of a projection of a point onto the onto the mat the linear subspace the principal subspace so this is the principal subspace here in dark and you take any point you project it here and the energy now the reconstruction error

Self-Supervised Learning


which is the distance between the original point on this projection on a principal subspace and so the the grayscale here represent that energy okay zero energy on the principal subspace and energy that goes quadratic quadratically as you move away obviously this is not a good representation of that data manifold this is the data okay so the data points are sampled from this spiral and PCA doesn't do a good job at this right k-means has a funny kind of energy function which is not directly doesn't have a sort of direct form but

Self-Supervised Learning


it's the it's the minimum of some more elementary energy function over some latent variable Z and we'll come back to models of this type so we have a energy function which is a reconstruction error so it's the squared distance between a data point Y and its reconstruction and its reconstruction is the product of a prototype matrix where the columns are bunch of prototypes multiplied by Z vector which is a latent variable and that Z vector is constrained to be a one hot vector so it's a vector with all zeros except it has a one at one

Self-Supervised Learning


location and to measure the reconstruction error of a particular point you figure out which prototype is closest to it and then the reconstruction error is the distance between the data point and the reconstruction and the the the closest prototype okay so the minimizing with respect to the Z vector will figure out which column of W which is which prototype is close as to why and that's the that's the one that produces the lowest reconstruction error and so now the energy function is defined as the minimum of Z where Z is a word of one of

Self-Supervised Learning


K code of this energy function now when you plot this energy will you train k-means on this data set where the data is sampled from this pink spiral and yet I think 2020 prototypes here you get a whole bunch of potential wells if you want so the quadratic bowls energy minima and the overall energy is the minimum of all of those twenty quadratic balls it looks really beautiful in two dimensions it doesn't scale very well within in high dimension because you know how do you populate a high dimensional space with photo types

Self-Supervised Learning


okay so those are merely two examples of classical and supervised Ling methods but here what I've done is list seven different classes or methods that ensure that the energy outside the region of of data is higher than on the of data so I mentioned the one at the top I build a machine so that the volume of low energy stuff is constant and that's the case for PCA for k-means gas mixture model square C etc another one is you make your energy function very flexible but you think of it as the log of some probability and you do maximum

Self-Supervised Learning


likelihood so automatically because your probability distribution needs to be normalized it's going to have the effect of pushing up the energy of stuff you don't observe but it's very difficult because you get the log of this normalization term the partition function which is generally intractable and so it's very hard to do this so that's called maximum likelihood and which when it's intractable you have to use approximation like variational approximations or or Monte Carlo methods or Markov chain Monte Carlo methods

Self-Supervised Learning


another technique which is quite popular is you push down on the energy of data points and then you push up on chosen locations outside and if you are familiar with Gans generated referral networks john chavis all networks our way of doing this so think of the object you're training it again as the discriminator not the generator the discriminator is the thing you're actually training and we think about the discriminator it is an energy function ok we move the exponential at the end of your audio discriminator it just

Self-Supervised Learning


produces a score think of this score as an energy that you're going to try to make large for bad samples and low for good samples okay good for you know low for examples you observe and bad for samples you don't observe and the question is how do you generate bad samples whose energy going to push up and the idea of gann is that you train a neural net which from a bunch of random numbers is going to produce a bad samples whose energy you're gonna push up that's your generator ok so you get and in fact a paper on this called

Self-Supervised Learning


energy based Ganso it's gonna be interpretation against in the context of energy based models and we talked about the other ones except except those two so this one denoising auto-encoder says I'm not going to train the system to actually compute an energy function it's physically but I'm gonna train a dynamical system to start from a point outside the region of data and then I'm going to train a neural net to map it back to the region of data okay so I take a training sample I corrupt it so now I take a noisy training sample and I

Self-Supervised Learning


train a neural net to map it back so my reconstruction error now is the distance between the noisy sample and the original one and so it's going to make the energy kind of be large outside the the region of data and I'll come back to this a little more explicitly but my favorite one is the last one and that last one says we're going to regularize some parameter inside the network so that the volume of stuff that is properly reconstructed is low okay so we're going to make the system pay for reconstructing too many things and how

Self-Supervised Learning


to do this and I'll come to this in a minute so if you are if you are probabilities and you go through this you do maximum likelihood you have you know you you your energy function as the normalized log of some probability distribution so you go through the gives distribution to turn it into a distribution into a normalized distribution and now you have a bunch of data points what you want to do is maximize the probability that your model gives to your data points so the product of the probability that your model gives

Self-Supervised Learning


to the data point or you want to minimize the negative log of the probability that your model gives to all the data points and that's here the objective function you see here so this is the negative log like if you want of a data point it's the negative log of the numerator also divided by data because it's easier minus the negative log of the denominator okay but there's two minuses so that gives us surprise so that's the objective function in each minimize if you are probably list and this will have the effect of pushing

Self-Supervised Learning


down on the energy of the data points and this will have the effect if minimize this you'll have to push up on the energy of every single point in your space okay because to make this small you have to make those energies high because it's negative exponential if you compute the gradient of that last function with respect to the parameters of your energy function you get the expression at the top here which tells you that a step of gradient is going to push down with a unit force on the data point and then the second term says I'm

Self-Supervised Learning


going to push up on every point in the space with a force that's proportional to the probability that my model gives to that point okay so points I have high probability gonna get pushed up really high points that I have low probability which means high energy I'm gonna push that not as not as hard the integral of the force is one and so when the only data point that has high probability which means low energy is the correct one then those two terms balance and the thing can converges if you want you can't actually compute this integral so

Self-Supervised Learning


very often you have to approximate it using Monte Carlo methods or variation of approximations and this is what a lot of people instead of probabilistic modeling do there's a lot of papers on this but you don't need to do all this ok so now let's talk about latent variable models so I talked about the k-means method where this is Z variable that you have to minimize over to get the energy of your of your system and it's kind of specific example of a kind of more general approach ignore the stuff on the on the right for

Self-Supervised Learning


now just look at the left side so you can either prefer the equations or the block diagram depending on whether you are a mathematician and computer scientist or an engineer but it's okay the block diagram now so you have data point Y that you want to reconstruct and your energy function is going to be the reconstruction error you're going to reconstruct this data point by running a latent variable through a decoder function think of it as on your own net in the simple sketch is just a simple matrix W code dictionary matrix and then

Self-Supervised Learning


you compute the square error so in this case the C cost function here is just a square error between the data points and the multiplication of a matrix by latent vector okay this is like a means except we don't constrain this vector to be one of K and there's a problem with this if you just use this without anything else if Z has the same dimension as Y your bigger or even slightly smaller there's always going to be a wire that's going to reconstruct ie it is always going to be a Z that is going to perfectly

Self-Supervised Learning


reconstruct any way you throw at it okay for a non degenerate version of the decoder nondegenerate matrix here is always going to be a Z that exactly we constructed the Y that's not good because that means every point in your white space is going to be exactly reconstructed your energy function is going to be flat equal to zero everywhere so the question now is how do you make sure the energy is high on points that you don't train on and this R of Z here is a regular razor that is going to make you pay for choosing a Z that's outside

Self-Supervised Learning


of kind of a small volume if you want okay and the usual trick something that gmer has worked on a lot is sparse coding so it's fast coding the the regular reservations the l1 norm of Z so it's the sum of the absolute values of the components with Z and the effect of this is basically to make the Machine want to make many components of the zero which is why it's called sparse coding it's you're trying to reconstruct a data point why as the product of a sparse vector with lots of zeros in it multiplied by a matrix okay so assuming

Self-Supervised Learning


W is known I'll give you a why you find the Z that minimizes the sum of those two terms it's going to give you a Z vector that is sparse has got lots of zeros okay and the region of space they have low energy other ones you know basically that energy once you minimize with respect to Z is the energy of every data point and because Z is constrained to be sparse it's going to be a small region of space that takes low energy and outside is going to be high energy okay so by adjusting the alpha coefficient here you can make the region

Self-Supervised Learning


of space that is properly we constructed as small as you want or or larger if you want the general form is something like this where Z equals some cost function that measures the discrepancy between the data point and the decoder function applied to Z the decoder function is trainable and then you have a regularizer that limits the information content essentially of z k-means does this implicitly by restricting Z to be a discrete variable okay PCA or things similar to this implicitly do this by limiting the dimension of Z

Self-Supervised Learning


ignore the bottom for now so if you apply sparse coding to this little spiral data set this is the energy function you get so every line that you you can you can sort of see here is a different linear subspace which is actually the selection of a different column or pair of columns of the W matrix and and the whole thing kind of fits the fits the data right okay the system is trained to just minimize the reconstruction error plus the rigor Iser on the data points and because it's got this regularizer as a consequence it gets high energy to stuff

Self-Supervised Learning


that is outside you apply this to em mist you get things like this so these are the columns of the W matrix and what sparse coding does in this case is that it reconstructs every digit in m nest as a linear combination of a small number of those guys okay because only a small number components of Z can be nonzero so a small number of columns will be selected so every digit is gonna be reconstructed as a linear combination of a small number of those you train the W matrix here in sparse coding but just with great understand you have to play a

Self-Supervised Learning


trick which is that you have to constrain the norm of those vectors to be within a sphere like less than 1 for example otherwise they blow up and the Z variable shrinks but that's not really interesting it's gonna be degenerate solution so you have to constrain the those vectors the columns of W to be small to be bounded and as a result when you train the system the the system identifies those pieces that need to be combined to form characters as kind of small pieces of strokes which is kind of a logical way of decomposing a character

Self-Supervised Learning


into elementary components some our colleagues at Facebook recently used one of those latent variable system but they kind of limited the information content of the little representation by kind of making it low dimensional they call that glow generation through latent optimization this is by version of skies around Lopes pass and shun and if you train on faces for example you get you know recon faces like this with you know relatively low dimensional latent vectors and you can sort of interpolate that latent

Self-Supervised Learning


space and sort of interpolate from you know one phase to another this is you know not as I mean this is a few years old but you know it's it's kind of at the time similar to the the Gans of the time there's been a lot of progress in Gans not so much in this particular approach but but it it's kind of interesting we're approaching the problem now there is an issue with sparse coding which is that if I give you a why you have to run an optimization algorithm to find the optimal Z that minimizes the energy that

Self-Supervised Learning


might be a little expensive even if you use Julian's code called spans it's still a little expensive so here's a trick and the trick is you're not going to make the Z variable a latent variable that you have to optimize over you're just going to make the output of an encoder so you're going to train a neural net here to predict what the optimal code is for sparse coding let's go as possible to encoder so you have a piece of data here you run into an encoder it predicts the value of a variable this variable is regular wise

Self-Supervised Learning


to be sparse through this RZ which could be another one norm and then you reconstruct the d adwords this is in the general form of regular risotto encoder in this particular form of an l1 regularizer that's called a sparse o2 encoder now these two forms of it a form where is just a neural net with an additional term in the in the objective function for training is another form where you still have Z as a written variable but now you have a third term in the energy that makes you pay for making z2 different from the output of

Self-Supervised Learning


the encoder and so the procedure is you give it a Y back within the sense you find a Z that minimizes the sum of three terms the reconstruction error the regularizer and the prediction error here which is the distance from Z to the output of the encoder which is a prediction of Z if you want then once you have that Z you do one step of gradient descent in the parameters of the encoder of the decoder so the decoder tries to get it put to get closer to why and the anchor get try to guess its I'd put the bar to

Self-Supervised Learning


get closer to the by minimizing this okay and there's couple papers on this that are about ten years old we used to call this predictive spas decomposition there's a form of equal lista so if you run this algorithm on natural image patches this is the running algorithm actually running and you start with random so what you see here each square is one column of the W matrix in the encoder I believe here but there is a similar one in the decoder which looks very similar and as learning proceeds of you train on more and more

Self-Supervised Learning


natural image patches what you see is a pattern of kind of features X appearing they end up being oriented edge oriented contour detectors you want something like this is a compositional form of this where the reconstruction now consists in taking a bunch of feature maps those are your latent variables convolving it with a bunch of kernels summing up the results and that's your reconstruction okay so instead of having scaler here you have future maps and instead of having columns of a matrix you have conditional kernels and you get

Self-Supervised Learning


really cool filters so those are the decoder filters these are the encoder filters for various number of various dimensions of the of the code here so get natural feature is emerging completely and you know unsupervised from training all natural image patches let me skip ahead a little bit okay so there is a formulation you've probably heard conservation electronic odor and when you look at those papers is a whole bunch of you know sort of vegetable bounds and all that stuff and it's very hard to understand intuitively

Self-Supervised Learning


what's going on so I'm going to kind of formulate this in the context of this energy based regularized o2 encoder model what a violet one color is and I'm sure for you is going to be enlightening if you haven't completely understood already with a version of the 20 color is so version auto encoder is an auto encoder you fit it the data a piece of data image patch or whatever running to an encoder you predict the code and then you add noise to that code in a particular way you add Gaussian noise to that code and then you run through a

Self-Supervised Learning


decoder and you have a reconstruction error and you minimize a reconstruction error by training this entire system now okay so in the space of codes in the z space every training sample is going to be a point okay and the house is going to be some sort of structure to that to those to those points you don't know if you add noise to each of those points which is what the factional encoded does you turn every single one of those points into a fuzzy ball right so you have a fuzzy sphere around every point now here's a problem if you give this

Self-Supervised Learning


data point and then you add noise and the noise puts it here when you reconstruct the system is going to think it was that point and so the reconstruction error is not going to be very good because it's going to confuse one you know one data point with another one so when you train the system the consequence of of this noise is that all those fuzzy balls are going to fly away from each other okay they're going to try to get as far from each other as possible to minimize the confusion and that doesn't help you it just makes the

Self-Supervised Learning


weights of the encoder larger but doesn't really help you so to prevent this from happening you're going to attach every every single one of those spheres with a spring it to the center okay so you tell us years you can't go too far you know you pay price for going too far and so the system is going to try to find a trade-off between you know pushing the balls far away from each other but not being able to do this is going to merge or or let some of the spheres interpenetrate as long as the reconstruction error that this causes is

Self-Supervised Learning


not too high and so in the end is going to try to find some sort of structure in the latent space that will sort of organize those points so that they you know basically capture the structure of the data and you can think of this as just another way of limiting the information content of the code so our regular riser RZ with sparse coding was limiting the information content of the code this is another way of limiting the information content of the code you can do it by imposing low dimension sparsity various other ways of this type or you

Self-Supervised Learning


can impose it by adding noise as long as you limit the norm of the other points now it's a bit more formulas in inversion autoencoder where the side of this of those balls is not fixed it can actually vary in all dimensions but there is a cost function that makes it makes you pay for making it significantly smaller than one and there's another term that makes sure the average the mean of all those points is actually centered on zero which is not with the spring analogy doesn't quite quite quite do you notice you go to

Self-Supervised Learning


encoder is this idea where you you take input data you corrupt it you ready to go to encoder and you reconstruct the original point without corruption and so visually it looks like this where this would be the data point every every one of those points is a is a a corrupted training sample if you want so it's still the spiral example and then you train a neural net to take you know you take a data point you corrupt it and then you train a neuron that to map from this to here this is the input this is the desired output very simple

Self-Supervised Learning


you transistor on that and that way you can plot is the blue points here are the outputs of that auto encoder neural net for every one of those golden points whatever right so I don't know this one maps to here and this one too there this one to here etc so you see that the system has kind of learned really it's going to similar representation here but you take every grid point and every point on the grid in the space and the blue points are the image of every single one of those points and they primarily cluster around the region of

Self-Supervised Learning


high data density which is what you want the energy function is just the square reconstruction error between the between every point in the space and the point that it maps to and those little vectors here can indicate the displacement if you want and the color indicates the energy function it's an issue which is that the energy is actually zero on this this is not a Ridge it's actually a valley and it's terrible okay so there's a it's a flaw of denoising auto-encoder it can create values in places that you shouldn't have

Self-Supervised Learning


won but it works really well for NLP so some of you may probably have heard of Bert okay it's gonna take on the world by storm in NLP and Bert is a special case of denoising auto-encoder where you give it a piece of text a window of a few hundred words and I know I'm at a time a window of a few hundred words umask so the the corruption consists in masking some of the words in that sentence or in that piece of text typically 15 percent of the words and then you train a giant neuron that transform a neural net which I'm not

Self-Supervised Learning


going to go into the explanation of of basically predicting the words that are missing and they're it's easy to handle the uncertainty in the prediction because at the output you have a big softmax that gives you a probability distribution over all words your dictionary so here I don't have the issue that you have with editing video where it's a high dimensional continuous space it's easy to handle uncertainty in discrete space which is why it works well in this case so you train the system to do reconstruction on tons of

Self-Supervised Learning


text billions of segments of text and in the end you use the internal representation learned by the network as input we to a downstream supervised task like you know natural language inference parsing winogradsky all kinds of stuff and this meets the record on just about it you know although the benchmarks in the glue so glue is in super glue our set of benchmarks in natural language understanding and systems based on bird the latest one is Roberta from Facebook actually has the record I think a few days ago Microsoft came up with an

Self-Supervised Learning


improvement on Roberta that actually can I brought the performance a little better it works really well doesn't work on images so if you use the same trick on images you block a piece of the image and you ask the system to reconstruct it doesn't quite work it doesn't give you features that are very useful in the end for for vision let's see I'm gonna stop here okay I just want to want to show you a simple a simple example of prediction and or certainty that that's that's interesting and it's an example of training one of those predictive model

Self-Supervised Learning


to be a forward model for a control problem of driving a car for example so if you drive a car you might want to be able to predict what the cars around you are going to do and it's not deterministic so if you are if you are this guy you have this car these are all the cars around you this is a little rectangle extracted around around yourself and it might be useful to predict what the cars around you are going to do if you want to plan a kind of trajectory that will avoid you know the probability of future accidents so

Self-Supervised Learning


one thing you can do is something like this where you take a few frames so you have the blue car the green cars our cars around you you're observing the world doing some stuff around you what you're going to train is one of those predicting model a big Commission on that with some latent variable to predict the next frame and the precise I'm going to go into the details with precise architecture it's basically very similar to a VA II in this case where you get the state of the world which is the environment of the car you run it to

Self-Supervised Learning


a couple you know few layers of a ComNet you add a latent variable here which goes through another couple layers of a neural net the certain variable is relatively low dimensional and then you run through a decoder to predict the next frame and to predict the value of the latent variable there is actually an encoder and of course the system could cheat because now it has the answer of the target and so you could just use the value of the target to predict the latent variable that wouldn't be very useful so you restrict information

Self-Supervised Learning


content in the latent variable because this looks like an auto encoder and you do this by the noise via e style essentially skipping details so what you get here is this is the kind of prediction you get if you don't use the latent variable you set it to zero all the time you can deterministic predictions but they become blurry there quickly after a while and those are multiple predictions that you get by making different samples of the latent variables right so you get different futures by getting different samples of

Self-Supervised Learning


the latent variable you can use this type of model as a forward model in a control system where you have the state of the world the action you take whether you turn the wheel without your accelerator brake you run this to your forward model gives you the next state it takes a random sample from the latent variable to make that prediction you can read this multiple time steps there's a cost function that measures if the car is in lane if it's too close to other cars it's differentiable so by back propagation you can propagate gradient

Self-Supervised Learning


to this entire thing and train another neuron that call a policy network to figure out what is the best action that should take so as to minimize the expected value of the cost and it's a it's not reinforcement running it's all differentiable so it's just back prop okay no no reinforcement there if you do this it doesn't work very well you have to regularize the system by making sure it stays in regions of the space where the ford model does a good prediction which are not going to go into the details off and then after you do this

Self-Supervised Learning


you run the the policy network so there's no planning necessary in the end the policy network has already kind of thought in his mind when it was training all the bad things that could happen and this is the the blue car here is driving itself in traffic and you have to realize that the traffic doesn't realize the car the blue car is here it doesn't see it is maybe a better example here so the yellow car is a real car the blue car started at the same location but decided to do something else so this is the one that we drive and here it's

Self-Supervised Learning


I just feel I feel the Ludvig has young legs that's all set anyway great to see everyone right so where to begin so this is this is joint work with a bunch of awesome people and I think really pushing on a lot of really interesting empirical work and I think I'm not gonna say that the talk will be uncontroversial so if I say a thing that makes you angry that's for me and if you say anything that's like well that's awesome that's from one of these fours alright so I know this is works up on deep learning I'd like to start with

Training on the Test Set and Other Heresies


machine learning that's why I talk about machine learning and I want to talk about kind of the conventional wisdom that we give to our undergraduates Jonathan here teaches hundreds of them and so I hope I don't know how many of these things you tell them one they're like some how to have a good predictor you have to balance bias and variance alright and that you really should not perfectly fit your training data that's a no-no t something we definitely teach them that in right yeah you have to oh he already knows you don't the punch


line this is the sad part and we know that high capacity models right wouldn't have really big ones shouldn't generalize that you got to keep them small because if you you know said on the first day trying to do one-dimensional polynomial regression right and optimize high precision non convex optimization that's hard and I think everybody in this room at this point knows that none of these are true right it's a it's weird right so we keep all kind of come to the terms that none of these things are true and yet we


teach in our undergraduate classes I'm trying it's really hard to teach them anything else to be honest nuance is really difficult at the undergraduate level especially when there are 700 of them so I think that's a kind of complicated thing but it is this weird thing that we're in the square in this weird regime where everybody here knows that none of these are true so how I want to push in a little bit on some other things that we tell people that also maybe aren't true kind of in this talk just in case you you don't


believe me what are we okay first of all what do I mean by machine learning I mean something very narrow right for me when I say machine learning yes I would upset some of my colleagues nobly Mike Jordan but I assess on my college might say this is all machine learning is it's just predict making predictions from examples I have a set of something I have a set of something else and I want to find prediction functions that map X to Y I know there's more than that but this is the backbone this is why all of you are here this is why they're


building our semester than this and so the whole idea is that how do we actually estimate these things we ask to make them from data we collect as many training data points as we can we ask them ate them from data and then we just hope that this works on other data a big hope big hope is that we have a lot of data if we collect enough of it we get all edge cases we can drive our cars whatever those cases are right and so that's the notion of what machine learning people mean by generalization I actually think generalization is one of


the machine learning is full of words that confuse the the listener they're just designed to confuse you so generalization when I say that to you that means that you know you learn how to throw a baseball and then you can throw a softball that is not what we mean in machine learning at all and machine learning mean if you throw a baseball you know how to throw a baseball as long as this regulation baseball made to regulation wait you're gonna be able to throw that baseball and that's not at all what would moral


people think right so the idea is that you give me an examples from a distribution I would like to find a good prediction function on these examples I have some loss function that I believe is a reasonable thing that I would like to minimize an expected value meaning that if I have some new thing that comes from this distributions I'd like to be able to do well that's not known and so what you do instead is you replace it with the sample average and the empirical risk and you minimize this and like the only thing we're allowed to do


what's wrong the only thing we're to do now is minimize this using one of two algorithm and they're like three things we can do we minimize it that way and then of course our generalization error is just the difference between these two things and so the question is you know if we can compute this one when is that a good or bad proxy for the other one right that's our whole that's what we mean by realization it's a very it's just a badly named term but it's very interesting couple problem and it's kind of like the core problem in machine


learning right and the core theorem of machine learning is that the the population error that we really care about is just equal to the training error plus the generalization error right sorry that's like the foundation of ten thousand ten thousand papers right we've all done that at one point in our lives right so we can measure this one and then we do a lot of thinking about why this should be small I mean this requires like the associative property there's stuff to be done here right I think we can equip a


little it about derive versus actually Matic right so I think right so what can we take away from this we know that if you have a small training error that means that the risk itself is just you're really only in leveraging generalization error right so if your training error for example is zero you're just hopefully know somehow that the generalization error is small and sorry and zero training error we know does not imply overfitting even though that's another thing that we do tend to sometimes gets lost in the weeds like


for example the the paper I really like which one test of time and we're at knits last year by um Bruce K and Mbutu right has this has this thing saying that you shouldn't train you shouldn't run an SGD too long because at some point all of these terms should be one over square root of n but we know that's not really true right I mean just because this is whatever squared event doesn't mean this one is one over square root of n and just cuz this is zero doesn't mean this one's small so you know we it's useful it's a useful way of


thinking about it but sometimes we over fit to what kind of these papers say all right so don't want to say it's not always true but it's a useful way of thinking about it another way that this one ends up sometimes is presented this is how I learned it not everybody possess it this way is that you decompose is this it's it's the same same proof though it's you did you put the error it's three parts right so there you compare your error of your estimator versus the error the best thing in that in the class of stuff


you've been looking at right and then you compare the error the best in the class - like what the true prediction function the best population risk monument monitor is and then you are stuck with some bias at the end which is this is just the irreducible error that you can never get away from and to whichever one you like more I don't care I like this one just because it allows me to attack this most my least favorite figure I'm kind of gonna be like the core the core of the talk today right so this is my least favorite


figure it's from hasty and tip shirani I like those guys buns in this figure we read way too much into this right so somehow the idea here is that we have to balance bias and variance in order to get good model complexity again you can do that you can do that but this is by no means by knowing is the only way to generalize and we know this okay I learned this because of deep learning and to be fair I learned this before and had forgotten it and I'm gonna talk to you about that in a second but I learned it again recently because of deep


learning that you can just get make these models gigantic and and they generalize so in a very unpopular paper that I know lots of people who are have my back tear right now don't like sorry maybe just one anyway so we read we read a bunch of experiments on this if our 10 dataset everybody's favorite dataset where we have chickens frogs deer in trucks and you know you rut so this is a 10 class classification problem it's relatively high dimensional has three thousand dimensions because the pixels 50,000 data points and if you take you


know what we found is that you can either get a loss to be nonzero here this is the loss is the log loss which is not the classification accuracy you can work really hard to get this to get your generalization error down and in this case the test error or you could just run it to zero essentially just taking this configuration and turning off all the regularization round to zero and while you see a drop in accuracy you don't see a gigantic drop so the test error increases but only by about 5% and moreover if you just pick a bigger model


now you're considerably better than the original Alex net if you pick an even bigger model you keep going down she wants here I'm sure he tried something even larger we just ran out of time so right so somehow here that the the the regularization parameters turned out to be just knobs that you can tune in terms of you could also tune architectures you could do lots of things and keep pushing this error down and indeed we saw the same thing on image net where here we have chicken frog Deere and truck much more clear right I


think at that point anyway and so we looked at an inception model inside Google I really don't like this experiment I just want to show it just to kind of give evidence that this happens and larger datasets the reason I don't like this experiment is that because of the way that these things were trained Inc site Google at the time we were able to run about six experiments six runs I think have we used I think one of the more valuable things that's happened to the community is this add-on benchmark and we just


used the DA benchmark we've been enabled around hundreds of experiments much better here but all we were able to do here was toggle flags inside some Google 3 models and so in particular you know in this case what we could do is we could turn off the LT regularization and we could turn off the the drop out sorry the UM data augmentation and you could still get perfect dieter palatial and even note that the top five accuracy here is only 19 point three sorry tight error top tight five errors only ninety point three you try to get to something


that has a 19 point three percent error it's really hard so it's significantly worse than what the inception model is getting but again it's not it's not catastrophic about this would have won this would have beat Alex net the first time around so still if it's good accuracy yeah let's ever talk like that regularization before my control variance so what yeah so okay there is talk of implicit regularization I guess what I would say is that we started to just look at more models when we started to do is stop looking at just this individual models


or for we just started download more this actually became a big trend in the research group um it's there's something that Becca roll-off started kind of kind of pushed us on this direction and the mood became sitting here kind of really pushed us even further in this context it huh Baz an experimental resource so you just get pull some some models and then see what happens and a bunch of things so this is the scatter pot I don't really buy that parameters the neural network is actually meaningful but you can see that essentially as you


make the number of parameters bigger the models continue to get bigger blue line is just the minimum error and the red is just a scatter plot isn't models at a particular model size I'm just taking the minimum error of the Reds yeah I mean like I'm not saying that they're all true I'm just saying you could do a pull request over here I go fine I say I can call Fernando get him on this one this is even better what you do here is just remove models and so now you see the trend this is there's no Alex net or vgd on here they kind of


ruin that but they're just up here some ways fundings like the Curt you really care about this lower envelope but this is the same thing kind of happens on imagenet you just keep making them bigger well you'll know he's like we have log axis on the X and and and and linear on the Y so semi-log this I pulled from a paper by some Google folks because only Google folks would think it would be fun to train something with 600 billion parameters million million million ok that's reasonable 600 billion was stupid but six our million perfectly


reasonable anyway so that's that one he's getting bigger here's your visa this is from misha's talk yesterday well I made it but it's fine it's the same same plot right we see the double descents this is now we're random feature models where we just keep adding random features and see how the accuracy goes and even though it does go up it kind of comes back down as you make the models bigger and bigger am i i'm just interpolating so this one is just interpolate I'm not doing this case is man we minimum euclidian arm


although although what's interesting is that I like this plot better is that if you don't do minimal Euclidean norm but you just go use read regression so you allow yourself a tunable parameter now that dip goes away no it just gets better oh so I don't know all right so this is not minimum nor mystique this is regression and then that bump goes away yeah you've picked the best Ridge you pick the best Ridge it's not a constable I picked the best one for each and you're gonna see why in a second this is the best one for each it's the only done


with a number of features give it like for that the best value goes down every number is so yes so sorry in this case I'm not taking the best one from here I'm not taking the best of these two it is just if I tuned this what's the best I can do just on this one fiber here is a paper something I stole from Peter this is from an inner UPS actually was nipped at the time but nerves tutorial from 20 years ago here he was doing boosting some of us what do you see here well I have first of all note the semi-log axis there we go semi log X


on the you know that's pretty neat and you add parameters I mean boosting right you increase your model size with every step and so now I have the models get bigger and bigger and bigger and the test error keeps going down I think the the so that's that was interesting right so that was small data machine learning but I still seem to have a thing this is another really interesting one which I pulled from a C ACM article by Bell and Koren describing how they won the Netflix prize and they saw exam the same thing that they have different kinds of


models but they kept making their models larger again we have semi log X on the routers linear in error and you see this thing kind of continue to keep going down to bigger and bigger you make your model so I think there is something interesting there so there you're seeing two things one making the model really huge doesn't doesn't you know it has a ton of capacity maybe you're controlling it with various kinds of regularization of some form or another but oh it just seems make it bigger worry about that later


does seem to be a good take home and the other thing is that you see significant diminishing returns right I mean if you have a log x axis I mean this doesn't mean that eventually you have to give up even Google eventually you have to give up so I'm not sure that this is necessary how to get a suit like that that irreducible error we'd like to get you no no no no no it's not a typo not a typo a hundred thousand million that's like Carl Sagan it's a hundred thousand million parameters that's a big deep I didn't make this plot they don't get


their ice log these need different things you have to read the caption that's not here sorry I maybe should've edited these things do not mean don't they're in the caption whatever those mean I don't know what those numbers are these number factors are you sure paper [Laughter] anyway so we could look it up and think see a cm it's cool they did a lot of good work let's go back to this one so look here's what I would like to say like there are crazy diminishing returns - it does seem making the model bare for


a fixed holdout set that we fix in time for the history of the universe does make that test error go down but that's also not generalization error alright if you do a holdout split or you take a train set and you take a holdout set and then you just fix that holdout set forever maybe what's happening here is you have this these giant models have enough fluctuations in them that they can you now over you could actually leak a lot to the test set and over fit to this one holdout set that you fixed forever so this leads to a question this


leads to a question we're all spend most of the rest of the time of the talk there it's perfect I mean I know we have planet I'm good so yeah that's the rest of the talk so maybe maybe what's happening here is you make these models really big and that allows you to overfit on this one holdout set and so there's only one way to check right which is make a new holdout set okay there a better way I don't know but we'd that's what we did that's what we did we made a new holdout set let me explain how so if here is progress on SIF r10


over time so if you just use raw pixels and do linear classification you get thirty seven percent accuracy I just realized we've switched from error to accuracy hopefully it will be clear from context of the same accuracy now we have 97.1% in 2017 and 2019 what is it lou big-nose these numbers ninety-nine point what nine point zero so we got you know there's still time to write more iclear papers everybody we've got ten more ten more ticks so this is like our deep revolution here all right this is the deep revolution happened in


2012 and you know we just keep making progress make them bigger get them more capacity make these models really large the shake-shake models are you.why rednecks also or just ones right I try to get them to fit brushes is this overfitting right is this overfitting right because this early part we can match with shallow Methos yeah actually you can get even get to like eighty I can't remember the number but there's there some work by chumki cada aleck agrawal greg valiant and leigh song like white 13 so if you just do random


features just you can get to about 85% accuracy using just dumb random features shallow stuff can get it was big so it's also a large shallow model can get to about 85 and so the question is are all we're doing here is just overfitting to the test set by graduate student design so we're gonna check and we're gonna check by building a new test set now what does that mean so it turns out that the syph are ten creation process is super well documented and it was documented by the folks at Toronto who made it in the first place


and so there there is a lot of details about how they did it and in particular where they got their images to begin with and that comes from one of my favorite data sets ever it's called the 80 million tiny images the tiny images data set and it's a was curated by Antonio - Rob Rob Ferguson bill Freeman and they made beautiful like scatter plots like all the images on the internet and these like mosaics and just seeing like how everything varies it's very cool kind of looking at what kind of things are out there that you can get


to get off the internet in 2008 so the reason why there were thumbnails is because they wanted to make something that you could store and then do all sorts of visualization and studies with so from these thumbnails they subsample 60,000 using a process that was very well detailed supply well detailed with human not experts but human laborers and they tried to get down to these ten classes so can we get iid resampling well there are you know there's out of 80 million images we only took 60,000 out so the hope is maybe we could sample


some more maybe not even that many more how many do we need to actually get something that would believe so I think Ludvig did his error bar calculation that said mm that's what he wanted 2,000 new ones and so and work that I did not want to get involved with Ludvig Becca and VY salt labeled I say mostly just Becca right this one for this one was looting in Becca labeled tens of thousands of images in in the tiny images data set and we got a new test set of size 2000 and again there find details about like what counts as a boat what counts to the


car and this kind of thing we drive to match as closely as possible so what did we see okay so the first thing we see is we take vgg 16 everyone's favorite network all right I guess anyway this is a big one and what we saw is a huge drop in accuracy right there's an 8 percent drop in accuracy from the first test set to the second test set which is much bigger than you would expect by a 1 over root N and some some reasonable capacity that's big mean this is we put a little confidence interval about where we should be this dashed line is the


reproducibility you'd hope that the confidence interval would hit the dash line that would mean okay there's you know these we'd say hey we're not having been overfitting at all well that's great that's great so it's clear what's gonna happen cuz right 85.3 was right in the ballpark of what people had seen you could do with just showering clearly what's gonna happen let me just I don't know if anybody's read this paper yeah but clearly what I thought would happen what I put my money on was that yeah now the shallow learning stuff will just be


here and we'll just see that we'll just get a saturation so we've just been adapting and we'll just see a saturation up here and everybody will be equal and right and that was like this big drop and so here here here's our random features that's not what we saw at all not what we saw at all I lost money I didn't lose money because all everybody else got the same it's good there was no house in this case right bigger drop 12% drop so that went from eighty five point six to seventy three point one bigger job present and the


shake-shake model only had a 4% drop this one that had super-high Agra so we saw exactly the opposite of what our hypothesis said right our hypothesis suggested that maybe the big models were able to just fluctuate themselves to overfit to there's one particular holdout set and if we draw a new holdout set we'll see some sign that they had adapted to it but we saw the opposite in fact the ones that have the better test error on the original test set have a better accuracy on the new testament oh did you have the original weights for


these models or did you have to retrain them we did well there was that we did retraining in the paper but in this case this was just original weights this is some summary training but again this is why github as an experimental resource is so powerful people put post the weights in the repos saves you tons of time so it actually is the contrary it's very you don't even need a GPU right thank you you can just download stuff that people already done change a parameter here do some kind experiments you already picked out like got a big


problem here right so there there was clearly this is not an iid resampling it's not ID resampling because that can Ludvig or not we don't know who the toronto folks were probably some paid undergraduates and they're not that kind of like who have never been to Toronto right yes so there is a there is a yeah so it's not perfectly ID right this is a great question but we'll come back to it let's come back on the empirical training possibly yeah but no you can't tell a difference if you train a classifier do you sample


test what white features oh just individual features to see if you see significant things we didn't do we did not do any of this yeah but let me keep going well then they're not from the same distribution right Marshall over it yeah we would hope so right I mean I that that you could probably check because there were some sub selected to begin with right so if too many were sub-select 'add then they're not going to be iid and that's something I think we could probably the dead part didn't vote human so that part


did not involve humans but not he's asking is just that probably the only thing that changed probably again being generous to us the only thing that's changing here is the labeling function not that of the human the thing is producing the why not the thing that's producing the X actually sight under why you only cycle you just want to know if you could do read you interleave the two test sets if you or sorry if you can interleave some of the stuff into the training data so we're gonna be then the entire and actually the fact that better models


have a smaller drop suggests that maybe this boils down to the test and it could be I mean what well so sorry what so what were you what are you suggesting can we just quit crossing it so this is actually really important this is really important the fact that you've already moved on from the fact that there's no adaptive overfitting is shocking look there's no adaptive overfitting we took this is rule one don't look at the holdout set more than once that's rule one we look at the holdout set fifty thousand trillion times at Google


every day fifty thousand trillion times and yet it doesn't matter the better you do on this holdout set this one stupid fixed holdout set like it doesn't matter that I think really before we talking to about why the drop happens just the fact that we don't see adaptive overfitting isn't it blew my mind I did not think that would happen I did not think that was or whether you see the single noun to productive or fitting into some others and so those are two different things which so far from your results in love


wait what does that mean might be your video yeah I cut her with Zico there's no adaptive or control and this gap is only because of the difference in the distribution yeah the thing is to be fair gonna be hard to get it's gonna be well I think so but right listen one second what is that I think it's gonna be hard to tell from safar 10 and I think what's interesting is if we go to this other data set that's more interesting anyway I mean maybe we can't a little bit but I think it's much more competitive yeah


yeah I'm getting there maybe I might be gay there we'll see how a lot of people want to stay through lunch yeah tell you it's not his hand up for a second here go ahead what it's gonna have a quick propose also epub Bank I knows we can have like 50 km yeah all these new data set and then test on all superpower tested and I don't know if we do get at leats I I don't think we get too concerned about mm emerges and some of 50 was a basically exposed so if you wanted to get another 50,000 mm just would actually be different


I think we could probably do more in 2000 but I'm not sure what the limit is yeah yeah that's actually in the paper you can see at the end but but also also the thing is I only have two Becca Becca and living now they have limited cycles you know this is a problem it's a problem it's a problem and certainly certain and I lost both of them that's really sad actually that's really sad yet both well I know loot big sting sorry good okay that was okay no first thing we have to do though we did these Becca she's graduated this year okay


also who cares about as if our time so far ten is this fun little thing that we all kind of like this we can train it and it certainly it's interesting because it's like the first non-trivial thing we can get to but you know what captured people's imagination was this image net data set questions do we do that one now that's harder because that's bigger there's more labels there's more and even like even here like this 1.2 million training images were not labeled in the lab they were labeled using Mechanical


Turk that's opportunity right so now we could try to reproduce this data set and this procedure using Mechanical Turk which makes it much more scalable and we can make much larger who can do much larger experiments so how do we get the xiety resampling of image that in the water but it's not as surprising as you say given the fact that we can we can drive particular models to zero laws something which we thought was impossible and that's okay this is like the community drive in there it's like community doing or repeating


on the test and it's kind of a same effect no I don't think so like why do it why don't we see a plateau anyway that's right that's right that's why anyway let's this is more interesting Misha the zipper tempting is so boring that meets let me show you this one this is much more interesting much more interesting I promise I promise much more interesting as if you want we can go into some more nitty-gritty with this one I have far more that I'm gonna be able to get through which is fine it's about no equations I'm no proof assemble class


like this so this this is kind of the interface that you would see if you were a mechanical turk being paid to label images on the internet which is basically how all of our everything is made to work now inside all the big tent companies they hire people and have them watch horrible disturbing content and flag it for their their algorithms is great wonderful world we live in so hey so we have so here's our images right these are supposed to be bow everybody can read the top a bow is actually is not quite clear a weapon for


shooting arrows composed of a curved piece of resilient wood with a top cord to propel the arrow and the task is click on every single image that has a boat that's what the Turker is asked to do click on every single image that has a bow so for example these three have both these ones don't actually these were three that were adversarially placed in here to make sure that the people that that the turgor z' weren't cheating by the so what we were eight what we did was we actually flag all of these nuke so we come in reproducing and


reproducing image that we tried to reproduce the query that was used to collect images reproduce the things that were allowed like including the date range from flickr reproduce the way that these things were labeled and we also through the old test set in here to see how those were labeled by these Turner's which is much much nicer than what we're able to do with with safar tender and what's interesting here is that there's super high variability in what everybody's like not everybody says the same thing obviously there's gonna be


disagreement right there's one you have this is site anybody who work in crowdsourcing knows this right you have the people who do these tasks see through they work too quickly or there's just new us right so for example this is this bow was selected a hundred percent of the time by every turn this one was 70% of our heroine from brave I mean there's a bow there it's a little bit hard to see you on this blown off screen right so there's a loss of reason it's not Center frame so the lot of reason to miss it this one now becomes a metaphysical


conversation about what is the bow good friend Conrad Curtin says yes so I mean neuroscientist so you know I don't know everybody has a good opinion about what that is and this one's wrong and then 20% of the people the boat it's a boat it's not the boat we want we want right ok so I think that's that's our issue right so this is not the boat we want and so we actually had like the histograms for every class for every class thousand classes we had histograms of the selection frequencies and so when we actually sampled our new test set we


tried to match the statistics of the the label from the original set with the new set also the big back end vise all looked at every single image and made sure it was correct in the new test which is clearly creating a distribution shift yes Alex told me it's Berg told me but I'm not anyway it's not true oh yeah you could go look look that's actually the other amazing thing about all these data sets how many people who ever looked at the images name isn't it I mean you don't need to you just watch the curve go down but like you know it's


like actually especially though it's like we kind of have decoupled ourselves from actually any of the domain expertise it's actually quite entertaining what you do yes it's quite a JD what do you do I grew up right ok so punch line right we see exactly the same thing we see exactly the same thing we saw in separate n a bigger drop technically speaking is the tempers dislike it 10 percent drop at the top here for the best ones but we still see a positive slope so the models that have that models that fared better on the original


set fared better on the new set but a little bit it's not for analysis it was first two part 10 but we definitely see a positive slope in our fit but we do still see the significant drop so what is how is the new legally we need you said you also labeled the ultimately just we labeled the old imagenet just to match the selection frequencies from our so okay there's was there a distribution shift already you are you already pointed out that Ludwig and Becca are not the same and then we matched their sampling frequencies but


you also in the process you also saying we sampled from we have a bigger pool we didn't have our new test set with a bigger pool of images and we sampled from that pool to match the statistics of the old Tessa oh just the statistics yes no no no not using in this evaluation say there are no old images in here really yeah yeah yes and there isn't same neighborhood yeah you can measure how will the old labels this might labeling take those images on those images take the predictor defined by the old labels yeah


accuracy on old accuracy images it doesn't write it has seven foundation just old validation is yeah it's noisy its noisy no man it's noisy it's still noisy again so the orange right here nothing too big nor think it's then but even if it doesn't matter it says you're taking the label that was as it appears in the chain sent every image on these the label yeah it was generated using some procedure yeah measure of how well those predictions predicted a pose that was assigned to me for your process that's not how these sets are built at


all hey Louie let me I got that I've got the okay it's okay I got the forest but I know I'm also used to this it's a procedure it's a procedure so so it's also true right that that none of there's no these data sets are not made by labeling images again like what happened right you show people you you query and then you test correctness as to whether or not they go in or out it's a weird it's a weird process the other thing that's really interesting by the way the other thing it's really interesting we take for granted the