get to ask a few more questions so the solution to this exercise which is we're looking at the value function of x times y where x and y are just scalars there is actually an equilibrium point where x is 0 and Y is 0 when when they're both 0 the each of them causes the gradient to go away on the and then we can look at the gradient descent dynamics by analyzing it as a


continuous-time system so if we actually evaluate the gradients DX DT is negative Y and dy DT is positive x the sign difference is because one of them is trying to minimize the value of function and one of them is trying to maximize it if we then go ahead and solve this differential equation to find the directions I guess there's a lot of different ways of doing it depending on

Ian Goodfellow: Generative Adversarial Networks (NIPS 2016 tutorial)


exactly which pattern matching technique you're most comfortable with my particular approach is to differentiate the second equation with respect to T and then I get that d squared Y DT squared is negative Y so I recognize from that that we're looking at a sinusoidal basis of solutions and from that you can guess and check the corresponding coefficients and we get


that we have this circular orbit where the only real thing that changes exactly what this the circle looks like is the initial conditions so if you initialize right on the origin you'll stay on the origin but if you initialize off the origin you never get any closer to it so a gradient descent goes into an orbit and oscillates forever rather than converging and then this is continuous


time gradient descent where we have an infinitesimal step size if we use a larger step size then it can actually spiral outward forever so there are actually conditions that you can check to see whether or not simultaneous gradient descent will converge or not and they involve complex eigenvalues of a matrix of second derivatives and I won't go into it because it's not really


the kind of thing that makes for a nice talk but the long and short of it is the the generative adversarial Nets game does not satisfy the main sufficient condition for convergence so that doesn't mean that they don't converge it means that we don't know whether they converge or not according to you the main criterion that we can look at and it seems like in practice they do


converge sometimes and they doubt other times and we don't have a great understanding of why they do or don't but the most important thing under said is that simultaneous gradient descent is not really an algorithm for looking for equilibria of game it it sometimes does that but it's it's not really its its purpose and the most important research direction in genitive


ever sonnets is to find an algorithm that does find equilibria in these high dimensional continuous non convex spaces it's important to mention that if we were able to optimize the generative adversarial network in function space if we were able to update the density function corresponding to the generator and and the discriminators beliefs about the generator directly then we can


actually use convexity in function space to prove that simultaneous gradient descent converges for that particular problem the reason that this breaks down is that we don't actually update the densities directly we update the G and D functions that do the sampling and and the ratio estimation and then on top of that we represent G and D using parametric functions deep neural


networks where the actual output values of G and D are very non convex functions of the parameters and so that causes us to lose all of our guarantees for convergence the main way that we see this affect the generative a dresser networks game is that we get behaviors like oscillation where the generator continually makes very different samples from one step to another but doesn't


ever actually converge to producing a nice consistent set of samples in particular the worst form of non convergence and one that happens particularly often is what we call mode collapse where the generator starts to make only one sample or one similar theme of related samples it usually doesn't output exactly the same image over and over again but it might do


something like every image it creates is a picture of the same dog and the dog is in different positions or has different objects in the background or we might see you know every sample it makes as a beach scene for example but it is essentially generating too few of things so the reason that mode collapse happens particularly often for the genitive adversarial Nets game is that the game


is a little bit pathological and the way that we specify the value function in particular if we look at the minimax version the min Max and the max min do different things if we do the min max where we put the discriminator in the inner loop and maximize over it there then we're guaranteed to converge to the correct distribution in practice we don't actually do the maximization in the


inner loop we do gradient descent on both players simultaneously if we put G in the inner loop that actually corresponds to a pathological version of the game where the generator learns to place all of its mass on the single point that the discriminator currently finds to be most likely so Luke Metz and his collaborators produced a really nice visualization of this in their recent


papers submitted to iclear where we have this target distribution shown in the middle of the slide which has several different modes in two-dimensional space and then over time we see how as we move left to right and train a generative adversarial Network we learn to sample from different modes of that distribution but we don't ever actually get multiple modes at the same time this


is because simultaneous gradient descent can sometimes behave a little bit like min Max and a little bit like max min and we're just unlucky enough that it often behaves more like max min and does the thing that we don't want some people have explained mode collapse in terms of the fact that we use the reverse KL loss that I described earlier when I said that I don't believe the reverse KL loss


it describes why we get sharp samples because the reverse KL loss would prefer to choose a single mode rather than averaged out two different modes it does superficially seem like it might explain why we get mode collapse but I don't think that it is actually the explanation in this case for one thing if we use the forward KL we still get mode collapse in many cases also the


reverse KL divergence does not say that we should collapse to a single mode it says that if our model is not able to represent every mode and to put sharp divisions between them then it should discard modes rather than blur modes but it would still prefer to have as many modes as the model can represent and with generative adversarial networks we usually see is a collapse to a much


smaller number of modes than the all can represent that makes me believe that the problem is really that we're doing max-min rather than that we're using the wrong cost we often see that generative adverts on networks work best on tasks that are conditional where we take an input and map it to some output and we're reasonably happy with the result as long as the output looks


acceptable and in particular we may not really notice if there's low diversity in the output so for example sentence to image generation as long as we get an image that actually resembles the sentence we're pretty happy with the output even if there isn't that much diversity in it Scott Reid and his collaborators have recently showed that for these sentence to image tasks


generative adversarial networks seem to produce samples that are much less diverse than those produced by other models in the panel on the right we can see how the sentence a man in an orange jacket with sunglasses and a hat skis down a hill gives three different images of a man in essentially the same pose when we use a generative adversarial Network but using the model developed in


this paper it's possible to get greater diversity in the output one way that we can try to reduce the mode collapse problem is to introduce what Tim Solomon calls mini-batch features these are features that look at the entire mini batch of samples when examining a single sample if that sample is too close to the other members of the mini batch then it can be rejected as having collapsed


to a single mode this procedure led to much better image quality on CFR 10 we're now able to see all the different ten classes of images in CFR 10 on the Left I show you the training data so you can see that this data is not particularly beautiful to start with it's 32 by 32 pixels so it's it's relatively low resolution you can see that there are things like cars


airplanes horses and so on in the panel on the right we have a Gantt rain'd with mini-batch features and it is now successfully able to generate many different recognizable classes like cars and horses and so on previous generative adverse own networks on CFR 10 would usually give only photo texture blobs that would look like regions of grass and regions of sky


regions of water but would not usually have recognizable object classes in them an image net the object classes are not as recognizable but if we go through and cherry-pick examples we can see some relatively nice recognizable images where we get many different kinds of animals like dogs and maybe koalas and birds and so on if we look at some of the problems that arise with this


sampling procedure we can see some of the amusing things that convolutional networks get wrong one thing in particular is that I think probably due to the way that pooling works in the convolutional network the network is usually testing whether some feature is absent or present but not testing how many times it occurs so we tend to get multiple heads in one image or animals


that have more than one face on the same head we also often get problems where the perspective of an image is greatly reduced and I think this might be due to the network not having enough long range connections between different pixels in the image but it's hard for it to tell the things like foreshortening ought to happen in particular the picture of the gray and orange dog looks literally like


a cubist painting to me where you know the Cubist's intentionally removed the perspective some of them also just look like we've taken an animal and skinned it and laid its fur out flat on the ground and then taken an axis aligned photo of it we also see a lot of problems where individual details are great but the global structure is wrong like there's this cow that is both


quadrupedal and bipedal there's a dog whose eyes are different sizes from each other and and a cat that has like a lamprey mouth we also often just see animals that don't really seem to have legs that they just sort of vanished into fur blobs that often conveniently end at the edge of the image so that the network doesn't need to provide the legs so did anybody notice anything that


actually looked real in in these samples Aaron yeah so the cat was the cat was real to test your discriminator network good job Aaron another really promising way to reduce the moat collapse problem besides many batch features is called unrolled gans this was recently introduced by Google brain and was submitted to iclear and I guess it's worth mentioning that a few other people had suggested


doing this for a few years beforehand so it's it is an idea that was floating around into ether a little bit I imagine some people in the audience are probably thinking like oh I told people about that but Brian was the first to go ahead and get it to really work really well revisiting the same visualization that we saw earlier there unrolled Gunn is able to actually get all the different


modes so the way that unrolling works is that to really make sure that we're doing min/max rather than max-min we actually use that maximization operation in the inner loop as part of the computational graph that we backprop through so instead of having a single fixed copy of the discriminator we build a complete tensor flow graph describing K steps of the learning process for the


discriminator so the generator nut is essentially looking into the future and predicting where the discriminator will be several steps later and because it's the generator looking into the future rather than the discriminator looking into the future we're actually setting a direction for that min max problem we're saying that it's max over the discriminator and the inner loop and


then men over the generator in the outer loop and that very elegantly gets us around the mode collapse problem another really big important research direction for generative address that works is figuring out how to evaluate them this is actually a problem that's broader than just generative address donuts it's a problem for generative models across the board models with good likelihood


can produce bad samples models with good samples can actually have a very bad likelihood and then even when we talk about good samples and bad samples there's not really a very effective way to quantify how good about a sample is there's a really good paper called a note on the evaluation of generative models the walks through a lot of corner cases to clearly explain all the


problems with the different metrics that we have available for today and then for genitive adverse so networks these problems are compounded by the fact that it's actually pretty hard to estimate the likelihood there is a paper based on estimating the likelihood in submission to I clear though so that problem might be cleared up pretty soon once once we have more


experience with that particular methodology another research frontier is figuring out how to use discrete outputs with generative ever sale networks I described earlier that the only real condition we impose on the generator network is that it be differentiable and that's a pretty weak criterion but unfortunately it means that we can't really generate sequences of characters


or words because those are discrete and if the output is discrete then the function isn't differentiable you can imagine a few ways around this one is you could use the reinforce algorithm to do policy gradients and use that to Train the generator network there's also the recently introduced techniques based on the Gumbel distribution for doing relaxations that allow you to Train


discrete variables or finally you could do the old-fashioned thing that we used to do I saw geoff hinton on thursday and he was mentioning to me how this reminds him a lot of the way that Boltzmann machines were really bad at generating continuous values so what we do there is we would pre process continuous values to convert them into a binary space and then we'd use Boltzmann machines from


there so you could do the same thing in Reverse with genitive adversarial nuts you could have a model that converts these binary values to continuous values and then use generative adverts or networks from there you could for example train a word embedding model and then have a generative adversarial network that produces word embeddings rather than directly producing discrete


words one very interesting extension of the discriminator is to actually make it recognize different classes and this allows us to participate in an important research area of semi-supervised learning with generative adversarial networks originally generative adversarial networks used just a binary output value that said whether things are real or fake but if we add extra


outputs saying which class they belong to and then having one fake class we are able to then take the and use it to classify data after we finished training the whole process and because it's learned to reject lots of fake data it actually gets regularize drooly well using this approach tim Salomon's and and i and our other collaborators in open area we're able to


set the state of the art on several different recognition tasks with very few labeled examples on em nist c fart n @ sv hn another important research direction is learning to make the code interpretable Peter Chen's info Gann paper here at nips actually shows how we can learn a code where different elements of the code correspond to specific semantically meaningful


variables like the position of an image another research Direction is connections to reinforcement learning recent papers have shown that generative address cell networks can be interpreted as an actor critic method or used for imitation learning or interpreted as inverse reinforcement learning finally if we're able to come up with a good algorithm for finding equilibria in


games we can apply that algorithm to many other places besides generative adversarial networks things like robust optimization literally playing games like chess and checkers resisting adversarial examples guaranteeing privacy against an attacker who wants to thwart your privacy and all of these different application areas are all examples of games that are rise in


artificial intelligence and might be improved by the same kinds of techniques that could help us to improve generative adversarial networks we're very close to out of time but I'll give you five minutes to do this exercise and I'll answer the last set of questions during the exercise this exercise is jumping back a little bit to earlier how I described that there's a different cost


function that you can use for maximum likelihood and generative adversarial networks and I think this is a really good closing exercise because it really drives home the point that the key mathematical tool generative a versatile networks give you is the ability to estimate a ratio and to see how the estimate ratio estimation works you are going to derive the maximum


likelihood learning rule in particular we have a cost function for the generator network which is an expectation of X sampled from the generator and then applying f of X and we want to figure out what f of X should be to make this cost function give us a maximum likelihood as a hint you should first start by showing the following that the derivatives with respect to the


parameters of the cost function are given by this expectation of f of X multiplied by the derivatives of the likelihood and if you'd like you could actually just take that as a given and skip to the last step at the very end what you do is you should figure out what f of X should be given this fact about the gradients if you can choose the right f of X you can


get the maximum likelihood gradient so I'll give you a few minutes to work on that and I'll take a few questions and then and then I'll conclude so in your previous about as a generative Network are you missions out there is a important assumption that is a function should be differentiable what if the function is not differentiable because in some area such


lacto Informatics the data is some categorical categorical label not numerical value so it's 94 insurable so in that use that acceleration how to generate a synthetic data and you using GT and network so there's there haven't been any papers actually solving that problem yet I talk about this a few slides earlier and my recommendations are to try the reinforce algorithm to do


policy gradients with discrete actions to try the concrete distribution and Gumbel softmax which are two papers that were recently released about how to train models with discrete outputs or to convert the problem into a continuous space where a generative address donuts can be applied so the variance and ganz is that it's very powerful in capturing the modes of the distribution


right but it's not really truly understanding what images are as in disease you know you start from zero to generate X right so the question is you know if you increase the systems increase the image size assumably the modes of the distribution going to increase exponentially so ultimately you know if you have a you know practically this may not be solution this may not be


a problem maybe we just care about hundred by hundred pixel images but this assume I'm interested in two thousand by 2,000 pixel images you know if I truly understand what images are how images are generated you know there is no difference between a hundred by a hundred and two thousand by two thousand I can you know that ultimate machine my question is about like way down the


future I mean at the end of the day you are capturing modes of the distribution but this mode is going to explode if you go to larger images so at some point you know the the modes of the model also have an exponential explosion as you use a bigger convolutional net so if I mean I mean I don't want to repeat the same structure I mean the question is the modes of the distribution right at the


end at the end of the day you are capturing the modes of the distribution yeah but a larger model can capture more modes I guess the nice thing about natural images is that when you increase the resolution you're looking at a different level of detail but within the same level of detail the same structure is repeated all across the image so let's say that we've been studying 64 by


64 images and we couldn't really see the individual firs ah like Harrison and animals fur and then we move up to a higher resolution we can see their fur at the higher resolution we don't need to relearn the distribution over images of fur at every pixel separately we we learn one level of detail that can be replicated across the whole image and we generate different Z values at every X


and y coordinate the that randomly decided you know fine details of the fur like which angle it should be pointed in and things like why do you think in practice ganz don't ask a love well when you go to larger images oh well you might be surprised by what comes in a few slides yeah I think I should probably move toward the conclusion now so recalling exercise 3


we're looking to design this f of X this cost function that's applied for every example generated by the generator in order to recover the maximum likelihood gradient we start by showing this property that we can write down the gradient of the generator in terms of an expectation where the expectation is we've taken with respect to generator samples and we multiplied f of X by a


likelihood gradient that's relatively straightforward to show the basic step is to turn the expectation into an integral use leibnitz's rule which means you have to make a few assumptions about the structure of the distribution involved and then finally we take advantage of our earlier assumption that the generator distribution is nonzero everywhere that allows us to say that


the derivatives of pg are equal to P G times the derivatives of log P G so that gives us this nice expression where we can get a gradients of the likelihood in terms of samples that came out of the generator but what would you really like is gradients of the likelihood in terms of samples that came from the data so the way that we're able to do that is important sampling we have this f of X


coefficient that we're able to multiply by each of the gradients and we can fix the problem that we're sampling from the generator when we want to sample from the discriminator by setting f of X to be P data over P generator and this means that we'll have kind of bad variance in our samples because we're sampling from the generator and then rewriting everything to make it look


like we sampled from the discriminator but in theory this is unbiased from there it takes a little bit of algebra to figure out exactly how we should take the discriminator and implement this ratio we recall that the optimal discriminator gives us this ratio of p data over p data plus p generator and doing a little bit more algebra we can rearrange that to say that we need to


set f of X to negative e to the logits this is maybe a lot to absorb right now but I think it's it's pretty intuitive once you've worked through it slowly on your own once and it gives you an idea of how you can take this ratio that the discriminator gives you and build lots of other things with it so to conclude the talk I'd like to show you some really exciting new


results that came out using generative adversarial networks and that kind of addressed the last question we had about whether a generative Ebersole networks scale to very large images a new model just came out last week I seem to have this curse that every time I have to give a talk about something an important new result comes out right as I have finished my slides so I desperately made


some new slides on the plane on the way here plug and play generative networks or generative models sorry make 256 by 256 high-resolution images of all thousand classes from imagenet and have very good sample diversity the basic idea is to combine adversarial training moment matching in a latent space do you know atom encoders and Monte Carlo sampling using the gradient and


the really cool thing is they also work for captioning or inverse captioning where you generate images by giving an input senate's that describes the image overall the basic technique is to follow a Markov chain that moves around in the direction of the gradient of the logarithm of P of x and y with with Y marginalized out you can use denoising auto-encoders to estimate the required


gradient but to make the denoising auto-encoder create really good images the auto encoder needs to be trained with several different losses one of those losses is the adversarial networks loss and that forces it to make images that look very realistic as well as images that are close to the original data in l2 space this confirms some of the tips that I gave earlier in the talk


for example on the tips and tricks section I said that you often get much better results if you include class labels we see here that plug-and-play generative models don't make nearly as recognized full of images if we generate samples without the class we also see that the adversarial loss is a really important component of this new system if you look at the reconstructions of the denoising


auto-encoder we begin on the left with the raw data in the middle we share the reconstructed image and on the right we show the reconstruction that you get if you train the model without the adversarial Network loss so adversarial learning has contributed a lot to the overall quality of this current state of the art model so in conclusion I guess I'd hope that everyone remembers that


generative adversarial networks are models that use supervised learning to approximate in intractable costs by estimating ratios and that they can simulate many different cost functions including the one that's used for maximum likelihood the most important research frontier in generative adversarial networks is figuring out how to find Nash equilibria in high


dimensional non convex continuous games and finally generate veteran-owned networks are important component of the current state of the art in image generation and are now able to make high resolution images with high diversity from many different classes and that concludes my talk and I believe that we're out of time for questions because we already took several of them in the


um thank you all for coming this is a massive room so today we will have six great invited talks panel discussion and a selection of posters and spotlight presentations I don't have much to say but welcome young good fellow from open AI he will be given the first talk of today an introduction to generative adversarial networks thank you good morning thank you everybody for coming

Introduction to GANs, NIPS 2016 | Ian Goodfellow, OpenAI


I guess I'll explain first a little bit what my goals for this this talk are I know there's a lot of different people here at the workshop and the main purpose of the talk is just to give everyone a little bit of context so that you know what adverse sail training is what generative adversarial networks are if you were at my tutorial on Monday you probably will have seen a lot of these


slides before but I'm also going to throw in a few new ideas just so that you feel like you've got something extra for your time but this talk is mostly for the people who have just arrived at the workshop and needed some context so this workshop is about adversarial training and the phrase adversarial training is a phrase whose usage is in flux and I don't claim exclusive


ownership of the phrase but to avoid confusion I thought I'd comment a little bit on how the phrase has been used before and how it's mostly used now so I first used the phrase adversarial training in a paper called explaining and harnessing adversarial examples and in that context I used it to refer to the process of training and neural network to correctly classify


adversarial examples by training the network on adversarial examples today other people have started using the phrase adversarial training for lots of different areas almost any situation where we train and model in a worst case scenario where the worst case inputs are provided either by another model or by an optimization algorithm so the phrase episode training now applies to lots of


ideas that are both new and old the way that we use the phrase adverse sail training now it could apply to things like and an agent playing a game against a copy of itself like Arthur Samuels checkers player back in the 1950s so it's important to recognize that when we use the phrase adversarial training today we're not only referring to things that were invented recently but the


usage has expanded to encompass a lot of older things that also had other names like robust optimization most of the day's workshop is about a specific kind of adverse ale training which is training of generative adversarial networks in the context of generative adversarial networks both both players in the game are neural networks and the goal is to learn to generate data that


resembles the data that was in the training set the reason that we call the training process for generative adverse cell network adversarial training is that the worst case input for one of these networks is generated by the other player and so one of the players is always trained to do as well as possible on the worst possible input it's worth mentioning that there are other works


going on in the space of adversarial training where the goal is still to train on adversarial examples inputs that were maybe created by an optimization algorithm to confuse the model and you will see some posters about that here there's also some work about that in the reliable ml workshop but I hope that clears up any confusion about the term adversarial training so


generative adversarial networks are mostly intended to solve the task of generative modeling the idea behind generative modeling is that we have a collection of training examples usually of large high dimensional examples such as images or audio waveforms most of the time we'll use images as the running scenario that we we show pictures of in slides because it's much easier easier to show a


picture of an image than to play an audio waveform but everything that we describe for images applies to more or less any other kind of data so there are two things you might ask for a generative model to do one is what we call density estimation we're given a large collection of examples we want to find the probability density function scribes as examples but another thing we


might do is try to learn a function or a program that can generate more samples from that same training distribution so I show that on the lower the lower row here where we have a collection of many different training examples in this case photos from the imagenet data set and we'd like to create a lot more of those photos and we create those photos in a random way where the model is actually


generating photos that have never been seen before but come from the same data distribution in this case the images on the right are actually just more examples from the image net data set generative models are not yet good enough to make this quality of images but that's the goal that we're striving toward the particular approach the generative adversarial Network to take


to generative modeling is to have two different agents playing a game against each other one of these agents is a generator network which tries to generate data and the other agent is a discriminator network that examines data and estimates whether it is real or fake the goal of the generator is to fool the discriminator and as both players get better and better at their job over time


eventually the generator is forced to create data that is as realistic as possible the data that comes to the same distribution as the training data the way that the training process works is that first we sample some image from the training data set like the face that we show on the Left we call this image X it's just the name of the input to the model and then the first player is this


discriminator network which you represent with a capital D the discriminator network is a differentiable function that has parameters that control the shape of the function in other words it's usually a neural network we then apply the function D to the image X and in this case the goal of D is to make D of X be very close to one signifying that X is a real example that came from the training


set in the other half of the training process we sample some random noise Z from a prior distribution over latent variables in our generative model you can think of Z as just a sort of randomness that allows the generator to output many different images instead of outputting only one realistic image after we've sampled the input noisy we apply the generator function just like


the discriminator the generator is a differentiable function controlled by some set of parameters and in other words it's usually a deep neural network after applying the function G to input noisy we obtain a value of x sampled in this case from the model like the face on the right this sample X will hopefully be reasonably similar to the data distribution but might have some


small problems with it that the discriminator could detect in this case we've shown a slightly grainy noisy image of a face suggesting that this brain and noise is a feature that the discriminator might use to detect the images faked we applied the discriminator function to the fake example that we pulled from the generator and in this case the discriminator tries to make its output D


of G of Z be near zero earlier when we use the discriminator and real data we wanted D of X to be near one and now the discriminator wants D of G of Z to be near zero to signify that the input is fake simultaneously the generator is competing against the discriminator trying to make D of G of Z approach one we can think of the generator the discriminator is being a little bit like


counterfeiters and police the police would like to allow people with real money to safely spend their money without being punished but would like to also catch counterfeit money and remove it from circulation and punish the counterfeiters simultaneously the counterfeiters would like to fool the police and successfully use their money but if the counterfeiters are not very


good at making fake money they'll get caught so over time the police learned to be better and better at catching counterfeit money and the counterfeiters learn to be better and better at producing it so in the end we can actually use game theory to analyze this situation we find that if both the police and the counterfeiters or in other words if both the discriminator and the generator


have unlimited capabilities the Nash equilibrium of this game corresponds to the generator producing perfect samples that come from the same distribution as a trading data in other words the counterfeit are producing counterfeit money that is indistinguishable from real money and at that point the discriminator or in other words the police can not actually distinguish


between the two sources of data and simply says that every input has probability one-half of being real and probability one-half of being fake we can formally describe the learning process using what's called a minimax game so we have a cost function for the discriminator and we call J superscript D which is just the normal cross entropy cost associated with the binary


classification problem of telling real data from fake data we have one mini batch of real data drawn from the data set and what a mini batch of fake data drawn from the generator and then if we use this minimax formulation of the game then the cost for the generator is just the negation of the cost for the discriminator the equilibrium of this game is a saddle point of a superscript


D and finding this saddle point resembles the process of minimizing the Jensen's Shannon divergence between the data and the model we can use that to actually prove that we'll recover the correct data distribution if we go to the equilibrium of the game we can analyze what the discriminator does and they play this game and we see exactly what it is that allows generative


adversarial networks to be effective the basic idea is that if you take the derivatives of the minimax games value function with respect to the outputs of the discriminator we can actually solve for the optimal function that the discriminator should learn this function turns out to be the ratio between P data of X and P data of X plus P model of X you can do a little bit of algebra on


that to rearrange it and you get P data of x over P model of X so we're learning a ratio between the density that the real data is drawn from and the density of the model currently represents estimating that ratio allows us to compute a lot of different divergences like the Jenson Shannon divergence and the KL divergence between the data and model that are used for training with


maximum likelihood so the key insight of generative Ebersole networks is to use supervised learning to estimate a ratio that we need to be able to do unsupervised learning there are also a variety of other papers by Shakir Muhammad and his collaborators and Sebastian knows and his collaborators that talk a lot about the different divergences that you can learn with these kinds of techniques and


how this estimation procedure compares to other techniques have also been developed in the statistical estimation literature previously but this is the basic idea right here is that we're able to learn this ratio so far I've described everything in terms of the minimax game I personally recommend that you don't use exactly that formulation you use a slightly different formulation


where the generator has its own separate cost and the idea is that rather than minimizing the discriminators pay off the generator should maximize the probability that the discriminator makes a mistake the nice thing about this formulation is that the generator is much less likely to suffer from the vanish and gradients problem but this is more of a practical tip and trick rather


than a strong theoretical recommendation and some of the other speakers you'll see today might actually give other advice so it's kind of an open question about exactly which tips and tricks work the best one of the really cool things about generative adversarial Nets is that you can do arithmetic on the z vectors that drive the output of the model we can think of Z as a set of


latent variables that describe what is going to appear in the image and so Alec radford the co-organizer of this workshop and his collaborators showed that you can actually take Z vectors corresponding to pictures of a man with glasses the Z vector for a picture of a man and the Z vector for a picture of a woman and if you subtract the vector for men from the vector for men with glasses


and you add the vector for women you'll actually get a vector that describes woman with glasses and when you decode small jitters of that vector you get many different pictures of a woman wearing glasses a lot of you may have seen a similar result before with language models where the word embedding for Queen could be used to do arithmetic where if you subtract off the word


embeddings for female and add the word embedding for male you get a vector that is very close to the word embedding for King in this case Alec and his collaborators have a slightly more exciting result because they not only show that the arithmetic works in vector space but also that the vector can be decoded to a high dimensional realistic image with many different pixels all set


correctly in the case of language modeling the final result was a vector that was very near the word for King but there was no need to decode that vector into some kind of extremely complicated observation set that corresponds to a king probably the biggest issue with generative adversarial networks and to some extent with other forms of adversarial training is that the


training process does not always converge most of deep learning consists of minimizing a single cost function but the basic idea of adversarial training is that we have two different players who are adversaries and each of them is minimizing their own cost function when we minimize a single cost function that's called optimization and it's unusual for us to have a major problem


with non convergence we might get unlucky and converge to a location that we don't like such as a saddle point with a high cost function value but we'll usually at least converge to some general region when we play a game with two players and each of them is simultaneously trying to minimize their own cost we might never actually approach the equilibrium of the game in


particular one of the worst forms of non convergence that we see with generative adversarial networks is what we call mode collapse or if you're in on a little joke in our first paper we also caught the Helvetica scenario sometimes the basic idea of the hind mode collapse is that when we use the minimax formulation of the game we'd really like to see is minimization over


G in the outer loop and maximization over D in the inner loop if we do this min max problem applied to the value function V we are guaranteed to actually recover the training distribution but if we swap the order of the mechs and the men we get a different result in fact if we minimize every G in the inner loop the generator has no incentive to do anything other than map all inputs Z to


the same output X and that output X is the point that is currently considered most likely to be real rather than fake by the current value of the generator so we really want to do min max and not max min which one are we actually doing the way that we train models we do simultaneous gradient descent on both players costs and that looks very symmetric it doesn't naturally


prioritize one direction of the min Max or max min in practice we find that we often see results that look an awful lot like Max min unfortunately with G in the inner loop so using some very nice visualizations from Luke Metz and his collaborator collaborators we see here that if we have a target distribution we'd like to learn with several different modes in two dimensions the


training procedure shown in the bottom row of images actually visits one mode after another instead of learning to visit all of the different modes so what's going on is that the generator will identify some mode that the discriminator believes is highly likely and place all of its maps there and then the discriminator learns not to be fooled by the generator going to that


one particular location and instead of learning that the generator ought to go to multiple locations the generator moves on to a different location until the discriminator learns to reject that one - one way that we can try to mitigate the mode collapse problem is with the use of what we call mini-batch features this is introduced in the paper that we presented on Monday night from


open AI where the basic idea is to add extra features to the discriminator so the discriminator can look at an entire mini batch of data and if all the different samples in the mini batch are very similar whether the discriminator can realize that mode collapse is happening and reject those samples is being fake on the CFR 10 dataset this approach allowed us to learn samples


that show all the different object classes in CFR 10 for the first time on the Left I show you what the training data looks like for CFR 10 you can see that it's not that beautiful to start with because there are only 32 by 32 pixel images so the resolution is very low on the right we see the samples that come from the model and you see that you can actually recognize horses ships


airplanes and so on and cars that we actually have the real object classes recognizably occurring within this data set an image net there's a thousand classes so it's much more difficult to resist the mode collapse problem an image that our model mostly produces samples that have kind of the texture of photographs but don't necessarily have rich class structure we do occasionally


get rich class structure if I show you some very cherry-picked examples we're able to make lots of different pictures of things like dogs spiders koalas bears and birds and so on we still see a lot of problems with the model though in particular we often see problems with counting we think that this might be something to do with the architecture of our convolutional


network that it's able to test whether a feature is absent or present but it doesn't necessarily test how many times that feature occurs so we see things like this giraffe head with four eyes this dog with something like six legs or this kind of three-headed monkey thing or you know stacks of puppies rather than a single puppy or a cat with one and a half faces we also often see


problems with perspective where the model generates images that are extremely flat in particular the image on the lower left looks to me like somebody skinned a dog you know like a bearskin rug and then took a picture with the camera looking straight down at it on the ground whether the picture in the lower middle looks to me literally like a cubist painting we're in the cubism


movement artists intentionally removed all the perspective from an image and rearranged the object to show us different parts from different angles but representing the entire thing is flat in many cases we see images that are really quite nice that have some problem with the global structure a lot of the time this just consists of images of animals where we don't actually get


to see their legs that they have a head and torso attached but they don't actually complete the legs anywhere and in my particular favorite generator samples so far on the lower left we have an image that we've actually named fall out cow where we have an animal that is both quadrupedal and bipedal it actually has legs and it has the right number of them but it has two different bodies so


what are some things that you can do with genitive adversarial networks there are just so many different things that it's a little bit hard to show all of them I showed a lot more in my tutorial but I can show you just a few really quick one thing that came out recently is image to image translation this is from the research group at Berkeley and the basic idea here is to take a


conditional generative adversarial Network and map from one domain to another it can do things like take images that say for every pixel where the different what kind of class should appear at each pixel and turn that into a photorealistic scene with the desired objects in the desired positions it can also take an aerial photo and turn it into a map where it can take a sketch of


an object and turn it into a photo of an object more recently there was a very exciting result that finally developed the ability to generate realistic samples on the image net data set from all 1,000 classes and with really good diversity this result is called plug-and-play generative models and it combines many different approaches to generative modeling including generative


adversarial Nets moment matching denoising auto-encoders and lunch main sampling the results are really excellent and we see lots of different very recognizable high quality images with all the right numbers of legs and everything so generative modeling has really come very far in just the last month actually and generative ever eternal nets are part of that progress I have a few comments


about exactly what it is that allows generally retro style networks to work well on kind of an intuitive level one of the main things that's really different about generative a dresser Nets compared to other approaches to machine learning is that they give a very nice way of telling the model that there are multiple correct answers so a lot of the time with supervised learning


we use something like mean squared error to tell the model loads output should have been so on the Left I show a little bit about what's wrong with the mean squared error training process suppose we have the blue dot at the bottom of the slide representing some input and we'd like to learn to map this input to some desired output suppose all the different green dots represent different


possible outputs that are all valid well in the training set suppose the the label that we had for this particular blue dot was the green dot on the far left but suppose that the model produced a green dot on the far right mean squared error will induce the red error arrow saying that instead of producing the dot on the right the model should have produced the dot on the Left which


is the one that appears in the training set that means that over time the blue dot will actually get mapped to something more like the mean of all these different green dots and that causes us to learn things like blurry images when we try to learn to predict different images associated with some input generative Everest or networks don't actually directly use a pair of


inputs and outputs to tell the model what it should do instead the discriminator learns how inputs and outputs can be paired and then the discriminator tells the model whether it did a good job or not so the discriminator what ideally learn that all of the different green dots are possible options and then when the generator produces the green dot on the right the discriminator says that that


was a good thing to do there are many different good things that the model can do and the discriminator will hopefully endorse all of them so we now have this mechanism for saying that many different outputs are possible instead of always steering the model toward one predefined answer we can see this especially in the context of next video frame prediction so this paper by Bill Lata and his


collaborators shows what happens when we use a few different kinds of models to predict the next frame in a video on the left I show the ground truth where we have this 3d rendered image of a person's head and you can see that the image is very sharp and and has a clearly visible ear using a model that was changed with mean squared error in the image in the middle we see that the


ear has vanished because the exact location of the ear is not especially predictable and when the model averages over many different possible places the year could go it vanishes similarly the eye has become blurry in the image on the right we see what happens when an adversarial loss is included in the training process in this case the model is now encouraged to produce samples


that actually look realistic and it knows that there are multiple different answers that are all possible so it's been able to choose one of the many different sharp images that could happen at the next time step it's also worth thinking about whatever sterols training looks like for people and really the way that the idea of adverse sail training emerged was that


economists and other researchers in those kinds of fields we're already working on thinking about the way that multiple different agents acting in a market have their behavior influenced by the process of them optimizing their own payoffs while all the other players optimize their paths so some things that I think could be interesting to look at from machine learning code of view are


weather cycles in markets can be explained by the failure of optimization algorithms to converge if we have trouble fitting generative Evaristo nuts and we have complete information just think about how hard it is to choose prices for goods when there are many more actors and when you don't have complete information about the market I'm sure this has been studied to some


extent but I think that bringing the economy the machine learning people together could find some more interesting ideas we didn't already know about we've also seen lots of cases of things like auctions that are designed to make sure that people pay the right price and that's more or less what I was thinking of when I designed generative adversarial Nets one last remit is that if we think about


the way that people learn researchers like Erikson have shown that the way to become really good at any particular task is to practice it a lot but also to do deliberate practice you're not just putting in a lot of hours you are specifically choosing subtasks within the skill that you're trying to get good at that are especially difficult for you and getting feedback from an expert who


coaches you you can think of adversarial training as capturing both of these aspects of developing a skill rather than just training and lots and lots of training examples your training on the worst case inputs that are really hard for the model and in the case of generative adverts or networks you have an expert the discriminator coaching the generator on what it should have done


instead so a lot of insights from human psychology and human learning are actually telling us how we can make machine learning more effective so in conclusion adversarial training is a way of training a variety of models in different ways that all involve working on a worst case input generative adversarial Nets are one of the most popular members of this framework and


they're based on using the estimate of a ratio of densities to do unsupervised learning and part of why they work so well is that they allow the model to have multiple correct answers and they draw on a lot of the ideas that help humans to learn really well I'm almost out of time but I think we might be able to have a few questions if that's okay with the organizers


you have several microphones on the sides okay yes I have caution first here over here so I secured armed for sequential things like video do you know any of any worked of using my recurrent network that actually can generate the risk so the sequence of data to generate our video or what kind of charge that would be yeah there is a paper about generating videos with generative


adversarial networks I forget the exact title off the top of my head okay there's also there's a paper here at this workshop today I called unrolled generative every cell networks and I know that one of their experiments involves using a recurrent network to generate Amnesty one pixel at a time so you could check out their spotlight and poster and how about generating language


sequence of words in discrete domain the thing that's difficult about that is is the discrete outputs which means that the generator is not differentiable so that's an open research area it might be solvable using things like the reinforce algorithm to do policy gradient on the parameters or using things like gun Bell softmax or the concrete distribution or it might be possible to generate word


embeddings from the generator and then decode them to discrete values instead of generating military values directly and how about a speech which is continuous we know that google's are wavenet use different mechanism of generating speech and using this gain framework can be used equally well yeah so before I left Google I suggested that they try generating continuous waveforms


it can and I don't actually know if they tried and again didn't work or if they just went straight to using Pixlr and n-type methods to generate the continuous waveform but I do think that the continuous waveform is the way to go with games if that did work the advantage of games would have over wavenet is that the gans can generate the sample much faster that


wavenet needs to pass through a neural net for every single sample of the audio so it's generating you know really thousands of samples using thousands of passes to the network it takes about two minutes to generate one sample one second about you okay though again could generate a long waveform in one shot it's interesting to see that wavenet didn't really have temporary effect I


use show over there using mean square error kind of criterion for the image so do you have an explanation why once brother the arm is not blur so the blurring effect is with mean square error in real valued spaces and they were using discrete spaces where they have a soft max distribution so their loss function is not actually mean squared error it's it's a categorical


hello there today we're looking at bootstrap your own latent a new approach to self supervised learning by researchers of deep mind and imperial college so not almost no day goes by where we don't hear some sort of new self supervised algorithm right here this paper on a high level tries to get rid of the necessary negative samples when doing the contrastive loss for self

BYOL: Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (Paper Explained)


supervised learning and they basically combined momentum contrast and seem clear and then remove the negative samples and that seems to work pretty well even though it's magic so yeah if you if you want to see how it's done stick around share the video out if you want other people to see how it's done and leave a comment this this one I really don't get what's going on so if you have ideas put


them there I'll I'll read them through it'll be fun alright so they say we introduced bootstrap your own latent or be all a new approach to self supervised image representation learning ok so image representation learning is the simple task of taking an image and then feeding it through a function which is usually like a neural network let's let's just say this is a neural network


and in fact all of these the community has sort of standardized this to be most of the time it's something like a ResNet 50 ok so what you want to do is you want to train a neural network like a ResNet 50 to give you a good representation of the image so this would be like H and H is a vector and H is a representation of this image and the representation should be


such that you can then take this representation and solve many tasks with it which either can be like linear you can put a linear classifier on top of the H or you can fine-tune the entire architecture to solve some other task the idea is if you have a large data set you may use this dataset to train these good representations of these images and then you can transfer learn transfer


this to a task where you might be not have as much data and because you don't have as much data it's not enough to completely train an architecture like this but it is enough to take an architecture that's been trained with the large data set and just adapt it to your small data set and that usually it tends to work pretty well this is called transfer learning this step here is


called fine-tuning sometimes and it's sort of the approach that comes from natural language processing from these big transformers like Bert where you first train on a really big data set that might not be the data set that you want in the end but it's really big so you can sort of learn a lot of things from that data set and then the only thing left to do is to fine tune it to


basically adapt it to the nuances of your data set but it will have learned most things already and that's called representation learning so the goal is to learn a good representation now this self supervise here is also important because representation learning can be as easy as if this here is image net the image net data set contains like a million images all with labels you can


simply train your ResNet 50 to predict the class this is the this is called supervised pre training or supervised representation learning and that works pretty well but you need a labelled data set in self supervised learning you do not need labels what you do is you do self supervision and self supervision it has many there are there many ways to do self supervision but what we'll see in


this particular paper is that you will take an image and you'll make a different variance of that same image so you'll take the image and you'll make many many variants of it well let's just say two so you have some procedure to sort of change the picture a little bit but it's essentially still the same and you do that through data augmentation so this could be a random crop or you color


jitter or you rotate it or something like this and then you exploit the fact that you know that these two things they should be still sort of the same image so once you send them through your through your encoder the representations of the two images they should be fairly close now let's actually read on right here Buhl relies on two neural networks referred to as online and target


networks that interact and learn from each other from an augmented view of an image we trained the online network to predict the target representation of the same image under a different dog mented view okay that's sort of what we saw so we have we have the same image under a different dog mented view so what does it mean what what I just said you make two versions of the same image one that


are slightly different and then their representation should be close now until this point we have always thought that this would degenerate because what if you think of this neural network that does this encoding to the hidden space this ResNet 50 right here if it wants to if you simply want to make the two representations close what's the best thing it can do you can simply map all


the hidden it can simply have the constant function H equals 0 or something like this just a constant function because then this loss here is always going to be 0 like perfect okay so no matter what image comes in if you always map it to the same thing you will always be close in representation space and therefore you always win that doesn't learn a really good


representation right so what people have done is they have included so-called negative samples where you'll say I'll take a different image from you know from this dataset but it's a different image than this image and I also do some maybe some data augmentation with that image and then I send this through the same encoder to also give me an H so this is the H let's call that H original this is


H plus because it's the same image but slightly differently augmented and this is H minus which is a different image and now the task is let's make those two very similar to each other but let's distance them from this other one so we want we want this to be as far away as possible and these two to be close to each other now the network can't simply map everything to a constant function


anymore right it needs to actually do something to make these be close together and this be far apart and the combination of this together with the augmentation procedure that goes into augmenting the images has been sort of a good combo to learn good representations and a lot of papers have alluded to the fact that this is so the negative samples are to do not have these


degeneracy right so to not have the simple solutions but the fact that the representation then is actually good like is good for image class image tasks down the line probably comes from the fact of these augmentations right here and there's a lot of evidence of from the fact that depending on which augmentations we choose these representations are going to be better


or worse for example random cropping of an image so the random sub like taking a random crop from the image tends to be very very beneficial because so here this is the same image twice right let's say we take a random crop here and one up here it's sort of maybe there's an overlap here in the middle right so it sort of needs to understand that these random crops sir sort of needs to


communicate between these two places in these random crops so the representation has to somehow make sure that the object that is overlapping here is somehow represented but it can't represent it just as a pixel value because it doesn't know where the crops come from so there's a lot of evidence that these representations are the thing that's responsible for making the


representations so good okay now this paper simply says do we really need these negative samples right here let's just get rid of them and with a couple of tricks this seems to work in here this is this is what seems like magic to me because as we go forward think of it nothing nothing keeps this model right here from doing the degenerate solution h equals constant


nothing right now for some reason it doesn't do that and I have the feeling that this is a super delicate balance that you have to do because when you train when you start out it's probably not the constant function right it's probably some some distribution and then simply by the fact that you train it and kind of keep it in the so this is certainly an optimal solution but you


might be like in some sort of local minimum once you start training and you simply don't get out of it during training and that's why the network has an easier time step by step as it updates itself in very small incremental steps it has an easier time actually going for the good representation then it has to see this solution right here and converge to that but yeah it seems


delicate so what are they doing they are taking that idea of taking an input image right here and so by the way why is it important that there are no negative samples because now the question is always or where do you get these negative samples from right should they be uniformly sampled should we keep a buffer should we order them there is this task of hard negative


mining where you say Oh any old negative won't do it's actually better if we take negatives that are you know just hard enough there is a curricular curriculum learning problems and so on so it would be best to actually just get rid of these negative things so that's why we want to get rid of them so that's the approach byl bootstrap your own latent there is the input image we take one


image at a time and you apply two different random augmentations to it right so you create two slightly different variants of that image through augmentation and again this can be something like a random crop it can be a horizontal flip randomly you color jitter you Solarize you blur and so on there are all these variants of data augmentation and the fact that down the


line that the representation of these two things has to be close to each other I think these random these augmentations here are responsible what the to make the to make the these augmentations are responsible to make the representations powerful okay the fact that later down the line the network has to sort of learn to ignore these it has to learn that oh you know it doesn't matter where


in the image this object is because it's been random cropped for different you know at different locations it doesn't matter where in the image this object is I simply need to have my hidden representation have this particular object in the image and that's what makes it powerful okay I've said that enough now then you have these two slightly different versions and then you map it through


your encoder okay let's go the top path first you see the bottom path has the same encoder but the parameters are different and this is going to be one of the crucial elements right here so this here are your actual parameters that you learn and this here are what are called the target parameters now after each and you can see this for all of these components


right here so what happens is that the target parameters are basically a copy of these what's what are called the online parameters okay so after each step you copy over from the online parameters you copy over to the target parameters you never learn the target parameters you simply copy them after each step now you don't copy them outright what you do is you do an


exponential moving average so the target parameters are always going to be sort of a lagging average of your online parameters and that idea comes from the momentum contrast principle where the reasoning sort of behind it is that you need a kind of a stable you kind of need a stable representation as a target but I think it hasn't been fully explored or explained why exactly that is so helpful


but we just know that if if we have the target to be not the same as the the online parameters but actually a kind of a stable version of the past of the online parameters then that tends to work well again it's kind of the same principle as with the augmentations with the augmentations we have two different versions of the same image and now with this procedure here we sort of have two


different versions of the same neural network but they're slightly different right and this idea you know has been around for much longer like the the first Q deep Q networks and so on they had the same principles where they had the the network that they actually learned and then the target network that is copied over every such-and-such episodes and so on so this this seems to


work seems to be a fundamental principle that seems to work all right so we take our two slightly different augmented versions of the same image and we run them through our two slightly different encoders to obtain two representations now this thing right here that's going to be our represent so after this procedure we discard the entire thing right here except that so


this here is your whatever your ResNet 50 okay after that follows a projection and the projection is is here to reduce the dimensionality and honestly I'm actually not sure why it is here because you can do it without like technically the algorithm doesn't require this projection so you can imagine the algorithm without the projection but just really quickly the projection


simply brings down the representation which is like 2048 dimensional that comes out of the resin at 50 it has it is a two layer neural network that first pumps this up to like four thousand and ninety two and then compresses it down to 256 dimensions okay so that's the projection Network again there is a part that's learned and then the target projector is simply the


exponential moving average of the online projector but again this is why exactly this is here probably simply because it works right but probably because there is no there is no distinction because you don't have different losses you simply back propagate through everything and then train everything so there is no logical distinction between the projection and the representation other


than you have a different dimensionality but maybe that's the point here that you make a different dimensionality even though you could you could do the rest in this 2048 space yeah so for now just this doesn't exist let's just say this doesn't exist and we just work with this representation here let's call this Z z prime okay so what happens is we take the representation and now we have one


neural network the predictor right here that takes the representation of one of the image versions and it simply tries to predict the representation of the other image version so what you want is that Q of Z equals Z prime okay and if we expand that is that Q of F of Z is equal to F target of Z prime and if we expand that even further you can see that Q I'll just write Q and F for now Q


of F of a which is an augmentation an augmentation of Z should be one bracket to bracket three bracket should be F of a of Z sorry not see that's the image X all right so this makes a lot of sense you're simply with Q since these are all different here so f is the target instead of these online parameters a is also different it's a different augmentation that you do but the X is


the same okay so the Q simply tries to somehow negate this augmentation and this difference between the target and the online parameters but you don't tell the queue which augmentation was used and you don't tell the Q what are the exact parameters of that network so what the Q has to do is it has to somehow it's like it's like a it has to take its best guess right so basically the Q is


trained to output the expected value of the representation right the expected of the representation f of a of X under all of the different possible image augmentations and that's why it learns to ignore these augmentations so your entire goal with these methods is you learn to ignore these augmentations so you want to learn some method that is independent of the augmentations so by


crafting the augmentations in a smart way we can make these representations contain a lot of semantic information because what we want to do with the augmentation is basically we want to destroy all the non segmenting information sorry non semantic information and random cropping is one of those methods horizontal flipping is one of those methods because we say well


whether an image goes left to right or right left most of the time the semantics are the same the pixels are different but the semantics are the same so by putting an augmentation in there we learn to ignore that augmentation because our representation now needs to be predictable right cuny we learn q to predict the the representation under the expectation of our augmentations and


that means it can't be dependent on one particular augmentation okay learns to ignore it so that's basically what's happening here again there is nothing keeping this from simply collapsing it to a trivial solution and it's probably a combination of the of the initialization and the learning procedure itself that it you know goes on in little little steps one


by one that keeps it in the realm of rather having to like it's easier to learn a good representation than it is to collapse to that to that solution okay so again components is image then you ought meant differently then you run it through different encoders but the encoders are similar in the fact that one is the exponential moving average of the other and then you try to predict


one from the other and that ultimately makes the representation be independent of the augmentation and that means that the representation can only include things that are not destroyed by the augmentations and if you construct the augmentations smartly that means you only retain the semantic information that's it so the loss function is pretty simple as you can see right here what


you want is and this bar is a normalization what you want is the l2 norm between the this representation be close to the cue of that representation so the cue simply tries to predict the other representation and you do that for both ways so you once stick the image in here and try to predict the other one and you do it vice versa so you get too lost components each time it's a


symmetric loss okay and that's it that's the method and they beat all the other self supervised methods and they get pretty close to the supervised supervised representation learning method as you can see right here as the number of parameters goes up in their model so one of them is resonate 50 but I'm gonna guess this one right here but you can also get to higher architectures


and then it appears to work even better and come even closer to this supervised baseline this could be because you know if you have more parameters technically in a supervised method you would also need more labeled images maybe and therefore it doesn't scale as well I don't I don't know there is a lot of unclarity in this research like all they show is that their numbers are good


which is cool right and it's cool that you don't need your you don't need the negative samples anymore and it actually doesn't collapse when you do that kind of stuff but there's a lot of I don't know there's a lot of things here for example we use a batch size of 4096 split over 512 TP uv3 course with this setup training takes approximately 8 hours for resin at 50 so they train eight hours on


512 TP use just imagine that so that's sort of crazy amount of computation again going into these models and then the second thing here is that you can see that there are some things missing right here and there are these all these annotations which probably means that they take these numbers from those papers now they allude to to the fact that they try to follow their protocol


as closely as possible but I mean that's never that's never given or almost never unless they release like the exact code and even then there are still going to be differences in even like you'd have to replicate the exact thing on the exact same number of TPU cores and whatnot so I I highly like these numbers seem to be I'm not sure especially if you then go and look and at some point


they actually do reproduce the same clear baseline so you can see right here that they have a own implementation of sim clear and they actually compare this to the numbers that they find in the same clear paper and you can see for example here there's like four percentage points that the the their implementation of seeing clear gains above this implementation and if you


look at this supervised baseline that's also from that paper and there is a graph further down where they also implement their own version of the their own version of the supervised baseline I forget here so you can see that between the supervised in that paper and the supervised of them sometimes there's like a giant gap right here for the same model it seems so all of these numbers um I'm not


sure you should put too much weight on the fact that this is now outperforming the other methods I would not put like unless this is like SuperDuper replicated very often I would not put a lot of weight on the fact that it is better what I would put a lot of weight on is the fact that it works at all and and achieves you know good performance and there is more they make they have


like experiments right here that show that their method they be a B y ou L is much more resistant to like changes in hyper parameters so here you can see that it falls off much later when you reduce the batch size which makes sense right because seem clear is one of these methods that uses negative samples and for negative samples it uses the other samples in the mini batch now if you


have less samples in the mini batch that means you have a less representative distribution of your entire data set as negative samples and therefore if you increase as decrease the mini batch then this drops off and also they show that for example their method is much more robust to the removal of a couple of these image augmentations so all of this I find actually pretty cool but the the


actual numbers here first I'm not super duper interested that they get like two or one points more in something but they do perform like a lot of experiments and that it shows that you can apply the method to different things it's not only like in one setting so that's pretty cool it works at least at the you can say it works at least as well as other methods and it is a lot easier because


you don't have this negative sample things now the last quarrel I have with the paper and where is it where is it somewhere they say that we release the code that they release the pseudocode they don't release the code they release the pseudocode in the appendix so I mean there are reasons why you sometimes want to release pseudocode and that's if Liane algorithm is so high level and so


simple in its high level T and so modular to be fleshed out that you can't like it makes more sense but here it's like pseudocode in Jax and and come on is it really that competitively advantageous to retain your code this is it's just not reproducible with this you know that they have like 50 billion hacks in their code and yeah so deep mind has this history of just not


releasing like publishing behind paywalls and and just giving pseudocode that has lots of mistakes in them like the new zero pseudocode you can't even like run it in its basic form if you fill in the things it's it's a it's a bit annoying in any way the method itself seems promising for representation learning as I said especially because it's pretty simple it


still heavily relies on these augmentation methods so and that's what they say right here nevertheless py will remind remains dependent on existing sets of augmentations that are specific to vision applications to generalize beyond to other modalities it is necessary to obtain similarly suitable augmentations for each of them designing such augmentations may require


significant effort and expertise there for automating the search for these augmentations would be an important next step to generalize to other modalities and I'm not sure if you can do this automating the search for these augmentations I guess you can do it if you have like a supervised data set and then you can search and then you can use those augmentations for the unsupervised


but it seems a bit bootstrap eeeh no pun intended right here I think the the power of the of these representations again comes from the fact that we have these augmentations carefully constructed so oh yes the last thing broader impact statement just read this like try to estimate the perplexity of this broader impact statement let's go the presented research should be


categorized as research in the field of unsupervised learning this work may inspire new algorithms theoretical and experimental investigation the algorithm presented here can be used for many different vision applications and a particular use may have both positive or negative impacts which is known as the dual use problem besides as vision datasets could be biased the


representation learned by Buell could be susceptible to replicate these biases like come up so people who advocated for making everyone do this is this what you wanted is this like is this a satisfactory result for you and if you have this as a reviewer is this okay or not I mean let's just cross out some words here blank that's black like field let's just put field or machine learning


why not machine learning machine learning this work inspire new algorithms yes the algorithm presented here can be used for many different machine learning applications and the particular use may have both men - yes besides as datasets could be biased representation learned by this paper could be susceptible to replicate these biases well there is a copy pasting that


you can apparently put into any and all papers that you write from that one and hey deepmind's doing it so you know there you go okay may be a bit cynical but I'm I like I told you this would happen I told you and you know okay so that was it for my comments right here they do have like a giant ton of experiments and I appreciate that right they really try to show that it works in


[Music] [Music] from the earliest writings of men we know that the human race has been comprised of the haves and the have-nots when I was a kid back during the Great Depression I was obsessed with a desire to know what invisible something separated the haves from the have-nots being a have nots I wanted to know why so few manage to be well-off financially in a country where success is available

Change Your Life in 19 Minutes with Earl Nightingale


to everyone for example in checking the Statistical Abstract of the United States published by the Bureau of the census I discovered just lately that only 10% of the men in this country 65 years of age and older have incomes of $6,000 or more a year more than 80 percent of all men 65 or older have incomes under four thousand a year only seven point six percent of incomes


between seven and ten thousand a year and only three point seven percent have incomes of ten thousand a year or more a man starts his working career in his 20s often earlier he's fortunate in that he lives in the free world he has better than 40 years to make the grade financially in the richest country on earth yet according with his statistics only about ten out of a hundred will be


financially secure by the time 65 rolls around and only about four men out of a hundred will be financially comfortable now why let me tell you how to find out for yourself conduct your own survey start down the street in your neighborhood on any Saturday or Sunday and ask the man of every house two questions the first question is what are you doing at the present time to


increase your income now that is how much do you want to earn and when you've evaluated the blanks there you get in response to that question asked question number 2 which goes how much money are you planning to be worth at age 65 and when the silence becomes too unnerving thank him and move on to the next house ask 50 men a hundred a thousand until you're completely convinced that the reason men


don't make more money during their working lives and the reason they're not financially independent by the time they're sixty-five is simply that they seldom if ever do any constructive thinking on either subject it's that simple unfortunately the reason is so easy to earn far more money than the average man earns in this country is that so few so very few are going about it the right


way this is a race without enough contestants to mother about the few who are really in the race can all be winners some will finish ahead of the others but even the man who finishes last in this race will be financially secure most people more than 90% aren't even in the race to prove it ask yourself the two survey questions up until the time you started listening


to this message what were your plans for increasing your income how much do you want to earn and how much money had you decided to be worth by the time you're 65 you see people who earn large incomes aren't lucky and they're not crooks as those without money are so fond of pretending nor are they end up with more brains or talent necessarily than their friends and neighbors nor are they privy


to our cult secrets and only a very few were lucky enough to have had rich fathers or grandfather's most of the people earning the big incomes today started the same way you and I did and most other people the only difference between the men who earned big incomes and those who earn small incomes is that those earning big incomes decided to earn more they're the people who made it


their business to earn more you see a woman who does not think about baking an apple pie for dinner tonight will never think of looking up the recipe for apple pie without the decision for pie there's no motivation for checking out the recipe a man who does not think about driving his car to st. Louis Missouri or Nacogdoches Texas will never get roadmaps which show how to get the same go as in


Nacogdoches and a man who never decides to earn more money will never think of learning how of looking up the rules burning more money you see people do what they make up their minds to do so get rid of the ancient superstition once and for all that people who are in big money are special people are lucky or get the breaks or had money to begin with or knew someone or are smarter or


anything else these are alibis they can all be disproved a thousand times the reason there are so many of these alibis around is that men who failed to make the grade financially are seldom honest enough to just admit that they really didn't try and keep trying so in order to justify their failure in order to remain seated they dream up and passed along these old


alibis we're all self-made but only the successful will admit it I want that occasion to visit Charleston South Carolina had never been there before so I hired a taxi to drive me around the historic old town I particularly wanted to see the battery with that famous shot was fired on Fort Sumter along this beautiful drive some of Charleston's oldest and finest homes


without over the bay I commented to my cab driver on what lovely homes they were and he said yes some of those homes have 40 rooms and then he thought a moment he said and every one of them is owned by a crook this is how the have-nots justify themselves and their lot in life I didn't say anything because I didn't feel I was entitled to advise him or try to straighten out his


thinking this is a free country where as long as he doesn't hurt others everyone has the inalienable right to be just as wrong as he wants to be as Thomas R Lounsbury the American scholar and educator put it we must view with profound respect the infinite capacity of the human mind to resist the inroads of useful knowledge my taxi driver and men and women like him all over the


world have been kidding themselves and holding themselves down and refusing the bounty and abundance of the world for centuries knowledge is available to everyone we can either listen to those qualified to teach us or we can go along with those ancient stumbling blocks we get from people who don't know any more than we do the truth incidentally about those homes along that beautiful Drive


is that they were built by the men and women who made the largest contribution to the city of Charleston in just a moment I'm going to give you the formula for getting rich but before I do I want to remind you of something before a jet pilot begins his takeoff from an airport he carefully goes over a checklist item by item he does this not only because it's required by law but because he


cannot afford to trust so important a job to his memory alone he has another checklist that he goes over just as carefully before he begins his lap down at his destination he does this without fail every time he takes off and every time he lands well I think living successfully is as important as flying an airplane and because of this I think each of us needs a checklist too


and that's why there's one included with this cassette we need a checklist to go over item by item before we take off in the morning and before we drop off to sleep every night so I want to recommend that you have fixed the checklist be a bathroom mirror stare at it as you brush your teeth in the morning and stare at it again as you prepare for bed at night go


over each item and as you do think of what each item represents and here's number one it's the formula for getting rich it also explains why you're in your present position whether you're earning six thousand a year or sixteen thousand or sixty thousand or six hundred thousand it applies to every adult whether he's employed or unemployed it applies to the richest man and to the


forest and every person in between and here it is our rewards in life will always be an exact proportion to our contribution our service now that's what the formula means as the first item on your checklist memorize it our rewards in life will always be in exact proportion to our contribution our service listen to it think about it until you know it emotionally as well as


intellectually it might give you some slight feeling of superiority to realize that there's probably not another man within a mile of where you live who knows it you can add it as a question on your survey if you want proof of that if you want it in another form here it is as it applies to a man's job it's the same thing really the same thing applies but you can express it differently the


money you're paid by the company you work for will always be in direct ratio to then what you do your ability to do it and the degree of difficulty involved in replacing you maybe you want to write the formula down in both of its forms and think about it until it's as much a part of you as your name the reason it isn't spelled out on your checklist is because you might not want everyone to


know what you're up to the checklist is valuable only to a person who knows what the words really indicate all right you've got the formula as you think about it its meaning will become clearer to you with the formula there are two rules which must be applied to properly use it this formula together with the two rules is your recipe or your roadmap to earning


all the money you really want now let's take a look at item number two on your checklist and I'm serious about your putting this checklist on your mirror you'll notice it's pressure sensitive item number two the gold mine the pulitzer prize-winning playwright Archibald MacLeish and is play the secret of freedom wrote the only thing about a man that is a man is his mind


everything else you can find in a pig or a horse strong words aren't they but as long as you live you will never hear a truer statement the key to every human being success lies in his mind the goldmine between his ears one idea can make you rich a lot of good ideas can move your steadily upward in the work you do and ideas are free and just think there's nothing now being done commercially that


will not be done better much better in the years ahead next years homes and most of what's in them will be better than this year's next year's cars will be better next year's manufacturing distributing marketing and selling and advertising should be better nothing is now being done as well as it must be done in the future and every innovation every new improvement will be


somebody's brainchild now what your especially how many good ideas have you come up with during the past year if you continue on as you have in the past where would you be and what will you be earning say a year from now five years from now every day of our lives we walk or drive buy more opportunity than we could develop in a lifetime in 50 lifetimes back in the twentieth Sinclair


Lewis wrote that you can kidnap a man blindfold and take him to any city in the country with a couple of notable exceptions put him in a chair in the downtown area take off his blindfold and he could sit there a week and not be able to tell you what town he's in the streets are all alike the buildings are all alike the businesses all look alike this is still largely true today the


reason for this being that most business men in this country are playing a game called follow the follower if a man goes into business no matter what line it happens to be the first thing he does is make certain that his place of business outside and inside looks exactly like every other place of business of that type in the country do you know why it's because he's been playing copycats since


he was a year old and does it without thinking about it for the same reason kids dress alike in school he wants to be one of the gang he doesn't necessarily examine all the business establishments in his field and pattern his on the one outstanding example the one that inspires him the one that he can really believe in he just does what everybody else in his


business is doing and by this simple process he guarantees his own mediocrity whose drum are you marching to if indeed you're marching to anyone's and why remember whatever you now do for a living will be done differently quite differently a few years from now never in the history of mankind have the opportunities for all of us been so great but the great majority of people


will be the beneficiaries of progress not those who bring it about which group you've rather belong to if you want to be a contributor you're not just a beneficiary here's the first rule it appears on your checklist as the gold mind so think think deliberately and with a purpose use the goldmine between your ears begin by thinking of a special time every day back during the


Depression in New York lumber dealer was growing rich while other lumber dealers were going broke when asked how he did it he said every evening when I get home I closed myself up in a quiet room sit in a comfortable chair and ask myself how will my business be conducted ten years from now then I try to do it now instead of competing with every other lumber dealer which is what they were


doing he was creating he was doing the very thing man was designed to do the very thing man does best a company growing at the rate of 10% a year will double its size in less than eight years but a man can improve his effectiveness 50% or a hundred percent a year or more the experts tell us that every one of us says with an in deep reservoir sir rebuttal - even genius that he


habitually fails to use well let's begin now to reach into these deep rich areas of pure net profit and use more more of our real abilities let's think here's the best way I've found to make yourself think start getting up a little earlier than you're accustomed to right off the bat this gives you extra time that 95% of the men in this country are not using it all one power earlier a day


gives you six and a half extra 40-hour weeks a year but at this time in the morning take a refreshing shower address get yourself a fresh hot cup of coffee if you're a coffee man and then sit down to a clean sheet of paper at the top of the paper write your financial goal now this is the amount of money per year you intend to earn soon incidentally you might like to keep this to yourself -


it's nobody's business with yours then start to think think about your goal and what it'll mean to you and your family then see how many ideas you can come up with to help you reach that goal ideas to improve what you now do for a living ways of increasing your contribution to match your income goal you know jobs don't have futures people do no matter what line of work you may be in there is


within it more than enough opportunity to last a lifetime you don't have to think of brand new ideas or revolutionary new ways of doing things although you well might come up with them think of ways of improving what is now being done if you are to increase your income by the amount you've specified you must find ways of increasing your contribution your service and the heat of this is to be