divorce so yes the take to to this rotation scheme and now I will talk about a couple of examples where the labels showed pretty pretty good results and one of their competitions is Canada model and education it was posted by keiko last year and the problem was to kind of quantify photos made by some camera into the camera it was taken by so it was a kind of multi conservation problem with ten classes and the classes are kind of Apple some soon some some kind of other devices and for example this particular image was taken by a HTC


One and yeah it's kind of impossible to say just looking but by eyes so it's another nano net network approach have been applied here for this competition and actually in this competition train that is data we're about the same size so so what about I guess 3/7 of images and we remember that do the labels show great great value when it have some kind of small small label data sets and large unlabeled data but the reason why syllabus are working here on this problem is this train and and test we'd actually also train images were taken by

How to cook pseudo-labels | by Yauhen Babakhin | Kaggle Days Dubai | Kaggle


a single physical device so for example this is green form but the test images for example might may be taken by the same model for example HTC or iPhone 4 but they were taken by different physical devices is this orange one and so the goal here was to some kind of use assisted predictions in terms of silver labels and probably if it could allow find as some particular features some particular ste facts that are specific for this country to first particle of polish model and the said that is ins ik handle learned for flow from from the


green model okay here is kind of a recipe what has been applied and they want what would have worked or for this particular competition so firstly we're just making a kind of classic classic approach we train multiple models and sample them and sample them so Lucas for example tried different architectures different training procedures and so on so welcome to the labels names this approach allows us to get 66 place on the quanta private leaderboard the next step is could be the kind of route would take take this to the labels


pretend pretend on them and printing on train data and it gives like a huge boost and we have receiving a top 20 position however as we've discussed there like distribution between train and test data it is probably the better idea to train on the unpure through the labels so the source step is kind of minute eliminates the second one and an instead of points Union on train data we train our model on pure silver labels so at this step we don't use as a the initial train data at all and in such case if it was to get even better


results and it achieves the same place or on the private leaderboard alright the next the next example is salt identification challenge it is the problem was in in semantic image segmentation so we were given some kind of images obviously some kind of or surface under earth and each pixel of the image was classified in the two classes whether it is sold on unsold and the goal was to build a model that predicts the salt deposits like some kind of mask masks represented all cones right yeah in this case in this competition as


a trained data contains only four four thousand images and the test data contains eighteen thousand images so it's really a perfect candidate for pop pseudo employment when I have this Assessor says what difference between labeled and another available data and again we start with a simple approach will train multiple models obtain filter labels and it gives in this case is a forty six position it's pretty pretty good I guess it was around three thousand participants in this competition and on the second stage we


apply our second recipe so we retain the model on CD labels twenty non-trained and obtain citizenship is a place in top ten and in order to achieve better results we just repeat this this is this these two steps multiple times so what does it mean so after second second step of ten you set up to open up a new set of see the labels we again train multiple models obtain new pursue the labels and also I can pretend which allows a French Union trade so repeaters just look multiple times allows to give the better results and a ship to touch


top one position for this competition and actually the skin looks likely cut off in this way so initially we have only label data between a model in this first round send a predictive see the labels on the unlabeled data select on the confident ones and retrain the model general repeats is this test for example K times and at each iteration which this improvement is a sports oh yes the sports improvement are kind of degrade so it's a smaller and smaller but it should iteration gives more and more information about


syllabus improves the calls up see the labels and achieve the better results also either board and actually one more dish that we are using here is that where a train is a model so at each stage we train the model from scratch so what what does it mean if we're trains we forgot to change model from with the first stage to the second and sort and so on we could finish with the situation when our like error propagates through each iteration so if I have made an error in the first round and out to the label I'll quit inaccurate then this error


will propagate through each round so with each round we're just starting from imagenet weights and start start start a model from scratch okay so yeah basically there is no kind of universal universal recipe of how to cook soup without the labels and how they could be applied because it's really well specific to data uuuu you're using ends it's a problem type you're dressing but you have some kind of building box that could be a temple night together with some kind of your sub some spices and and some ideas that could improve your


performance patron performance of your models yes this approach could be applied in both competitions and in real machine learning projects and it really performs very well when it have very small label did the data the data set but unlabeled data is available kind of in fact in in in yeah so unlevel did this is a lots of unlabeled data and when when what is a good idea what when to use this substitute the labels it could be also quite in some kind of final stage of the competition when you kind of stuck with the ideas and your


um thank you all for coming this is a massive room so today we will have six great invited talks panel discussion and a selection of posters and spotlight presentations I don't have much to say but welcome young good fellow from open AI he will be given the first talk of today an introduction to generative adversarial networks thank you good morning thank you everybody for coming I guess I'll explain first a little bit what my goals for this this talk are I know there's a lot of different people here at the workshop and the main

Introduction to GANs, NIPS 2016 | Ian Goodfellow, OpenAI


purpose of the talk is just to give everyone a little bit of context so that you know what adverse sail training is what generative adversarial networks are if you were at my tutorial on Monday you probably will have seen a lot of these slides before but I'm also going to throw in a few new ideas just so that you feel like you've got something extra for your time but this talk is mostly for the people who have just arrived at the workshop and needed some context so this workshop is about adversarial training and the phrase adversarial


training is a phrase whose usage is in flux and I don't claim exclusive ownership of the phrase but to avoid confusion I thought I'd comment a little bit on how the phrase has been used before and how it's mostly used now so I first used the phrase adversarial training in a paper called explaining and harnessing adversarial examples and in that context I used it to refer to the process of training and neural network to correctly classify adversarial examples by training the network on adversarial examples today


other people have started using the phrase adversarial training for lots of different areas almost any situation where we train and model in a worst case scenario where the worst case inputs are provided either by another model or by an optimization algorithm so the phrase episode training now applies to lots of ideas that are both new and old the way that we use the phrase adverse sail training now it could apply to things like and an agent playing a game against a copy of itself like Arthur Samuels checkers player back in the 1950s so


it's important to recognize that when we use the phrase adversarial training today we're not only referring to things that were invented recently but the usage has expanded to encompass a lot of older things that also had other names like robust optimization most of the day's workshop is about a specific kind of adverse ale training which is training of generative adversarial networks in the context of generative adversarial networks both both players in the game are neural networks and the goal is to learn to generate data that


resembles the data that was in the training set the reason that we call the training process for generative adverse cell network adversarial training is that the worst case input for one of these networks is generated by the other player and so one of the players is always trained to do as well as possible on the worst possible input it's worth mentioning that there are other works going on in the space of adversarial training where the goal is still to train on adversarial examples inputs that were maybe created by an


optimization algorithm to confuse the model and you will see some posters about that here there's also some work about that in the reliable ml workshop but I hope that clears up any confusion about the term adversarial training so generative adversarial networks are mostly intended to solve the task of generative modeling the idea behind generative modeling is that we have a collection of training examples usually of large high dimensional examples such as images or audio waveforms most of the time we'll use


images as the running scenario that we we show pictures of in slides because it's much easier easier to show a picture of an image than to play an audio waveform but everything that we describe for images applies to more or less any other kind of data so there are two things you might ask for a generative model to do one is what we call density estimation we're given a large collection of examples we want to find the probability density function scribes as examples but another thing we might do is try to learn a function or a


program that can generate more samples from that same training distribution so I show that on the lower the lower row here where we have a collection of many different training examples in this case photos from the imagenet data set and we'd like to create a lot more of those photos and we create those photos in a random way where the model is actually generating photos that have never been seen before but come from the same data distribution in this case the images on the right are actually just more examples from the image net data set


generative models are not yet good enough to make this quality of images but that's the goal that we're striving toward the particular approach the generative adversarial Network to take to generative modeling is to have two different agents playing a game against each other one of these agents is a generator network which tries to generate data and the other agent is a discriminator network that examines data and estimates whether it is real or fake the goal of the generator is to fool the discriminator and as both players get


better and better at their job over time eventually the generator is forced to create data that is as realistic as possible the data that comes to the same distribution as the training data the way that the training process works is that first we sample some image from the training data set like the face that we show on the Left we call this image X it's just the name of the input to the model and then the first player is this discriminator network which you represent with a capital D the discriminator network is a


differentiable function that has parameters that control the shape of the function in other words it's usually a neural network we then apply the function D to the image X and in this case the goal of D is to make D of X be very close to one signifying that X is a real example that came from the training set in the other half of the training process we sample some random noise Z from a prior distribution over latent variables in our generative model you can think of Z as just a sort of randomness that allows the generator


to output many different images instead of outputting only one realistic image after we've sampled the input noisy we apply the generator function just like the discriminator the generator is a differentiable function controlled by some set of parameters and in other words it's usually a deep neural network after applying the function G to input noisy we obtain a value of x sampled in this case from the model like the face on the right this sample X will hopefully be reasonably similar to the data distribution but might have some


small problems with it that the discriminator could detect in this case we've shown a slightly grainy noisy image of a face suggesting that this brain and noise is a feature that the discriminator might use to detect the images faked we applied the discriminator function to the fake example that we pulled from the generator and in this case the discriminator tries to make its output D of G of Z be near zero earlier when we use the discriminator and real data we wanted D of X to be near one and now the discriminator wants D of G of Z to be


near zero to signify that the input is fake simultaneously the generator is competing against the discriminator trying to make D of G of Z approach one we can think of the generator the discriminator is being a little bit like counterfeiters and police the police would like to allow people with real money to safely spend their money without being punished but would like to also catch counterfeit money and remove it from circulation and punish the counterfeiters simultaneously the counterfeiters would like to fool the


police and successfully use their money but if the counterfeiters are not very good at making fake money they'll get caught so over time the police learned to be better and better at catching counterfeit money and the counterfeiters learn to be better and better at producing it so in the end we can actually use game theory to analyze this situation we find that if both the police and the counterfeiters or in other words if both the discriminator and the generator have unlimited capabilities the Nash equilibrium of this game corresponds to


the generator producing perfect samples that come from the same distribution as a trading data in other words the counterfeit are producing counterfeit money that is indistinguishable from real money and at that point the discriminator or in other words the police can not actually distinguish between the two sources of data and simply says that every input has probability one-half of being real and probability one-half of being fake we can formally describe the learning process using what's called a minimax game so we have a cost function for the


discriminator and we call J superscript D which is just the normal cross entropy cost associated with the binary classification problem of telling real data from fake data we have one mini batch of real data drawn from the data set and what a mini batch of fake data drawn from the generator and then if we use this minimax formulation of the game then the cost for the generator is just the negation of the cost for the discriminator the equilibrium of this game is a saddle point of a superscript D and finding this saddle point


resembles the process of minimizing the Jensen's Shannon divergence between the data and the model we can use that to actually prove that we'll recover the correct data distribution if we go to the equilibrium of the game we can analyze what the discriminator does and they play this game and we see exactly what it is that allows generative adversarial networks to be effective the basic idea is that if you take the derivatives of the minimax games value function with respect to the outputs of the discriminator we can actually solve


for the optimal function that the discriminator should learn this function turns out to be the ratio between P data of X and P data of X plus P model of X you can do a little bit of algebra on that to rearrange it and you get P data of x over P model of X so we're learning a ratio between the density that the real data is drawn from and the density of the model currently represents estimating that ratio allows us to compute a lot of different divergences like the Jenson Shannon divergence and the KL divergence between the data and


model that are used for training with maximum likelihood so the key insight of generative Ebersole networks is to use supervised learning to estimate a ratio that we need to be able to do unsupervised learning there are also a variety of other papers by Shakir Muhammad and his collaborators and Sebastian knows and his collaborators that talk a lot about the different divergences that you can learn with these kinds of techniques and how this estimation procedure compares to other techniques have also been developed in the statistical estimation


literature previously but this is the basic idea right here is that we're able to learn this ratio so far I've described everything in terms of the minimax game I personally recommend that you don't use exactly that formulation you use a slightly different formulation where the generator has its own separate cost and the idea is that rather than minimizing the discriminators pay off the generator should maximize the probability that the discriminator makes a mistake the nice thing about this formulation is that the generator is


much less likely to suffer from the vanish and gradients problem but this is more of a practical tip and trick rather than a strong theoretical recommendation and some of the other speakers you'll see today might actually give other advice so it's kind of an open question about exactly which tips and tricks work the best one of the really cool things about generative adversarial Nets is that you can do arithmetic on the z vectors that drive the output of the model we can think of Z as a set of latent variables that describe what is


going to appear in the image and so Alec radford the co-organizer of this workshop and his collaborators showed that you can actually take Z vectors corresponding to pictures of a man with glasses the Z vector for a picture of a man and the Z vector for a picture of a woman and if you subtract the vector for men from the vector for men with glasses and you add the vector for women you'll actually get a vector that describes woman with glasses and when you decode small jitters of that vector you get many different pictures of a


woman wearing glasses a lot of you may have seen a similar result before with language models where the word embedding for Queen could be used to do arithmetic where if you subtract off the word embeddings for female and add the word embedding for male you get a vector that is very close to the word embedding for King in this case Alec and his collaborators have a slightly more exciting result because they not only show that the arithmetic works in vector space but also that the vector can be decoded to a high dimensional realistic


image with many different pixels all set correctly in the case of language modeling the final result was a vector that was very near the word for King but there was no need to decode that vector into some kind of extremely complicated observation set that corresponds to a king probably the biggest issue with generative adversarial networks and to some extent with other forms of adversarial training is that the training process does not always converge most of deep learning consists of minimizing a single cost function but


the basic idea of adversarial training is that we have two different players who are adversaries and each of them is minimizing their own cost function when we minimize a single cost function that's called optimization and it's unusual for us to have a major problem with non convergence we might get unlucky and converge to a location that we don't like such as a saddle point with a high cost function value but we'll usually at least converge to some general region when we play a game with two players and each of them is


simultaneously trying to minimize their own cost we might never actually approach the equilibrium of the game in particular one of the worst forms of non convergence that we see with generative adversarial networks is what we call mode collapse or if you're in on a little joke in our first paper we also caught the Helvetica scenario sometimes the basic idea of the hind mode collapse is that when we use the minimax formulation of the game we'd really like to see is minimization over G in the outer loop and maximization


over D in the inner loop if we do this min max problem applied to the value function V we are guaranteed to actually recover the training distribution but if we swap the order of the mechs and the men we get a different result in fact if we minimize every G in the inner loop the generator has no incentive to do anything other than map all inputs Z to the same output X and that output X is the point that is currently considered most likely to be real rather than fake by the current value of the generator so we really want to do min max and not max


min which one are we actually doing the way that we train models we do simultaneous gradient descent on both players costs and that looks very symmetric it doesn't naturally prioritize one direction of the min Max or max min in practice we find that we often see results that look an awful lot like Max min unfortunately with G in the inner loop so using some very nice visualizations from Luke Metz and his collaborator collaborators we see here that if we have a target distribution we'd like to learn with several


different modes in two dimensions the training procedure shown in the bottom row of images actually visits one mode after another instead of learning to visit all of the different modes so what's going on is that the generator will identify some mode that the discriminator believes is highly likely and place all of its maps there and then the discriminator learns not to be fooled by the generator going to that one particular location and instead of learning that the generator ought to go to multiple locations the generator


moves on to a different location until the discriminator learns to reject that one - one way that we can try to mitigate the mode collapse problem is with the use of what we call mini-batch features this is introduced in the paper that we presented on Monday night from open AI where the basic idea is to add extra features to the discriminator so the discriminator can look at an entire mini batch of data and if all the different samples in the mini batch are very similar whether the discriminator can realize that mode collapse is


happening and reject those samples is being fake on the CFR 10 dataset this approach allowed us to learn samples that show all the different object classes in CFR 10 for the first time on the Left I show you what the training data looks like for CFR 10 you can see that it's not that beautiful to start with because there are only 32 by 32 pixel images so the resolution is very low on the right we see the samples that come from the model and you see that you can actually recognize horses ships airplanes and so on and cars that we


actually have the real object classes recognizably occurring within this data set an image net there's a thousand classes so it's much more difficult to resist the mode collapse problem an image that our model mostly produces samples that have kind of the texture of photographs but don't necessarily have rich class structure we do occasionally get rich class structure if I show you some very cherry-picked examples we're able to make lots of different pictures of things like dogs spiders koalas bears and birds and so on


we still see a lot of problems with the model though in particular we often see problems with counting we think that this might be something to do with the architecture of our convolutional network that it's able to test whether a feature is absent or present but it doesn't necessarily test how many times that feature occurs so we see things like this giraffe head with four eyes this dog with something like six legs or this kind of three-headed monkey thing or you know stacks of puppies rather than a single puppy or a cat with one


and a half faces we also often see problems with perspective where the model generates images that are extremely flat in particular the image on the lower left looks to me like somebody skinned a dog you know like a bearskin rug and then took a picture with the camera looking straight down at it on the ground whether the picture in the lower middle looks to me literally like a cubist painting we're in the cubism movement artists intentionally removed all the perspective from an image and rearranged the object to show us


different parts from different angles but representing the entire thing is flat in many cases we see images that are really quite nice that have some problem with the global structure a lot of the time this just consists of images of animals where we don't actually get to see their legs that they have a head and torso attached but they don't actually complete the legs anywhere and in my particular favorite generator samples so far on the lower left we have an image that we've actually named fall out cow where we have an animal that is


both quadrupedal and bipedal it actually has legs and it has the right number of them but it has two different bodies so what are some things that you can do with genitive adversarial networks there are just so many different things that it's a little bit hard to show all of them I showed a lot more in my tutorial but I can show you just a few really quick one thing that came out recently is image to image translation this is from the research group at Berkeley and the basic idea here is to take a conditional generative adversarial


Network and map from one domain to another it can do things like take images that say for every pixel where the different what kind of class should appear at each pixel and turn that into a photorealistic scene with the desired objects in the desired positions it can also take an aerial photo and turn it into a map where it can take a sketch of an object and turn it into a photo of an object more recently there was a very exciting result that finally developed the ability to generate realistic samples on the image net data set from


all 1,000 classes and with really good diversity this result is called plug-and-play generative models and it combines many different approaches to generative modeling including generative adversarial Nets moment matching denoising auto-encoders and lunch main sampling the results are really excellent and we see lots of different very recognizable high quality images with all the right numbers of legs and everything so generative modeling has really come very far in just the last month actually and generative ever eternal nets are part of


that progress I have a few comments about exactly what it is that allows generally retro style networks to work well on kind of an intuitive level one of the main things that's really different about generative a dresser Nets compared to other approaches to machine learning is that they give a very nice way of telling the model that there are multiple correct answers so a lot of the time with supervised learning we use something like mean squared error to tell the model loads output should have been so on the Left I show a little


bit about what's wrong with the mean squared error training process suppose we have the blue dot at the bottom of the slide representing some input and we'd like to learn to map this input to some desired output suppose all the different green dots represent different possible outputs that are all valid well in the training set suppose the the label that we had for this particular blue dot was the green dot on the far left but suppose that the model produced a green dot on the far right mean squared error will induce the red error


arrow saying that instead of producing the dot on the right the model should have produced the dot on the Left which is the one that appears in the training set that means that over time the blue dot will actually get mapped to something more like the mean of all these different green dots and that causes us to learn things like blurry images when we try to learn to predict different images associated with some input generative Everest or networks don't actually directly use a pair of inputs and outputs to tell the model


what it should do instead the discriminator learns how inputs and outputs can be paired and then the discriminator tells the model whether it did a good job or not so the discriminator what ideally learn that all of the different green dots are possible options and then when the generator produces the green dot on the right the discriminator says that that was a good thing to do there are many different good things that the model can do and the discriminator will hopefully endorse all of them so we now have this mechanism


for saying that many different outputs are possible instead of always steering the model toward one predefined answer we can see this especially in the context of next video frame prediction so this paper by Bill Lata and his collaborators shows what happens when we use a few different kinds of models to predict the next frame in a video on the left I show the ground truth where we have this 3d rendered image of a person's head and you can see that the image is very sharp and and has a clearly visible ear using a model that


was changed with mean squared error in the image in the middle we see that the ear has vanished because the exact location of the ear is not especially predictable and when the model averages over many different possible places the year could go it vanishes similarly the eye has become blurry in the image on the right we see what happens when an adversarial loss is included in the training process in this case the model is now encouraged to produce samples that actually look realistic and it knows that there are multiple different


answers that are all possible so it's been able to choose one of the many different sharp images that could happen at the next time step it's also worth thinking about whatever sterols training looks like for people and really the way that the idea of adverse sail training emerged was that economists and other researchers in those kinds of fields we're already working on thinking about the way that multiple different agents acting in a market have their behavior influenced by the process of them optimizing their own


payoffs while all the other players optimize their paths so some things that I think could be interesting to look at from machine learning code of view are weather cycles in markets can be explained by the failure of optimization algorithms to converge if we have trouble fitting generative Evaristo nuts and we have complete information just think about how hard it is to choose prices for goods when there are many more actors and when you don't have complete information about the market I'm sure this has been studied to some


extent but I think that bringing the economy the machine learning people together could find some more interesting ideas we didn't already know about we've also seen lots of cases of things like auctions that are designed to make sure that people pay the right price and that's more or less what I was thinking of when I designed generative adversarial Nets one last remit is that if we think about the way that people learn researchers like Erikson have shown that the way to become really good at any particular


task is to practice it a lot but also to do deliberate practice you're not just putting in a lot of hours you are specifically choosing subtasks within the skill that you're trying to get good at that are especially difficult for you and getting feedback from an expert who coaches you you can think of adversarial training as capturing both of these aspects of developing a skill rather than just training and lots and lots of training examples your training on the worst case inputs that are really hard for the model and in the case of


generative adverts or networks you have an expert the discriminator coaching the generator on what it should have done instead so a lot of insights from human psychology and human learning are actually telling us how we can make machine learning more effective so in conclusion adversarial training is a way of training a variety of models in different ways that all involve working on a worst case input generative adversarial Nets are one of the most popular members of this framework and they're based on using the estimate of a


ratio of densities to do unsupervised learning and part of why they work so well is that they allow the model to have multiple correct answers and they draw on a lot of the ideas that help humans to learn really well I'm almost out of time but I think we might be able to have a few questions if that's okay with the organizers you have several microphones on the sides okay yes I have caution first here over here so I secured armed for sequential things like video do you know any of any worked of using my recurrent


network that actually can generate the risk so the sequence of data to generate our video or what kind of charge that would be yeah there is a paper about generating videos with generative adversarial networks I forget the exact title off the top of my head okay there's also there's a paper here at this workshop today I called unrolled generative every cell networks and I know that one of their experiments involves using a recurrent network to generate Amnesty one pixel at a time so you could check out their spotlight and


poster and how about generating language sequence of words in discrete domain the thing that's difficult about that is is the discrete outputs which means that the generator is not differentiable so that's an open research area it might be solvable using things like the reinforce algorithm to do policy gradient on the parameters or using things like gun Bell softmax or the concrete distribution or it might be possible to generate word embeddings from the generator and then decode them to discrete values instead


of generating military values directly and how about a speech which is continuous we know that google's are wavenet use different mechanism of generating speech and using this gain framework can be used equally well yeah so before I left Google I suggested that they try generating continuous waveforms it can and I don't actually know if they tried and again didn't work or if they just went straight to using Pixlr and n-type methods to generate the continuous waveform but I do think that the continuous waveform is the way to go


with games if that did work the advantage of games would have over wavenet is that the gans can generate the sample much faster that wavenet needs to pass through a neural net for every single sample of the audio so it's generating you know really thousands of samples using thousands of passes to the network it takes about two minutes to generate one sample one second about you okay though again could generate a long waveform in one shot it's interesting to see that wavenet didn't really have temporary effect I


use show over there using mean square error kind of criterion for the image so do you have an explanation why once brother the arm is not blur so the blurring effect is with mean square error in real valued spaces and they were using discrete spaces where they have a soft max distribution so their loss function is not actually mean squared error it's it's a categorical cross entropy okay and but that's still no excuse let's ask another question and you can check with you later yeah any other questions okay thank you


thank you thank you very much for the warm introduction it's an honor to be here and participate in this lecture series I want to first clear up any possible misconceptions I have one kid just one and he has one head this was this two-headed thing was taken with photo booth on the iPad and it's his favorite thing to do is to take pictures of himself with two heads so anyway that's impossible and he's thinking about it okay all right all right okay so so I have two goals for this talk and they're kind of conflicting a little bit so the first

Thinking Algorithmically About Impossibility


goal is I want to tell you about the state of the art in in circuit complexity lower bounds in what particular areas of certain complexity at least and the second goal is to give a general thought for a scientific audience so I'm I'm going to try to navigate these two things and hopefully it will turn out okay all right let's begin so as a young computer science theorist one is taught that there are roughly two kinds of people there are algorithms people and there are complexity people okay the algorithms designers asked the following very


positive upbeat questions they asked when can we solve problems quickly what can we do with computers when can we find a good algorithm what what does good mean good could mean many sorts of things generally speaking it means that your algorithm is time efficient it runs very fast okay on the other hand there's this other group of people called complexity theorists okay the complexity theorists are negative nancies they ask where our problem is difficult to solve okay when when can we prove the problem is not easy okay in more slightly more


technical terms when can we prove a lower bound on the amount of time that we need to solve a problem or the amount of resources needed to be consumed in order to solve a problem so so this sort of lower bound thing versus algorithms think it's going to be pretty prevalent this talk so here are some intuitions about theory that you build up again as a young computer scientist you learned that the elegant design and the complexity theorists have opposing goals and tasks okay certainly the way I presented it I mean one is trying to say


there's an algorithm the other ones trying to say nope you know so so the way I presented there really a opposite to each other okay and the second thing you learn when way or another so the algorithms tribe has an easier life somehow somehow designing algorithms is just much easier than proving lower balance and this intuition is really really strong I mean you want to say look if I design an algorithm I just have to give you one piece of code it's just going to work and solve the problem no matter what but if just to show a


lower bound I have to show that no matter what code you wrote down even code I will never see like just there's just only so much code I could see in my life oh well no matter what you write down it's just not going to work it's just not going to solve the problem fast enough so just from that point of view it seems designing Aram's is easier okay so some of our lower bound says have you have to reason about all possible algorithms right so I assume many of you know about algorithms so I'll spend some time talking more about lower bounds at


least the beginning so the most famous lower bound question the Grandmamma of them all is P equals NP okay and just to give you a very high-level idea of what P and NP are not to go in the definitions NP is the set of problems where verifying a solution to the problem is easy and P are the problems where finding a solution is easy to do okay and so by easy we mean the following technical sense of Paul meal time so there's some algorithm that on inputs of length n runs in some polynomial and an amount of steps think


of this P of n is N squared or in cubed something like that so this is what we mean by easy most of the things that you see when you're learning our than they're easy sorting easy so this problem was articulated in 1970s by Steve cook and the carp and also in the Soviet Union by leonid 11 okay so the rather than going into definitions and things like this I will just give you an illustration of this and probably the best illustration of the difference between P and NP is the problem of factoring numbers so suppose I give you


this number okay it's 232 digits and I want you to I want to know like x and y so the x times y equals this number okay well turns out there are two such numbers okay X of this length Y of this length all right now the problem of giving such a number what are x and y right so this is an EMPI problem because if I gave you x and y to begin with if I gave you the solution of the problem the X and the y then it's easy to check that next times y equals the given number you can definitely multiply these things in less


than a second on your cellphone check they equal the number okay what about finding the x and y given just this number what can you do well I gave you a very special number here actually so this is the RSA 768 number it took two years and a team of number theorists and programmers to find this X&Y a lot of famous people crushed really really hard to find this X&Y despite the fact that once it's given to me takes no time at all to verify their items okay so in so MP is this kind of problem where you I give you the number and I want to know the


two the two factors x and y well with P you just you know I'm just I just want to like find the answer okay so the problem of people's emptiness asking whenever it's possible to check that x times y equals a number could I find could I do the inverse and figure out x and y efficiently as well okay so this is a very stark contrast between these two kinds of problems okay so P equals NP is asking a verifying a solution a problem is easy to do then can we find a solution the problem also easily so here's another number okay


it's been slightly longer it's 240 decimal digits alright so we don't actually don't know how to factor this number well okay as far as we know perhaps our friends the NSA have factored this number a long time ago okay and and if you want to you know a bigger list of numbers that we don't know how to factor there's a whole page and Wikipedia with lots and lots of numbers going up about six hundred digits big lists of numbers okay so if we could factor integers easily so this problem is looks a simple number theoretic problem but you


probably know that if we could factor these interests easily we could break all our essay based cryptography so we could break anything that says there's just a little lock here on your web browser yeah forget it it's not really I mean this is crack it open okay well HTTPS SSL totally broken okay these are as same numbers actually is a great resource for someone like me if someone send emails me and say they have a proof people's MP I just forwarding this link and say I'll be waiting you know just all you got to do


just tell me these you know short 100 something digit numbers from everything in this list okay then oh then I read your paper okay so so that that makes them so very very makes it easy for me right okay so we believe the answer is no okay hope this is you know somewhat convincing just this this problem of multiplication versus factoring a number but how that we mathematically prove that how could we prove there's just no way to do it there's no way to break a number in the protectors we have to show that some


problem that's easy to verify it somehow impossible to efficiently solve okay so this requires us to somehow reason about all easy programs one way or another so let me emphasize it P where's MP is only one of many questions of this kind many of which just I simply can't mention this talk I just want to mentioned the most prominent one there but they all sort of hint or at the same kind of thing reasoning about efficient computation we don't know how to do it okay so where lower bounds good for so they're impossibilities they'll say you


can't do something well the first sort of very useful way in which the lower bounds were uses for algorithm engineering so if I prove that a lower bound on a particular mathematical formalization or a problem that just often just means that the way you set the problem formally is not the right sort of way it's math it just not going to help you if you try for all this if you set if you wrote the problem down in this way so so you could try something else you try to read for me at the problem go back to it and and see if you


can steer yourself away from things which MP hard and things which are more like P but lower bounds have actually been extremely useful in other areas such as machine learning theory useful in in learning functions for which you know lower bounds for those kinds of functions for cryptography as as demonstrated with RSA so you sort of need lower bounds just to get off the ground you need to be able to say that you know I have have concocted a problem efficiently but you you can't crack it you can't you know turn around and solve


it for me and also more grounds come up in an unexpected way and the construction of so-called pseudo-random generators so these come up when you have a randomized algorithm arete the album which tosses coins and you can argue that you can mathematically prove that okay if you have uniform independent random coins then this algorithm is going to work well maybe maybe you know you can't rely on independent random coins well lower bounds can actually be used to remove the randomness from algorithms replace randomized armed systematically with


something that just uses a purely deterministic sequence of coin flips and still works well this is a very nice connection and sort of unexpected when you first think about it but one day I really want to try at home is that we don't really know what all lower bounds are good for because we have no idea what the proofs of them look like okay I mean this almost all this is just from sort of knowing just that the lower bound holds what you can do but what if you actually had a mathematical proof something that


actually did the reasoning and you could study it then there's no telling what we could we could learn about computation that way and so lower balance to me are one of the great scientific mysteries of our time there are conjectures about them everywhere ok Internet security relies on them being true but knowing them for sure is mostly inaccessible to us ok so so yeah this is just a great mystery why is easy computation so hard to understand okay why are lower bound so hard to prove well why are we so stuck here so


actually we have a pretty good understanding of why we're stuck so there are many known nogo theorems in complexity theory with various technical names cultivation natural proofs algebra zation so on what they say in a nutshell is that the common proof techniques that we use are just not going to be enough to prove things much much weaker than p9 wimpy things much weaker than this just like very simple things we can't we can't prove allowance for and this has cost a great deal of weeping and wailing and gnashing of teeth a great pessimism


in complexity theory and so the big question here is how will we make progress how do we get smarter how are we going to get off the ground and start really proving that some problems are hard to solve ok so so this is what I'm going to try to say something about okay I mean obviously I don't have all the answers but I would like to say something that I hope can be conveyed to everyone yeah well sort of the main insights in work that I participate in so far is in connections between arms and lower bounds so we're


talking about how arm designs are the easier life then lower bound improvers in fact we are deep connections between the two subjects that we just don't understand yet we know for sure that they are there we have sort of glimpses of connections we but there is certainly something more under the hood that we've got to know and so I'll tell you about the glimpses that we have so the main idea I guess the one thing I would like to you to try to take away from this talk is this thing in this box designing algorithms is as hard as proving lower


bounds sometimes now this is one of those ideas that depending on who you talk to is either complete heresy or they thought of it already okay so that means it's a good idea from from my from my informal judgment okay so yeah so it's a so it but it's sort of provocative so it's very provocative actually because if because lower bound designer doesn't have it as hard as the album designer okay so somehow the design event certain algorithms is going to actually give you lower bounds so maybe the our designer doesn't have


it so easy after all it just they managed to solve other problems or maybe it's just psychological maybe we just are wired to think about algorithms and some you know more easily than lower bounds but there's not so much different between them in the end so here's here's just that my vague attempt to draw some parallel here so a typical resultant album design says here is an algorithm a that solves some problem on all possible instances of the problem now it's actually amazing how many statements of this form hold but


the fact that like you can have a algorithm that say solves a a problem all possible graphs no matter which graph you see even graphs you will never be able to see you can still prove it's going to work and give you the right answer for this graph okay on all possible instances okay now a typical theorem from lower bounds will say something like here is a proof P that this problem can't be solved on all possible by all above two albums of some class some weak class of algorithms that I've sort of box over here and I can


rule all of them out from solving this problem here so what I would like to promote is to draw a parallel between the algorithm here in the album design problem and the proof here in lower bounds and all possible instances of the problem in the album design problem with all possible albums with some class okay so you think about this it sort of automatically suggests the kind of algorithms we want to design so the kind of problems we want to solve our problems which take some kind of algorithm as input itself and perform


some computation so we wanted to solve some kind of problem that is taking an algorithm at an input we have to be very careful of how we do this I mean like if you just took the usual kind of algorithm as input like the sort of terrain type algorithm then there are lots of results an undergrad computability theory that just says no matter what you try to do analyzing the things it's not going to work so it's make this kind of connection tractable we have to slightly change the model of algorithm we're looking at and


this is what I'm going to talk about okay so so I have a very high level very high level here's the specific kind of connection that I'm going to talk about so suppose you have there exists some non-trivial circuit analysis our okay so think of this is just some albums some fixed program maybe a Turing machine what have you it receives as input some circuit encoded in some nice way some logical circuit and for all logical circuits it can do the some kind of non-trivial analysis on this thing let's say it can determine whether that this


circuit is satisfiable in other words whether the circuit is computing a trivial function or this computing something interesting let's suppose I can do this in some nice way something just better than trying all possible inputs to the circuit and seeing what happens if you can set this kind of thing of it you can solve this kind of album design problem then you can say there is a function f compute with respect to certain Turing machines for which for all circuits they cannot compute this function for all circuits


that cannot compute this function so you get circuit complexity lower amounts so from the design of particular kinds of algorithms problems ones which take as some kind of algorithm or circuit as input and analyzing them non trivially you can then prove that for whatever it is I analyzed non-trivial e I now have a new function that can't be computed by it so that this is a very very counterintuitive if you look at it in the wrong way if you look at it it's quite a far away maybe it's so slightly more into okay so this is the basic kind


of connection that I and others have been able to draw and we've been able to actually prove lower new lower bounds new limits on computing using this kind of connection so here's an outline for the rest of the talk I'll give you a very quick introduction to just boolean circuits and sort of the problems we're thinking about here I'll compare circuits to algorithms in the usual sense that people learn them and then I'll talk briefly at the end about how circuit analysis algorithms can imply circuit limitations okay all right so


bullion circuits look you know something like this so this is a boolean circuit which is taking three bits of input okay it's taking the aunt of each pair of bits and then taking the or of those bits and so it's outputting a and B or B NC or ANC okay in general a circuit of size s for us will take a fixed number of zero one inputs and it will output let's say a single bit you can generalize it to outputting mobile bits for SS a single bit then for F times it's going to take two previously computed bits maybe some


input bits such as here a and B and you compute a function of those two bits and that will be some new bit that I can use in subsequent computations such as here I'm taking this or if those things okay so just to give an example this above circuit on three inputs outputs one if only if at least two the inputs of one okay now the size of the complexity of this circuit is going to be like four four gates in it and so this is how we can measure computation but for finite functions so it's important here that the number of bits of input is something


finite okay and then we can talk about if I want to compute some given function how many times do I have to take some previously computed bits and give up another but before I finally get the output of the function I want okay and so trying to minimize this as the number of little no gates you have in here is the core problem in certain complexity all right so most functions most require humongous circuits okay so a very old theorem of Shannon and matched by loop on/off says it with high probability if you randomly choose a function by just


sort of flipping a bunch of coins generating a function by coin flipping this will require a very large circuit but still gonna nearly a size like to the N over N okay it's another number of inputs to this thing so this is just saying I take a bit zero one and you know I take n of those bits and I output a single bit okay so in fact every function has a circuit of size about to the you know gram okay so this think of it this way so both um I'm flipping two to the N points and I generate some very very long two to the N bit string this


is just defining some function f okay what this is saying is that there's no way in general with high probability to compress this 2 to the N bit thing and a small very small representation this is basically proved by a counting argument is too many small series too few small circuits and too many randomly chosen functions so a random one is just not going to be represented by something small there's a simple incompressibility type argument so okay the universe is littered with you know these really really hard functions from the


circumflex II point of view my question is which natural functions ones that we actually want to solve exhibit this kind of exponential behavior they require a large number of gates in order to compute them okay so this is a very central question in certain place you just trying to understand which functions can be solved at all we know that random ones cannot can we can we get is there some function of interest you know that we that we that doesn't have that we cannot to solve the small sir okay that's it so that's a sorry


complex see that show so the kinds of things I want to do is circuits I is I want to analyze them so I'm going to talk about computational problems that take a circuit encode it in some way as an input and compute something on that circuit so the circuit analysis problem is a problem where the input is a circuit so here we have some logical circuit written down in some encoding the P that our program can read and we want to output some property of the function computed by this circuit so we want to know something about what


function is it computing in you know inside this little description so the canonical example of this is a circuit satisfiability problem also known as circuit set okay it's really like the simplest circuit analysis problem in a particular sense here you're given a logical circuit see so you wrote you've written down some description of some circuits got wired together in some funky way and you want to know is it computing the all zeroes function is it the case that no matter what input I give it is always going to print zero or


is there some input which will make it print one okay that would usually when something makes it print one we call it a satisfying assignment and that's the origin of satisfiability but really want to know Jesus is it computing a trivial function or not so just tell me that okay so a circuit set is a so-called np-complete problem and so it's very unlikely to be solvable efficiently okay unless P equals NP okay like this so probably we're not going to be able to solve this circuit analysis problem very well but we could ask still if there is


some way to solve it faster than the obvious self what is the obvious algorithm just try every possible input to the circuit okay so here we're leveraging the fact that there's a finite number of inputs so the circuit has a finite number of bits coming in so you could just if it's got n bits coming in there to the impossibility is just try them all see if any of them make this thing print one okay so we can answer this something faster this obvious brute-force search our okay we and maybe this is an interesting


question maybe it isn't I always thought it was a very interesting question turns out it actually is interesting for other reasons it actually is connected to circuit complexity but just the question of it you have us you have a silly algorithm you have a obvious algorithm is it the best you can do this is just a very fundamental question by itself all right so that's a quick introduction to circuits okay now I want to talk about how circuits compare to algorithms so in the usual everything model where you think of you know writing a single


program it's going to solve a problem you have some finite description of some program and this program can take an arbitrarily long inputs and still solve the problem okay if you give it an input of length 10 no problem input of length of thousand no problem a million no problem 100 million no problem this thing always solves the problem single thing okay that is in contrast with the circuit model where we only allow you to take in fixed length inputs okay so so if we're thinking about how our circuits can relate to one another we've


got this sort of correctness type mismatch where algorithms can take an arbitrarily long input and circuits can only take in the fixed length inputs and there's a very easy way that complexity theorist came up with sort of unifying the two and the idea is to talk about a computational model called a circuit family the circuit family has is an infinite collection of circuits infinitely many and you have a circuit on one bit of input a circuit on two bits of input circuit on ten bits of input for every possible size of input


you've got a circuit with that many inputs okay so for each end you have a circuit C sub n that's going to bring you on on inputs of lengths in and so it's obvious what I'm going to do and I want to compute something I'm you give me an input I measure its length I feed it to whatever the circuit is in the collection that matches that length as I run it it gives me some answer okay that's how the circuit family would compute on on some problem that would normally be solved by now okay all right so in this model the circuit family


model programs have infinitely long descriptions a priori okay there is nothing bounding what these circuits could be you get a separate program every single time here input changes so it's possible that you know if your input if your input is you know a thousand there's something really clever you can do just when in facility two thousand one thousand and one you gotta restart everything do something completely different and get something you know very good okay so that there's this infinite description can be really


really powerful okay but our notion of efficiency here is this class P poly so efficiency for algorithms is just the class P P poly is a set of problems solvable with one of these infinite circuit families so separate circuits ease of in for each input length where for every end the size of the in circuit is some polynomial in it this polynomial is fixed once and for all let's say it's N squared and so this circuit sees up a thousand be a thousand squared sighs okay so this is our notion of efficiency here for this or infinite


model okay and the idea is that each circuit here it's going to be small relative to its input but you get a separate circuit for each input length that's sort of the extra thing you get you get this infinite description okay so this circuit family right here alright that's a notion of P poly so you anything programs have infinite link descriptions huh excuse me why study this model okay Theory dork that's all fine and good but why don't you go play off in theory land well all the rest of us go change the world okay okay well I have


an answer for you all right okay so so proving limitations on what circuit families can compute is a step towards a non asymptotic complexity theory a complexity it doesn't talk about polynomials or Exponential's or whatever when have you talked about numbers how big does the computation get if I want to run it and if I want to run it on inputs of length a million can I fit it inside the known universe so so concrete limitations on computing within the known universe is would be is the the thing you would like to have that's sort


of the ideal thing you would like to have something not as I'm sorry we don't care about polynomials exploration we just want to know how big is the computation gonna get so think something like any computer solving most instances might tend to for a bit problem these at least 10 to the 125 bits to be described so if you've got you know this kind of statement then your problem is just simply not solvable period okay so actually such a statement was proved by Meyer and stockmeyer in the 70s for a particular logic problem


and they derived it we by reverse engineering a circuit lure about because circuit lower bounds are actually explicitly considering the trade-offs between the size of an input and the size of the computation you got a throw at the input to solve whereas Peters MP is actually not talking about this time at a fixed algorithm that's gonna work on all inputs okay and so the universe turns out stores a less than 10 to the 125 bits this is the this is the famous bekenstein bound in 1970s let's derive from that and so what


you're saying is that any computer solving most instance my problem just won't fit within the known universe okay that's a pretty good lower bound okay so that's the kind of thing we would like to have and so the circuit family is just some stepping stone to getting some non asymptotic things were just talking about numbers that is the ultimate goal we are really this is des soon okay all right so a lot of algorithms compared to circuit families okay so we define this thing how does it compare to the uniform


one program during low so if you got a program that runs and say T of n time when inputs of length n so in the squared time then it's well-known that you can always get a circuit family that will do the same thing and it has about TN in size as well there's some extra little factors don't see it in size so you can simulate efficient albums with efficient circuit families okay so time will scale the size all right so so that's all fine good in fact if you flip random coins in your algorithm and it gives you the right answer with high


probability you will still be able to get infinite family no circuits has about the same size so in a sense you can remove the RAM this when you allow for an infinite circuit fan this is nothing more than the statement okay in complexity theory terms that bvp is in P Polly randomized phone real-time as efficient bonafide circuit changes okay suppose I'm just looking at circuit families I want to know can they be simulated by algorithms well turns out there is a family we in fact every circuit has size about in some linear


size circuits so there's definitely in ppalli there's only a linear size as so as the input gross it's only just grows linearly in the transit slice however that family has no algorithm at all okay there is just simply no algorithm that will solve the problem that is solved by the circuit family okay so you you can't like hope to make them equivalent okay the main key here is that the circuit family gets an infinite description you get a different circuit for every input link and here you have a finite description a single element right and


this is just a statement of some undecidable problems in fact many of them are in P poly okay so undecidable meaning there's no our at all that's just technical so finally I want to talk about okay suppose you've got a really complicated algorithm what what can you say about the size of the circuit family solving it so let's suppose I have some album running in exponential time so to the in step so given an in bit input it takes to the in step so like something just solving the trivial rm4 circuit set so I'm just


going through all possibilities through the the inputs and just seeing what happens okay so it's actually open it's an open question whether every exponential time algorithm I'm saying extraordinary amount of time can be solved by circuits with a circuit family with N squared size every circuit has this is this is a really really remarkably open question so is it somehow possible that extremely long time consuming computation can be split up into implementing tiny little you know chunks we're on every single


input length when I solve that problem optimally I actually get only N squared amount of computation this is possible it's actually possible we don't believe it's true but it's still a possibility we cannot rule it out and this is saying in complexity terms that X in ppalli is an open question so exponential time being in poly size there is the open question okay so just to give you just a even more crazy way in which we still don't know how like the power circuit families talk about exponential time versus shallowness some of you


might have attended ms or a workshop on neural net okay this is also we don't want shallow Nets here so this is a very very shallow neural networks so suppose you've got an algorithm and this algorithm is solving a problem that solvable in so-called non-deterministic exponential time so this is like some just gigantic unfathomable class so this is problems where the solution takes exponentially many bits to describe and then verifying the solution is exponentially so this is just some gigantic class of problems ok the


solutions themselves are gigantic and then verifying them takes a gigantic amount of time ok so this is a huge class of problems so it's actually possible this is gigantic class problems every single problem in this class could be again simulated by an infinite family of circuits one for every input length with the following property so this is zooming in ok so every circuit looks like the following very very simple object okay we've got N squared neurons here and so in some layer we've got n inputs so this is the amp circuit so


it's going to take n inputs and output a bit and then from those n squared neurons they're gonna give me a 0 or 1 and they want to take one more neuron some function of those and output 0 1 ok so if each neuron we're just computing some linear form of the input so there's some weights here being some real weights being multiplied to 0 one values we're checking whether that exceeds some particular threshold value let's say T part if it does we we fire it doesn't you know what if it doesn't we fire as you just this very


simple activation function okay so these are neural networks with one hidden layer small neural networks with one hidden layer there should be a very very weak class ok in fact if the weights are really small the weights are like minus 1 1 we do have very strong Louisville but if we allow the weights to be anything we want this is still possible it's a gigantic gap in our knowledge so we don't yet understand very simple neural networks okay just even things like just one hidden layer just one hidden layer with


a simple activation function it's so so I like to just give an analogy here of an alien brain versus implementing for libraries okay so so think of this not as a central time thing is as you got some super advanced you know alien brain no no think of it is like something times 100 that okay something just just some brains and solving problems unfathomable to you okay okay but it's one brain it's just one of them okay and that you want to know if there are any mini little fly brains you know just just one just two layers of depth you


know anslee many of them though which will also solve the problem no matter what this no matter what the the big alien brain okay so this so the only advantage you have is that there are implementing our brains and there's a different for each input length is that enough almost certainly not but it's the implementing that's that trips us up alright okay so now I want to talk and remaining turn about how certain analysis algorithms can actually imply circuit limitations so circuit islands algorithms can imply


cellular balance and particular situations and in a particular sense this was well known a long time ago there's a theorem of car flipping and Meyer from 1980 which says suppose we had extremely efficient circuit now serums like basically perfect sureños arms okay so then there are problems solvable by hours in exponential time say to the end time long enough time that cannot be solved the polynomial size circuit family so this would resolve this X and P poly question we were talking about really okay so just a notation if P equals


empty say circuit side is in P that is a good circuit an office algorithm then we would have it this class X was not in people okay this is a very interesting implication very interesting that that an algorithm analysis could even the other P would be okay that's part not gonna be there but some error analysis can actually prove a lower bound can actually lead to a lower bound at all okay but the thing is we don't believe that hypothesis is true so it's kind of like saying if pigs can fly then pigs can wink I mean like we we don't believe


I bought is this we expect the conclusion to happen okay so so it's it's you know it's it could be useful but it seems to be a limited utility it seems like we have to assume you've got like this amazing like circuit analysis algorithm let's go study that I mean okay maybe extra people not in people ollie happens but okay this is you know by far the more interesting of the of the tooth to me okay so so what we want to do this you know take the wings off this i'll just dude and you know bring him back down to earth and like maybe


maybe he can like you know like a lot no you know something that is actually possible and then you know we can prove something that we expect anyway yeah so a lower a circuit lower about that that's what we would like to do alright and so we were able to do this in particularly restricted situations so this is work of myself and many others who are working on this too and and a very high level what we can say is it a slightly faster algorithm for the circuit set problem just slightly faster than exhaustive search already implies


lower balanced against circuits solving these problems gigantic class non-deterministic exponential time this class were which like it's possible that even neural Nets is one hidden layer could just solve the class so in pictures what we're saying is that suppose you know I'll give you just some arbitrary circuit well and there's some way to inspect this circuit and you can find an input which makes it print one whenever it exists yeah and the one extra thing I want to say is instead of taking say the exponential


cost of two to the end time as you would have exhaustively you take to the end over end of the ten time okay and you do this for all polynomial size circuits so let's just say you do this for every circuit in particular s you shave off some polynomial into the tenth is enough okay for for our purposes suppose you can do that just a tiny sliver off exhaustive search then you'll be able to prove for the same circuit class that there it that it cannot compute sorry in X cannot be so so X is not in people okay so so what's the intuition behind


this the there's two part the basic intuitions what is it faster circuit set algorithms uncover a weakness in and what circuits can do so in particular it says it high-level that small circuits can't offer you skate the all zeros function they can't hide it from you so I suppose this where a black box okay and you can't actually appear inside the circuit and look at what's going on the only thing you can do is take inputs stick them in and get outputs that's why that's the only thing you could do and now you want to solve this a problem you


want to know there's an input which makes the circuit print one right then you can prove that you you would need to call this black box at least to the end times to know for sure by there ways you could trip up any particular strategy for querying this black box getting inputs and outputs without looking in it so that you've got to take to the end time so if we can solve the problem into the end over a ten time then there is some fundamental way in which we are opening up the guts of this circuit and getting some advantage over a black box


and turns out this little advantage this weakness in circuits of the algorithm is uncovered is already enough to start proving the lower bounds alright so that's it that's the first intuition for why is such an implication should even hold the second intuition is that faster circuits set algorithms show a strength of faster than to the N algorithm so they saw some tasks that you didn't think could be done namely find an input which makes the circuit print one it does it faster than to the end so there's some nice algorithm that can


efficiently tell us when a given circuit computes the all zeroes function so so given these two intuitions you can think of this problem in general as some kind of gain between ours and in circuits there's a circuit set problem that algorithms would like to solve and there are circuits that are you know inherently devious and try you know trying to fool any given algorithm and wired in some way to keep the algorithms from telling whether it's and when we can be exhaustive search we are winning the game the algorithm is winning the


game and no matter what circuits given it can somehow drill down through the back black box and and get the answer okay so with the algorithm winning the game we running a lesson to the end time we hope to say circuits or a week less than to the entire albums are strong and somehow we're going to turn this into some function in non interesting exponential time that doesn't have your smallest area so now the circuits algorithm is showing that circuits are weak and this theorem is more less making that formal ok so we can actually


apply this kind of theorem to prove lower bounds for restricted classes of circuits see fancy see ok I won't go into what the definitions of these things are but the the way in which they're proof lays out exactly as you might expect we show that faster circuits at algorithms for this for particular classes can imply circuit lower bounds for that particular class ok so just some generic kind of theorem if see circuits sat on circuits with n inputs can be solved and to the N over in ten steps then this class in X or


some function this big class in X it doesn't have circuits all that kind whatever it is that we prove this circuit settling for okay and the second step is just design the element now this is just an in conditional sort of thing so just we can prove for many interesting certain classes ones will renew no lower bounds whatsoever we can solve the SAP problem faster than exhaustive search and then plugging that into the above connection you get lower bounds for them so you can improve over exhaustive search in some


cases and so far this is kind of the only way we know how to prove certain lower balance is through the design of the right kind of set over all right so I'd like to conclude with the following challenge so how do we become smarter about computation so the outsiders it may seem that we already know maybe too much about computation like computers are getting smarter and smarter than us all the time yet from the theory level we actually don't know precisely how powerful that we are understanding this is really coarse so the dirty secret we


still don't know too much about the limits of computers so how can algorithms help prove lower bounds so this is just in general a direction I would like people to think more about what kinds of algorithms could help to lower bounds because if we can set things up in this way then the lower bound problem does not become so intimidating anymore it just becomes the problem of designing the right kind of our something that we have done as computer scientists for many years and then how can lower bounds help design


algorithms I didn't have time at all here to talk about how that how the connections of that form but there are there are many such connections namely in this thing I briefly mentioned about how lower bounds can imply D randomization removing the roundness from algorithms and making them fully deterministic there are many more connections to be film I'm sure so just earlier this year my student Brin more Chapman and I showed that lower bound circuit lower bounds can in a particular sense be equivalent to a particular


design problem so you prove a lower bound circuit lower bound if and only if you desire test data for testing whether a given function of given circuit computes a particular function okay so trying to minimize the test data you need to test whether a circuit computes of function is actually equivalent to proving a lower bound so this is very nice in the sense that that it gives a very constructive way of thinking about how you prove an impossibility result a very algorithmic way how you do and so in general I think we will make serious


progress by studying alguns and complexity as a whole and not as competing fields for anything we're really doing I really think we're doing a lot of the same thing more than we think okay that's all I have to say thank you sorry can you repeat it again right here oh so like the work of prasada you know yeah I mean so I think what's going on there is they're explaining the fact that I'm at least to me is that improving something is np-hard is actually an algorithms problem and so it's like from my understanding is you


know either something can be done or it can't be done you get and you get a hardness reduction so it's so I mean in one case you get an algorithm or you get a hard introduction I mean in both cases you get some kind of algorithm but but I don't know I mean or any of them unconditional I don't know I need a lower bound you get unconditional yeah yeah I think it's a different kind of instance it's very interesting kind of instance but yeah I think I think it's different from what I'm trying to say but that's I didn't think I thought


you still need unique games are or P not equal NP or something like this so I mean I would like to say something stronger than this so it's not just algorithm or hardness reduction yeah yes yes yes yeah yeah yeah yeah yeah so that's what makes the problem so immensely difficult is that it could just be anything in you know in and the hardest case is where each one is sort of incompressible by itself yeah if they're uniform if there's an algorithm generating those then you know this kind of dichotomy I was talking about becomes


a completely different picture it was a very different picture I mean there's still interesting questions I mean the open questions around it but the picture is very different see I thought it was very careful not to say that I'm not trying a equals not hmm oh so yeah I mean even at the undergrad automata Theory level like when I teach them my hole in the road theorem I say well this means that for every language either it's got a fire automaton it's regular or there's this weird infinite object called distinguishing set and


this proves that there's no final total and so in that case it is some sort of constructive way in which you're showing that something is not regular something does not have a DFA so yeah it does come up elsewhere I just yeah I only have so much time yeah yeah there's a whole community of people that would disagree with I mean that solving is quite an enterprise actually yeah yeah oh well I mean there are things like this so-called exponential time hypothesis is false so then for example counting independent sets if it's in sub


[Music] reading the war between human and artificial intelligence our deep learning systems are beginning to surpass humans sorry [Music] jurgen schmidhuber the swiss AI lab ids IA on the 9th of november 1989 I saw the Berlin Wall fall on TV if you ask me when did you ever have tears in your eyes this is the first event that comes to my mind when I was a boy I wanted to maximize my impact on the world and I was smart enough to realize that I'm not very smart and so it became clear to me that I have to build a

How AI Is Beginning To Surpass Humans | Jürgen Schmidhuber


machine and artificial intelligence that learns to become much smarter than I could ever hope to be such that it can learn to solve all the problems that I cannot solve myself such that I can retire and my first publication on that dates back 30 years today 1987 might promises was about solving the grand problem of AI not just building something that learns a little bit here and a little bit over there but also learns to improve the learning algorithm itself and it learns the way it lands away learns recursively and I'm still


working on the same thing and I'm still saying the same thing and the only difference is that more people are listening because on the way to that goal my team has developed learning methods which are now on 3,000 million smartphones what you see behind me are the logos are the five most valuable companies of the Western world Apple Google Microsoft Amazon Facebook and all of them claim that AI is central to what they are doing and all of them are using heavily the deep learning methods as they are called now that we have developed in our


little labs in Munich and in Switzerland since the early 90s in particular something called the long short-term memory has anybody in this room ever heard of the long short-term memory ends up fel SEM has anybody in this room never heard of the LS TM and okay I see we have a third group in this room who didn't understand the question the lsdm is an artificial neural network which has recurrent connections and it's a little bit inspired by the human brain in your brain you've got about 100 billion little processors and they are


called neurons and each of them is connected to maybe 10,000 other neurons on average and some of these neurons are infant neurons where video is coming in through the cameras and audio is coming into the microphones and tactile information is going in through the pain sensors and some of the neurons are output neurons and they move the finger muscles and speech mushrooms and in between are these hidden neurons we're thinking is taking place and they all connected and each connection has a strength which says how much does this


neuron over here influence this neuron over here at the next time step and in the beginning all these connections are random and the network knows nothing but then over time it learns to improve itself and it learns to do so of all kinds of interesting problems such as driving a car just from examples from training examples and you may not know the lsdm but all of you have it in your pockets on your smartphone because whenever you take out your smartphone and you do the speech recognition and you say okay guru show


me the fastest way to the station then it's recognizing your speech and what's happening there's an lsdm in there which gets about 100 and puts per second from the microphone and they are streaming in memories of past inputs are circling around these these recurrent connections and from many training examples it has learned to adjust these internal connections such that it can recognize what you're saying that's now on 2 billion Android phones it's much better than what Google had before 2015 here is the basic LS TM cell I don't have time to


explain it but here are also the names of the brilliant students in my lab who made that possible how are the big companies using it well speech recognition is just one of many examples if you're on Facebook is anybody on Facebook ok are you sometimes using the translate function where you can translate text from other people yes again whenever you do that you are waking up a long short term memory and lsdm which has learned from scratch to translate sentences into equivalent sentences in different languages and


Facebook is using that a system which has LSD a Metascore for about 4 billion translations per day that's about 50,000 per second and another another 50,000 and the second and another 50,000 if you have an Amazon Alexa it's talking back to you it sounds like a female voice it's not a recording it's an STL SDM which has learned to sound like a female voice to see how much lsdm is per meeting the modern world just look at what all these Google Data Centers are doing now 30% 29% as of 2016 of the awesome computational power for


inference in all these Google Data Centers was used for lsdm the big Asian card companies such as Samsung are also using it and and just a couple of months ago Samsung became the most profitable company in the world for the first time what can be learned from that if you want your company to be among the most profitable profitable ones better.you is lsdm now we started this type of research a long time ago in the early 90 years and by the way you are a large audience by my standards but back then few people were interested in artificial


intelligence and I remember I gave a talk and there was just one single person in the audience a young lady I said young lady it's very embarrassing but apparently today I'm going to give this talk just to you and she said ok but please hurry I am the next speaker [Applause] since then we have greatly profited from the fact that every five years computers are getting ten times cheaper that's an old trend much older than Moore's law and goes back at least to 1941 when here invests a team Conrad Sousa built the first working


program controlled computer and 30 years later for the same price we could do 1 million times as many operations per second because he could do only one operation per second roughly and now it's 75 years later we can do roughly a million billion instructions per second for the same price and it's not clear that this trend is going to break soon because the physical limits are much further out there if this trend doesn't break then within the near future we are going for the first time we are going to have little computational devices that


can compute as much as a human brain we don't have that yet but soon it will be possible if that trend doesn't break them it will take only 50 more years such that for the same price you can compute as much as all 10 billion brains on the planet and there will not be only one little device like that but many many many everything is going to change by 2011 computers were fast enough to allow us for the first time to have superhuman performance at least and limited domains through these deep learning networks back then that


was 2011 so computers were about 20 times more expensive than today today we can do 20 times as much for the same price and and that was already good enough to do superhuman traffic sign recognition which is important for self-driving cars and ten years ago five years ago when computers were about ten times more expensive than today they were already fast enough to make us win these medical imaging competitions what you see behind me is a slice through the female breast tissue and our network which started as


a stupid Network had no idea of anything just learned to recognize cancer by imitating a human doctor a histology and out competing all the other competitors back then soon all of healthcare soon all of medical diagnosis is going to be superhuman it is going to be so good that it's going to be mandatory at some point we can also use Alice TM and things like that to control robots but we don't only have systems that slavishly imitate human teachers know we also have systems that invent their own goals we call that artificial curiosity


artificial creativity systems that like little babies learn to invent their own experiments to figure out how the world functions and what you can do in it and systems that set their own goals are required to become smart because if they don't have the freedom to do that they are not going to become more and more general problems over solving one new self-invented problem after another on the other hand it's hard to predict what they are going to do but you can steer them in the not-so-distant future I guess we will for the first time have AI


on the level of small animals we don't have that yet but it's not going to take so many years once we have that it may need it may require just a few additional decades to reach human level intelligence why because technological evolution is maybe a million times faster than biological evolution because the dead ends are weeded out much faster and it took 3.5 billion years to go from zero from nothing to a monkey but just a few tens of millions of years afterwards to go from the monkey to human level intelligence we have a company that is


trying to make that a reality it's called Nissan's pronounced Nissan's like in English but spelled in a different way and and this company is trying to him to build the first the general-purpose AI that really deserves the name many people think there is this insurmountable wall between today's special purpose a tires which do for example the speech recognition etc and translation and and the universal or a general purpose AI or intelligence of humans but mr. Gorbachev we are going to tear down this one and there is no doubt


in my mind that within not so many decades for the first time we are going to have superhuman decision-makers in many many domains super-smart ARS which are as I told you not just going to be slaves of humans they are going to do their own thing in many ways and and they are going to realize what we have realized a long time ago which is that most resources are not in our thin film of biosphere now they are out there in space so of course they are going to expand out there in space where most of the resources are and through billions


of self-replicating robot factories they are going to colonize the solar system and within a few hundred thousand years they are going to cover the entire galaxy with senders and receivers such that they can travel the way they are traveling in my lab today which is by radio from sender to receiver now nobody knows anything about the details of how all of that is going to happen but it's the only logical thing because you still need resources and torm terms of Matan energy so the only way is to move outwards what's happening now is much


more than another Industrial Revolution this is something that transcends humankind and biology itself a new type of life is going to expand from this little planet in a way where humans cannot follow well that's okay we don't have to believe we are going to stay the crown of creation we don't believe we have to stay the crown of creation but you still can see beauty in being part of something of some grander scheme that goes that leads the universe from less complexity to higher complexity it's a privilege to live at a time when we can


hello everyone today we're looking at bird pre-training of deep bi-directional transformers for language understanding by Jacob Devlin and then why Chung Kenton Lee Cristina Tata Nova these are people from Google AI language so you're about to see the most hyped model currently so basically Bert is a model that takes as an input language sub token sequences and outputs various things so it can it can be made to do various things almost any NLP tasks with basically little training because the Bert model comes pre trained on a very

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding


large corpus and we're gonna see how that's done all right so the paper introduces basically the kind of current state of the art of language models and they say okay what they want to do new is they want to do bi-directional training I'm going to go down here and see their comparison right so here they compare three models and these are representative of three types of models so first here is for example the the open AI transformer so this is a this is the classic or one of the classic transformer models we've talked


about transformers before and the attention is all you need video so what a transformer does is it uses attention and if for those who forgot what attention is if you have like a token sequence ABCDE then a classic model to use that would be an LST M so the other stem would go here it would like have a vector representation a hidden state and then it would take this a it would take this hidden state and compute a new hidden state and then it will go on and take be and incorporate this into the hidden state the hidden state kind of always


stays the same size but the the recurrent model will update the hidden state as it goes over the input sequence so this is one way of dealing with language but people have kind of done another way and that's the attention based mechanism is where basically for each of these you compute a compute a vector independently of each other so each one has a vector representation and then you have a vector representation of what you of what you want which is called an attention head and you can have multiple of these but in the


simplest case let's just say we are looking for the subject in this sentence ABCDE is a sentence and one of the words is the subject of the sentence then we could have a vector here that's called a query rec vector so these are called these are called values V and this is called a query Q and then these vectors are the same size report at this you're gonna compute the inner product with each of these so the inner product you wanna you wanna do okay I already screwed this up you're actually computing two vectors


for each token so honesty but this is this is not too important for this step one is the key and one is the value all right value and this is called the key and you have your query Q and you compute the inner products actually with the key sorry values aren't too important for what I want to demonstrate but you compute key with query all right and that gives you basically for each key it's gonna give you an output and so you're gonna have you're gonna have a for this ABCDE you're gonna have like this much inner product this much inner


product this much this much this much inner product so after maybe the softmax you have like a nice distribution and then you can say AHA here this is the the biggest the biggest alignment of the particular key with my query and that query is which one is the subject of course you're gonna train all these queries and keys producing procedures so this is this is a tension mechanism and if you then want that that's where the value comes in you can if if your queries not only which ones the subject but it's actually generically read that


okay I'm gonna extract some information from some token that I'm going to use later then you would actually take B say aa B is the best one okay I'm gonna take the value of B you're basically gonna get a cab it a weighted average of the values according to these values here right so this is very shortly what attention is if you if you want a lengthy explanation go to the attention is all you need video right so open that I cheap T uses attention here and it's a it's a left to right transformer that's what it says here and what that means is


because also step-by-step but in each step it uses attention so here is the input tokens and as you can see it goes in this direction so each one of these and these are multiple layers of attention so you can also layer these of course so each one of the attention intermediate steps can only attend to whatever is to the left of it right you can see this here so it goes step by step and it goes left to right basically so it can it can kind of take the sequence in as a left to right input basically what that means is whenever