also show the effect of sample size we show the accuracy of the network's chosen by our method for each n so that's the sample size we list the optimal accuracy for sample sizes 10 and hundred and random selection over the whole benchmark so in this case they have the the optimal one which I guess they just draw 10 samples and then take the best one so they train all of them


and then take the best one you can see that already gets you to the 93 and whereas in their case sometimes when they add more they get worse so here they get better but then they get worse again so they comment on this right here we observe that the sample size does not have a large effect on the accuracy of our method but note that as sample size increases our method suffers from a

Neural Architecture Search without Training (Paper Explained)


small amount of noise increasing the gap between our score and the optimal result and of course the key practical benefit is execution time so again they are massively faster than the other methods but to me it seems you could just think of combining these methods right you combine this with this in that what you want to do is actually actively search for the best ones but by doing so you


could if you could pretty quickly weed out the bad ones using this method down here you might already have like a big speed up because again with comparison to this random ones what appears to happen is that the get good at finding you know you're 90% architecture but then they fail to differentiate the top performance performers from each other where you'd really have to train the network to find


out what's you know which one's better so yeah here they say they visualize the trade-off between search time and accuracy for C for 10 for different NES algorithms on dnas benchmark by removing the need for training our method is able to find accurate networks in seconds instead of hours and here you can see the accuracy and here you can see the time and all the the good ones are


either way over here or here and there's is almost at zero while being quite close to the accuracy of the other ones all right yeah that was that was this paper again I think this is pretty valuable if you are especially if you're in a new domain where you might not know what kind of network to build you might just be able to write a little script that generates networks run it through


this algorithm and at least you get an idea of which ones are certainly not worth considering and then you can simply select one of the other ones it doesn't you know often it doesn't need to be the best ones and you can then tweak it a little bit manually the ones you found may be you see some regularity and yeah that was my two cents on this paper I hope you liked it if you did


Translator: Michele Gianella Reviewer: Saeed Hosseinzadeh When I was a boy, I wanted to maximise my impact on the world, and I was smart enough to realise that I am not very smart. And that I have to build a machine that learns to become much smarter than myself, such that it can solve all the problems that I cannot solve myself, and I can retire. And my first publication on that dates back 30 years: 1987.

True Artificial Intelligence will change everything | Juergen Schmidhuber | TEDxLakeComo


My diploma thesis, where I already try to solve the grand problem of AI, not only build a machine that learns a little bit here, learns a little bit there, but also learns to improve the learning algorithm itself. And the way it learns, the way it learns, and so on recursively, without any limits except the limits of logics and physics. And, I'm still working on the same old thing,


and I'm still pretty much saying the same thing, except that now more people are listening. Because the learning algorithms that we have developed on the way to this goal, they are now on 3.000 million smartphones. And all of you have them in your pockets. What you see here are the five most valuable companies of the Western world: Apple, Google, Facebook, Microsoft and Amazon.


And all of them are emphasising that AI, artificial intelligence, is central to what they are doing. And all of them are using heavily the deep learning methods that my team has developed since the early nineties, in Munich and in Switzerland. Especially something which is called: "the long short-term memory". Has anybody in this room ever heard of the long short-term memory,


or the LSTM? Hands up, anybody ever heard of that? Okay. Has anybody never heard of the LSTM? Okay. I see we have a third group in this room: [those] who didn't understand the question. (Laughter) The LSTM is a little bit like your brain: it's an artificial neural network which also has neurons, and in your brain, you've got about 100 billion neurons. And each of them is connected


to roughly 10,000 other neurons on average, Which means that you have got a million billion connections. And each of these connections has a "strength" which says how much does this neuron over here influence that one over there at the next time step. And in the beginning, all these connections are random and the system knows nothing; but then, through a smart learning algorithm,


it learns from lots of examples to translate the incoming data, such as video through the cameras, or audio through the microphones, or pain signals through the pain sensors. It learns to translate that into output actions, because some of these neurons are output neurons, that control speech muscles and finger muscles. And only through experience, it can learn to solve all kinds of interesting problems,


such as driving a car or do the speech recognition on your smartphone. Because whenever you take out your smartphone, an Android phone, for example, and you speak to it, and you say: "Ok Google, show me the shortest way to Milano." Then it understands your speech. Because there is a LSTM in there which has learned to understand speech. Every ten milliseconds, 100 times a second,


new inputs are coming from the microphone, and then are translated, after thinking, into letters which are then questioned to the search engine. And it has learned to do that by listening to lots of speech from women, from men, all kinds of people. And that's how, since 2015, Google speech recognition is now much better than it used to be. The basic LSTM cell looks like that:


I don't have the time to explain that, but at least I can list the names of the brilliant students in my lab who made that possible. And what are the big companies doing with that? Well, speech recognition is only one example; if you are on Facebook - is anybody on Facebook? Are you sometimes clicking at the translate button? because somebody sent you something in a foreign language


and then you can translate it. Is anybody doing that? Yeah. Whenever you do that, you are waking up, again, a long short term memory, an LSTM, which has learned to translate text in one language into translated text. And Facebook is doing that four billion times a day, so every second 50,000 sentences are being translated by an LSTM working for Facebook; and another 50,000 in the second; then another 50,000.


And to see how much this thing is now permitting the modern world, just note that almost 30 percent of the awesome computational power for inference and all these Google Data Centers, all these data centers of Google, all over the world, is used for LSTM. Almost 30 percent. If you have an Amazon Echo, you can ask a question and it answers you. And the voice that you hear it's not a recording;


it's an LSTM network which has learned from training examples to sound like a female voice. If you have an iPhone, and you're using the quick type, it's trying to predict what you want to do next given all the previous context of what you did so far. Again, that's an LSTM which has learned to do that, so it's on a billion iPhones. You are a large audience, by my standards:


but when we started this work, decades ago, in the early '90s, only few people were interested in that, because computers were so slow and you couldn't do so much with it. And I remember I gave a talk at a conference, and there was just one single person in the audience, a young lady. I said, young lady, it's very embarrassing, but apparently today I'm going to give this talk just to you.


And she said, "OK, but please hurry: I am the next speaker!" (Laughter) Since then, we have greatly profited from the fact that every five years computers are getting ten times cheaper, which is an old trend that has held since 1941 at least. Since this man, Konrad Zuse, built the first working program controlled computer in Berlin and he could do, roughly, one operation per second.


One! And then ten years later, for the same price, one could do 100 operations: 30 years later, 1 million operations for the same price; and today, after 75 years, we can do a million billion times as much for the same price. And the trend is not about to stop, because the physical limits are much further out there. Rather soon, and not so many years or decades,


we will for the first time have little computational devices that can compute as much as a human brain; and that's a trend that doesn't break. 50 years later, there will be a little computational device, for the same price, that can compute as much as all 10 billion human brains taken together. and there will not only be one, of those devices, but many many many.


Everything is going to change. Already in 2011, computers were fast enough such that our deep learning methods for the first time could achieve a superhuman pattern-recognition result. It was the first superhuman result in the history of computer vision. And back then, computers were 20 times more expensive than today. So today, for the same price, we can do 20 times as much.


And just five years ago, when computers were 10 times more expensive than today, we already could win, for the first time, medical imaging competitions. What you see behind me is a slice through the female breast and the tissue that you see there has all kinds of cells; and normally you need a trained doctor, a trained histologist who is able to detect the dangerous cancer cells,


or pre-cancer cells. Now, our stupid network knows nothing about cancer, knows nothing about vision. It knows nothing in the beginning: but we can train it to imitate the human teacher, the doctor. And it became as good, or better, than the best competitors. And very soon, all of medical diagnosis is going to be superhuman. And it's going to be mandatory, because it's going to be so much better than the doctors.


After this, all kinds of medical imaging startups were founded focusing just on this, because it's so important. We can also use LSTM to train robots. One important thing I want to say is, that we not only have systems that slavishly imitate what humans show them; no, we also have AIs that set themselves their own goals. And like little babies, invent their own experiment


to explore the world and to figure out what you can do in the world. Without a teacher. And becoming more and more general problem solvers in the process, by learning new skills on top of old skills. And this is going to scale: we call that "Artificial Curiosity". Or a recent buzzword is "power plane". Learning to become a more and more general problem solvers


by learning to invent, like a scientist, one new interesting goal after another. And it's going to scale. And I think, in not so many years from now, for the first time, we are going to have an animal-like AI - we don't have that yet. On the level of a little crow, which already can learn to use tools, for example, or a little monkey. And once we have that, it may take just a few decades


to do the final step towards human level intelligence. Because technological evolution is about a million times faster than biological evolution, and biological evolution needed 3.5 billion years to evolve a monkey from scratch. But then, it took just a few tens of millions of years afterwards to evolve human level intelligence. We have a company which is called Nnaisense


like birth in [French], "Naissance", but spelled in a different way, which is trying to make this a reality and build the first true general-purpose AI. At the moment, almost all research in AI is very human centric, and it's all about making human lives longer and healthier and easier and making humans more addicted to their smartphones. But in the long run, AIs are going to - especially the smart ones -


are going to set themselves their own goals. And I have no doubt, in my mind, that they are going to become much smarter than we are. And what are they going to do? Of course they are going to realize what we have realized a long time ago; namely, that most of the resources, in the solar system or in general, are not in our little biosphere. They are out there in space.


And so, of course, they are going to emigrate. And of course they are going to use trillions of self-replicating robot factories to expand in form of a growing AI bubble which within a few hundred thousand years is going to cover the entire galaxy by senders and receivers such that AIs can travel the way they are already traveling in my lab: by radio, from sender to receiver.


Wireless. So what we are witnessing now is much more than just another Industrial Revolution. This is something that transcends humankind, and even life itself. The last time something so important has happened was maybe 3.5 billion years ago, when life was invented. A new type of life is going to emerge from our little planet and it's going to colonize and transform the entire universe.


The universe is still young: it's only 13.8 billion years old, it's going to become much older than that, many times older than that. So there's plenty of time to reach all of it, or all of the visible parts, totally within the limits of light speed and physics. A new type of life is going to make the universe intelligent. Now, of course, we are not going to remain the crown of creation, of course not.


hi there today we're looking at Excel net generalized auto regressive pre-training for language understanding by Jill and yang and other people from Carnegie Mellon University as well as Google brain so this is kind of a the elephant in the room currently as Excel net is the first model to beat Bert which was the previous state of the art and a lot of NLP tasks to be burped at a

XLNet: Generalized Autoregressive Pretraining for Language Understanding


lot of these same NLP tasks so the chief state of the art result on 18 of of 20 tasks I believe maybe they test one they outperform Bert on 20 the chief state of the art on 18 including things asked question answering natural language inference sentiment analysis and so on so those are kind of remarkable results and even more remarkable is that the architecture of the network is actually


very very similar to Bert the kind of new introduction is a a pre-training a different free training procedure and we'll look into that so let's actually jump into their main points straight away what they go into is there are two kinds of currently used pre training methods for these NLP test and both or can be understood as kind of a language modeling one so language modeling for those you don't


know is predict the next word in a sequence so if I give you the sequence here unsupervised representation learning has been and then ask you what's next and then you're supposed to say highly right those language modeling in in a nutshell so what they what they differentiate are two kinds of language modeling the first one they say is order aggressive language modeling


now what auto regressive language modeling does is exactly what we've looked at I give you unsupervised learning has been you were supposed to predict highly and then in the next step I give you unsupervised representation learning has been highly and you're supposed to predict success and so on so in the next step I'm gonna give you the entire sentence up until


here and you're supposed to do predict in autoregressive because each token can look at the kind of previous ones in the in the sequence so when you sorry you can't see that when you predict when you predict you you can always kind of order aggressively look at what the previous ones were when including what you've previously predicted of course during training this is a teacher first as I


said so you put the actual words there this is auto regressive modeling in contrast to what they call Auto encoding and Auto encoding is what birth does and this is the following so in contrast to that let's say I have the same sequence unsupervised representation learning has been highly successful in the domain of yeah something and then I say okay I give you the sequence but I am going to


delete this and this right and now I ask you to predict these two right so you can see the the task is slightly different as you now have access to all of the sequence basically except the ones that you are asked to predict but you're you kind of asked to predict yet them not in any order but you're asked to predict them at the same time basically so at the same time you're


you're asked to predict this word and this word and so the first kind of this Auto regressive language modeling has been used by transformer models until birth and then basically bert really pushed this auto encoding language model pre-training which made it so successful and now this paper excel net wants to like combine the best of both of them and in order to understand what's the


best of both of them so what's good at birth we've already seen it can actually draw information from all of the context of the words it's trying to predict but what is the kind of pitfall of birth and they they actually put this really nicely in an example they give way further down where they say comparison to but I don't know why that is not like also in the introduction but here they


have the sentence New York is a city right New York is a city this one and you're asked to predict these two words and if you now compare birth to what xl9 does if so the context is is a city and you're asked to predict New York what birth does is it simply masks out the two words and says here please fill in these two words now this translates to the kind of objective being separated in


the two words such that the prediction of York here is completely independent of the prediction of new so if you know of any other city that is made of two words for example San Francisco or Los Angeles then these would be as valid and any mixture would be as valid so you might there might end up with laws York is a city and that would be perfectly fine for birth because while it's


predicting loss is a perfectly fine prediction for the first word of a two-word City and York is a perfectly fine prediction for the last word of a two-word City right so these are the kind of mistakes that bird can get into by not being order aggressive by basically predicting all of these tokens at the same time independently of each other whereas x-l net what they will do


is we specify an order let's say okay first I will predict the word noon for the first word new something is a city and then when I predict York I will actually take into account the I previously have predicted the word new so um that's the main advantage at that autoregressive training has over Auto encoding now what are the pitfalls the pitfalls or if you have this this


sentence let's look at it I'll write it down New York is a city right if you have this sentence and let's say yeah actually you're not you're not asked to predict you and your crew you're asked to predict the word a here a right you're asked to predict that in order regressive style or a city it's a better example the two words I said in order regressive style if you predict the word


a you can only ever look at what comes before hand whereas if Bert were to predict a just the word a it would be able to look at all of it that's not predict City so you see the kind of auto regressive model is bound to the order of the of the factorization of the sentence that's right it's bound to the order in which it has to predict the tokens so here if it's predicting a


you can only look at stuff that comes before it because it needs to do it in order right once it gets to city you can actually look at the entire sentence here but um before that it only ever has partial information about the about the context so actually it wouldn't be much better if I had said we're trying to predict these two words is and a right and once I predict so so Bert would


actually have access to the word City here whereas the auto regressive models only have access to the ones before it I hope that makes it clearer so the main idea in excel net is where did where does this order dependence come in the autoregressive model the order dependence actually comes from the factorization of the sentence of the of the language model so in a language


model we're actually trying to assess the probability distribution of sentences here X is a sentence right and this can be naturally factorized into a product over the words where the probability of each word is only dependent on the words before it so this is a this is an equality is not an approximation this is an equality the probability of a sequence can be


decomposed into a product of probabilities like this exactly so this here is exactly what these auto regressive models implement each word is predicted from the words before it right there are other kinds of autoregressive models that also do the other direction where here they say okay the probability of a sentence is a product and each word is predicted from the words after it but


it kind of is the same problem you only ever have access into the one direction basically however you define the order of decoding you only ever have access from a given word to what was before it in the order so the main idea of excel net is they say hey why don't we consider all possible orderings right I mean that that's kind of a that's it's an idea so let's go back to our thing


here they say why don't we consider all possible orderings so basically what we will do is if this sample comes up New York is a city all right what I can do is I can define an ordering let's say I always want to predict two words so the bird typically masks out about 15% of its input to be predicted and here let's say we'll mask out 20% which two words so of this sequence will mask


two words and ask the model to predict it that that will be our our pre training objective the first time this sample comes up from the data set I might specify the order just classically right just one two three four five all right I'll predict the last two words I'll kind of mask them out right I give the model New York is and then I could let it predict a and


then in the next step I'll give it New York is a and let it predict City cool so now if the pitfall is the word a here only has access to things before it and not to city itself City has access to everything all right so but then I continue training and the next set time this sample right it's in my data set New York is the city the next time it comes up I simply go for a different


order let's say one two three four five right so now again I'm asked and asking to predict the last two tokens which here our city and York so in the first step I would give it is a and new and I will ask it what's here and I'll ask you to predict city and then in the second step I'll also give it that and I'll ask it okay now what's here given all of that right so new is a city all right


you're asked to predict the missing word so that that's pretty so the first step its new is a hmm and you Resta predicted the second and then the second step is new is the city and the rescue protect you first so now as you can see while predicting City here all of a sudden we didn't no longer in this ordering we don't have access to the world York so we'll have to learn to predict


City from the rest of the context now even more even more if we now decide decide on a different ordering again one three four five so now well actually first step is to ask New York City please predict this thing here all right yeah you might train the model to predict is and then the second step you say New York is City please predict it now we see before before when we are at were asked to


predict the word a it only had access to things to the left of it and the very first example but now it actually has access to the entire context so the the the idea is as we sample this data point multiple times and each time we decide on a different ordering duty code for each the prediction of each token token sorry will actually have seen many many parts many different variants of the


context and in expectation will actually have seen all of the context just like Bert but will always having have done it in an order aggressive way so basically you get all the advantages of being order aggressive namely that you are able to decode step by step while always referring to everything in front of you in the ordering so the predictions are not independent but you also get the


benefit of Bert that it's able to basically look at all of the rest of the context in expectation in order to make this prediction so this is this is the main idea of of excel net they formalize this jump up again they formalize it in saying okay what Bert does here is it actually seek it factorized law probability of a sentence into this sum so the product in the law becomes sum


into the sum of log probabilities of no sorry this is auto regressive confused ah into the words conditioned on everything in front of them what bird does is it actually approximately factorizes the law of probability into each word and then everything in the context everything that's not masked in the context and this is only an approximate factorization because you're


basically dropping away all these mask tokens and um what they do now is they do the same as the AR as their auto regressive models here they decompose to log probability into a sum of log from abilities over each of the words given all the words before it but now not before it in the sequence but before it in and chosen permutations Z and Z is sampled uniformly from the set of all


possible permutations so in expectation they'll see all of the context so this is the this is the main thing they show this here in a kind of a picture with so here is the neural network this is the input layer then these are the hidden layers as the attention layers go up and up here you're asked to predict the the token so here you're always asked to predict X 3 so there is no there's never


going to be any awake here since if you knew X 3 you would be able trivially to predict X 3 all right so in the in the first example the factorization order chosen at random is 3 2 4 1 now you asked to predict X 3 and we know okay we should only we should only do this with things that are before it in the permutation order well here are since X 3 is the first in the permutation order


we actually don't we don't have anything to go on we wait Stickley asked to predict x3 from scratch as if it were the start of a sentence so we'll basically tell the model I have a sentence that goes please predict the third right it's a hard task yeah by the way you're always able to look at this memory thing here don't worry about this for now this is just this is a an augmentation they do


on top of their idea this is not the core idea so okay but now the second time this sample comes up from the training set we decide on a different order so the order here is 2 4 3 1 now again we're asked to predict x3 and we're allowed to look at everything before it so 2 & 4 as you see here there are weights from x2 and x4 into this column that finally is then 8 asked to


predict x3 so this is also this is now an easier task right you're allowed to look at the word to the left and to the right if you have the following permutation order 1 4 2 3 you're actually allowed to look at all of the other words because x3 is at the end of the permutation order in order to produce x-ray so all of these four and the fourth thing is a similar so all of


these four things will appear during training and you will learn from them so in expectation C you will basically have seen all variants of different of different versions of the context which which helps a lot apparently right so in the in order to achieve this they had to make some architectural changes to the the model namely what you want to do is in a single pass through the model here


you not only want to predict one token but you want to do many predictions this helps training a lot so vert and naturally always does like the 15 we must add 15% of the tokens or so what was that like 40 50 tokens so it masks them and it predicts them all at the same time now you would like to do this here as well you would like to predict all at the same time the ones that you're asked


to predict but of course the problem is for here if you're asked if in this factorization order 2 4 3 1 if you're asked to predict X 3 you're allowed to look at X 2 and X 4 if you're asked to predict X 1 you're allowed to look at X 2 X 4 and X 3 so if you only have a single pass through the model the question is do you now input X 3 or do you not because the prediction of X 3 is


not allowed to look at X 3 while the prediction of X 1 is allowed to look at X 3 so they do an architectural change in order to achieve both things so that you can do have a single pass through the walk through the model but the prediction of each token only depends on the things in front of it in the permutation order and they do this by having these kind of two stream


is masked to stream attention where they basically have not only not one hidden representation like in classic transformers but they have at each step two hidden representations one they call H only called G so the HS are initialized with the embeddings of the tokens and the g's are just initialized randomly and then they get transformed and the point is the h of the next layer


is always able to look at everything in front of it including its own its own H basically it's the one layer down its own position one layer down while the G is only allowed to look at the a it's allowed to look at the ages but the H is from before right so so all the G's here are only ever able to look at the H is from before the current position whereas the H is always allowed here to look at


the same but also at the H at the current position and now at the last layer you simply ask the model to predict the token from just the G and you can easily see that this results in this model only oh yeah only attending to things before it okay the G by the way can also look at the G of the current layer so that's that's also nothing but it cannot look at that


at the age so there's never any information flowing from the current from the current word embedding of the token you're trying to predict to the prediction layer so basically that that means the model can't just look like you you're not telling the model the answer yet you're still able to feed to predict multiple things in a single pass through the model formally this is described


here in the attention layer so they divide how they produce the queries and how they produce the keys and values usually the queries and the keys and values are produced from the same hidden representation but here they produce the keys and values from the h's in both cases but to update the G's they produce the queries from the last layers G and do produce HS they produce the queries


from the last layer HS and most importantly when they produce the keys and values the H is they'll look at here to update the G you're only allowed to look at H is before you in the permutation order but to update the H you're allowed to look at everything before including the position you're currently after so that's kind of the that's a it's an engineering solution to


the problem introduced by their augmentation I think it's a pretty pretty neat solution pretty cool so the rest of the paper here is incorporating ideas from transformer Excel so transformer Excel is one of these classic transformers that that is like this AR so this Auto regressive style of transformer but that has a few improvements over the classic vanilla


transformer and they incorporate a number of things here namely first of all they incorporate this memory thing so the memory thing allows you to input longer sequences let's say our our transformer input length is maximum of five tokens what the transformer Excel allows you to do is you input five tokens and then you save you do your transformer thing you encode it and they


save something into this memory and then when you input the next five tokens your transformer is then allowed to look at the memory of the last sequence right and also update it so that that that's kind of these this memo oxy Sawyer so you're always allowed to look at these men blocks from last sequence and then the hidden representations here of this sequence they will actually be stored in


the member lok for the next sequence this is kind of a trick to carry over information it's not the deep updating the memory part isn't learned with the objective to make the next prediction better but it's just some information it's a kind of gradient free information to provide to the next step and it apparently helps you can incorporate longer sequences into this transformer


excel so they take this over and implement this into excel net they also do relative position encodings relative segment and codings I won't go into this too much more here because it's not the main idea basically so they do experiments and they compared to birth architecture with the same basically same architecture the same number of parameters and/or the years and they beat Burt in all of these


kind of NLP tasks or most of I think they said in 20 they reach new state of the art in 18 NLP tasks so apparently their method works very well so what they do here is a last thing I find important is an ablation study of the effects of their improvements so they wear because kind of my problem is I never know like they have this new idea okay we do these random permutations but


then they also say oh and also we include memory from XL net and we do relative positioning coatings and so on to me these kind of papers of course you reach better numbers you get a new state of the art so it's kind of a landmark paper but to me a paper should more be like a single thing so whatever your idea is this your idea is these or drinks and whatever you need to do to


make that work okay fine but then why why the additional transformer Excel things it's it's really then hard to estimate how much of the improvement comes from your ID and how much of the improvement simply comes from the fact that you already put these other things actually have nothing to do with it so I appreciate these kind of analyses called ablation studies where they kind of try


to take away the memory and these things and kind of look at what it's doing to the model and use you see here kind of degrades down here as for example this con degrades as you take stuff away while still being more kind of more successful than Burt so that that I would say also yeah here is more unclear but also kind of seems to degrade a bit and while being more successful than


Bert I appreciate this there's some and of really trying to show that your gains really come from your new idea and not from some other stuff all right so the last thing I want to mention actually is this thing so someone claiming or calculating that it costs two hundred and forty five thousand dollars to train the Excel net model the way they describe it in the paper I'm sure that's


gonna be brought down because it was brought down that like the time the train was brought down with Bert as well but this is just I mean this is crazy this is just training it um it it kind of gives large questions about the state of research and the ability for kind of let's say more academic players to participate in research on the one hand of course we like of course these


machine learning it's a buzzword but I would also claim it's a lot more than just a buzzword how many have experience with machine learning it's a small portion area it's great to see when I started learning machine learning I found it difficult to understand how the different algorithm worked and what the main difference between them was I find it really difficult understand where I

Machine learning algorithms, choosing the correct algorithm for your problem - Joakim Lehn


should start to learn machine learning but in this lightning speech you hopefully will learn fundamental difference between different categories of algorithms this will helpful both beginners with also those who there who have a little experience with machine learning all right so real quick about myself my name is Joakim Lyon I'm a consultant here in Oslo for the Nordic consulting firm


knowit's and I've been fascinated with machine learning for the last couple of years doing some project both personally and professionally and I must say there are many different algorithms out there just take a look at this this is a small portion of really great algorithms so where they begin obviously some algorithms are more suited for certain problems that others and the result will


vary greatly of how well your algorithm is suited for a problem but luckily you can divide these algorithms into four different categories of machine learning so the four different categories is supervised learning and supervised learning semi-supervised learning and reinforcement learning and when you face a machine learning problem it is important to understand which category


it fits into so today we're going to explore these four different categories before we get through really good stuff I need to explain two key words there in machine learning you have something called features this is basically a property of your training data and a label is the output you get from your model after training it so you could say features input levels output it's


partially true because you can also have labels on your input data I'm gonna explain that with an example let's say you want the machine learning algorithm to estimate the height of a person based on age and gender then age and gender are features and the height you want to find is the label and if you have a training set with a lot of people with their height


corresponding to age and gender then you have a labeled training set so the first category is called supervised learning and in supervised learning you have training data that consists of a set of training examples you have a labeled training set and the basic idea is to find the most optimal model parameters to predict unknown labels on other objects let's look at a few examples


let's say I want to estimate the value of a used car based on age mark the mileage you name it then a machine learning algorithm can do this pretty well if you give it a training set with a lot of Sol cars with the corresponding value another example could be a mail it spammers it not spam a machine learning algorithm can do this if it has a large training sets a great


algorithm within the supervised domain is called decision trees the reason I picked this one is because it's more intuitive than most others so in decision trees you have nodes and in every node you choose the best split between a lot of features and you make this procedure recursively until we finish with the stopping criteria again I'm gonna illustrate this with an


example let's say you want to find out we should accept a new job offer the first thought might be well how much is the salary is it above some threshold if it's not you're definitely not gonna take the job but if it is then do you have to commute for a long while do they are free coffee we'd have a food table I don't know you might ask yourself questions like this but at some point


you can end up and of stopping leaf and I'm gonna either accept or decline the job offer so that was supervised learning next category is called unsupervised learning and then unsupervised learning you only have input data and no corresponding output variables you have no labels in your training sets and the goal phone supports learning is to model the underlying structure or distribution in


the data in order to learn more about the data so the algorithms are left on their own to discover and present the interesting structures in the data let's look at a couple examples let's see you want to group customers by their purchasing behavior people buy item 810 to buy item B then obviously you should recommend these items to people interested in one of them


another example could be in Netflix Netflix video recommendation system it recommends TV series movies and whatnot and they do this by using a series of unsupervised learning algorithms a great algorithm here is called k-means not for Netflix maybe but for other purposes and here we try to divide all the data into K clusters and you select random K points in new clusters and in the


Centers and the cluster of other objects are defined by the closest cluster Center you tune this algorithm by selecting the K number of clusters you can use this algorithm for many things let's say you have not hotel chain anyone to open a new hotel in a city where do you place your hotel hopefully you start off by looking at potential sites gather a lot of data


like we see close to downtown the restaurants nearby is it easy to get to the hotel and so on and maybe hopefully from all this data and algorithm can find clusters in this data to show interesting spots for your tab so most unsupervised learning third category of problems falls between unsupervised and supervised problems and it's called semi-supervised learning


here we have partially labeled data and many real problems fall into this category because it can be really expensive or at least time-consuming to try to label all the data let's save a million pictures and you're gonna label them takes too much time unlabeled data however is cheap and usually easy to collect and store so here a mixture of techniques from the


unsupervised unsupervised domain can be used an example here could be as I already mentioned if you have a photo archive you might label some of the images like there's a cat in this picture people skiing topless person on the beach I don't know and from this label pictures you have a lot of unlabeled pictures and you can try to label those with an algorithm so


the last category of problems fall into reinforcement learning and it's not like any of the supervised learning algorithms because you don't have any label data and you don't have any unlabeled data usually you don't have any training data the idea is to create a software agent and it's got some states and it's gonna perform some action in environments the environment is gonna either punish


it or reward it somehow and they can end up in new state and you do this recursively and you can imagine this by saying you're a robot and you wake up and in a strange place you can perform activities and you're gonna get rewards from the environment so after you get more rewards you get more clever and your actions get more complex and you're training to behave the most effective


way on each step so this is kind of a human way to learn and human way to think and we made some incredible progress within the reinforcement domain last years as first speaker mentioned alphago was a great great example here they managed to be the best player in the Google ingo and the reason why I bring this up is because it made some moves that humanity has never seen


before and they now teach some of the moves it did during the game at go schools in China they have go schools in China which is surprising and I find it really interesting that humans can now learn from machine and not just the other way around another really cool example I think is from pretty recent time from open AI they managed to create an AI that could beat some of the best


players in the world in dota 2 and dota 2 is a real-time game so the world was quite shocked to see that this happen already we fought this was the years and years until it could happen because it's vastly more complex than traditional board games like chess and this is my personal dream project I'm really hoping I can beat myself but by creating an AI that can beat me in Mario Kart and not


to brag but I'm quite good at Mario Kart so I'm not sure if I'm program skills is cute enough we'll see so hopefully the last 10 minutes you learned what's here in supervised learning all there is labeled and the algorithm learns to predict the output from the input data when unsupervised learning all there is unlabeled and the algorithm learns the inherent structure from the input data


semi-supervised learning have some data labeled some unlabeled mostly unlabeled and a mixture of supervised and unsupervised techniques can be used reinforcement learning is an area of machine learning concerned with how a soft region out to take action in environment so as to maximize some notion of cumulative reward so as I said in the beginning machine learning might


well uh hello and welcome to this uh richard m carp distinguished lecture uh my name is peter barton i'm the associate director of the simons institute for the theory of computing uh thanks for joining us we established the richard m cup series to celebrate the role of simon's institute founding director dick carp in establishing the field of theoretical computer science

Insights on Gradient-Based Algorithms in High-Dimensional Learning


formulating central problems and contributing amazing results in the areas of computational complexity and algorithms the series features visionary leaders in tcs and is geared towards a broad scientific audience we're grateful to the many contributors to the richard m carp fund who've made this series possible so i'm delighted to welcome our speaker today lenka stevarova


lenka is a researcher at cnrs working in the institute of theoretical physics in cea uh paris clay she has a background in physics and is famous for the application of methods of statistical physics to problems in machine learning and signal processing in inference and optimization link is the recipient of the cnrs bronze medal in 2014 the philippe meyer prize in theoretical


physics in 2016 and the irene julio curie prize in 2018. the talk today is entitled insights on gradient-based algorithms in high-dimensional learning so please join me in welcoming lenka's devarova thank you peter and i will share my screen so that you see the slides i prefer it and i'm really really honored to be giving this lecture especially given the influence that you know being part


of one of the programs at simon's institute four years ago it had on my career and i enjoyed it so immensely and it's amazing what the simuls institute is doing so first thing i should do is to correct my affiliation so it's only a second seminar i'm giving and a third week i'm spending at my new affiliation that is epfl so not anymore in france but in a neighboring country


switzerland and i will be telling you about work that i you know i have recently did a lecture in this simon's institute bootcamp for the program of this semester where kind of a lot of the works that seemed like a statistical physics voodoo maybe 20 years ago actually have been established rigorously and part of the program is about it and it's pretty and it's


very exciting so for this very special lecture i decided to go back to results from physics where most of it is not established vigorously and is waiting for the mathematical inputs and works and and that's something that was going on in the past two years with the list of collaborators that i give here the the main among them are the two students highlighted in


in blue stefano mannelly sarah and francesca and stefano is among the panelists so if you have some clarification questions or questions you can he's able to answer them even during the talk without interrupting it so please don't hesitate so this is the list of six papers from the past two years on which this talk is based and the talk will be about gradient


descent based algorithms or stochastic gratities and base algorithms that you know pictorially are the workhorse of machine learning that is really everywhere these days so they are really worth understanding and studying in in more detail in particular in deep learning we have the empirical observation that local or even global minima with bad generalization error actually


do exist there are many kind of works uh going towards showing like something like that empirically one of them that i like quite a bit is this this paper by dimitry akioptas and his collaborators where he starts by interpolating and fitting random labels in the neural network and then he puts back the real labels little by little and he shows that the gradient descent


actually doesn't go that far away from the point where it interpolated random labels and it generalizes pretty bad much worse than it would if you just initialize it randomly so that really tells us something notable about how this optimization landscape looks like and we really need to understand how comes that the gradient-based algorithms initialize randomly


are able to avoid the bad minimum and so the goal here you know it's it's pretty much clear these days that this will this cannot happen just by studying the landscape that it really what matters is you know even the initialization so what matters is the whole trajectory that the algorithm is taking so we want to understand the trajectory and these non-convex high dimensional


problems and just two points to make to set the talk you know in practice the number of samples is limited so i don't want to be working in some limit where the number of samples is is unreasonably large and also constants do matter so i don't want to be talking working with with some rates without log factors and with arbitrary constants so in order to be able to do something


like that to keep in mind finite sample complexity and constants i need to make some simplification somewhere so for the purpose of you know the work that i'm describing in this talk this will be on the side of the data so i will not be assuming any kind of really generic data set i will be working with synthetic models for data for which we can say something


so the first such model on which will be the first say 20 minutes of the talk will be the spiked matrix tensor model which is the optimization you can think of optimizing the loss function that is written here it has two parts one so so the variable over which you are optimizing is the x that is living on an n-dimensional sphere and n will be large that will be the limit we will be


interested in high dimensional and then the way the loss function depends on the x is through the matrix y that is created from some ground through x star plus a load of noise so x star x star transpose more precisely and it also depends on this tensor t that has order p and is created by taking an outer product p times of the same vector x star and adding a lot of noise and then the goal


of the you know interference problem here is to find back the vector x star by minimizing the loss function written over here so why this model so that also kind of sets again what what i'm aiming to to achieve so this small because it's high dimensional and on convex that's kind of what makes a study of gradient descent non-trivial it's an interference problem meaning


that what we are interested in is a correlation with the ground true signal x star we are not really interested in the optimization problem per sec so this is similar to the machine learning with neural networks where we always solve it by optimization but we are really interested in the generalization error in something slightly different than the value of the


loss function itself and the third and fourth point is that this model has interesting computational properties and the dynamics of the gradient is unsolvable and this is something that i will show you to persuade you of that so the statistical physics must come at some point in and this is where it does so you just rewrite the same model with the noises of the with the variances of the two


gaussian noises that i considered just rescaled a little differently the way we usually do in physics and then i take this loss function that was sum of two squares and developed the squares and realized that some terms just don't depend on the x over which i'm optimizing and some terms depend on it in a trivial way if the x lives on the sphere they're just equal


to some constant so the only non-trivial term that matters is the term here i called h of x that if i look back in statistical physics is exactly the hamiltonian of something that is called the spherical mix b spin glass so those in the audience that know about spin glasses have seen this smile because that's the one one of those that is most often studied in the field of


statistical physics of disordered systems and so what we will be so we will be using that but what we'll be interested in is at one hand the you know when i say gradient-based algorithms i will be speaking at this in this part about mainly two one of them will be the launcher algorithm with the aim of actually estimating the ground through x star in a base optimal way which would


be done by writing the posterior measure and computing its marginals and this corresponds to writing the boltzmann measure of the corresponding statistical physics problems and sampling sampling it at temperature one and that's exactly what the longevity algorithm aims at and the second estimator i will be looking at is the kind of more common one maximum likelihood estimator that is


computing the minimizer of that loss function of the or the ground state of the statistical physics model and that's what the gradient descent or flow aims at so just to get a little bit more familiar with this model so if you listened to the bootcamp lecture you would you know we told you about set of tools that you can use to actually describe what's happening


in a model such as this one from the point of view of information theory what is possible statistically and from the point of view of approximate message passing algorithm which ends up to be the best we know for this type of problem and this phase diagram summarizes of what's going on so i will just explain it and then in this talk we are interested what the


gradient descent is doing so that will be the new part so on the axis here we have the variances of the noises delta 2 is the noise added to the matrix and delta p is the noise added to the tensor so the bigger the noise the harder this inference problem will be and for instance if the delta p was infinity that would be effectively as if the tensor was not there


it's not giving you any information so in that case you are in the case of spiked matrix factorization that is problem widely studied in statistics and you know it has the bbp phase transition and that's precisely what the value lambda 2 equal to 1 corresponds to so that's what distinguishes the phase where the spike is impossible to recover from the easy phase where if you


only had the matrix you recover the spike simply by spectral methods looking at the spectrum of the matrix then if the matrix was not there that is that delta 2 would be infinity 1 over delta 2 would be 0 then you only have the spiked tensor model which you know information theoretically is solvable at at some point uh highlighted with the red line here but even if the noise is smaller than


that it's algorithmically hard and it's also a problem that have been studied so in order to make the the kind of computational question more interesting i mix them and if i mix them then you see what's going on there is this algorithmically hard phase appearing that we believe cannot be entered by any polynomial algorithm that's a conjecture and now all i want to be telling you


about how is how bad and decent and longivan dynamics fits in this diagram you know does it do as good as the approximate message fasting does it do worse why so to define what i mean by you know more precisely was the longevity algorithm and the gradient flow it's simply the derivative so i will be working with the continuous time version here because that's the one that i know


how to analyze so it's the time derivative of the x that's the variable over which i'm optimizing is simply equal to minus the gradient of the hamiltonian or the loss function plus a term that corresponds to weight decay or spherical constraint it would be called in physics plus noise that either is there and has a covariance proportional to a constant that is called temperature in physics and if


that constant t is equal to 1 then this is the logic algorithm that is guaranteed at exponentially large times to sample the boltzmann measure and to solve the problem optimally but we will not be looking at exponentially large times because that's untractable our question will be what happens attractable time so that will be constant on constant times logarithm of


the dimension or something of that kind so that you know we can wait for such a long time and then if we simply don't put this additional noise there so this constant t is zero then this is the gradient flow so so going you know how how that model is solvable so in statistical physics of disordered systems this this works cited here is very well known it's basically the reference work


that we have in physics to understand what's going on in materials such as structural classes and it so happens that this work actually looked at a model very much related to the one we are studying here it's it's exactly the same one except that it didn't have this ground truth vector x star so it's exactly the same loss function but the tensor and the matrix is created


without this ground true planted in but that's you know that's that's a complication of the model that can be worked out and you know this theory from this paper can be generalized and this is what we did i will not be going into details of the of the derivation that would be very lengthy but if you are interested in the details actually just um two months ago there


has been a wonderful lecture by uh my co-author francesco urbani that you can watch at the at the leisure's website so this dynamical mean field theory that describes in a closed form what the gradient flow or the longevity algorithm i have the two versions here is doing is a set of equations that close on three parameters this function c of two times that is a


correlation function this function c bar of one time that is a correlation of where the gradient flow is at the given time and the ground true vector x star and a so called response function r of again two parameters and in the limit when the size of the system goes to infinity these functions in the algorithm evolve following this set of pretty ugly looking equations but the


kind of important thing here is that we started with a high dimensional problem the n corresponds to the dimension was very large and the closed equations that we wrote they are just on scalar variables these functions corresponding to two times but the dimension is not there anymore so we described the complicated high dimensional dynamics with the effective


set of equations that are just color equations and so since they are you know scalar they're also simple to solve so we can plug them in in a computer program and solve them and yeah i will be i will be going through several open problems during the talk so the first one of them is you know proof that the dynamics gradient flow and launch of the dynamics in this model


indeed follows these equations and there has been a related work in the past where you know this proof has been done but again for the version where there is not the spike so the equations are not are not exactly the same so this you know this is something that is quite probably not so complicated to to generalize these proofs including this the spine but it hasn't been done yet so i will not be


talking about that instead i will be talking about what happens if we solve these equations what insight can we get about the behavior of this optimization problem so this is depicted here so as a function of the iteration time i am plotting the correlation with the ground roof and at the i start randomly so at the beginning it's zero and then it is growing event or not but


here it is growing and depending on the value of the noise so there's the delta p here so darker line here is larger delta p so larger noise is harder and indeed you are seeing that when it goes up the value at which it saturates is lower for the larger noise so that's intuitive should be lower correlation because it's higher noise so it's a harder problem but what's not intuitive is that it


actually for larger noise the correlation the good correlation with the ground truth is attained earlier whereas for smaller values of the noise it takes longer to find it so this is non-intuitive nevertheless this is what is happening here that's the property of the launch of algorithm in this problem and in the inside i'm just comparing to the very same lines for the approximate


message passing algorithm which is another iterative but not gradient based algorithm that one that one behaves in the intuitive way the easier ones get there earlier but not for the longevity so if i collect this information i can actually extrapolate the value of the noise at which the time to get a good correlation would diverge and if i plots in the phase diagram i showed before


where this happens i actually get that the easy regime that is easy for the other algorithms say approximate message passing has actually part the one that is colored uh orange green here that is hard for the longest of algorithm where the london algorithm you run it full time that is some proportional to the dimension maybe with some polylog factors and it's still


stuck at completely zero correlation with the ground truth and then if you are above this line where it is really only green then it reaches the optimal correlation so you can do exactly the same thing for the gradient flow and you will get you know another curve in this phase circum which is a bit higher so the fact that it is higher is expected because this is a high


dimensional problem with a lot of noise the maximum likelihood estimator here is not optimal the optimal one is the one that samples the posterior so in a sense by running the gradient flow we are aiming to solve the wrong problem so no wonder that we do a bit worse so that's not surprising but you know it's it's a non-trivial curve in this diagram so can we explain it can we kind


of understand intuitively where it comes from so kind of the popular explanation of why for some parameters the gradient flow would be working and why for others it will not be working will be kind of this this cartoon with spurious local minima that either are there or are not there since there are no spurious local minima then the gradient flow has basically no


choice than to good for the good one then to go for the good one and if they are spurious local minima then then it's a high dimensional problem there will typically be exponentially many of them so the intuition is that it will just fall into one of the exponentially many and not the good one so in this model this is actually the model is so kind of basic that we


can we have access to actually counting exactly how many minima they are at the given value of the energy of the loss and this is done by the so-called cats rise approach and so here again i'm not giving the derivation here just the resulting formula that is telling us of you know that the entropy is always number of something that is exponentially numerous so is the


logarithm divided by the size of the system and its number of what it's number of the local minima that have a given correlation with the ground truth as the parameter m at the given value of the loss corresponding to the matrix e2 and corresponding to the tensor e and again this is a result you know resulting from a series of works where these kind of methods were developed


so here what i'm showing you is the annual entropy that is the expectation of the of of the number of those minima but actually at zero correlation with the ground truth this also is the is the quenched so is also the expectation of the logarithm so we actually know when they are and when they are not spruce locally minima and if we collect it from this formula


back to my face diagram we are getting the purple line here so the purple line means that above it with high probability the only minima that is there is the one that correlates with the signal and below it there are exponentially many spruce local minima not correlating with the signal and yet you see that these are not the same lines as the one starting from which the


gradient flow is working so there is a region between the purple and the green line where they are exponentially many first local menu with no correlation to the signal yet the gradient flow happily manages to ignore them and finds the good one so how is this possible so to understand how is this possible in this model we need to dig a little bit more into what is


happening with the algorithm and we actually can look at the at the following plot that is showing us how does the loss function the e on the y axis change as we iterate as a function of the iteration time t and we find out that either for a high value of the noises the dynamics is stuck at some value of the loss that seems to be you know pretty flat starting from sometime about


100 to 100 here or it's actually stuck at that value but then somehow escapes from it and reaches good correlation with the signal that is the dashed line that's the magnetization and when we actually investigate whether that value at which it is stuck corresponds to something we find that yes that it interestingly corresponds to the value of the loss that it would reach if the signal was


not at all there if the if the x star was not in the in the mouse so just to the non-planted map and this is a value of um of energy that was studied a lot in physics that has a name that's the threshold energy and studying the non-planted system we actually can compute that value so so here we make a hypothesis we say okay let's assume that the dynamics goes to


this threshold energy and then what matters is whether the minima that lie at the typical ones that lie at that energy not lower one not higher one that one whether those are stable or not towards the signal and that stability decides whether you get you stay there or whether you go to gain some correlation with the signal and what i'm saying in words here we can


actually put into equations so here the first equation is where about are the threshold states and the second equation is telling us you know derived both from the gates rise approach and directly also from the dynamic domain field theory but again the details are not shown here is the condition for uh the lowest eigenvalue of the corresponding hessian of the minima being having an


eigenvector that points towards the signal or does not so if i put these two together i actually get the third expression here a conjecture for where is the line above which the gradient descent or lounge of our dynamics depending on what the parameter t here will work and so this leads me to the following conjecture it you know the contractor is that gradient


flow with random initialization finds you know in time that is it finds the optimal correlation with the signal in time that is proportional to the to the input size which is n to the power p times some polynomial of log n finds the optimal solution and if the noise is bigger than that then it does not and if i plot this expression into so so again open problem prove this conjecture


and if i plot this expression into my face diagram that is the blue line you see that that one is perfectly agreeing with the points that i got previously by numerically solving this integral differential dynamical mean field equation so so this this seems to be explaining whether or not the gradient flow work works and i can do exactly the same thing just plug t one instead of zero in


the same expression and get the same result for the lonzeva algorithm that has an interesting point that actually the line that corresponds to this uh to this threshold it's set to it it reaches the line lambda two equal one at uh sorry delta tweak one at delta p equal to so there is a three critical point so if delta p is bigger than two there is no loss of a hard phase anymore


but maybe let's go back to this popular explanation what was wrong with that you know absence or presence of the spruce local minima at least in this particular model the correct explanation based on the landscape and kind of intuition about how the landscape looks like is the following one it's not the presence or absence of spurious local minima nor their number


what it really is is the fact that the dynamics goes to the highest lying minima that happens to be the threshold state and what decides whether it finds the solution or not is that these highline states have a negative direction in the in the hashem towards the solution or not and if they do then the algorithm goes to the solution even though there still may be


exponentially many spruce local minima at lower energy so they're not really spruce because the gradient flow just never ever sees them with probability that is that is one up to some exponentially small factor so here i should be about a middle of the talk and i want to conclude about the spike matrix model so i showed you you know this is i think the first time that we have a close form


conjecture for the threshold of what gradient-based algorithm is able to do including the constant in a high-dimensional non-convex interference problem and the question would be you know can we apply the same methodology to something that looks more like a supervised neural network simple one we also show that the gradient flow is worse than the larger algorithm that itself is expected but


they are both worse than the approximate message passing there is quite a considerable gap so is there some generic kind of a strategy that we that we can make them work as well as the approximate message passing or at least closer so that would be question related to the second point and question really and the third point is i showed you that gradient flow


sometimes work even when the spruce local minima are present we showed that using the cast race approach but what about stochastic gradient descent so far i was only talking about gradient no stochastic so let's say a year ago i would have stopped here and say that you don't know the the green questions would be open but today actually i do have an answer to


each of them so i will start with the first one so is the same methodology applicable to some simple neural networks and in statistical physics when we kind of set up a model for a data so that you can keep track of constants and and not only rates and finite sample complexity the kind of popular model in which something like that can be done at least in simple neural


networks is this teacher student setting so now i'm switching the mall no more spiked matrix tensormal in the talk now i'm going towards this teacher student neural networks where at the input i put id data not only iad sample from sample but also the components of every sampler id so that's of course you know not real data do not like look like that but that's part of the simplifying


assumptions here then i take a neural network like for instance the one here i generate the weights of the neural network in some again random way i let this teacher neural network generate the labels y using those ground through ways w star and then i hide the w stars i never show to the student network the w stars i just show to the studio network as traditionally


the set of samples x and y and i will have n samples each sample will live in dimension p so before p was the order of the tensor now p will be the dimension till the end of the talk and then the student may or may not not know the architecture of the teacher network i will actually be telling you about both cases in this talk and the question is what is the


generalization error that the gradient descent is reaching depending on the number of samples that it got from the teacher so this is a setting of a neural network that have been studied in physics for 30 years kind of the most common example would be this teacher student perceptron where the nonlinearity that the teacher is using is just a sign but with just a sign and no


constraints on the weights w this becomes a convex optimization problem so today we are interested in intrinsically non-convex optimization problems so in order to make it more interesting and intrinsically non-convex we will actually be looking today at the face retrieval where the teacher instead of using a sign uses an absolute value on the scholar product of the samples and the ground


true base or the teacher weights w star so the labels here will not be just binary this we will be looking at this regression problem the data will be generated as is written here the input is gaussian the labels are obtained as the absolute value of the of the scalar product and the neural network then sees the set of samples and tries to regress the y or the x on the y


and so what do we know again again without yet talking about gradient descent what do we know about this problem information theoretically and in terms of the approximate message passing that also here is conjecture to be made the best of the of the polynomial algorithms so here i am showing you the the mean square error of recovering the w star which is you know very related to the


generalization error just like this plot as a function of the alpha which is the ratio between the number of samples and the dimension and both number of samples and dimension are large i'm in the high dimensional limit and the ratio is some small constant here you know between 0.3 and 1.2 so information theoretically the generalization error can be zero start as long as soon as you have


more samples than [Music] than is the dimension in this problem that corresponds to the orange line now algorithmically you need slightly more samples than the dimension about 13 percent more for the approximate message passing to work and be able to generalize perfectly in this problem so now we will be looking at what gradient d sent us and how it compares


to to what to do this so gradient descent on watch loss which loss functions so corresponding to the phase retrieval the natural loss function is the one i write here that would correspond to in a sense to to to neural network with no hidden variable one hidden variable with quadratic activation function so that's an actual you know i just square the label sends instead of putting absolute


value i put a square here and then i'm looking at the the performance of the gradient flow so just to you know set up the stage of what is known and what kind of we can expect so as i said if ignoring gradient descent we know that starting from one the problem is solvable information theoretically and starting from 1.13 by some very adapted algorithm to this problem for the


gradient flow what we know is is this work that that popped here that vigorously shows that randomly initialized gradient descent will need uh we'll need the dimension times some polynomial of the log of the dimension samples in order to be able to solve the problem so there is quite a big gap between 1.13 and poly alpha it is 1.13 and alpha that is some polynomial of logarithm of


the dimension so as a physicist we always try to look numerically what's actually going on so numerically if we are looking what's the fraction of success of writing this in terms of solving this problem as the dimension is growing so here the capital n is actually what i call the p is the dimension so we are seeing that at alpha that is around six or seven it's already solving


the problem almost always so so can we understand that a bit more theoretically not just running the gradient decent that's of course nice but that's that's not just satisfactory so we take lessons from the spiked two plus piece per mole that i showed you and we kind of ask ours okay could it be could it be happening similarly as there could it be that the gradient flow first


goes to the threshold states and then what matters is a kind of bvp-like transition of the hessian of these threshold states that drives the success versus failure and we just test this numerically whether it looks like true and it actually does in the sense that if we look at the non-planted phase retrieval that would be the right hand side here that defines the value of the


loss function that i call the threshold value and then if you look at the dynamics of the gradient flow in the planted version we see that it's quite possible that it's again going to the threshold and then away from it or not or staying stuck there so we again hypothesize that this is actually the mechanism and put it into equations this time the equations are slightly more complicated


but we can still do it actually using a recent random matrix theory results from works of ulu and square collide collaborator lee and also the fact that the threshold states are marginal meaning that the lowest eigenvalue corresponding to them is stuck to zero and this if we combine it gives us an expression of what should be the threshold above which the gradient descent


works as a function of this probability distribution of the true labels y and the labels y hat that the gradient descent is currently estimating so that probability distribution is still something pretty not non-intuitive to capture but in the within the within the theory of one-step replica symmetry breaking that is again one of the methods coming from statistical physics


we actually can estimate this probability distribution between the joint the true label and the label that is currently estimate currently estimated by the grading descent and this is shown in this picture i show so in the left hand side i actually show the loss the value of the threshold energy of the threshold loss as it comes from simulations that is the purple line and as it comes


from this one rsp theory they are not exactly equal here that's not the conjecture here is that this is not exact but they are close so so we use this as an approximation on the right hand side i am showing again numerically obtained moments of the distribution on which the formula depends and the moment is computed from the one rsp theory and the agreement is pretty good so when we put


these things together you know ignoring the little differences this actually leads us to an estimation of the gradient decent threshold that is about 13.8 so if i put it back onto this axis i showed you that the numerics this constant starting from which gradient descent is working looks like 7 from approximated theory we get something like 13.8 so we are not sure where the discrepancy


comes from whether it is finite size effects and the numerics would actually converge to the 13.8 or whether it is the small difference between the exact result and the one rsv approximation so both is both is possible but what is nevertheless clear is that it seems that it's a constant the the polylog of b is not needed so here is another open problem proof


that you know any constant times p is actually a sufficient number of samples for randomly initialized gradient descent to solve phase retrieval in time that is p times times some polylog p so in the time the polylog p is not avoidable because otherwise you're just stuck at kind of zero correlation but in the number of samples the conjecture from from our work is that it should be


avoidable and that the true constant is somewhere around 7 or 13. but what about the gap between the the performance of the approximate message passing and of the grand descent there is you know still big difference between say one and ten so can we somehow close that gap can we do something generic that would diminish that gap so that's the question for the next a


few slides and that that corresponds to to this you know when i was concluding about the spike matrix model that was the second input so that's the second point to which we are doing and surprisingly or not we will do that by over parametrization so let's still look at the phase retrieval so the problem the regression problem we are trying to solve here is still the same


so phase retrieval with random gaussian data and the teacher coming from a gaussian and generating the labels this didn't change but what changes now is the loss function so now the loss function that i will be considering doesn't correspond anymore to simple the simplest neural network with no hidden unit or one hit unit that's the same now the neural network will have m


hidden units and i will be working in a regime where the number of hidden units is bigger than the dimension p so this is the over parametrized two-layer neural network i will be optimizing over the weights of the first layer this matrix w and the second layer will be fixed the weights of the second layer will be fixed to one over m or to one and i use the scaling one over


m so i'm not really learning here the second layer but that would know the conjecture kind of is that that wouldn't change much in the in the overall message of this so again i'm just running gradient flow on this loss function with a random initialization so how does this behave so this is a wide over parameterized two-layer neural network does this solve the phase you


achieve or not and this is from a paper that that we that came up in june with a colleague from uiu eric vanden eiden and same student as stefano sarah mannelly where uh in two theorems we kind of answered provide some answer to this question so the first theorem is it is purely geometric it is telling us that if you are looking at the loss function as i just defined


it then if alpha is so alpha again was the ratio between the number of samples and the dimension so if alpha is smaller than 2 then this loss function has many spiritual minima and if alpha is bigger than two then the probability that the only local minima that is there corresponds to the ground roof that would be this a star that is just the teacher vector times is transposed is


actually the only one so we believe that this is actually with probability one but what we could prove is this is only with positive probability but there is something clearly happening about the threshold alpha equal to and when we and this is purely geometric no gradient descent yet but when we put this together with our second theorem about the gradient descent


that tells us that in terms of this parameter a that is the weight matrix times its transpose the gradient descent always goes to global minima then putting these two together actually if there is only one global minima corresponding to the to the ground true with finite probability well then the gradient descent also goes there so this means that the gradient descent


solved this problem by optimizing this loss function corresponding to the over parameterized neural network starting from alpha equal to and here is just a little plot that you know that shows that just running numerically writing descend on relatively small systems is pretty consistent with that result so if i put that back into onto undo the axis of alpha i obtain that by using over


parameterized nail network i can push down the threshold at re starting from which the gradient flow is working down to two so not yet to the 1.13 of amp but much lower than if i was not over parameterizing so the conclusion is here that over parameterized neural networks need fewer samples and this is a quantification of how much fewer samples in this particular model


and the open problem would be and that i i really don't know the answer like is there a neural network architecture maybe if you make it deeper or over parametrized differently for which the plane randomly slice guardian descent would just need less than alpha 2 so less than 2p samples so i think that's an interesting kind of concrete question for this particular model


and i i might have time for the third point stop me if i don't but i wanted to mention the third point about what can we say about the stochastic gradient descent so far i was only talking about gradients and or more precisely gradient flow because i was always considering the continuous time version because that's the one that is easier to analyze so what about stochastic gravities so


first of all a reminder was what's stochastic it's you know the same thing but we are taking the samples one by one so when we say stochastic writing decent in the literature we mean usually one of the two following things so either we mean the online stochastic grading descent where each iteration uses a fresh sample and never uses a sample that was ever seen before


and i like to call it online stochastic running decent that one is simpler to analyze but it's also less interesting because it minimizes directly the population loss and there is no notion of the generalization gap the training test are the same so a lot of the mysteries about how comes the train error can be so much smaller than the test error that we are kind of asking and deep


learning cannot really be answered by looking at the online stochastic gravity scene it's also not used in practice what's used in practice is multipass stochastic gradient descent where we use one or few samples at the time but we reuse the samples many times and this is much harder to analyze this has much less kind of existing theory but that's the one we want to look at


because it's used in practice and it can access you know non-trivial generalization gap then so can we do that so first of all the first step above which i didn't even talk before because it was obvious how bright and descends in the limit of small learning rate becomes gradient flow for the stochastic gradient descent is not so clear what is the limit of the infinitely


small learning rate that actually is well defined so so i'm just explaining it on this slide so if i define stochastic gradient descent using this variable s of t that would be you know one for some samples and zero for some other samples so if i do what is usually done in stochastic gradient descent is that every at every time step i randomly choose who is in the batch and


who is not in the batch well then this doesn't have a well-defined limit of the learning rate going to zero it doesn't really have a gradient flow limit so this is not so nice for the dynamic community of theory so we instead define a slightly different version of stochastic iron descendant we call persistent stochastic variant descent where we as before have some fraction of


samples that are in the batch but instead of reshuffling the batch at every time step randomly we actually decide at each time step whether we keep or not the the sample in the batch and we we keep the we keep that sample with some typical time that we call here the persistence time following the rule that is written here so if we do it this way then we can take


the limit of the learning rate to zero and it has actually a well-defined stochastic guide in flow limit so that is the that is the that is the dynamics that's that i will be analyzing on a model that here will be um slightly different so it's not the phase retrieval or the model on which we will be analyzing this it's uh it's just a gaussian mixture a supervised


learning of a gaussian mixture so in the two cluster case that is on the left here i have two clusters one cluster is plus labels plus one cluster is labels minus and i'm trying to separate them so that's very simple i can just imagine there is some hyperplane in the middle and separating them but the clusters are really noisy so i'm in a regime where i will not be


able to separate perfectly and yet so so this will even lead to a convex problem so that's more for kind of a comparison but the one that will be interestingly non-convex is the three cluster case where i have three gaussian clusters two on the periphery and one in the middle but the two on the periphery they have the same label so this is a data set that is no longer


linearly separable so actually to be able to to to to have some meaningful learning the loss function that i will be using for these three clusters is is actually is actually kind of you know using the the structure of the data set and i will be doing logistic regression but not directly on the data points but on the you know as specified here on this c c my


mu okay but whether it is the two cluster case or the three cluster case how do we describe the full trajectory of the granules the complexity is really not so much helping us to describe the the full trajectory so this will be again done with the dynamical mean fee theory but this time with a little bit more advanced version of it because for the perceptron uh


case the the the simple one that we used before is is not quite working it's not it's not so the equations do not close so simply but this period is the same here we start with this high dimensional markovian dynamics of a strongly correlated system and the dynamic domain field theory maps it into non-markovian so dynamics with memory but of one single degree of freedom


so this is where we lose the high dimension and get a system that we can actually plug into computer and analyze and here uh rd you know is that one dimensional system but this is a little you know i want to get to the results or conclude so i will not be explaining this in detail but again i was mentioning this lecture by pierre francesco obani in lesos where he actually derives this equation


in details peter put on his video does that mean i should be concluding maybe sorry you have five minutes five minutes great that's fine so let me just like okay this i will not be explaining every detail but let me just explain what this is so here i actually the claim is that the dynamics in this of this classifier in for the data set that is this that is this high dimensional gaussian mixture


behaves in the same way as the following scholar stochastic process for this variable h of t t is just the iteration time that and the stochastic process actually has several types of noise so it has for instance a noise that corresponds to the regularization lambda which is just the ordinary reach legalization but it also has a noise that plays exactly the same role of the rich


regularization but that came from the dynamics that is not there explicitly so this is some kind of implicit rich regularization that comes from this variable s of t that was the variable in the in the stochastic guiding descent that was deciding which sample is there and which sample is not there so that's that's kind of an interesting you know interpretation of


what this stochastic model stochastic process actually is this is a interpretation of how kind of the implicit regularization might be coming out in these type of problems the second term is directly the noise coming from the stochastic grading descent because you might be saying that here i have batches that are still extensively big so maybe the noise doesn't matter so much so it


actually does it's still explicitly there even in this effective dynamics and then there is a dynamical noise here that is just gaussian with some covariance matrix mc that is consistently computed you know for via set of closed equations and then there is some memory kernel here mr that also needs to be consistently computed from a set of equations so i put these


in very small because you know these are these are kind of hard to grasp in just like one minute but it's again some some work that comes from recent works in statistical physics and that can be directly adapted to this problem as we did in this paper here again open problem would be of course challenge once once one goes in detail through what this color stochastic process is can we


actually prove that it is equivalent to what the grad what the stochastic gradient flow is doing in in this problem and you know once you compute it numerically all these uh quantities uh from these equations then you can compute everything else including the training loss the test laws the generalization here the corresponding accuracy is this is this is highlighted in this set of


equations so with that what you can do is you can plot for instance a picture like that where i plot the generalization error and the training error in the intersect the generalization here in the main part is a function of the time and now the points that's just running the stock the persistent stochastic gradient descent numerically on this data set so this is just a plain simulation of


this simple neural network and the lines that's a result that i get from the dynamic community fury so it is you know you can see that it is describing the whole trajectory what is the generalization error at any given time for the persistent stochastic guiding descent so you see that at large times it is going somewhere but at intermediate times there would be


some early stopping to do here and depending on the batch size and order on the persistence time is not exactly the same curve and quite interestingly even the the orange points actually would be the normal stochastic gradient descent and the line corresponding that is plotted there is is in a sense not quite justified by our theory we just kind of ad hoc discretize the theory in


the same way we would discretize to do the canonical stochastic descent it still seems working which is a bit puzzling to us it really kind of we don't really know why should it but so we didn't maybe even need to do this this persistent stochastic guidance descent for this theory to work or maybe yes maybe there is some small error that doesn't show up on you know on this comparison with numeric


so this we don't know yet and you can you know look at other things for instance this um this case of two clusters is quite interesting because if you don't do the stochastic gradient descent but fulbright guiding this and that would be this this picture that is of course just a special case of the equations that i just wrote it has this specular behavior that if


you initialize so r is the variance at initialization if you initialize at zero with really small variance then after one iteration you reach the base optimal error and then the gradient descent is actually driving you away from it and training accuracy is growing you know this is a regime where you're interpolating the training accuracy goes to one but the test error is getting worse


so actually after one iteration you were perfectly optimal but then the gradient is driving you away from the perfect generalization not perfectionist the optimal generalization point so that's a kind of specular property of this particular two cluster model that we that we discovered in this paper and and you can look at other things such as you know looking


how different the dynamics is when you are changing the batch size and comparing how the dynamics changes when you're changing the ridge legalization of the loss and this is what's on these two pictures so i see that as i'm changing the batch size the time scale where i start to decrease is changing because the number of iterations i need with smaller batch size is


bigger so this intuitive but otherwise if the curves look kind of comparable to the ones if i was adding more and more regularization in the sense that the blue ones that is just gradient descent no real backsides one is as if i was regularizing only a little in this case and regularizing a lot is actually corresponding to the smallest batch slice in this in this


picture but you know that's that's just like observing how the how the curves look like so there's no like formal statement here so this was the last figure that i wanted to show you and just to conclude so you know i was telling you about this dynamical mean field theory and the results coming from it that is able to track the full trajectory of the grounding descent or


the stochastic realities and for a range of our synthetic models for the data and there are of course many directions in which this this we would want to uh extend it including all the open problems that i stated we would like to have more more math and rigor into that but also deduce more of insights just by looking of what the dynamic and your equations are telling us as a


function of all the hyper parameters and the nature of the noise that i was describing in the in the equations and we can look at other data models these are not the only ones for which this this this can be written and we can look at networks that actually have hidden variables and of variants of the gradient descending stochastic guardian descent that have momentum for instance


etc so this is hopefully to come and i just flashed back the list of papers from the beginning that that i covered in this store and open for the discussion if there is still time for the discussion thanks very much lenko that was that was a fascinating uh talk we've actually run over time so um we probably should take questions offline but um you know thank


i am sapna sharma and i am here to present a paper i am here along with ragini to present a very interesting paper published by six google researchers in may 2019 the researchers are david birthlord nicholas scarlini ian goodfellow avital oliver nicholas papernote and colin raffle the title of the paper is mix match a holistic approach to semi-supervised learning

MixMatch: A Holistic Approach to Semi-Supervised Learning


so next slide please so we will be covering the background and purpose to see why mix match was [Music] required the key terms to understand the algorithm we will be giving a brief description of the algorithm and the results of the experiments performed by the authors so we are all aware that there uh we are all aware of the three main classes of the machine learning that is