moment you have the answers within you present to me the hunch the inspiration and I will act upon it with calm faith and it will be reveal to you to the extent that you have the connection and the relationship between the conscious and subconscious super conscious it will be revealed to you right that in there and the key is take action on it speed of implementation if you get that hunch you have to act upon it right away because if you don't act upon it right away more doubt will set it more indecision will set it when you act upon


it you move everything forward and it might not necessarily directly produce the result but in some way somehow it will be brought forth because you're moving forward the next ideal will show up the next thing and when you connect the dots looking backwards then it will make sense to you there are infinite ways number 12 infinite ways of bringing forth your vision found by listening to your own inner voice the inner voice speaks from the super conscious mind which is the source of all that exists and your subconscious and even if you don't

18 Secrets That Lie Hidden In Your Subconscious Mind (Neville Goddard, Napoleon Hill)


believe in the super conscious it will still speak from the subconscious because the subconscious is a storehouse of so many reference experiences that you have had in your life and there's tons of them you have to believe and have faith and confidence in yourself that's right recommend watching the video I did on self confidence formula by Napoleon Hill when a person has self-confidence and faith in themselves they tap into their subconscious and their subconscious expresses you'll notice that people who have a high


degree of real core confidence are more likely in flow more often they are not in their head they're not stuck an inharmonious in the relationship between the conscious and the subconscious you have to trust that exists within you and you have an inner voice and it can speak to you for me I build a connection with my inner voice to the point my inner voice speaks to conversations but for you it might be a hunch and inspiration but the more you honor it the clear that inner voice becomes the more pronounced it becomes and again


that's why Steve Jobs said don't let the noise of other people's opinions drown out your own inner voice because all is one mind the inner voice is connected to the inner voice of everybody else The Sixth Sense probably is the medium of contact between the finite man or finite mind of man and the infinite intel and infinite intelligence and for this reason it is a mixture of both the mental and the spiritual it is believed to be the point in which the mind of man connects to the universal mind all well-being is materialized through


alignment with consciousness rising and experienced through our five senses guided by our six now this will make sense to you when you have raised your consciousness and I recommend studying the work of David Hawkins letting go and power versus force in the levels of consciousness when you work on releasing the lower consciousness level based thinking of fear anger resentment hatred Envy you move into a place of bliss joy unconditional love acceptance real reason will understand you seek to understand and then what happens is that


those experiences those thoughts project outwards via your subconscious mind and materialize into form and as we said guided by the sixth sense the voice with it lifting up your thoughts to higher degree no matter what you find as far as the statement it can always be Allah elevated to a higher level number 14 the most powerful televisions a great quote that I put together my Instagram that came to me as a result of a conversation I had with my superconscious the most powerful television is our own imagination in which you create worlds


where you are the hero and all supporting you to bring forth your well-being into this planet with such vivid detail that it rewires your neurology in your brain to automatically generate actions to bring it forth you do not need anyone else to tell you what your vision has to be a lot of us are programmed by the information that we're consuming and the question we have to ask consciously and it's where the conscious mind has to step in and say is this information programming our subconscious mind for abundance for wellbeing for joy


for peace or is it not and we have a choice right then and there and we always have to remember this the imagination in our mind can be the greatest source of entertainment joy happiness bliss inspiration in your own mind and you control it and you have access to it and you can cultivate it and you can build a relationship with it and just like how you consume information that information goes into your subconscious mind and projects outwards to materialize into form so will what will be in your imagination it


will do the same thing the question is do you want to have control of that what programs your subconscious mind so Napoleon Hill says before you can put any portion of this philosophy into successful use you must be prepared to receive it the preparation is not difficult it begins with study analysis and understanding of the three enemies which you shall have to clear out these are indecision doubt and fear the sixth sense will never function while these three negatives or any of them remain in your mind the members of this unholy


trio are closely related where one is found the other two are close at hand so this is one of the biggest insights that I'm going to share with you what I've found with working with the subconscious mind you can categorize all subconscious mind reprogramming to the removal of indecision doubt and fear which are $0.05 based input data and meaning gave not the data but the meaning so it's you've heard this said before it's not what happens due to how you respond to it it's not what happens to you it's the meaning you give to it and that's why I


recommend man's search for meaning and watch the video I did last on mental chemistry which is exactly about this transmutation now when fear doubt and indecision sets in your subconscious mind they will begin to project outwards to materializing to form to reflect fear doubt and indecision and fear when it's projected outwards in the external world cripples us because it shows up and it's scary because it's revealing what's within which is scary and it can cause us to remain there now there's always a way out we'll always find a way out but


the key is this if you want to continue to unravel that what is within your vision as you do the work to remove it at an early stage fears which is a net result of indecision and doubt number 15 the discipline of taking rapid action on an idea that you're curious about will not only give you empirical data but train your ability to trust yourself and act in spite of doubt fear and hesitation while cultivating wisdom and decisiveness ok discipline of taking rapid action will cut through the dis indecision and doubt right then and


there it's a habit see overthinking is going to lead to fear that's it when you start overthinking you have initially embraced a slight seedling of indecision in doubt that's why right then and there we could ask the subconscious on what we need to do and take the action right then and there and if you look at anybody that has created success one of my favorite programs that I've ever was part of us called get altitude by eben pagan and he said in the beginning he said one of the commonalities that he has founded he's a very extensive reach


a researcher and he a lot of his information I hold true as foundations fundamentals for building success especially in entrepreneurship he said the one commonality the one thing that successful entrepreneurs have in common is speed of implementation and I did that discussion on opportunity but I've been paying in I recommend you watch it he said get version one out in the marketplace as fast as possible and yet again then get it up to version three as fast as possible out into the marketplace take the action


now this is a practice it's a way of living when you get an idea to act upon it right away you are cultivating decisiveness and you're getting data you cannot get data from the external where everything is optimization you cannot get data from the external world unless there's some action taken in the external or otherwise you get stuck in your head in theory the beautiful thing is the more you do this with repetition the more you honor your inner voice because you'll start to see that some way somehow it works out number 16 the


sixth sense is a conversation with infinite intelligence that shows up when you are in a vibrational match in your imagination which is clear of thoughts containing fears doubts and indecision so by taking rapid action by honoring your inner voice by trusting it and not being afraid of the six basic fears as a result of thing so what happens fear of poverty fear of criticism fear of ill health fear of loss of love with someone fear of old age fear of death hold somebody back from taking action that's because in the earlier stages they


encourage indecision and doubt based thoughts that that became fear in those areas they encouraged them to the point it became fear and you don't want it to get to that point and if you want to build a connection to the superconscious mind you have to do this work you have to release those fears you have to release the early stages of it then take action because you have to get into the vibrational match of the success which is a really high vibration which is a confidence based vibration a person that believes in themselves has confidence


creates success look at anybody that has produced any results that you admire doesn't necessarily be money or finance whatever you will find within them a high degree of confidence and whether they can articulate what what it is I'm talking about right here and a lot of them I've talked to have never articulated before and I've shared this information with with them and they say this it said Wow you are able to articulate what I have been feeling and so they might not necessarily be able to explain what we're talking about here


but they are experiencing this in some shape or form but the key is this they get access to their subconscious and superconscious by releasing the fears doubts and indecision so thus that's why I put huge emphasis when it comes to programming the subconscious mind one of the big elements that we want to keep into consideration is noting where the fears doubts and indecision so up and work on releasing them number 17 flow brings forth results in harmony with our vision because the feeling experienced while in


flow is the mental alchemy that is projected outwards to materializes form it is then experienced back through our five senses as validation of the feeling of flow within flow is where we want to be at flow is where challenge meets skill it is where your being and living a purposeful life it is when you're progressively moving forward towards the realization of your definite chief aim it is when you're living your vision creating your vision and it will be brought forth and then you move on to the next one and the next one so watch


the video I did on mental chemistry watch the discussion I did on flow by me nitric sent behind flow is a very important element to keep into consideration when a person is in flow they're also very light-hearted when a person is not in flow they're stuck in their head they're angry that's because they have given in to fear fear brings more forth those lower levels of the emotional scale the level that you want to get at is the higher levels of the emotional scale okay watch the emotional scale discussions that are probably on


YouTube on Esther and Jerry Hicks maybe I'll do a discussion on that the emotional guidance scale they even talk about it they said being in those higher vibrations to higher levels of the emotional scale will allow you to connect to infinite intelligence to your voice within the superconscious and that is experience when you're in flow when you experience challenges while you're in flow you're able to overcome it because the idea has come from within we have the subconscious there's a harmonious relationship between the


conscious and the subconscious and I'll even add to it and say the superconscious as well and number 13 always remember this you are the conduit that streams infinite intelligence we all have access to infinite intelligence it is accessible to all of us and the link is within it is within you just like any conduit energy flows from side one side to another within its casing fully perfect protected from external contaminants to smoothly carry out its meaningful journey in all its purity all the way to its destination you are being


guided you are protected you are being supported on your vision and you have a choice to honor that by taking inventory of what is within your subconscious mind that denies it and releasing that and as you release it you'll find yourself more in flow flow is the energy that goes through the conduit it is where you are on purpose now this doesn't mean you have to be on flow all the time but it does mean this the more you embrace flow the more you honor your flow the more you honor these things that we're talking about the more you'll find that


you're living purposefully the more you'll find you'll be able to come up with saw and solve problems in creative ways never even thought of before because you have a very harmonious relationship between the conscious - subconscious and superconscious and the energy flows through you protected you will be guided your sixth sense will warn you you will have a heightened degree of sensitivity for vibes and energy you will learn to trust and honor that what you feel and I'll speak from experience I've had this many times I've


well uh hello and welcome to this uh richard m carp distinguished lecture uh my name is peter barton i'm the associate director of the simons institute for the theory of computing uh thanks for joining us we established the richard m cup series to celebrate the role of simon's institute founding director dick carp in establishing the field of theoretical computer science formulating central problems and contributing amazing results in the areas of computational complexity and algorithms the series features visionary leaders in

Insights on Gradient-Based Algorithms in High-Dimensional Learning


tcs and is geared towards a broad scientific audience we're grateful to the many contributors to the richard m carp fund who've made this series possible so i'm delighted to welcome our speaker today lenka stevarova lenka is a researcher at cnrs working in the institute of theoretical physics in cea uh paris clay she has a background in physics and is famous for the application of methods of statistical physics to problems in machine learning and signal processing in inference and optimization link is the recipient of the cnrs bronze medal


in 2014 the philippe meyer prize in theoretical physics in 2016 and the irene julio curie prize in 2018. the talk today is entitled insights on gradient-based algorithms in high-dimensional learning so please join me in welcoming lenka's devarova thank you peter and i will share my screen so that you see the slides i prefer it and i'm really really honored to be giving this lecture especially given the influence that you know being part of one of the programs at simon's institute four years ago it had on my career and i enjoyed it so


immensely and it's amazing what the simuls institute is doing so first thing i should do is to correct my affiliation so it's only a second seminar i'm giving and a third week i'm spending at my new affiliation that is epfl so not anymore in france but in a neighboring country switzerland and i will be telling you about work that i you know i have recently did a lecture in this simon's institute bootcamp for the program of this semester where kind of a lot of the works that seemed like a statistical physics voodoo


maybe 20 years ago actually have been established rigorously and part of the program is about it and it's pretty and it's very exciting so for this very special lecture i decided to go back to results from physics where most of it is not established vigorously and is waiting for the mathematical inputs and works and and that's something that was going on in the past two years with the list of collaborators that i give here the the main among them are the two students highlighted in in blue stefano mannelly sarah and


francesca and stefano is among the panelists so if you have some clarification questions or questions you can he's able to answer them even during the talk without interrupting it so please don't hesitate so this is the list of six papers from the past two years on which this talk is based and the talk will be about gradient descent based algorithms or stochastic gratities and base algorithms that you know pictorially are the workhorse of machine learning that is really everywhere these days so they are really worth


understanding and studying in in more detail in particular in deep learning we have the empirical observation that local or even global minima with bad generalization error actually do exist there are many kind of works uh going towards showing like something like that empirically one of them that i like quite a bit is this this paper by dimitry akioptas and his collaborators where he starts by interpolating and fitting random labels in the neural network and then he puts back the real labels little by little


and he shows that the gradient descent actually doesn't go that far away from the point where it interpolated random labels and it generalizes pretty bad much worse than it would if you just initialize it randomly so that really tells us something notable about how this optimization landscape looks like and we really need to understand how comes that the gradient-based algorithms initialize randomly are able to avoid the bad minimum and so the goal here you know it's it's pretty much clear these days that this


will this cannot happen just by studying the landscape that it really what matters is you know even the initialization so what matters is the whole trajectory that the algorithm is taking so we want to understand the trajectory and these non-convex high dimensional problems and just two points to make to set the talk you know in practice the number of samples is limited so i don't want to be working in some limit where the number of samples is is unreasonably large and also constants do matter so i don't want to be talking


working with with some rates without log factors and with arbitrary constants so in order to be able to do something like that to keep in mind finite sample complexity and constants i need to make some simplification somewhere so for the purpose of you know the work that i'm describing in this talk this will be on the side of the data so i will not be assuming any kind of really generic data set i will be working with synthetic models for data for which we can say something so the first such model on which will be


the first say 20 minutes of the talk will be the spiked matrix tensor model which is the optimization you can think of optimizing the loss function that is written here it has two parts one so so the variable over which you are optimizing is the x that is living on an n-dimensional sphere and n will be large that will be the limit we will be interested in high dimensional and then the way the loss function depends on the x is through the matrix y that is created from some ground through x star plus a load of noise


so x star x star transpose more precisely and it also depends on this tensor t that has order p and is created by taking an outer product p times of the same vector x star and adding a lot of noise and then the goal of the you know interference problem here is to find back the vector x star by minimizing the loss function written over here so why this model so that also kind of sets again what what i'm aiming to to achieve so this small because it's high dimensional and on convex that's kind of what makes a study of


gradient descent non-trivial it's an interference problem meaning that what we are interested in is a correlation with the ground true signal x star we are not really interested in the optimization problem per sec so this is similar to the machine learning with neural networks where we always solve it by optimization but we are really interested in the generalization error in something slightly different than the value of the loss function itself and the third and fourth point is that this model has interesting computational


properties and the dynamics of the gradient is unsolvable and this is something that i will show you to persuade you of that so the statistical physics must come at some point in and this is where it does so you just rewrite the same model with the noises of the with the variances of the two gaussian noises that i considered just rescaled a little differently the way we usually do in physics and then i take this loss function that was sum of two squares and developed the squares and realized that some terms just don't depend on the


x over which i'm optimizing and some terms depend on it in a trivial way if the x lives on the sphere they're just equal to some constant so the only non-trivial term that matters is the term here i called h of x that if i look back in statistical physics is exactly the hamiltonian of something that is called the spherical mix b spin glass so those in the audience that know about spin glasses have seen this smile because that's the one one of those that is most often studied in the field of statistical physics of disordered


systems and so what we will be so we will be using that but what we'll be interested in is at one hand the you know when i say gradient-based algorithms i will be speaking at this in this part about mainly two one of them will be the launcher algorithm with the aim of actually estimating the ground through x star in a base optimal way which would be done by writing the posterior measure and computing its marginals and this corresponds to writing the boltzmann measure of the corresponding statistical physics problems and


sampling sampling it at temperature one and that's exactly what the longevity algorithm aims at and the second estimator i will be looking at is the kind of more common one maximum likelihood estimator that is computing the minimizer of that loss function of the or the ground state of the statistical physics model and that's what the gradient descent or flow aims at so just to get a little bit more familiar with this model so if you listened to the bootcamp lecture you would you know we told you about set of tools


that you can use to actually describe what's happening in a model such as this one from the point of view of information theory what is possible statistically and from the point of view of approximate message passing algorithm which ends up to be the best we know for this type of problem and this phase diagram summarizes of what's going on so i will just explain it and then in this talk we are interested what the gradient descent is doing so that will be the new part so on the axis here we have the variances of the noises delta 2 is the


noise added to the matrix and delta p is the noise added to the tensor so the bigger the noise the harder this inference problem will be and for instance if the delta p was infinity that would be effectively as if the tensor was not there it's not giving you any information so in that case you are in the case of spiked matrix factorization that is problem widely studied in statistics and you know it has the bbp phase transition and that's precisely what the value lambda 2 equal to 1 corresponds to so that's what


distinguishes the phase where the spike is impossible to recover from the easy phase where if you only had the matrix you recover the spike simply by spectral methods looking at the spectrum of the matrix then if the matrix was not there that is that delta 2 would be infinity 1 over delta 2 would be 0 then you only have the spiked tensor model which you know information theoretically is solvable at at some point uh highlighted with the red line here but even if the noise is smaller than that it's algorithmically hard and it's


also a problem that have been studied so in order to make the the kind of computational question more interesting i mix them and if i mix them then you see what's going on there is this algorithmically hard phase appearing that we believe cannot be entered by any polynomial algorithm that's a conjecture and now all i want to be telling you about how is how bad and decent and longivan dynamics fits in this diagram you know does it do as good as the approximate message fasting does it do worse why so to define


what i mean by you know more precisely was the longevity algorithm and the gradient flow it's simply the derivative so i will be working with the continuous time version here because that's the one that i know how to analyze so it's the time derivative of the x that's the variable over which i'm optimizing is simply equal to minus the gradient of the hamiltonian or the loss function plus a term that corresponds to weight decay or spherical constraint it would be called in physics plus noise that either is there and has a


covariance proportional to a constant that is called temperature in physics and if that constant t is equal to 1 then this is the logic algorithm that is guaranteed at exponentially large times to sample the boltzmann measure and to solve the problem optimally but we will not be looking at exponentially large times because that's untractable our question will be what happens attractable time so that will be constant on constant times logarithm of the dimension or something of that kind so that you know we can wait for such a long


time and then if we simply don't put this additional noise there so this constant t is zero then this is the gradient flow so so going you know how how that model is solvable so in statistical physics of disordered systems this this works cited here is very well known it's basically the reference work that we have in physics to understand what's going on in materials such as structural classes and it so happens that this work actually looked at a model very much related to the one we are studying here it's it's exactly the same one except


that it didn't have this ground truth vector x star so it's exactly the same loss function but the tensor and the matrix is created without this ground true planted in but that's you know that's that's a complication of the model that can be worked out and you know this theory from this paper can be generalized and this is what we did i will not be going into details of the of the derivation that would be very lengthy but if you are interested in the details actually just um two months ago there has been a wonderful lecture by


uh my co-author francesco urbani that you can watch at the at the leisure's website so this dynamical mean field theory that describes in a closed form what the gradient flow or the longevity algorithm i have the two versions here is doing is a set of equations that close on three parameters this function c of two times that is a correlation function this function c bar of one time that is a correlation of where the gradient flow is at the given time and the ground true vector x star and a so called response function


r of again two parameters and in the limit when the size of the system goes to infinity these functions in the algorithm evolve following this set of pretty ugly looking equations but the kind of important thing here is that we started with a high dimensional problem the n corresponds to the dimension was very large and the closed equations that we wrote they are just on scalar variables these functions corresponding to two times but the dimension is not there anymore so we described the complicated high dimensional dynamics with the effective


set of equations that are just color equations and so since they are you know scalar they're also simple to solve so we can plug them in in a computer program and solve them and yeah i will be i will be going through several open problems during the talk so the first one of them is you know proof that the dynamics gradient flow and launch of the dynamics in this model indeed follows these equations and there has been a related work in the past where you know this proof has been done but again for the version where there is not


the spike so the equations are not are not exactly the same so this you know this is something that is quite probably not so complicated to to generalize these proofs including this the spine but it hasn't been done yet so i will not be talking about that instead i will be talking about what happens if we solve these equations what insight can we get about the behavior of this optimization problem so this is depicted here so as a function of the iteration time i am plotting the correlation with the ground roof


and at the i start randomly so at the beginning it's zero and then it is growing event or not but here it is growing and depending on the value of the noise so there's the delta p here so darker line here is larger delta p so larger noise is harder and indeed you are seeing that when it goes up the value at which it saturates is lower for the larger noise so that's intuitive should be lower correlation because it's higher noise so it's a harder problem but what's not intuitive is that it actually for larger noise the correlation the


good correlation with the ground truth is attained earlier whereas for smaller values of the noise it takes longer to find it so this is non-intuitive nevertheless this is what is happening here that's the property of the launch of algorithm in this problem and in the inside i'm just comparing to the very same lines for the approximate message passing algorithm which is another iterative but not gradient based algorithm that one that one behaves in the intuitive way the easier ones get there earlier but not for the longevity


so if i collect this information i can actually extrapolate the value of the noise at which the time to get a good correlation would diverge and if i plots in the phase diagram i showed before where this happens i actually get that the easy regime that is easy for the other algorithms say approximate message passing has actually part the one that is colored uh orange green here that is hard for the longest of algorithm where the london algorithm you run it full time that is some proportional to the dimension maybe


with some polylog factors and it's still stuck at completely zero correlation with the ground truth and then if you are above this line where it is really only green then it reaches the optimal correlation so you can do exactly the same thing for the gradient flow and you will get you know another curve in this phase circum which is a bit higher so the fact that it is higher is expected because this is a high dimensional problem with a lot of noise the maximum likelihood estimator here is not optimal the optimal one is the one


that samples the posterior so in a sense by running the gradient flow we are aiming to solve the wrong problem so no wonder that we do a bit worse so that's not surprising but you know it's it's a non-trivial curve in this diagram so can we explain it can we kind of understand intuitively where it comes from so kind of the popular explanation of why for some parameters the gradient flow would be working and why for others it will not be working will be kind of this this cartoon with spurious local minima that either are


there or are not there since there are no spurious local minima then the gradient flow has basically no choice than to good for the good one then to go for the good one and if they are spurious local minima then then it's a high dimensional problem there will typically be exponentially many of them so the intuition is that it will just fall into one of the exponentially many and not the good one so in this model this is actually the model is so kind of basic that we can we have access to actually counting exactly


how many minima they are at the given value of the energy of the loss and this is done by the so-called cats rise approach and so here again i'm not giving the derivation here just the resulting formula that is telling us of you know that the entropy is always number of something that is exponentially numerous so is the logarithm divided by the size of the system and its number of what it's number of the local minima that have a given correlation with the ground truth as the parameter m at the given value of the loss


corresponding to the matrix e2 and corresponding to the tensor e and again this is a result you know resulting from a series of works where these kind of methods were developed so here what i'm showing you is the annual entropy that is the expectation of the of of the number of those minima but actually at zero correlation with the ground truth this also is the is the quenched so is also the expectation of the logarithm so we actually know when they are and when they are not spruce locally minima and if we collect it from this formula


back to my face diagram we are getting the purple line here so the purple line means that above it with high probability the only minima that is there is the one that correlates with the signal and below it there are exponentially many spruce local minima not correlating with the signal and yet you see that these are not the same lines as the one starting from which the gradient flow is working so there is a region between the purple and the green line where they are exponentially many first local menu with no correlation to the signal yet


the gradient flow happily manages to ignore them and finds the good one so how is this possible so to understand how is this possible in this model we need to dig a little bit more into what is happening with the algorithm and we actually can look at the at the following plot that is showing us how does the loss function the e on the y axis change as we iterate as a function of the iteration time t and we find out that either for a high value of the noises the dynamics is stuck at some value of the loss that seems to be you know pretty flat


starting from sometime about 100 to 100 here or it's actually stuck at that value but then somehow escapes from it and reaches good correlation with the signal that is the dashed line that's the magnetization and when we actually investigate whether that value at which it is stuck corresponds to something we find that yes that it interestingly corresponds to the value of the loss that it would reach if the signal was not at all there if the if the x star was not in the in the mouse so just to the non-planted map


and this is a value of um of energy that was studied a lot in physics that has a name that's the threshold energy and studying the non-planted system we actually can compute that value so so here we make a hypothesis we say okay let's assume that the dynamics goes to this threshold energy and then what matters is whether the minima that lie at the typical ones that lie at that energy not lower one not higher one that one whether those are stable or not towards the signal and that stability decides whether you get you stay there


or whether you go to gain some correlation with the signal and what i'm saying in words here we can actually put into equations so here the first equation is where about are the threshold states and the second equation is telling us you know derived both from the gates rise approach and directly also from the dynamic domain field theory but again the details are not shown here is the condition for uh the lowest eigenvalue of the corresponding hessian of the minima being having an eigenvector that points towards the


signal or does not so if i put these two together i actually get the third expression here a conjecture for where is the line above which the gradient descent or lounge of our dynamics depending on what the parameter t here will work and so this leads me to the following conjecture it you know the contractor is that gradient flow with random initialization finds you know in time that is it finds the optimal correlation with the signal in time that is proportional to the to the input size which is n to the power p times some polynomial


of log n finds the optimal solution and if the noise is bigger than that then it does not and if i plot this expression into so so again open problem prove this conjecture and if i plot this expression into my face diagram that is the blue line you see that that one is perfectly agreeing with the points that i got previously by numerically solving this integral differential dynamical mean field equation so so this this seems to be explaining whether or not the gradient flow work works and i can do exactly the same


thing just plug t one instead of zero in the same expression and get the same result for the lonzeva algorithm that has an interesting point that actually the line that corresponds to this uh to this threshold it's set to it it reaches the line lambda two equal one at uh sorry delta tweak one at delta p equal to so there is a three critical point so if delta p is bigger than two there is no loss of a hard phase anymore but maybe let's go back to this popular explanation what was wrong with that you know absence or


presence of the spruce local minima at least in this particular model the correct explanation based on the landscape and kind of intuition about how the landscape looks like is the following one it's not the presence or absence of spurious local minima nor their number what it really is is the fact that the dynamics goes to the highest lying minima that happens to be the threshold state and what decides whether it finds the solution or not is that these highline states have a negative direction in the in the hashem towards the


solution or not and if they do then the algorithm goes to the solution even though there still may be exponentially many spruce local minima at lower energy so they're not really spruce because the gradient flow just never ever sees them with probability that is that is one up to some exponentially small factor so here i should be about a middle of the talk and i want to conclude about the spike matrix model so i showed you you know this is i think the first time that we have a close form conjecture for the threshold of what


gradient-based algorithm is able to do including the constant in a high-dimensional non-convex interference problem and the question would be you know can we apply the same methodology to something that looks more like a supervised neural network simple one we also show that the gradient flow is worse than the larger algorithm that itself is expected but they are both worse than the approximate message passing there is quite a considerable gap so is there some generic kind of a strategy that we that we can make them work as well as


the approximate message passing or at least closer so that would be question related to the second point and question really and the third point is i showed you that gradient flow sometimes work even when the spruce local minima are present we showed that using the cast race approach but what about stochastic gradient descent so far i was only talking about gradient no stochastic so let's say a year ago i would have stopped here and say that you don't know the the green questions would be open but today actually i do have an answer to


each of them so i will start with the first one so is the same methodology applicable to some simple neural networks and in statistical physics when we kind of set up a model for a data so that you can keep track of constants and and not only rates and finite sample complexity the kind of popular model in which something like that can be done at least in simple neural networks is this teacher student setting so now i'm switching the mall no more spiked matrix tensormal in the talk now i'm going towards this teacher student neural networks


where at the input i put id data not only iad sample from sample but also the components of every sampler id so that's of course you know not real data do not like look like that but that's part of the simplifying assumptions here then i take a neural network like for instance the one here i generate the weights of the neural network in some again random way i let this teacher neural network generate the labels y using those ground through ways w star and then i hide the w stars i never show to the student network the w


stars i just show to the studio network as traditionally the set of samples x and y and i will have n samples each sample will live in dimension p so before p was the order of the tensor now p will be the dimension till the end of the talk and then the student may or may not not know the architecture of the teacher network i will actually be telling you about both cases in this talk and the question is what is the generalization error that the gradient descent is reaching depending on the number of samples that


it got from the teacher so this is a setting of a neural network that have been studied in physics for 30 years kind of the most common example would be this teacher student perceptron where the nonlinearity that the teacher is using is just a sign but with just a sign and no constraints on the weights w this becomes a convex optimization problem so today we are interested in intrinsically non-convex optimization problems so in order to make it more interesting and intrinsically non-convex we will actually be looking today at the face


retrieval where the teacher instead of using a sign uses an absolute value on the scholar product of the samples and the ground true base or the teacher weights w star so the labels here will not be just binary this we will be looking at this regression problem the data will be generated as is written here the input is gaussian the labels are obtained as the absolute value of the of the scalar product and the neural network then sees the set of samples and tries to regress the y or the x on the y and so what do we know again again


without yet talking about gradient descent what do we know about this problem information theoretically and in terms of the approximate message passing that also here is conjecture to be made the best of the of the polynomial algorithms so here i am showing you the the mean square error of recovering the w star which is you know very related to the generalization error just like this plot as a function of the alpha which is the ratio between the number of samples and the dimension and both number of samples and dimension


are large i'm in the high dimensional limit and the ratio is some small constant here you know between 0.3 and 1.2 so information theoretically the generalization error can be zero start as long as soon as you have more samples than [Music] than is the dimension in this problem that corresponds to the orange line now algorithmically you need slightly more samples than the dimension about 13 percent more for the approximate message passing to work and be able to generalize perfectly in this problem so now we will be looking at what


gradient d sent us and how it compares to to what to do this so gradient descent on watch loss which loss functions so corresponding to the phase retrieval the natural loss function is the one i write here that would correspond to in a sense to to to neural network with no hidden variable one hidden variable with quadratic activation function so that's an actual you know i just square the label sends instead of putting absolute value i put a square here and then i'm looking at the the performance of the gradient flow


so just to you know set up the stage of what is known and what kind of we can expect so as i said if ignoring gradient descent we know that starting from one the problem is solvable information theoretically and starting from 1.13 by some very adapted algorithm to this problem for the gradient flow what we know is is this work that that popped here that vigorously shows that randomly initialized gradient descent will need uh we'll need the dimension times some polynomial of the log of the dimension samples in order to be able to solve the problem


so there is quite a big gap between 1.13 and poly alpha it is 1.13 and alpha that is some polynomial of logarithm of the dimension so as a physicist we always try to look numerically what's actually going on so numerically if we are looking what's the fraction of success of writing this in terms of solving this problem as the dimension is growing so here the capital n is actually what i call the p is the dimension so we are seeing that at alpha that is around six or seven it's already solving the problem almost always so so can we understand


that a bit more theoretically not just running the gradient decent that's of course nice but that's that's not just satisfactory so we take lessons from the spiked two plus piece per mole that i showed you and we kind of ask ours okay could it be could it be happening similarly as there could it be that the gradient flow first goes to the threshold states and then what matters is a kind of bvp-like transition of the hessian of these threshold states that drives the success versus failure and we just test this numerically


whether it looks like true and it actually does in the sense that if we look at the non-planted phase retrieval that would be the right hand side here that defines the value of the loss function that i call the threshold value and then if you look at the dynamics of the gradient flow in the planted version we see that it's quite possible that it's again going to the threshold and then away from it or not or staying stuck there so we again hypothesize that this is actually the mechanism and put it into equations this time the


equations are slightly more complicated but we can still do it actually using a recent random matrix theory results from works of ulu and square collide collaborator lee and also the fact that the threshold states are marginal meaning that the lowest eigenvalue corresponding to them is stuck to zero and this if we combine it gives us an expression of what should be the threshold above which the gradient descent works as a function of this probability distribution of the true labels y and the labels y hat that the gradient


descent is currently estimating so that probability distribution is still something pretty not non-intuitive to capture but in the within the within the theory of one-step replica symmetry breaking that is again one of the methods coming from statistical physics we actually can estimate this probability distribution between the joint the true label and the label that is currently estimate currently estimated by the grading descent and this is shown in this picture i show so in the left hand side i actually show


the loss the value of the threshold energy of the threshold loss as it comes from simulations that is the purple line and as it comes from this one rsp theory they are not exactly equal here that's not the conjecture here is that this is not exact but they are close so so we use this as an approximation on the right hand side i am showing again numerically obtained moments of the distribution on which the formula depends and the moment is computed from the one rsp theory and the agreement is pretty good so when we put


these things together you know ignoring the little differences this actually leads us to an estimation of the gradient decent threshold that is about 13.8 so if i put it back onto this axis i showed you that the numerics this constant starting from which gradient descent is working looks like 7 from approximated theory we get something like 13.8 so we are not sure where the discrepancy comes from whether it is finite size effects and the numerics would actually converge to the 13.8 or whether it is the small difference


between the exact result and the one rsv approximation so both is both is possible but what is nevertheless clear is that it seems that it's a constant the the polylog of b is not needed so here is another open problem proof that you know any constant times p is actually a sufficient number of samples for randomly initialized gradient descent to solve phase retrieval in time that is p times times some polylog p so in the time the polylog p is not avoidable because otherwise you're just stuck at kind of zero correlation but in the number of samples


the conjecture from from our work is that it should be avoidable and that the true constant is somewhere around 7 or 13. but what about the gap between the the performance of the approximate message passing and of the grand descent there is you know still big difference between say one and ten so can we somehow close that gap can we do something generic that would diminish that gap so that's the question for the next a few slides and that that corresponds to to this you know when i was concluding about the spike matrix model that was the second


input so that's the second point to which we are doing and surprisingly or not we will do that by over parametrization so let's still look at the phase retrieval so the problem the regression problem we are trying to solve here is still the same so phase retrieval with random gaussian data and the teacher coming from a gaussian and generating the labels this didn't change but what changes now is the loss function so now the loss function that i will be considering doesn't correspond anymore to simple the simplest neural network with no


hidden unit or one hit unit that's the same now the neural network will have m hidden units and i will be working in a regime where the number of hidden units is bigger than the dimension p so this is the over parametrized two-layer neural network i will be optimizing over the weights of the first layer this matrix w and the second layer will be fixed the weights of the second layer will be fixed to one over m or to one and i use the scaling one over m so i'm not really learning here the second layer but that


would know the conjecture kind of is that that wouldn't change much in the in the overall message of this so again i'm just running gradient flow on this loss function with a random initialization so how does this behave so this is a wide over parameterized two-layer neural network does this solve the phase you achieve or not and this is from a paper that that we that came up in june with a colleague from uiu eric vanden eiden and same student as stefano sarah mannelly where uh in two theorems we kind of answered


provide some answer to this question so the first theorem is it is purely geometric it is telling us that if you are looking at the loss function as i just defined it then if alpha is so alpha again was the ratio between the number of samples and the dimension so if alpha is smaller than 2 then this loss function has many spiritual minima and if alpha is bigger than two then the probability that the only local minima that is there corresponds to the ground roof that would be this a star that is just the teacher vector times is transposed is


actually the only one so we believe that this is actually with probability one but what we could prove is this is only with positive probability but there is something clearly happening about the threshold alpha equal to and when we and this is purely geometric no gradient descent yet but when we put this together with our second theorem about the gradient descent that tells us that in terms of this parameter a that is the weight matrix times its transpose the gradient descent always goes to global minima then putting these two together actually


if there is only one global minima corresponding to the to the ground true with finite probability well then the gradient descent also goes there so this means that the gradient descent solved this problem by optimizing this loss function corresponding to the over parameterized neural network starting from alpha equal to and here is just a little plot that you know that shows that just running numerically writing descend on relatively small systems is pretty consistent with that result so if i put that back into onto undo the


axis of alpha i obtain that by using over parameterized nail network i can push down the threshold at re starting from which the gradient flow is working down to two so not yet to the 1.13 of amp but much lower than if i was not over parameterizing so the conclusion is here that over parameterized neural networks need fewer samples and this is a quantification of how much fewer samples in this particular model and the open problem would be and that i i really don't know the answer like is there a neural network architecture


maybe if you make it deeper or over parametrized differently for which the plane randomly slice guardian descent would just need less than alpha 2 so less than 2p samples so i think that's an interesting kind of concrete question for this particular model and i i might have time for the third point stop me if i don't but i wanted to mention the third point about what can we say about the stochastic gradient descent so far i was only talking about gradients and or more precisely gradient flow because i was always considering the


continuous time version because that's the one that is easier to analyze so what about stochastic gravities so first of all a reminder was what's stochastic it's you know the same thing but we are taking the samples one by one so when we say stochastic writing decent in the literature we mean usually one of the two following things so either we mean the online stochastic grading descent where each iteration uses a fresh sample and never uses a sample that was ever seen before and i like to call it online stochastic


running decent that one is simpler to analyze but it's also less interesting because it minimizes directly the population loss and there is no notion of the generalization gap the training test are the same so a lot of the mysteries about how comes the train error can be so much smaller than the test error that we are kind of asking and deep learning cannot really be answered by looking at the online stochastic gravity scene it's also not used in practice what's used in practice is multipass stochastic gradient descent


where we use one or few samples at the time but we reuse the samples many times and this is much harder to analyze this has much less kind of existing theory but that's the one we want to look at because it's used in practice and it can access you know non-trivial generalization gap then so can we do that so first of all the first step above which i didn't even talk before because it was obvious how bright and descends in the limit of small learning rate becomes gradient flow for the stochastic gradient descent is


not so clear what is the limit of the infinitely small learning rate that actually is well defined so so i'm just explaining it on this slide so if i define stochastic gradient descent using this variable s of t that would be you know one for some samples and zero for some other samples so if i do what is usually done in stochastic gradient descent is that every at every time step i randomly choose who is in the batch and who is not in the batch well then this doesn't have a well-defined limit of the learning


rate going to zero it doesn't really have a gradient flow limit so this is not so nice for the dynamic community of theory so we instead define a slightly different version of stochastic iron descendant we call persistent stochastic variant descent where we as before have some fraction of samples that are in the batch but instead of reshuffling the batch at every time step randomly we actually decide at each time step whether we keep or not the the sample in the batch and we we keep the we keep that sample with some typical time that we call here


the persistence time following the rule that is written here so if we do it this way then we can take the limit of the learning rate to zero and it has actually a well-defined stochastic guide in flow limit so that is the that is the that is the dynamics that's that i will be analyzing on a model that here will be um slightly different so it's not the phase retrieval or the model on which we will be analyzing this it's uh it's just a gaussian mixture a supervised learning of a gaussian mixture so in the two cluster case


that is on the left here i have two clusters one cluster is plus labels plus one cluster is labels minus and i'm trying to separate them so that's very simple i can just imagine there is some hyperplane in the middle and separating them but the clusters are really noisy so i'm in a regime where i will not be able to separate perfectly and yet so so this will even lead to a convex problem so that's more for kind of a comparison but the one that will be interestingly non-convex is the three cluster case where i have three gaussian clusters


two on the periphery and one in the middle but the two on the periphery they have the same label so this is a data set that is no longer linearly separable so actually to be able to to to to have some meaningful learning the loss function that i will be using for these three clusters is is actually is actually kind of you know using the the structure of the data set and i will be doing logistic regression but not directly on the data points but on the you know as specified here on this c c my mu okay but whether it is the two cluster case or


the three cluster case how do we describe the full trajectory of the granules the complexity is really not so much helping us to describe the the full trajectory so this will be again done with the dynamical mean fee theory but this time with a little bit more advanced version of it because for the perceptron uh case the the the simple one that we used before is is not quite working it's not it's not so the equations do not close so simply but this period is the same here we start with this high dimensional markovian dynamics of a strongly


correlated system and the dynamic domain field theory maps it into non-markovian so dynamics with memory but of one single degree of freedom so this is where we lose the high dimension and get a system that we can actually plug into computer and analyze and here uh rd you know is that one dimensional system but this is a little you know i want to get to the results or conclude so i will not be explaining this in detail but again i was mentioning this lecture by pierre francesco obani in lesos where he actually derives this equation


in details peter put on his video does that mean i should be concluding maybe sorry you have five minutes five minutes great that's fine so let me just like okay this i will not be explaining every detail but let me just explain what this is so here i actually the claim is that the dynamics in this of this classifier in for the data set that is this that is this high dimensional gaussian mixture behaves in the same way as the following scholar stochastic process for this variable h of t t is just the iteration time that


and the stochastic process actually has several types of noise so it has for instance a noise that corresponds to the regularization lambda which is just the ordinary reach legalization but it also has a noise that plays exactly the same role of the rich regularization but that came from the dynamics that is not there explicitly so this is some kind of implicit rich regularization that comes from this variable s of t that was the variable in the in the stochastic guiding descent that was deciding which sample is there and which sample is not


there so that's that's kind of an interesting you know interpretation of what this stochastic model stochastic process actually is this is a interpretation of how kind of the implicit regularization might be coming out in these type of problems the second term is directly the noise coming from the stochastic grading descent because you might be saying that here i have batches that are still extensively big so maybe the noise doesn't matter so much so it actually does it's still explicitly there even in this effective


dynamics and then there is a dynamical noise here that is just gaussian with some covariance matrix mc that is consistently computed you know for via set of closed equations and then there is some memory kernel here mr that also needs to be consistently computed from a set of equations so i put these in very small because you know these are these are kind of hard to grasp in just like one minute but it's again some some work that comes from recent works in statistical physics and that can be directly adapted


to this problem as we did in this paper here again open problem would be of course challenge once once one goes in detail through what this color stochastic process is can we actually prove that it is equivalent to what the grad what the stochastic gradient flow is doing in in this problem and you know once you compute it numerically all these uh quantities uh from these equations then you can compute everything else including the training loss the test laws the generalization here the corresponding accuracy is this


is this is highlighted in this set of equations so with that what you can do is you can plot for instance a picture like that where i plot the generalization error and the training error in the intersect the generalization here in the main part is a function of the time and now the points that's just running the stock the persistent stochastic gradient descent numerically on this data set so this is just a plain simulation of this simple neural network and the lines that's a result that i get from the dynamic community fury


so it is you know you can see that it is describing the whole trajectory what is the generalization error at any given time for the persistent stochastic guiding descent so you see that at large times it is going somewhere but at intermediate times there would be some early stopping to do here and depending on the batch size and order on the persistence time is not exactly the same curve and quite interestingly even the the orange points actually would be the normal stochastic gradient descent and the line corresponding


that is plotted there is is in a sense not quite justified by our theory we just kind of ad hoc discretize the theory in the same way we would discretize to do the canonical stochastic descent it still seems working which is a bit puzzling to us it really kind of we don't really know why should it but so we didn't maybe even need to do this this persistent stochastic guidance descent for this theory to work or maybe yes maybe there is some small error that doesn't show up on you know on this comparison with numeric


so this we don't know yet and you can you know look at other things for instance this um this case of two clusters is quite interesting because if you don't do the stochastic gradient descent but fulbright guiding this and that would be this this picture that is of course just a special case of the equations that i just wrote it has this specular behavior that if you initialize so r is the variance at initialization if you initialize at zero with really small variance then after one iteration you reach the base optimal error


and then the gradient descent is actually driving you away from it and training accuracy is growing you know this is a regime where you're interpolating the training accuracy goes to one but the test error is getting worse so actually after one iteration you were perfectly optimal but then the gradient is driving you away from the perfect generalization not perfectionist the optimal generalization point so that's a kind of specular property of this particular two cluster model that we that we discovered in this paper and


and you can look at other things such as you know looking how different the dynamics is when you are changing the batch size and comparing how the dynamics changes when you're changing the ridge legalization of the loss and this is what's on these two pictures so i see that as i'm changing the batch size the time scale where i start to decrease is changing because the number of iterations i need with smaller batch size is bigger so this intuitive but otherwise if the curves look kind of comparable to the ones if i


was adding more and more regularization in the sense that the blue ones that is just gradient descent no real backsides one is as if i was regularizing only a little in this case and regularizing a lot is actually corresponding to the smallest batch slice in this in this picture but you know that's that's just like observing how the how the curves look like so there's no like formal statement here so this was the last figure that i wanted to show you and just to conclude so you know i was telling you about this dynamical mean


field theory and the results coming from it that is able to track the full trajectory of the grounding descent or the stochastic realities and for a range of our synthetic models for the data and there are of course many directions in which this this we would want to uh extend it including all the open problems that i stated we would like to have more more math and rigor into that but also deduce more of insights just by looking of what the dynamic and your equations are telling us as a function of all the hyper parameters and


the nature of the noise that i was describing in the in the equations and we can look at other data models these are not the only ones for which this this this can be written and we can look at networks that actually have hidden variables and of variants of the gradient descending stochastic guardian descent that have momentum for instance etc so this is hopefully to come and i just flashed back the list of papers from the beginning that that i covered in this store and open for the discussion if there is still time for the


hi there today we're looking at manifold mix-up better representations by interpolating hidden states by because Verma a tall number of big names on this paper as you can see and I also saw this at ICN also I was intrigued by it they proposed manifold mix-up which is sort of a regularizer of neural networks specifically of supervised learning and it's actually a pretty simple concept and they kind of show that it has some nice properties and outperforms of the regularizer z-- so what's the problem the problem is that if you look at this

Manifold Mixup: Better Representations by Interpolating Hidden States


spiral problem here which is often kind of used to to show properties and neural networks what you have are blue points and the blue points on our one class and the red points are another class you see the two classes here are in this kind of spiral pattern it's just the data space is just two-dimensional so you see here this is one class this is the other class this is pretty difficult for a model to learn because of course the easy models would be like linear classifiers but there's no way to like put a line through this such that one


classes on one side mostly so neural networks if you train them they will give you something like you see here they will try to kind of bound the regions with the red points from the blue points but then gets you know is it there's some weird things like here is a weird thing here is a weird thing so you'd imagine a correct model would actually classify this area as blue but the the neural network has no concept of of let's say that the spiral should continue that thus it it simply sees our here's blue here's blue here's a bit of


a gap in the training data so yeah it in this case it assigns a red class to it this is one problem that the decision boundaries are rather not say squiggly and irregular and the second one if you look at the actual colors full blue means very confident blue class full red means very confident red class and in between you kind of see going into the the white so if you look very closely I can actually zoom in more here if you look very closely you'll see that the blue gets lighter and lighter until it reaches white and from here the red goes


lighter and lighter until it reaches white and white means not confident white means like 5050 they see the that area of not confident is actually very small right if you consider a point here is actually still very confident that it's a blue point and the area of Nan confidence is very small even though maybe as as humans we would judge like a relatively large band in the middle to be not confident like if we get a point like this and the third problem is that you can see in multiple locations like here or here or here that the decision


boundary is very close to the data points unnecessarily close so especially if you look here the decision boundary could be much more optimally placed probably something like this right given the training data but the neural networks because the only C training data they they have no basically no incentive to do this all right one might think of you know something like a support vector machine that actually has an incentive to to put the decision boundary away from the from the training data but the neural networks currently


there are not SVM's they're basically logistic regressions and as such have no no incentive to do this so this these are the problems the other problems are this is the input space if you look at the hidden space so they build neural networks specifically they have like the 2d input and then that goes through a bunch of layers and then at one point there's a bottleneck layer was just two hidden nodes and then I guess that goes again and then it goes into a classifier so in this bottleneck layer they analyze the hidden


representations of the data points and in this case for this spiral dataset what happens is so in red you see again the red classes in blue the blue class it's 2d so you can plot it what it does is it bunches up the hidden representations fairly fairly so it bunches them kind of up it spreads them out in directions here here here most are bunched up here and it does these kind of weird arrangements here with the pockets of those and of course the neural network is powerful enough such that it can actually you know separate


all of this from each other but it's not ideal and the black dots they represent kind of points in between or points from the input space that are not part of the training data so they say they sample uniformly in the range of the input space you see that the black dots are all over the place right some are confidence blue some are confident red some are like somewhere right what you would expect from a good model is that if you input something that's kind of in-between or not really sure not even part of the input distribution that it


assigns like a low confidence to it that it says well I'm not sure about this this must be somewhere in the middle so just to jump and jump forward to the results what does manifold mix up do without knowing what it is in the same data set it gives you a picture like this you see the decision boundaries are much more smooth right the region of no-confidence or of low confidence indicated by the light color is here is much larger and also the decision boundary here we had specifically this data point here you


see the decision boundary is pushed away well you could argue about that particular point but the decision boundary is generally pushed away from the data points you also see no more kind of these squiggles here it doesn't happen in in here also if you look at the hidden representations the hidden representations now are spread out the classes are bunched up so not all the points are bunched up but the the points of individual classes are bunched up together and the randomly sampled points are in the middle as they should be si


only confident red is down here confident blue is up here and everything in between is unconfident and third if you look at the singular value decomposition of the hidden layer and that's kind of a measure of how spread out in the different dimensions a dataset is you see that the manifold makes up here in green it concentrates or it it real owers the singular values of the kind of lower indexes so the first singular value is large which means that there is like a dominant direction in the in the data and this is


done for each class separately as I understand it it puts a lot of weight on the first singular vector and then it pushes down the contributions of the other singular vector which means that the data set that is analyzed is is concentrated in two fewer directions of variance this is layer one and here is layer three means so you see it happens in both that the manifold makes up compared to the baseline model does this so now you might ask what is manifold mix-up it's actually a pretty pretty simple concept right here is another


comparing it to other kind of regularization techniques and showing that none of them really does this so manifold mix-up is this basically what you do is when you train a neural network you have input data and you take mini batches of input data specifically you take two mini batches x and y and X prime Y Prime all right and then what you do is if I have the draw the neural network here so here is the inputs like a picture of a cat [Music] it goes through layers right and then what you do is you say at some


particular you say stop stop right you take the representation up you and you do this with two different mini batches so here is this is cat one down back here is cat two dog that's a captain you pass it in right here you take it out here you pass it through the network and you take it out so you now have two different forward paths of two different mini batches and then you define a lambda and I guess they randomly sample a lambda in zero one right in the range of 0 1 so this is a mixing coefficient and then you mix you


say lambda times hidden representation of batch 1 plus 1 minus lambda of hidden representation of batch 2 and that is what you pass through the rest of the network right so basically you forward propagate to different batches until a certain layer here then you mix them with a random coefficient and then you pass it through the rest and then the only thing you also have to do is then at the end if you think of the labels of these two things you want to mix the labels in the same fashion so you want to mix lambda


times y of batch 1 plus 1 minus lambda of Y of batch 2 and and then this is your training signal for whatever comes out here right so it's it's um these are these are one hot labels so if it's class three its zero zero one zero zero and if Y two is class five its zero zero zero zero one and then you simply mix the two alright and that becomes your training signal so in a practical example if let's just have a mini batch size of one so just one sample if this is cat and this is dog you would pass them forward right you would mix so in


the hidden representation it would kind of become a cat dog maybe you do it 50/50 but then you would also mix the labels of cat and dog 5050 and tell the net well this is a mixture of 50% cut 50% dog and then you would train the network to predict that 50/50 coefficient so they do this the question is at which layer do you do this and they simply I think for each mini batch sample one hidden layer at random they might have someone waiting or something but the way they describe it is they simply sample one layer or mean per mini


batch and then do the mixing there and then you can actually backprop through everything everything is differentiable this mixing is differentiable so you can backdrop through any of everything and there's even you know a kind of an engineering trick to only use a single mini batch by mixing it with itself so that's that's pretty neat so this is manifold mix up as you can see here is the that's kind of the description you mix the hidden representations with lambda and you mix the labels with the same lambda and that will become your


actual training signal all right so they give some theory to it that it flattens representations and specifically they say under some conditions namely if the network is large enough so if the dimension of the hidden representation is of a certain size then if you optimize this manifold mix up like if you optimize over every London over the entire training data set what you will end up is actually a linear rally near function of the input this is not too surprising that if you because what you do is you mix linearly


this mixture happens in a linear fashion so if you optimize for and you not only optimize for the training set but you optimize for every possible mixture of the training set a linear mixture your minimization your minimizer function will actually become a linear function it's not surprising but they have a formal proof of this and they also have a proof that if certain assumptions are given then the minimizer's if you apply the minimizer's the hidden representations will actually fall on a low dimensional subspace which is also


not surprising but it's kind of the theoretical an analogue to what they show with the singular value distribution that it basically suppresses low singular values that means the data set is much more into a single direction the hidden representations sorry all right so this the theory part is you can you can read it if you if you want to it's yeah it's it's - the results are to be expected I would say from what they do and the last thing they give a pictorial example of why none fold mixup flattened representations so


both of these things the fact that the minimizer's will become linear functions and the fact that the singular value spectrum is more concentrated on the first thing you'll evaluate a shion's are flattened and here is a pictorial representation so in this case what happens if you if you basically have these four data points a 1 a 2 B 1 and B 2 where a 1 and a 2 are blue class and B 1 and B 2 or red class and if you now look at an interpolation point between the two so if you look at this interpolation point between a 1 and B 2


what happens is that in this case this should be 50/50 blue and red but if you now look at the points that it where it's not interpolated on this is very close to a 2 in this case it probably should be more like 95 blue and 5 red do they say here well if you use manifold mix up to learn the network what you'll actually do is you say ok actually this hidden representation needs to be pushed outward and you will achieve something over here where any mixture of two points of the opposite class will actually give you a 50/50 so all the


midpoints here will give you a 50/50 mixture between the labels which basically means what you end up with is a line between this data and this data and it means that basically the network becomes more linear and the representations become more flat because flat is the optimal if you distribute our flat all the distances to the line are the same and this objective is optimized and this is basically my my kind of biggest problem with the method is that it it kind of mixes the input with a linear function where we know


that that is kind of not the shape of the true data manifold the input manifolds as you can see here the input manifold here isn't linear or flat it's actually very very tangled and we know that neural networks as you continue in the layers will flatten those representations because ultimately at the end it needs to classify the dataset linearly because the last layer is a softmax layer but the the idea that you could apply this to any layer seems a bit shady to me of course it works and they show it works and it's really nice


that it works but apply applying this to low layers and neural networks seems a bit not principled to me so I think this is not the end of the story of this line of work and there is kind of more that can be done in a more principled fashion but in any case they show that this actually works in terms of performance on generalization on kind of standard data sets so they have results on C 4 10 and C 4 100 which are famous image data sets and they show that they have a regularizer outperforms others and they also show that they can withstand


one-step single step adversarial attacks more it kind of better so they have a better perform performance against single step adversarial attacks after regularizing mostly again giving kind of an idea that the if you push if you push it if you have a two points this is X this is X X 1 X 2 they're of different classes if you put the decision boundary really close to X 2 then an adversarial attack can simply move the point across the decision boundary with a very small step but if you actually have the decision boundary


pushed away from both data points then the an adversarial attack must go a very long way to the decision boundary and thus if you limit the size of adversarial attacks which is what you usually do you can maybe not reach this decision boundary and thus you mitigate some of the problem so it's pretty cool I think yeah there's work to be done but I think this is pretty cool it's implemented pretty easy I I've seen there's a lot of libraries already available with it in and yeah won't hurt to add this to your code and make your


hi there today we're looking at neural architecture search without training by Joseph Miller Jack Turner amis Torquay and Eliot J Crowley on a high level this paper performs neural architecture search by looking at the correlation matrices of the Jacobian of the of the data when you pass it through the network and it does so at initialization so you pass the data look at the Jacobian and if it's very correlated then the network is bad and if it's very uncorrelated then the network is good and by simply observing that they can

Neural Architecture Search without Training (Paper Explained)


already achieve a very good score on neural architecture search benchmark alright that was a high level and maybe a bit too simplified but that's sort of what's going on ok let's dive in so what's neural architecture search neural architecture search is the discipline of you are given a data set let's say here we have a data set which could be something like C 410 which is an image data set and you are given a sort of a training procedure let's say Adam or SGD for 100,000 steps or something like this with many batches of size 64 ok and


you're given a loss function which the loss function here could be the cross entropy between the outputs of the network which we'll call L and the label Y and your task is now to find a neural network architecture that conforms to these specifications but gives the lowest possible loss or de sorry the highest possible validation accuracy in this case so this here would be like the Train and then you'd have the test accuracy or the validation accuracy okay so you could decide well I'm gonna go with you know first like three


convolutional layers each one having like a real ooh non-linearity but you could also say well I'm going to build like a skip connection from here to here you could also say that I'm going to down sample by you could have maybe a bigger stride and so on so the kernel size of the convolution you can vary until now people have done this by hand right in effect we all use like the same 10 to 20 different architectures so if it's an image problem we tend to go for like a res net or a wide ResNet like a vgg style architecture someone has come up


with those at some point with each of those discover that it works well and we don't really do much exploration we simply kind of use the same things over and over and the the truth is that there might be much better architectures that were simply not exploring right there might be much better building plans for networks that we don't know of that might perform a lot better with the same data and the same training so neural architecture searches the process of automatically searching for these better architectures of course that's a


combinatorial problem but the idea is that you know you can actually learn to construct good architectures and by doing so you can you can sort of speed up this process that is manual otherwise and the idea behind it is there some regularity of when an architecture is good there's some like high level of pattern that you as a human maybe cannot really grasp but like a machine can figure out which architectures are good and which ones aren't so there have been a few inventions in this in this area but they are mostly costly that's what


they say here the time and effort involved in hand designing deep neural networks is immense this has prompted development of neural architecture search techniques to automate this design however neural architecture search algorithms tend to be extremely slow and expensive they need to train vast numbers of candidate networks to inform the search process so what neural architecture search methods do is what they'll have is they'll have something like a controller in the controller itself of course is going to be a neural network


so there'll be this thing that will be the controller and the controller will emit like a building plan so the controller will emit like a building plan for this network right here and then you train the entire thing once through for the entire hundred thousand steps and then you observe the final validation accuracy which might be something like eighty percent and then you know okay this is eighty percent so you feed the eighty percent into your controller and the controller out puts the next building plan that it thinks


will score higher and then you train the entire thing again and you maybe observe a 70% accuracy you again feed that in right and the controller realizes oh I may have done something wrong let me try something else and does it yet if this looks like reinforcement learning to you that's because this is reinforcement learning so there really is see here the controller would be the agent the percentages here the accuracies would be the reward and the invasions would be basically this thing here this thing would be the actions but sometimes it's


the observations and you need to score the different things okay so the problem of course with this is that the reinforcement learning requires a lot of data it requires a lot of steps to converge because the signal from the reward is just so weak you simply get one number for your action and you don't know what you can change to make it better you simply have to try so you need a lot of steps but this thing here is mighty slow because each each single step in your reinforcement learning procedure involves training an entire


neural network for like this many steps ok so all of this is ginormously slow and resource intensive and that of course blocks a lot of research because you know we started with the plan to automate this part right here but automating it itself is super expensive so they go for a different solution they say this could be indeed if we could infer at Network sorry if we could infer a networks trained accuracy from its initial state okay it seems a bit out there but let's let's give them benefit of the doubt in


this work we examine how the linear maps induced by data points correlate for untrained network architectures in the NASA bench 201 search space and motivate how this can be used to give a measure of modeling flexibility which is highly indicative of a network strained performance we incorporate this measure into a simple algorithm that allows us to search for powerful networks without any training in a matter of seconds on a single GPU okay and they have the code available right here if you want to go and check that out so let's go in let's


go into that the claims are pretty big and the reasoning behind the claims is the following observation you can already sort of see in this graphic right here we'll we'll go over what it means in one second but what they do is they take different networks in this search space and the search space in this case is given by this benchmark so this benchmark basically has a long list I think of architectures that you could consider actually so it's a it's a constructive list so they don't actually give you the list but they give you like


a a way to construct architectures and they took those architectures and they rank them by how well they score on C 410 so there are very good architectures which are here there are good ones there are mediocre ones and then the bad ones okay and you can see that the histograms here of whatever they measure they look quite different so the histograms with the good ones they all have kind of spiky around zero and the histograms of the bad ones all sort of look spread out so this is the measure that they're going to propose is they have some sort


of number some sort of histogram that they produce and if the histogram is very spiky and close together around zero then they conclude that this network is good and if the histogram is very spread out like this they conclude that the network is bad now these histograms as you might expect they are computed not from the final trained Network but they are computed from the initial Network so here they show at least you know in this case it seems to be that there is a general correlation between the trained accuracy and how


this histogram looks and we're going to explore what they do so it's essentially it's pretty easy they compute the linear map around each data point so what is that if you imagine a neural network as a nonlinear function which I guess you should because it is so let's imagine it as like a nonlinear function from X to Y what they'll do is simply they'll look at a given date training data point which could be here right this could be the X and this could be the the Y and in fact let's look at it in lost landscape not even in Y but


in L in terms of the loss because we don't need necessarily a single label this could be for unsupervised this could be for anything okay so it Maps a data point to a loss now what we'll do is we'll simply linearize the function around that point which means we'll just freeze all the nonlinearities in place and that will give us this linear function right here okay we just observe that this linear function can exist it's the tangent to the lost landscape and it's at a particular data point right it's in data space not in in weight


space then we look at a different data point so we look at this data point right here another data point what's the linear function around this one is sort of like whoops T is like that and then around this one is like this okay so this is one function now let's look at a different function right here so L X and we'll look at this function the linear function okay so for some reason this is like this and if we consider two data points their linearization is very similar now imagine that these two have been produced by the same sort of neural


networks it's just the architecture is a little different but they have been produced like they have the same number of parameters in the neural network which neural network would you prefer remember you can in by training the neural network you can actually shape this loss function you can kind of shape that around so which one would you prefer I personally would prefer the top one because the top one already tells me that hey you know I might have 10 parameters here and this already sort of looks like each of the 10 parameters is


doing something so if I then go into my 10 parameters and I you know turn this knob right here then I might you know up this bump or down this bump or do something with it but the sort of frequencies curvature the randomness of the function the way that it fluctuates tells me that all of the different parameters must have some sort of effect right because it's of quite an expressive function whereas if I have the same number of parameters for a function like this this sort of tells me well maybe only one of the when we only


one of the weights is actually doing something maybe only one of the dimensions is doing something this seems odd right that even though I've initialized it randomly a super regular function like this comes out so maybe all of the all of these parameters down here they don't do anything or it is so somehow the signal doesn't get through so that's I they don't explicitly say it in these terms but this is how I make sense of this what they're saying is that if you look at the linearizations of the function and you look at the the angle right here


so the angle in this case is that and in this case is that and in this case is that so you look at the slope here and the slope is basically the gradient of these linearized functions and what you want to do is you want to look at the correlation between those of the different data points so here you have three angles one is very short one is very bit longer like this and or no even like this and one is even over ninety degrees like that they are not correlated at all right they're all very different however the angles here they're all


quite the same as you can see so what they propose is the following let's send all the data points or in that case all the data points in a particular mini-batch let's send them through the function and let's calculate their linearizations so the linearization is nothing else than you send them through the network to obtain the F value for the x value and then you calculate the gradient with respect to the input now you have to get used to this a bit because usually we calculate the gradient with respect to the weight but


now we calculate the gradient with respect to the input which if this is a linear function so if you have a live f of X equals WX like a linear function then this gradient del F del X would just give you the W will give you the slope of the linear function and the same in the neural network when you linearize it alright so we're going to obtain all these linearizations and that gives us the this matrix J right here and what we can do is we can then observe the covariance matrix of J of all these linearizations the covariance


matrix simply tells you how to data points vary with each other and in fact they don't look at the covariance matrix but they look at the correlation matrix which is simply the scaled covariance matrix so one entry in this covariance matrix so you have n data points and this gives you a matrix that's n by n and that particular entry here like the entry IJ would simply state how does the angle of data point I correlate with the angle of data point J okay that's the that's the covariance matrix and now the hypothesis is if all


of these data points are sort of independent like in our very expressive function here then the these correlations they should not be high in fact most data points should be rather uncorrelated however in this case right here if the function is sort of kind of degenerative or something not very expressive then all of these all of these angles or of these linearizations should be highly correlated and that's what you see in this graph right here this right here now is these correlation histogram of the correlations between


local linear maps across all pairs of items in a mini batch of C 410 training data each pod is a histogram for a single untrained na s bench 201 architecture so remember the expressivity is important because we want to train that function and therefore it's important that every parameter does something and if it's degenerate we can't train it well and that's I find that's the reasoning they they sort of say this but not I might make I might make the wrong sense out of it here but it seems to me like that's what's actually going on so you can see


this is simply these matrix values rolled out and then plot it as a histogram so what does it mean when the histogram is like super spread out like this it means that there are a lot and I think down here our axes yes there are a lot of data points that correlate highly or anti correlate highly with each other okay which means that exactly this degeneracy happens either too high or too negative high correlation means that they're very much they're kind of the same thing so there is if you have as many parameters as


data points that means that one parameter can potentially serve these two data points or these two that are correlated by one or negative one you don't need both parameters and therefore you have a lot of parameters doing nothing whereas over here with the good networks you can see that this spikes around zero meaning that the data points are not correlated or the linearizations around the data points are not correlated and therefore you can sort of shape the function around each data point however you want which we sort of


know that neural networks what they do is they're so over expressive that they're actually able to shape the functions around the data points without necessarily looking at other data points nearby and that expressivity is what what you want and that expressivity is what this in part measures okay so they make a they have some experiments here where they validate this so for all these architectures in this benchmark and maybe I should tell you what show you what the benchmark looks like so the benchmark has this particular form this


particular form there's this skeleton and in this skeleton there is this block and it's always repeated and you're basically your task is to determine what this block should be so this block has an input node a and an output node D and two intermediate nodes and what you have to do is basically you have to determine these connections right here so there are six connections and for each one you have the option of putting different things there like you can see you put can put a convolution you can put the identity function which is a skip


connection zero eyes I'm I don't maybe that's the zero function so it basically means nothing I'm not so sure honestly but you could technically put a convolution here and here right or and or different convolutions or things like this so there are these 15,625 possible cells okay so the nurse benchmark contains 15,625 possible architectures that you'll have to search and they take these architectures and they plot now they plot for each architecture the validation accuracy after training and the training protocol