is to solve the riddles of the universe and and that means you have to become a physicist however then I realized that there's something even grander you can try to build a machine that isn't really a machine any longer that learns to become a much better physicist than I could ever hope to be and that's how I thought maybe I can multiply my tiny little bit of creativity into infinity but ultimately that creativity will be multiplied to understand the universe around us that's that's the the curiosity for that mystery that that


drove you yes so if you can build a machine that learns to solve more and more complex problems and more and more general problems older then you basically have solved all the problems at least all the solvable problems so how do you think what is the mechanism for that kind of general solver look like obviously we don't quite yet have one or know how to build one who have ideas and you have had throughout your career several ideas about it so how do you think about that mechanism so in the 80s I thought about how to build this

Juergen Schmidhuber: Godel Machines, Meta-Learning, and LSTMs | Lex Fridman Podcast #11


machine that learns to solve all these problems I cannot solve myself and I thought it is clear that has to be a machine that not only learns to solve this problem here and problem here but it also has to learn to improve the learning algorithm itself so it has to have the learning algorithm in a representation that allows it to inspect it and modify it such that it can come up with a better learning algorithm so I call that meta learning learning to learn and recursive self-improvement that is really the pinnacle of that why you then not only


alarm how to improve on that problem and on that but you also improve the way the machine improves and you also improve the way it improves the way it improves itself and that was my 1987 diploma thesis which was all about that hierarchy of metal or knows that I have no computational limits except for the well known limits that Google identified in 1931 and for the limits our physics in the recent years meta learning has gained popularity in a in a specific kind of form you've talked about how that's not really meta learning with


Newall networks that's more basic transfer learning can you talk about the difference between the big general meta learning and a more narrow sense of meta learning the way it's used today the ways talked about today let's take the example of a deep neural networks that has learnt to classify images and maybe you have trained that network on 100 different databases of images and now a new database comes along and you want to quickly learn the new thing as well so one simple way of doing that as you take the network which already knows 100


types of databases and then you would just take the top layer of that and you retrain that using the new label data that you have in the new image database and then it turns out that it really really quickly can learn that to one shot basically because from the first 100 data sets it already has learned so much about about computer vision that it can reuse that and that is then almost good enough to solve the new task except you need a little bit of adjustment on the top so that is transfer learning and it has


been done in principle for many decades people have done similar things for decades meta-learning true mental learning is about having the learning algorithm itself open to introspection by the system that is using it and also open to modification such that the learning system has an opportunity to modify any part of the learning algorithm and then evaluate the consequences of that modification and then learn from that to create a better learning algorithm and so on recursively so that's a very different animal where


you are opening the space of possible learning algorithms to the learning system itself right so you've like in this 2004 paper you described get all machines and programs that we write themselves yeah right philosophically and even in your paper mathematically these are really compelling ideas but practically do you see these self referential programs being successful in the near term to having an impact where sort of a demonstrates to the world that this direction is a is a good one to pursue in the near term yes we had these


two different types of fundamental research how to build a universal problem solver one basically exploiting [Music] proof search and things like that that you need to come up with asymptotic Liam optimal theoretically optimal self-improvement and problems all of us however one has to admit that through this proof search comes in an additive constant an overhead an additive overhead that vanishes in comparison to what you have to do to solve large problems however for many of the small problems that we want to solve in our


everyday life we cannot ignore this constant overhead and that's why we also have been doing other things non universal things such as recurrent neural networks which are trained by gradient descent and local search techniques which aren't universal at all which aren't provably optimal at all like the other stuff that we did but which are much more practical as long as we only want to solve the small problems that we are typically trying to solve in this environment here yes so the universal problem solvers like the girdle machine


but also Markos who does fastest way of solving all possible problems which he developed around 2012 - in my lab they are associated with these constant overheads for proof search which guarantee is that the thing that you're doing is optimal for example there is this fastest way of solving all problems with a computable solution which is due to Marcus Marcus jota and to explain what's going on there let's take traveling salesman problems with traveling salesman problems you have a number of cities in cities and you try


to find the shortest path through all these cities without visiting any city twice and nobody know is the fastest way of solving Traveling Salesman problems tsps but let's assume there is a method of solving them within n to the 5 operations where n is the number of cities then the universal method of Marcus is going to solve the same trolley salesman problem also within n to the 5 steps plus o of 1 plus a constant number of steps that you need for the proof searcher which you need to show that this particular class


of problems that Traveling Salesman salesman problems can be solved within a certain time bound within order into the five steps basically and this additive constant doesn't care for in which means as n is getting larger and larger as you have more and more cities the constant overhead pales in comparison and that means that almost all large problems I solved in the best possible way our way today we already have a universal problem solver like sound however it's not practical because the overhead the constant overhead is so large that for


the small kinds of problems that we want to solve in this little biosphere by the way when you say small you're talking about things that fall within the constraints of our computational systems thinking they can seem quite large to us mere humans right that's right yeah so they seem large and even unsolvable in a practical sense today but they are still small compared to almost all problems because almost all problems are large problems which are much larger than any constant do you find it useful as a person who is dreamed of creating a


general learning system has worked on creating one has done a lot of interesting ideas there to think about P versus NP this formalization of how hard problems are how they scale this kind of worst-case analysis type of thinking do you find that useful or is it only just a mathematical it's a set of mathematical techniques to give you intuition about what's good and bad mm-hmm so P versus NP that's super interesting from a theoretical point of view and in fact as you are thinking about that problem you can also get


inspiration for better practical problems always on the other hand we have to admit that at the moment as he best practical problem solvers for all kinds of problems that we are now solving through what is called AI at the moment they are not of the kind that is inspired by these questions you know there we are using general-purpose computers such as recurrent neural networks but we have a search technique which is just local search gradient descent to try to find a program that is running on these recurrent networks such


that it can or some interesting problems such as speech recognition machine translation and something like that and there is very little theory behind the best solutions that we have at the moment that can do that do you think that needs to change you think that world change or can we go can we create a general intelligence systems without ever really proving that that system is intelligent in some kind of mathematical way solving machine translation perfectly or something like that within some kind of syntactic definition


of a language or can we just be super impressed by the thing working extremely well and that's sufficient there's an old saying and I don't know who brought it up first which says there's nothing more practical than a good theory and um yeah and a good theory of problem-solving under limited resources like here in this universe or on this little planet has to take into account these limited resources and so probably that is locking a theory in which is related to what we already have sees a sim totally optimal


comes almost which which tells us what we need in addition to that to come up with a practically optimal problem so long so I believe we will have something like that and maybe just a few little tiny twists unnecessary to to change what we already have to come up with that as well as long as we don't have that we mmm admit that we are taking sub optimal ways and we can y'all not Verizon long shorter memory for equipped with local search techniques and we are happy that it works better than any competing method


but that doesn't mean that we we think we are done you've said that an AGI system will ultimately be a simple one a general intelligent system will ultimately be a simple one maybe a pseudocode of a few lines to be able to describe it can you talk through your intuition behind this idea why you feel that uh at its core intelligence is a simple algorithm experience tells us that this stuff that works best is really simple so see asymptotic team optimal ways of solving problems if you look at them and just a few lines of code it's really


true although they are these amazing properties just a few lines of code then the most promising and most useful practical things maybe don't have this proof of optimality associated with them however they are so just a few lines of code the most successful mmm we can neural networks you can write them down and five lines of pseudocode that's a beautiful almost poetic idea but what you're describing there is this the lines of pseudocode are sitting on top of layers and layers abstractions in a sense hmm so you're


saying at the very top mmm you'll be a beautifully written sort of algorithm but do you think that there's many layers of abstractions we have to first learn to construct yeah of course we are building on all these great abstractions that people have invented over the millennia such as matrix multiplications and real numbers and basic arithmetic and calculus and derivations of error functions and derivatives of error functions and stuff like that so without that language that greatly simplifies our way our thinking about


these problems we couldn't do anything so in that sense as always we are standing on the shoulders of the Giants who in the past simplified the problem of problem solving so much that now we have a chance to do the final step the final step will be a simple one oh if we if you take a step back through all of human civilization in just the universe in check how do you think about evolution and what if creating a universe is required to achieve this final step what if going through the very painful and an inefficient process of evolution is


needed to come up with this set of abstractions that ultimately to intelligence do you think there's a shortcut or do you think we have to create something like our universe in order to create something like human level intelligence hmm so far the only example we have is this one this universe and you live you better maybe not but we are part of this whole process right so apparently so it might be the key is that the code that runs the universe as really really simple everything points to that possibility because gravity and


other basic forces are really simple laws that can be easily described also in just a few lines of code basically and and then there are these other events that the apparently random events in the history of the universe which as far as we know at the moment don't have a compact code but who knows maybe somebody and the near future is going to figure out the pseudo-random generator which is which is computing whether the measurement of that spin up or down thing here is going to be positive or negative underlying quantum mechanics


yes so you ultimately think quantum mechanics is a pseudo-random number generator monistic there's no randomness in our universe does God play dice so a couple of years ago a famous physicist quantum physicist Anton Zeilinger he wrote an essay in nature and it started more or less like that one of the fundamental insights our theme of the 20th century was that the universe is fundamentally random on the quantum level and that whenever you measure spin up or down or something like that a new bit of information enters the history of


the universe and while I was reading that I was already typing the responds and they had to publish it because I was right that there's no evidence no physical evidence for that so there's an alternative explanation where everything that we consider random is actually pseudo-random such as the decimal expansion of pi supply is interesting because every three-digit sequence every sequence of three digits appears roughly one in a thousand times and every five digit sequence appears roughly one in ten thousand times what do you really would


expect if it was run random but there's a very short algorithm short program that computes all of that so it's extremely compressible and who knows maybe tomorrow somebody some grad student at CERN goes back over all these data points better decay and whatever and figures out oh it's the second billion digits of pi or something like that we don't have any fundamental reason at the moment to believe that this is truly random and not just a deterministic video game if it was a deterministic video game it would be


much more beautiful because beauty is simplicity and many of the basic laws of the universe like gravity and the other basic forces are very simple so very short programs can explain what these are doing and and it would be awful and ugly the universe would be ugly the history of the universe would be ugly if for the extra things the random the seemingly random data points that we get all the time that we really need a huge number of extra bits to destroy all these um these extra bits of information so as long as we don't have evidence


that there is no short program that computes the entire history of the entire universe we are a scientists compelled to look further for that Swiss program your intuition says there exists a shortest a program that can backtrack to the to the creation of the universe so the shortest path to the creation yes including all the entanglement things and all the spin up-and-down measurements that have been taken place since 13.8 billion years ago and so yeah so we don't have a proof that it is random we don't have a proof


of that it is compressible to a short program but as long as we don't have that proof we are obliged as scientists to keep looking for that simple explanation absolutely so you said simplicity is beautiful or beauty is simple either one works but you also work on curiosity discovery you know the romantic notion of randomness of serendipity of being surprised by things that are about you kind of in our poetic notion of reality we think as humans require randomness so you don't find randomness beautiful you use you find simple determinism


beautiful yeah okay so why why because the explanation becomes shorter a universe that is compressible to a short program is much more elegant and much more beautiful than another one which needs an almost infinite number of bits to be described as far as we know many things that are happening in this universe are really simple in terms are from short programs that compute gravity and the interaction between elementary particles and so on so all of that seems to be very very simple every electron seems to reuse the same sub program all


the time as it is interacting with other elementary particles if we now require an extra Oracle injecting new bits of information all the time for these extra things which are currently no understood such as better decay then the whole description length our data that we can observe out of the history of the universe would become much longer and therefore uglier and uglier again the simplicity is elegant and beautiful all the history of science is a history of compression progress yes so you've described sort of as we build up


abstractions and you've talked about the idea of compression how do you see this the history of science the history of humanity our civilization and life on earth as some kind of path towards greater and greater compression what do you mean by there how do you think of that indeed the history of science is a history of compression progress what does that mean hundreds of years ago there was an astronomer whose name was Keppler and he looked at the data points that he got by watching planets move and then he had all these data points and


suddenly turnouts that he can greatly compress the data by predicting it through an ellipse law so it turns out that all these data points are more or less on ellipses around the Sun and another guy came along whose name was Newton and before him hook and they said the same thing that is making these planets move like that is what makes the apples fall down and it also holds form stones and for all kinds of other objects and suddenly many many of these compression of these observations became much more compressible because as long


as you can predict the next thing given what you have seen so far you can compress it you don't have to store that data extra this is called predict coding and then there was still something wrong with that theory of the universe and you had deviations from these predictions of the theory and 300 years later another guy came along whose name was Einstein and he he was able to explain away all these deviations from the predictions of the old theory through a new theory which was called the general theory of relativity which


at first glance looks a little bit more complicated and you have to warp space and time but you can't phrase it within one single sentence which is no matter how fast you accelerate and how fast are hard you decelerate and no matter what is the gravity in your local framework Lightspeed always looks the same and from from that you can calculate all the consequences so it's a very simple thing and it allows you to further compress all the observations because suddenly there are hardly any deviations any longer that you can measure from the


predictions of this new theory so all of science is a history of compression progress you never arrive immediately at the shortest explanation of the data but you're making progress whenever you are making progress you have an insight you see all first I needed so many bits of information to describe the data to describe my falling apples my video are falling apples I need so many data so many pixels have to be stored but then suddenly I realize no there is a very simple way of predicting the third frame in the video from the first tool and and


maybe not every little detail can be predicted but more or less most of these orange blocks blobs that are coming down they accelerate in the same way which means that I can greatly compress the video and the amount of compression progress that is the depth of the insight that you have at that moment that's the fun that you have the Scientific fun that fun in that discovery and we can build artificial systems that do the same thing they measure the depth of their insights as they are looking at the data which is


coming in through their own experiments and we give them a reward an intrinsic reward and proportion to this depth of insight and since they are trying to maximize the rewards they get they are suddenly motivated to come up with new action sequences with new experiments that have the property that the data that is coming in as a consequence are these experiments has the property that they can learn something about see a pattern in there which they hadn't seen yet before so there's an idea of power play you've described a training general


problem solver in this kind of way of looking for the unsolved problems yeah can you describe that idea a little further it's another very simple idea so normally what you do in computer science you have you have some guy who gives you a problem and then there is a huge search space of potential solution candidates and you somehow try them out and you have more less sophisticated ways of moving around in that search space until you finally found a solution which you consider satisfactory that's what most of computer science is about


power play just goes one little step further and says let's not only search for solutions to a given problem but let's search two pairs of problems and their solutions where the system itself has the opportunity to phrase its own problem so we are looking suddenly at pairs of problems and their solutions or modifications are the problems over that is supposed to generate a solution to that new problem and and this additional degree of freedom allows us to build Korea systems that are like scientists in the sense that they not only try to


solve and try to find answers to existing questions no they are also free to impose their own questions so if you want to build an artificial scientist we have to give it that freedom and power play is exactly doing that so that's that's a dimension of freedom that's important to have but how do you are hardly you think that how multi-dimensional and difficult the space of them coming up in your questions is yeah so as as it's one of the things that as human beings we consider to be the thing that makes us special the intelligence that makes us


special is that brilliant insight yeah that can create something totally new yes so now let's look at the extreme case let's look at the set of all possible problems that you can formally describe which is infinite which should be the next problem that a scientist or power-play is going to solve well it should be the easiest problem that goes beyond what you already know so it should be the simplest problem that the current problems all of that you have which can already sold 100 problems that he cannot solve yet by


just generalizing so it has to be new so it has to require a modification of the problem solver such that the new problem solver can solve this new thing but the old problem solver cannot do it and in addition to that we have to make sure that the problem solver doesn't forget any of the previous solutions right and so by definition power play is now trying always to search and this pair of in in the set of pairs of problems and problems over modifications for a combination that minimize the time to achieve these criteria so as always


trying to find the problem which is easiest to add to the repertoire so just like grad students and academics and researchers can spend the whole career in a local minima hmm stuck trying to come up with interesting questions but ultimately doing very little do you think it's easy well in this approach of looking for the simplest unsolvable problem to get stuck in a local minima is not never really discovering new you know really jumping outside of the hundred problems the very solved in a genuine creative way no


because that's the nature of power play that it's always trying to break its current generalization abilities by coming up with a new problem which is beyond the current horizon just shifting the horizon of knowledge a little bit out there breaking the existing rules search says the new thing becomes solvable but wasn't solvable by the old thing so like adding a new axiom like what Google did when he came up with these new sentences new theorems that didn't have a proof in the phone system which means you can add them to the


repertoire hoping that that they are not going to damage the consistency of the whole thing so in the paper with the amazing title formal theory of creativity fun in intrinsic motivation you talk about discovery as intrinsic reward so if you view humans as intelligent agents what do you think is the purpose and meaning of life far as humans is you've talked about this discovery do you see humans as an instance of power play agents yeah so humans are curious and that means they behave like scientists not only the official scientists but


even the babies behave like scientists and they play around with toys to figure out how the world works and how it is responding to their actions and that's how they learn about gravity and everything and yeah in 1990 we had the first systems like the hand would just try to to play around with the environment and come up with situations that go beyond what they knew at that time and then get a reward for creating these situations and then becoming more general problem solvers and being able to understand more of the world so yeah


I think in principle that that that curiosity strategy or sophisticated versions of whether chess is quiet they are what we have built-in as well because evolution discovered that's a good way of exploring the unknown world and a guy who explores the unknown world has a higher chance of solving problems that he needs to survive in this world on the other hand those guys who were too curious they were weeded out as well so you have to find this trade-off evolution found a certain trade-off apparently in our society there are as a


certain percentage of extremely exploitive guy and it doesn't matter if they die because many of the others are more conservative and and and so yeah it would be surprising to me if if that principle of artificial curiosity wouldn't be present and almost exactly the same form here in our brains so you're a bit of a musician and an artist so continuing on this topic of creativity what do you think is the role of creativity and intelligence so you've kind of implied that it's essential for intelligence if you think of


intelligence as a problem-solving system as ability to solve problems but do you think it's essential this idea of creativity we never have a program a sub program that is called creativity or something it's just a side effect of when our problem solvers do they are searching a space of problems or a space of candidates of solution candidates until they hopefully find a solution to have given from them but then there are these two types of creativity and both of them are now present in our machines the first one has been around for a long


time which is human gives problem to machine machine tries to find a solution to that and this has been happening for many decades and for many decades machines have found creative solutions to interesting problems where humans were not aware of these particularly in creative solutions but then appreciated that the machine found that the second is the pure creativity that I would call what I just mentioned I would call the applied creativity like applied art where somebody tells you now make a nice picture off of this Pope and you will


get money for that okay so here is the artist and he makes a convincing picture of the Pope and the Pope likes it and gives him the money and then there is the pure creative creativity which is more like the power play and the artificial curiosity thing where you have the freedom to select your own problem like a scientist who defines his own question to study and so that is the pure creativity of UL and opposed to the applied creativity which serves another and in that distinction there's almost echoes of narrow AI


versus general AI so this kind of constrained painting of a pope seems like the the approaches of what people are calling narrow AI and pure creativity seems to be maybe I'm just biased as a human but it seems to be an essential element of human level intelligence is that what you're implying to a degree if you zoom back a little bit and you just look at a general problem-solving machine which is trying to solve arbitrary problems then this machine will figure out in the course of solving problems that it's good to be


curious so all of what I said just now about this prewired curiosity and this will to invent new problems that the system doesn't know how to solve yet should be just a byproduct of the general search however apparently evolution has built it into us because it turned out to be so successful a pre-wiring a buyer's a very successful exploratory buyers that that we are born with and you've also said that consciousness in the same kind of way may be a byproduct of problem-solving you know do you think do you find it's


an interesting by-product you think it's a useful by-product what are your thoughts on consciousness in general or is it simply a byproduct of greater and greater capabilities of problem-solving that's that's similar to creativity in that sense yeah we never have a procedure called consciousness in our machines however we get as side effects of what these machines are doing things that seem to be closely related to what people call consciousness so for example in 1990 we had simple systems which were basically recurrent networks and


therefore universal computers trying to map incoming data into actions that lead to success maximizing reward in a given environment always finding the charging station in time whenever the battery's low and negative signals are coming from the battery always finds the charging station in time without bumping against painful obstacles on the way so complicated things but very easily motivated and then we give these little a separate we can all network which is just predicting what's happening if I do that in that what will happen as a


consequence of these actions that I'm executing and it's just trained on the long and long history of interactions with the world so it becomes a predictive model loss of art basically and therefore also a compressor our theme observations after what because whatever you can predict you don't have to store extras or compression is a side effect of prediction and how does this record Network impress well it's inventing little sub programs little sub Network networks that stand for everything that frequently appears in


the environment like bottles and microphones and faces maybe lots of faces in my environment so I'm learning to create something like a prototype face and a new face comes along and all I have to encode are the deviations from the prototype so it's compressing all the time the stuff that frequently appears there's one thing that appears all the time that is present all the time when the agent is interacting with its environment which is the agent itself so just for data compression reasons it is extremely natural for this we can


network to come up with little sub networks that stand for the properties of the agents the hand you know the the other actuators and all the stuff that you need to better encode the data which is influenced by the actions of the agent so they're just as a side effect of data compression during problem-solving you have inter myself models now you can use this model of the world to plan your future and that's what yours have done since 1990 so the recurrent Network which is the controller which is trying to maximize reward can use this model as


a network of the what is this model network as a wild this predictive model of the world to plan ahead and say let's not do this action sequence let's do this action sequence instead because it leads to more predictor to rewards and whenever it's waking up these layers of networks let's stand for itself and it's thinking about itself and it's thinking about itself and it's exploring mentally the consequences of its own actions and and now you tell me what is still missing missing the next the gap to consciousness yeah hi there there isn't


that's a really beautiful idea that you know if life is a collection of data and in life is a process of compressing that data to act efficiently you in that data you yourself appear very often so it's useful to form compressions of yourself and it's a really beautiful formulation of what consciousness is a necessary side-effect it's actually quite compelling to me you've described our nen's developed LST aims long short-term memory networks the there type of recurrent neural networks they have gotten a lot of success recently so


these are networks that model the temporal aspects in the data temporal patterns in the data and you've called them the deepest of the Newell networks right so what do you think is the value of depth in the models that we use to learn since you mentioned the long short-term memory and the lsdm I have to mention the names of the brilliant students of course that's worse first of all and my first student ever set for writer who had fundamental insights already in this diploma thesis then Felix Kias had additional important contributions Alex


gray is a guy from Scotland who is mostly responsible for this CTC algorithm which is now often used to to train the Alice TM to do the speech recognition on all the Google Android phones and whatever and Siri and so on so these guys without these guys I would be nothing it's a lot of incredible work what is now the depth what is the importance of depth well most problems in the real world are deep in the sense that the current input doesn't tell you all you need to know about the environment mm-hmm so instead


you have to have a memory of what happened in the past and often important parts of that memory are dated they are pretty old and so when you're doing speech recognition for example and somebody says eleven then that's about half a second or something like that which means it's already fifty-eight time steps and another guy or the same guy says seven so the ending is the same Evan but now the system has to see the distinction between seven and eleven and the only way I can see the differences it has to store that fifty steps ago


there wasn't or a nerve eleven or seven so there you have already a problem of depth fifty because for each time step you have something like a virtual a layer and the expanded unrolled version of this Riccar network which is doing the speech recognition so these long time lags they translate into problem depth and most problems and this world Asajj that you really have to look far back in time to understand what is the problem and to solvent but just like with our CMS you don't necessarily need to when you look


back in time remember every aspect you just need to remember the important aspects that's right the network has to learn to put the important stuff in into memory and to ignore the unimportant noise so but in that sense deeper and deeper is better or is there a limitation is is there I mean LCM is one of the great examples of architectures that do something beyond just deeper and deeper networks there's clever mechanisms for filtering data for remembering and forgetting so do you think that that kind of thinking


is necessary if you think about LCM is a leap a big leap forward over traditional vanilla are nuns what do you think is the next leap hmm it within this context so LCM is a very clever improvement but LCM still don't have the same kind of ability to see far back in the future in the in the past as us humans do the credit assignment problem across way back not just 50 times steps or a hundred or a thousand but millions and billions it's not clear what are the practical limits of the lsdm when it comes to looking back already in 2006 I


think we had examples where it not only looked back tens of thousands of steps but really millions of steps and who won Paris artists in my lab I think was the first author of a paper where we really was a 2006 or something had examples word learn to look back for more than 10 million steps so for most problems of speech recognition it's not necessary to look that far back but there are examples where it does now so looking back thing [Music] that's rather easy because there is only one past but there are many possible


futures and so a reinforcement learning system which is trying to maximize its future expected rewards and doesn't know yet which of these many possible future should I select given this one single past it's facing problems that the LCN by itself cannot solve so the other sim is good for coming up with a compact representation of the history so far of the history and observations in action so far but now how do you plan in an efficient and good way among all these how do you select one of these many possible action sequences that a


reinforcement learning system has to consider to maximize reward in this unknown future so again it behaves this basic setup where you have one week on network which gets in the video and the speech and whatever and it's executing actions and is trying to maximize reward so there is no teacher who tells it what to do at which point in time and then there's the other network which is just predicting what's going to happen if I do that then and that could be an LCM Network and it allows to look back all the way to make better predictions


of the next time step so essentially although it's men predicting only the next time step it is motivated to learn to put into memory something that happened maybe a million steps ago because it's important to memorize that if you want to predict that at the next time step the next event you know how can a model of the world like that a predictive model of the world be used by the first guy let's call it the controller and the model the controller and the model how can the model be used by the controller to efficiently select


among these many possible futures so naive way we had about 30 years ago was let's just use the model of the world as a stand-in as a simulation of the wall and millisecond by millisecond we planned the future and that means we have to roll it out really in detail and it will work only as the model is really good and it will still be inefficient because we have to look at all these possible futures and and there are so many of them so instead what we do now since 2015 and our cm systems controller model systems we give the controller the


opportunity to learn by itself how to use the potentially relevant parts of the M of the model network to solve new problems more quickly and if it wants to it can learn to ignore the M and sometimes it's a good idea to ignore the the M because it's really bad it's a bad predictor in this particular situation of life where the control is currently trying to maximize r1 however it can also allow and to address and exploit some of the sub programs that came about in the model network through compressing the data by predicting it so it now has


an opportunity to reuse that code the ethnic information in the modern are trying to reduce its own search space such that it can solve a new problem more quickly than without the model compression so you're ultimately optimistic and excited about the power of ära of reinforcement learning in the context of real systems absolutely yeah so you see RL as a potential having a huge impact beyond just sort of the M part is often develop on supervised learning methods you see RL as a four problems of cell traffic cars or any kind of applied


cyber BOTS X that's the correct interesting direction for research in your view I do think so we have a company called Mason's Mason's which has applied to enforcement learning to little Howdy's there are DS which learn to park without a teacher the same principles were used of course so these little Audi's they are small maybe like that so I'm much smaller than the real Howdy's but they have all the sensors that you find the real howdy is you find the cameras that lead on sensors they go up to 120 20 kilometres an hour if you if they want


to and and they are from pain sensors basically and they don't want to bump against obstacles and other Howdy's and so they must learn like little babies to a park take the wrong vision input and translate that into actions that lead to successful packing behavior which is a rewarding thing and yes they learn that they are salt we have examples like that and it's only in the beginning this is just the tip of the iceberg and I believe the next wave of a line is going to be all about that so at the moment the current wave of AI is about passive


pattern observation and prediction and and that's what you have on your smartphone and what the major companies on the Pacific of em are using to sell you ads to do marketing that's the current sort of profit in AI and that's only one or two percent of the world economy which is big enough to make these company is pretty much the most valuable companies in the world but there's a much much bigger fraction of the economy going to be affected by the next wave which is really about machines that shape the data through our own


actions and you think simulation is ultimately the biggest way that that though those methods will be successful in the next 10 20 years we're not talking about a hundred years from now we're talking about sort of the near-term impact of RL do you think really good simulation is required or is there other techniques like imitation learning you know observing other humans yeah operating in the real world where do you think this success will come from so at the moment we have a tendency of using physics simulations to learn


behavior for machines that learn to solve problems that humans also do not know how to solve however this is not the future because the future is and what little babies do they don't use a physics engine to simulate the world no they learn a predictive model of the world which maybe sometimes is wrong in many ways but captures all kinds of important abstract high-level predictions which are really important to be successful and and that's what is what was the future thirty years ago when you started that type of research


but it's still the future and now we are know much better how to go there to to move there to move forward and to really make working systems based on that where you have a learning model of the world a model of the world that learns to predict what's going to happen if I do that and that and then the controller uses that model to more quickly learn successful action sequences and then of course always this crazy thing in the beginning the model is stupid so the controller should be motivated to come up with experiments


with action sequences that lead to data that improve the model do you think improving the model constructing an understanding of the world in this connection is the in now the popular approaches have been successful you know grounded in ideas of neural networks but in the 80s with expert systems there's symbolic AI approaches which to us humans are more intuitive in a sense that it makes sense that you build up knowledge in this knowledge representation what kind of lessons can we draw in our current approaches mmm


for from expert systems from symbolic yeah so I became aware of all of that in the 80s and back then a logic program logic programming was a huge thing was inspiring to yourself did you find it compelling because most a lot of your work was not so much in that realm mary is more in learning systems yes or no but we did all of that so we my first publication ever actually was 1987 was a the implementation of genetic algorithm of a genetic programming system in prologue prologue that's what you learn back then which is a logic programming


language and the Japanese the anthers huge fifth-generation AI project which was mostly about logic programming back then although a neural networks existed and were well known back then and deep learning has existed since 1965 since this guy and the UK and even anko started it but the Japanese and many other people they focus really on this logic programming and I was influenced to the extent that I said okay let's take these biologically inspired rules like evolution programs and and and implement that in the language which I know which


was Prolog for example back then and then in in many ways as came back later because the Garuda machine for example has approved search on board and without that it would not be optimal well Marcus what does universal algorithm for solving all well-defined problems as approved search on board so that's very much logic programming without that it would not be a Centanni optimum but then on the other hand because we have a very pragmatic is also we focused on we cannula networks and and and some optimal stuff such as gradient based


search and program space rather than provably optimal things the logic programming does it certainly has a usefulness in when you're trying to construct something provably optimal or probably good or something like that but is it useful for for practical problems it's really useful at volunteer improving the best theorem provers today are not neural networks right no say our logic programming systems and they are much better theorem provers than most math students and the first or second semester on but for reasoning to for


playing games of go or chess or for robots autonomous vehicles that operate in the real world or object manipulation you know you think learning yeah as long as the problems have little to do with with C or improving themselves then as long as that is not the case you you just want to have better pattern recognition so to build a self-driving car you want to have better pattern recognition and and pedestrian recognition and all these things and you want to your minimum you want to minimize the number of false


positives which is currently is slowing down self-driving cars in many ways and and all that has very little to do with logic programming yeah what are you most excited about in terms of directions of artificial intelligence at this moment in the next few years in your own research and in the broader community so I think in the not so distant future we will have for the first time little robots that learn like kids and I will be able to say to the robot um look here robot we are going to assemble a smartphone it's takes a slab of plastic


and the school driver and let's screw in the screw like that no no not like that like so hmm not like that like that and I don't have a data glove or something he will see me and he will hear me and he will try to do something with his own actuators which will be really different from mine but he will understand the difference and will learn to imitate me but not in the supervised way where a teacher is giving target signals for all his muscles all the time no by doing this high level imitation where he first has to learn to imitate


me and then to interpret these additional noises coming from my mouth as helping helpful signals to to do that Hannah and then it will by itself come up with faster ways and more efficient ways of doing the same thing and finally I stopped his learning algorithm and make a million copies and sell it and so at the moment this is not possible but we already see how we are going to get there and you can imagine to the extent that this works economically and cheaply it's going to change everything almost all our production is going to be


affected by that and a much bigger wave much bigger ai wave is coming than the one that we are currently witnessing which is mostly about passive pattern recognition on your smartphone this is about active machines that shapes data Susy actions they are executing and they learn to do that in a good way so many of the traditional industries are going to be affected by that all the companies that are building machines well equip these machines with cameras and other sensors and they are going to learn to solve all kinds of problems


through interaction with humans but also a lot on their own to improve what they already can do and lots of old economy is going to be affected by that and in recent years I have seen that all the economy is actually waking up and realizing that those vacations and are you optimistic about the future are you concerned there's a lot of people concerned in the near term about the transformation of the nature of work the kind of ideas that you just suggested would have a significant impact of what kind of things could be automated are


you optimistic about that future are you nervous about that future and looking a little bit farther into the future there's people like you la musk - a rustle concerned about the existential threats of that future so in the near term job loss in the long term existential threat are these concerns to you or yalta mele optimistic so let's first address the near future we have had predictions of job losses for many decades for example when industrial robots came along many people many people predicted and lots of jobs are


going to get lost and in a sense say were right because back then there were car factories and hundreds of people and these factories assembled cars and today the same car factories have hundreds of robots and maybe three guys watching the robots on the other hand those countries that have lots of robots per capita Japan Korea and Germany Switzerland a couple of other countries they have really low unemployment rates somehow all kinds of new jobs were created back then nobody anticipated those jobs and decades ago I already


said it's really easy to say which jobs are going to get lost but it's really hard to predict the new ones 30 years ago who would have predicted all these people making money as YouTube bloggers 200 years ago 60% of all people used to work in agriculture today maybe 1% but still only I don't know 5% unemployment lots of new jobs were created and Homo Luden's the the playing man is inventing new jobs all the time most of these jobs are not existentially necessary for the survival of our species there are only


very few existentially necessary jobs such as farming and building houses and and warming up the houses but less than 10% of the population is doing that and most of these newly invented jobs are about interacting with other people in new ways through new media and so on getting new high types of kudos and forms of likes and whatever and even making money through that so homo Luden's the playing man doesn't want to be unemployed and that's why he is inventing new jobs all the time and he keeps considering these jobs as really


important and is investing a lot of energy and hours of work into into those and new jobs it's quite beautifully put were really nervous about the future because we can't predict what kind of new jobs would be created but your ultimate ly optimistic that we humans are so Restless that we create and give meaning to newer in your jobs telling you likes on faith things that get likes on Facebook or whatever the social platform is so what about long-term existential threat of AI where our whole civilization may be swallowed


up by this ultra super intelligent systems maybe it's not going to be smaller DUP but I'd be surprised if B were B humans were the last step and the evolution of the universe you you've actually at this beautiful comment somewhere that I've seen saying that artificial quite insightful artificial general intelligence systems just like us humans will likely not want to interact with humans they'll just interact amongst themselves just like ants interact amongst themselves and only tangentially interact with humans hmm and it's quite


an interesting idea that once we create a GI that will lose interest in humans and and have compete for their own Facebook Likes on their own social platforms so within that quite elegant idea how do we know in a hypothetical sense that there's not already intelligent systems out there how do you think broadly of general intelligence greater than us how do we know it's out there mmm how would we know it's around us and could it already be I'd be surprised even with within the next few decades or something like that we we


won't have a eyes that truly smarts in every single way and better problem solvers and almost every single important way and I'd be surprised as they wouldn't realize what we have realized a long time ago which is that almost all physical resources are not here and this biosphere but for thou the rest of the solar system gets 2 billion times more solar energy than our little planet there's lots of material out there that you can use to build robots and self-replicating robot factories and all this stuff and they


are going to do that and there will be scientists and curious and they will explore what they can do and in the beginning they will be fascinated by life and by their own origins and our civilization they will want to understand that completely just like people today would like to understand how life works and um and also the history of our own existence and civilization and also on the physical laws that created all of that so they in the beginning they will be fascinated my life once they understand that I was


interest like anybody who loses interest and things he understands and then as you said the most interesting sources information for them will be others of their own kind so at least in the long run there seems to be some sort of protection through lack of interest on the other side and now it seems also clear as far as we understand physics you need matter and energy to compute and to build more robots and infrastructure and more AI civilization and III ecology is consisting of trillions of different types of AIS and and so it seems


inconceivable to me that this thing is not going to expand some AI ecology not controlled by one AI but one by trillions of different types of AI is competing and all kinds of quickly evolving and disappearing ecological niches in ways that we cannot fathom at the moment but it's going to expand limited by Lightspeed and physics it's going to expand and and now we realize that the universe is still young it's only 13.8 billion years old and it's going to be a thousand times older than that so there's plenty of


time to conquer the entire universe and to fill it with intelligence and senders and receivers such that AI scan trouble the way they are traveling in our labs today which is by radio from sender to receiver and let's call the current age of the universe one Eon one Eon now it will take just a few eons from now and the entire visible universe is going to be full of that stuff and let's look ahead to a time when the universe is going to be one thousand times older than it is now they will look back and they will say look almost


immediately after the Big Bang only a few eons later the entire universe started to become intelligent now to your question how do we see whether anything like that has already happened or is already in a more advanced stage in some other part of the universe of the visible universe we are trying to look out there and nothing like that has happened so far or is that her do you think we'll recognize it or how do we know it's not among us how do we know planets aren't in themselves intelligent beings how do we know ants


seen as a collective are not much greater intelligence in our own these kinds of ideas no but it was a boy I was thinking about these things and I thought hmm maybe it has already happened because back then I know I knew I learned from popular physics books that the structure the large-scale structure of the universe is not homogeneous and you have these clusters of galaxies and then in between there are these huge empty spaces and I thought hmm maybe they aren't really empty it's just that in the middle of that some AI civilization already has


expanded and then has covered a bottle of a billion light-years diameter and is using all the energy of all the stars within that bubble for its own unfathomable purposes and so it always happened and we just failed to interpret the signs but then alarmed effect gravity by itself explains the large-scale structure of the universe and that this is not a convincing explanation and then I thought maybe maybe it's the dark matter because as far as we know today 80% of the measurable matter is invisible and we know that because otherwise our galaxy


or other galaxies would fall apart they would they are rotating too quickly and then the idea was maybe all us he is AI civilizations and hourly out there they they just invisible because they are really efficient in using the energies at their own local systems and that's why they appear dark to us but this is awesome at a convincing explanation because then the question becomes why is there are there still any visible stars left in our own galaxy which also must have a lot of dark matter so that is also not a


convincing thing and today I like to think it's quite plausible that maybe are the first at least in our local light cone within a few hundreds of millions of light years that we can reliably observe is there exciting to you it will might be the first and it would make us much more important because if we mess it up through a nuclear war then then maybe this will have an effect on the on the on the development on of the entire universe so let's not mess it up let's not mess it up Union thank you so much for talking


hi there today we're looking at back propagation in the brain by Timothy Lilly corrupt Adam Santoro Luke Morris Colin Ackerman and Geoffrey Hinton so this is a bit of an unusual paper for the machine learning community but nevertheless it's interesting and let's be honest at least half of our interest comes from the fact that Geoffrey Hinton is one of the authors of this paper so this is a paper that basically proposes a hypothesis on how the algorithm of back propagation works in the brain because previously there has been a lot

Backpropagation and the brain


of evidence against there being something like back propagation in the brain so the question is how do neural networks in the brain learn and they they say there there can be many different ways that neural networks learn and they list them up in in this kind of diagram where you have a network and it maps from input to output by having these weighted connections between neurons so the input is two-dimensional and then it maps using these weights to a three-dimensional hidden layer and usually there is a nonlinear function

Backpropagation and the brain


somewhere at the output here of these so they they do a weighted sum of the inputs and then they do a nonlinear nonlinear function and then they propagate that signal to the next layer and till then to finally to the output all right so how do these networks learn the one way of learning is called hebbian learning the interesting thing here is that it requires no feedback from the outside world basically what you want to do in hebbian learning is you want to update the connections such that they kind of match their own

Backpropagation and the brain


previous outputs or even increase their own previous outputs so you propagate a signal and then maybe this neuron spikes really hard and this Spike's really low then if you propagate the signal again right then you want to match that those those activations or if you if you properly similar signals no feedback required so basically it's a self amplifying or self dampening process the ultimately though you want to learn something about the world and that means you have to have some some feedback from outside right so with

Backpropagation and the brain


feedback what we mean is usually that the output here let's look this way the output here is goes into the world let's say this is a motor neuron right you do something with your arm like you hammer on a nail and then you either hit the nail or you don't let's say you don't hit the nail so after it looks like crooked there you have feedback right so feedback usually in the form of some sort of error signal right so feedback it can be like this was good or this was bad or it can be this was a bit too much to the left or so on the important part

Backpropagation and the brain


is you get kind of one number of feedback right how bad you were and now your goal is to adjust all of the individual neurons or weights between neurons such that the error will be lower so in hebbian learning there is no feedback it's just simply a self reinforcing pattern activation machine in the first in these kind of first instances of perturbation learning what you'll have is you'll have one single feedback and that you can see this is a diffuse cloud here what you're basically saying is that every single neuron is

Backpropagation and the brain


kind of punished let's say the the feedback here was negative one that means every single neuron is is punished for that so how you can imagine something if you have your input X and you map it through through your function f then the function f has a way to w1 and so on right so you map X through it right and then you get feedback of negative 1 and then you map X with a little bit of noise plus M right da-da-da-dah and you get a feedback of negative 2 right then you you that means that the direction of this noise was probably a bad direction

Backpropagation and the brain


so ultimately you want to update X into the direction of negative that noise by modulated of course by by some some factor here that's that it kind of tells you how bad it was so this could be the negative 2 minus negative 1 now that makes big sense No yes that would be no it would be negative 1 minus negative nevermind so basically with a scalar feedback you simply tell each neuron what it did right or sorry if if the entire network right the entire network did right or wrong so the entire network will lead to

Backpropagation and the brain


this feedback you don't have accountability of the individual neurons all you can say is that whatever I'm doing here is wrong and whatever I'm doing here is right so I'm gonna do more of the right things now in back propagation it is very different right in back propagation what you'll do is you'll have your feedback here let's say that's negative 1 and then you do a reverse computation so the forward computation in this case was this weighted sum of this layer now usually layer wise reverse computation which

Backpropagation and the brain


means that you know how this function here this output came to be out of the out of the inputs and that means you can inverse and you can do an inverse propagation of the error signal which is of course the gradient so this would be your your you you would derive your error by the inputs to the layer right so this basically tells in the back propagation algorithm you can exactly determine if you are this node how do I have to adjust my input weights how do I have to adjust them in order to make this number here go down right and

Backpropagation and the brain


then because you always propagate the error according to that what you'll have in each in each layer is basically a vector target so it's no longer just one number but each layer now has a target of vectors and it says okay these are the outputs that would be beneficial please this layer please change your outputs in the direction of negative two negative three plus four so you see this is so the negative two would be this unit the negative three would be this unit and the plus four would be this unit so each unit is instructed

Backpropagation and the brain


individually to say please this is the direction that each unit should change in in order to make this number go lower you see how this is much more information than the perturbation learning in the perturbation learning all the units simply know well the four was bad and now is better so let's you know change a bit and here you have detailed instructions for each unit because of the back propagation algorithm so ultimately people have kind of thought that since back propagation wasn't really possible with biological

Backpropagation and the brain


neurons that the brain might be doing something like perturbation learning but this paper argues that something like back propagation is not only possible but likely in the brain and they proposed this kind of backdrop like learning with the feedback network so they basically concern all the they differentiate hard between these two regimes here in this hand you have the scalar feedback which means that the entire network gets one number as a feedback and the each neuron just gets that number and here you have

Backpropagation and the brain


vector feedback where each neuron gets an individual instruction of how to update and they achieve this not by back propagation because still the original formulation of back prop as we use it in neural networks is not biologically plausible but they achieve this with this backdrop like learning with the feedback network and we'll see how this does but in in essence this feedback network is constructed such that it can give each neuron in the forward pass here detailed instructions on how to update itself right so yeah they have a

Backpropagation and the brain


little bit of a diagram here of if you do hebbian if this if this is an error landscape if you do have you in learning you basically you don't care about the error you're just reinforcing yourself if you do perturbation learning then you it's very slow because you don't have a detailed signal you just you just rely on this one number it's kind of if you were to update every single neuron in your neural network with reinforcement learning considering the output the of the neural networks or the error considering that the reward not using

Backpropagation and the brain


back row and then with back probably have a much smoother much faster optimization trajectory so they looked at this and they they come to some some conclusions first of all so here's here's back prop basically saying back prop as we said you have the forward pass and there you simply compute these weighted averages and you you also pass them usually through some sort of nonlinear activation right and the cool thing about this is in artificial neural networks is that once the error comes in you can exactly reverse that so you can

Backpropagation and the brain


do a backward pass of errors where you can propagate these errors through because you know it's kind of invertible the function doesn't have to be invertible but that the gradients will flow backwards if you know how the forward pass was computed so first of all they go into a discussion of back prop in the brain how can we even expect that and one cool piece of evidence is where I find is that they cite several examples where they use artificial neural networks to learn the same tasks as humans right and or as as animal

Backpropagation and the brain


brains and then I have no clue how how they measure any of this but then they compare the hidden representations of the living neural networks and the artificial neural networks and it turns out that the these the networks that were trained with backpropagation x' then networks that were not trained with backdrop so basically that means if you train a network with backprop it matches the biological networks much closer in how they form their hidden representations and they they do a number they cite the number of

Backpropagation and the brain


experiments here that show this so this gives you very good evidence that if the hidden representations they look as if they had been computed by backdrop and not by any of these scaler update algorithms so it is conceivable that we find backprop in the brain that's why they go here next they go into problems with backdrops so basically why why would we why so far have we believed that back prop isn't happening in the brain so now let's I want to highlight two factors here that that I find a thinker suffice state they have more but first

Backpropagation and the brain


of all back prop demands synaptic symmetry in the forward and backward paths right so basically if you have a neuron and it has output to another neuron what you need to be able to do is to pass back information along that neuron so it kind of has to be a symmetric connection idea of the forward and the backward pass and these need to be exact right and this is just not if you know how neurons are structured they have kind of input dendrites and then there's this accent act action potential and along the axon the signal travels

Backpropagation and the brain


and the back traveling of the signal just I think is very is very very very slow if even possible and so it's generally not invertible or inverse compute capable so this is one reason why that prop seems unlikely and then the second reason here is error signals are signed and potentially extreme valued and i want to add to that they also just talk about this somewhere that error signals are of a different type right that's a different type so first let's see what signed error signals are signed yes we need to be

Backpropagation and the brain


able to adjust neurons in a specific directions right if you look at again what we've drawn before here we said here this is how these neurons must must update so the first neuron must must decrease by two this must decrease by three and this must increase by four now in background we need this but in if if we assume that there is something like a reverse computation or signaling here happening then we still have the problem that usually these output signals are in the form of spiking rates which means that over time right so if a neuron

Backpropagation and the brain


wants to if a neuron has zero activation there's just no signal but if a neuron has a high activation it spikes a lot if has a low activation it kind of spikes sometimes well what he can do is negative spike right like zero is as low as it goes so the the thought that there are signed information in in the backward pass is inconceivable even if you have something like a second so you can imagine here instead of this backward connection because of the symmetry problem we have some kind of second neural network that goes in this

Backpropagation and the brain


direction still you'd have the problem that here you can only have positive signal or a zero and they might be extreme valued which okay it can't be really encoded with the spiking because they are they're limited in the range they can assume but they are also of a different type and I'm what I mean by that is basically if you think of this as a programming problem then the forward passes here are our activations right and the backward passes here they are deltas so in the backward passes view either propagate deltas or you

Backpropagation and the brain


propagate kind of directions so the activations are sort of impulses whereas the backward signals are this isn't how you need to change their their gradients ultimately so it's fundamentally a different type of data that is propagated along would be propagated along these directions and that makes it very unlikely because we are not aware as this paper says that the that neural networks that neurons can kind of switch the data type that they're they're transmitting all right so then the paper goes into their end

Backpropagation and the brain


grad hypothesis and what this is the hypothesis basically states that the brain could implement something like neural networks by using by using an approximate backdrop like algorithm based on autoencoders and I want to jump straight into the algorithm no actually first they do talk about autoencoders which which I find very interesting so if you think of autoencoders what is an autoencoder an autoencoder is a network that basically starts out with an input layer and then has a bunch of hidden layers and at the end it tries to

Backpropagation and the brain


reconstruct its own input right so you feed a data in here you get data out here and then your error the error signal it will be your difference to your original input now the usually when we train autoencoders in deep learning we also train this by back prop right we see then this error here and this goes back but if you just think of single layer autoencoders so um let's let's go over here single layer auto-encoder with let's say the the same number of the same number of units in this in this layer what you'll have is so this this

Backpropagation and the brain


is input this is output and this is the hidden layer right you'll have a weight matrix here and you'll probably have some sort of nonlinear function and then you have another weight matrix here and they call them W and B another way to draw this is I have weight matrix going up then I have a nonlinear function going transforming this into this signal and then I have the be going back right so I'm drawing I'm drawing it in two different ways up here or over here and with the second way you can see that it is kind of a forward backward algorithm

Backpropagation and the brain


where now the error if you look at what is the error here the error is the difference between this and this and the difference between this and this and the difference between this and this right and you can train an autoencoder simply by saying W please make sure that the that the the the input here gets mapped closer to the output and to be the same thing this will become clear in a second so but basically sorry this I mean the the hidden representations you'll see basically the idea is that you can train an autoencoder only by

Backpropagation and the brain


using local update rules you don't have to do back prop and that's what this algorithm is proposing namely if you think of a stack of autoencoders this this this transforming one hidden representation into the next right this is the feed-forward function what you can do is you first of all you can assume that for each of these functions here you have a perfect inverse right you can you can perfectly compute the inverse function that's this this G here of course this doesn't exist but assume you have it what you then could do is you could if

Backpropagation and the brain


if you knew in one layer and on the top layer of course you know if you knew that okay I got this from my forward pass but I would like to have this this is my desired output right so in the output layer you get this this is your error signal if you knew you you you could compute an error right here this is what you do in the output right now in back prop we would back propagate this error along the layers but now we don't do this instead of what we do is we use this G function to invert the F function right and by that what we'll

Backpropagation and the brain


say is what hidden representation in layer two what should the hidden representation have been that in order for us to obtain this thing right so the the claim here is if in layer two we had had H two as a hidden representation then we would have landed exactly where we want it right that's what this G function does because here we use F so had we had F h2 and used F on it we would be exactly where we want instead we had h2 here and used F on it and then we landed here where we don't want so this is where we want we would want to

Backpropagation and the brain


be in layer two and this is where we were so again we can compute an error here again instead of back propagating that error what we'll do is we'll use the inverse of the forward function in order to back propagate our desired hidden representation and you can see there is of course a relationship to the true back prop here but the the important distinction is we are not trying to back propagate the error signal we're trying to invert the desired hidden states of the network and then in each layer we can compute from the forward pass we can

Backpropagation and the brain


compute the difference to the desired hidden state and thereby compute an error signal and now we have achieved what we wanted we want an algorithm that doesn't do back prop that only uses local information in order to compute the error signal that it needs to adjust and by local I mean information in the same layer and also the data type that is propagated by F is activations right of hidden representations and by G is also activations of hidden representations both of them are always positive can be encoded by spiking

Backpropagation and the brain


neurons and so on so this algorithm achieves what we want they go bit into detail how the actual error update here can be achieved and apparently neurons can achieve you know in the same layer to to adjust themselves to a given desired activation so this algorithm achieves it of course we don't have this G we don't have it and therefore we need to go a bit more complicated what they introduces the this following algorithm the goals are the same but now we assume we do not have a perfect inverse but we have something that is a bit like an

Backpropagation and the brain


inverse so we have an approximate inverse and they basically suggest if we have an approximate inverse we can do the phone so G G is now an approximate inverse to F what we can do is this is our input signal right we use F to map it forward to this and so on all the way up until we get our true or error right here this is our error from the environment right this is the nail being wrong and then we do two applications of G right so this is an application of F we do to applet of g1 we applied g2 this to what we got

Backpropagation and the brain


in the forward pass right and this now gives us a measure of how bad our inverse is right so if G is now an approximate inverse and this now we see here oh okay we we had a ch2 in the forward pass and we basically forward passed and then went through our inverse and we didn't land quite exactly where we started but we know that okay this this is basically the difference between our our inverse our forward inverse H and our true H and then we also back project using G again the desired outcome so we invert the desired outcome

Backpropagation and the brain


here now before we have adjusted directly these two right because we said this is what we got this is what we want but now we include for the fact that G isn't a perfect inverse and our assumption is that G here probably makes about the same mistakes as G here so what we'll do is we'll take this vector right here and apply it here in order to achieve this thing and this thing is now the corrected thing our corrected to desired hidden representation correct for the fact that we don't have a perfect inverse and now again we have

Backpropagation and the brain


our error here that we can locally adjust again all the signals propagated here here and here are just neural activations and all the information required to update a layer of neurons is now contained within that layer of neurons right and and this goes back through the network so this is how they achieve how they achieve this this is a bit of a close-up look and here are the computations to do this so basically for the forward updates you want to adjust W into the direction of the H minus the H tilde and the H tilde in

Backpropagation and the brain


this case would be this the the hidden representation that you would like to have so you will update your forward forward weights into the direction such that your hidden representations are closer sorry that your forward haven representation is closer to your backward hidden representation and the backward updates now your goal is to get a more a better to make G so sir W here is our W or the weight of F and B or the weights of G so in the backward updates your goal is to make G a better inverse right so what you'll do is again you'll

Backpropagation and the brain


take the difference between now you see the difference here here here right not the same error so here you will you in the W update use what we labeled error here in the G update you use this error here so this is the error of G so when you update the function G you want to make these two closer together such that G becomes a better inverse right because you're dealing with an approximate inverse you still need to obtain that approximate inverse end and this here is how you learn it this algorithm now achieves what we wanted right

Backpropagation and the brain


local updates data types check signed check and so on I hope this was enough clear in essence is pretty simple but it's pretty cool how they work around this they call this a different story with propagation and not these these kind of papers I don't think they invented this maybe I'm not sure maybe they did maybe they didn't and this paper just kind of frames it in this hypothesis it is unclear to me I am not familiar with this kind of papers so sorry if I miss attribute something here all right then they go into into how could these

Backpropagation and the brain


things be implemented biologically and they go for some evidence and they also state that we used to look at neurons basically in this way where you had input and feedback here very simple simplistic view of neurons whereas nowadays even the company computational community views neurons in a more differentiated way where you have for example different regions here on the soma that can be separated from each other and you have inter neuron interference and so on I'm not qualified too much to comment on this stuff but I

Backpropagation and the brain


yeah so I think since I'm one of the organizers I'll actually take the opportunity to thank all the speakers and all of you for attending it's been a lot of fun to hear the wide range of perspectives and topics this week and of course also thanks to the Simons Institute for for hosting us the date I noticed is wrong here actually obviously it's November 20th not October 20th I haven't been sleeping much lately so so I'm gonna I think in the interest of time I'm gonna skip over some of the introductory material here we've heard a

Dynamic Pricing in Ride-Sharing Platforms


lot about ride-sharing platforms already so let me just skip that I think at this point in the week between the the talk that was given by Christos Co on Monday and then also the industrial visitors day we had last week we all are familiar with uber and lyft and sidecar and platforms like this so what's our goal in our work oh and I should actually start preface this by saying this is joint work with Sid Banerjee who's an assistant professor at Cornell he was doing a postdoc with me and Carlos raqami who's a student of mine at


Stanford and also Sid did an internship with lyft and with their data science team and so some of the inspiration for this work came from from talking and working with them so I guess what but I'm interested in getting across to you and what I found exciting about this problem is that there was a combination of maybe three things that we needed to somehow include in one model and and then use that to actually say something useful about the you know the strategy that the platform takes and that's basically that there's passengers and


drivers who are strategic the platform is setting you know a pricing rule a decision for how transaction prices are set on each on each interaction and then there's this kind of underlying queueing dynamic that governs you know the number of rides that are requested and the number of drivers that are available and in isolation we have a lot of you know different models and in various literature's that tell us about any one of these three problems one of the things that makes us really interesting to me is the fact that all three kind of


come together in one place so this would just be I think you know a pure theoretical exercise it would be fun to build a model of something like that now of course you want to do that with some purpose in mind in our case the motivation for wanting to do this in the first place is that we sort of wanted to try to understand what the advantages were of using a dynamic pricing policy over static price policy and I'll explain more what I mean by that as we go on so I'll skip this side as well I just want to briefly


point out that there's a wide range of literature in very different communities actually that touches on this so at the same time that you know we've been thinking about matching markets whether an econ or in in the UC community it's been interesting to see in the applied probability community there's been a huge amount a huge surge of interest in models of queuing systems with matching behavior and so I think you know for those of you that are working on matching markets I would strongly recommend sort of looking into some of


the some of the work that I think really starts with the Donna and bison goes from there there's a lot of work on strategic queuing models two-sided platforms revenue management so like I said you know there's there's a lot there and and and and one of the things that made this fun is sitting at an intersection of those of those topics okay so let me tell you a bit about the model so the model is something where we need to capture three features so one is that I need to be able to say something about you know the platform's goals


right you know how is it setting a pricing policy I need to be able to tell you what the incentives are of passengers and what the incentives are our drivers and that's all sort of the strategic aspect of the model I also want to be able to tell you you know exactly even fixing all of this you know how how does the system evolve what are the dynamics of drivers and passengers okay so some an apology I want to interject here is that I think this is a talk I personally view as sort of at the tip of a very large iceberg and in many ways


I think the work that we were doing raises more questions than it answers so I'm happy with some of the answers we got but I also want to be very kind of explicit with you where I think there's important things missing okay so some of them are on this slide and some of them all mentioned as we go through so one of them is that we're gonna focus on just a single block of time and what do I mean by a block I mean something like let's say rush hour or you know maybe for those of you that were here for Christmas goes talk a


block of time might be you know a window of time around when bars close on a Friday or Saturday evening so why is that important in many ways what I want to focus on here is not I want to avoid talking about predictable changes in demand okay so everybody knows there's going to be more demand around rush hour than there is in the afternoon alright so I want to avoid that and indeed the platform's avoid that so even even before surge pricing became sort of something which is changing at a minute-to-minute basis you


know it was the case that they would know in advance that there's going to be greater demand let's say around rush hour or bars closing and and the the surge market multiplier would be higher in those intervals so when I talk about static pricing I don't mean a fixed price over the entire week I mean static over something like you know a few hours a block of time okay the next thing is in the talk I'm only going to focus on a single region now of course you know cities are not just a single region and I think even Chris on Monday mentioned


that you know the pricing involves multiple neighborhoods in in the work that we've done in the paper we do have the main insights that we have do generalized to networks there's some sort of exceptions to that but I'm not going to devolve that in the talk I think one important problem that my focusing on a single region completely ignores and even the work on networks completely ignores is I guess there's two issues that that that sort of eliminates right so one is the notion of an estimated time arrival or ETA and I


think as you heard from Chris and as anyone who actually uses uber or lyft would know you know in addition to whether or not their search pricing you're very sensitive to when you think you're actually going to get a ride note so what the ETA actually is so sort of by fixing the network and you'll see more in the models sort of how this plays out now one of the things we don't actually have much to say about is how passengers are sensitive to etas and sort of in particular you know one thing that might mean is that the design of


the regions themselves like what you consider to be a region is not a topic that we address at all right so you so you may want to make your regions more granular so that you're able to better you know acclimatize a better match supply and demand on a very local scale but as you do that you will also want to make sure that drivers who are nearby are able to move in or out okay so we're not really we're not really accounting for those effects in the kinds of network model that we build and I think so so thinking about ETA s and and how


the platform actually designs you know designs it's it's topology I think that's I think that's a really interesting direction of work it's something sedan I've talked about we haven't really done much with it yet and find the last thing I'll mention is the objective function I'm going to focus on is is throughput the rate of completed rides there's at least three objectives that you you know might care about there's throughput there's profit and then there's welfare so we we have results for throughput that sort of have no


qualification whatsoever we have results for profit when the system is supplied limited in a sense I'll make precise and then there's similar numerical results for welfare but the theory there is actually a bit more challenging and so I'm not going to claim anything for that in the talk okay so let me start by modeling sort of the strategic side of the problem and this is another point which we'd say you know very little about we don't say anything in our paper on and I think it's an interesting issue is that we're going to just assert that


the platform takes a fixed fraction of every dollar that's spent and you know I think that's fairly consistent with how the platform's work today but it's kind of interesting I mean it's a mechanism designer you might actually think this is one of the very first things you would want to design you'd want to optimize over so our work does not optimize over this it kind of holds that as an exogenous constant I think one of the most interesting things to do is to kind of a lot of questions during chrises talk on Monday I think alluded


to this that it's interesting to think about whether the platform should be sort of varying the the share that it takes perhaps you know based on the state of the system there's a good reasons why most platforms don't really change this on a very fast timescale you know I mean this is the type of thing that would be updated you know over months or something like that if at all and it's really a I think I think it's it's more of a sort of cultural issue right i mean i think i think drivers and passengers are not drivers would not be


very happy if they if they kind of were subject to to a rapidly varying share of earnings it becomes very unpredictable from their perspective of course the platform needs both drivers and passengers and it uses pricing to align the two sides one note on terminology in the platform's literature you know in economics if you use the word pricing you have to be a little clearer about you means so pricing might mean the fee structure which is the gamaheer for example or pricing might mean actually setting the transaction price and one of


the reasons that ride-sharing platforms were fun to think about in contrast to other kinds of platforms that I find interesting is because they actually directly set the transaction price so that like that's why this is an interesting question you know if you come compare that with something like let's say op work or Airbnb they have a lot of influence on what the transaction prices might be like but they don't actually directly set them okay so here when I use the word pricing I actually mean the platform is setting the


transaction price and so they're able to use their mechanism for setting the transaction price to align the two sides so the first bit of notation is that the way I'm going to model the platform setting the transaction price is just as a function of the number of available drivers remember I'm focused on only one region there's a number of available drivers right now in that region I'll imagine there's some function that the platform uses to map the number of available drivers to the price that you're going to get charged if you if


you take a ride right now okay all right next qualifier so when I say price you know I'm saying you're setting the transaction price and again it's not directly the transaction price that's being set in in the ride-sharing platforms it's a multiplier on the transact on a base price okay so the way these platforms are work they'll have a published formula that's time and distance dependant that's on their website and if you use a fare estimator you can actually directly calculate this so that's what they call the base price


and any price manipulation that's happening on a faster time scale is happening through a multiplier once the lift calls it prime time pricing uber calls it surge pricing and so you know what's happening is that they will they will tell you that there's some percentage that's going to get added on top of the base price because of that sort of current state of the market so when I use the word price in the talk what I'm really talking about is this multiplier okay that raises yet another interesting question which is of course


you know Los Angeles and San Francisco are very different markets and in Los Angeles in particular you know there may be a very large cost to pulling drivers from you know that look available from further away to come pick up a ride and that ride you know that cost as part of a distance dependent so these kinds of things you know it's often a question that comes up here is well why is this also not a matter of manipulation not just a multiplier on type on top of a formula but why not actually just directly be sort of manipulating prices


in a way that that you're not just picking a single multiplier regardless of what the time the distance is going to be and again something I'm not touching I think this is another one of those things just thinking about from regulatory perspective you know ride-sharing platforms are fighting against the taxi industry and that's you know the taxi sort of industry has you know published fare schedules like this it's already challenging enough to convince the public that you know surge pricing or primetime pricing is palatable I find


that shocking I have to confess but that that's the way it is so given that that's hard enough I think varying this is also going to be you know politically even more difficult but that's said from a you know market design standpoint I think it's a reasonable question to ask okay so that's the platform yeah multiplier number of available drivers yes I'm gonna have a model where the state of the system is the number of available drivers and this doesn't do that that's right yeah I think there's a lot of different things that are being


left out here so that's part of it and as you're gonna see in a second in I'm taking in my talk in extreme view where drivers are making entry decisions over longer time skills now if you think about some of the tools that platforms are using and and you know Chris talked about some of these on Monday another thing that's left out here is not just sort of the number of available drivers right now but let's say a forecast that I have of how many available drivers there will be all that kind of stuff is just left out of it I


think there's a lot of interesting things to do there in terms of the richness of the pricing policy okay so with the passengers what do passengers do act fairly simple model of passengers every passenger is one ride so passenger equals one ride request I don't model any sort of longitudinal behavior of the passenger it's just simple and basically I'm just gonna model the passengers as entering if their price exceeds if their reservation value exceeds the current price in the system okay so to model that I think of every every ride request


is being drawn iid from some valuation distribution and if if the value that's drawn is bigger than the current price they enter there's some eggs agenda straight of what I'll call app opens Chris talked about the same thing and that's basically like I open the app and looked do I want to request a ride and then look at the price and determine whether I actually request a ride so from that it's pretty easy to work out based on the pricing you know formula what is the rate of ride requests it's the exogenous rate times the tail probability that the


valuation exceeds the the current price okay so this F bar is the tail tail CDF of the evaluation distribution right so that's the passengers it's relatively simple the drivers I think is where it gets a little bit more interesting this is maybe the one place where our motivation for choosing this model kind of came from what what seemed to us to be like a natural distinction between drivers and passengers I would say that I think of this as kind of a stylized extreme point and there you know especially as the platform's changed the


technology that they use to induce drivers to enter or exit I think we can get we can bisque and get a lot more interesting but let me tell you what this is basically the point that that we make here is that we think of drivers is making decisions on just a substantively different time scale than passengers okay so if a driver is thinking about whether to drive for example in the early days of lyft you know you had like a booking calendar when you would essentially say like when you wanted to be on or off the platform