about transformers? Because couldn't I use any deep learning thing or why transformers?


Predicting the rules behind - Deep Symbolic Regression for Recurrent Sequences (w/ author interview)

Well, first of all, like previous experience, I mean, Guillaume and Francois have been working


on these transformers. They've basically always been good at the problems we've given them.


Likely, I mean, one natural justification is that as we saw for the outputs, you can represent


math as a language in a very easy way. It's actually, we can see here that it's much harder


to represent the inputs as tokens, but the formulas themselves are very easy to represent


as a language with this Polish notation thing. So it's very natural to use transformers because


they're our best models to deal with language. So yeah, I think that's the main reason.


And yeah, I'm not sure what else we could particularly, I mean, we could use like RNNs,


et cetera, but these days transformers are so powerful. I mean, these models we used,


we didn't even, as I was saying before, we didn't have to tune them much. We just basically took


the same architecture that was used in the paper two years ago. We didn't even have to change the


learning rate. Like it's pretty amazing how easy it is to train these things.


Okay. Yeah. So the transformers are a natural way to deal with sequences and from text learning,


we kind of know this, but we always learn sort of on human text, right? And that has a particular


structure. And I want to think if I look at these sequences, there are almost like, there's so many


symbolic formulas that could possibly explain these sequences. And yeah, I can, you say you make,


you want maybe the simplest sequence or you don't want your formulas to blow up. That's,


you even generate only formulas that are, let's say relatively simple. So there's clearly a bias


towards simplicity, but still there are a lot of things that explain the same sequence. So I'm


thinking more, is it like if, when we as humans do these tasks, is it like a property of humanity


and civilization that we kind of come up with the same sequences that the person who made the riddle


came up with? Is it because we kind of think alike, right? Because of whatever society or our


environments that shaped us, or is there like a property of math that says, well, if actually,


if you look for the simplest sequence, it is kind of defined, even though there are infinite


possibilities. Like, do you know a little bit what I mean? Is it more like a property of humanity or


of mathematics? I think it's probably two different things. So as far as humans is


concerned, indeed, we tend to prefer simplicity. That's like our OCam's razor principle. We like


going for the compressing information and going for the simplest representation. In terms of our


algorithm here, we didn't put at all this simplicity inductive bias from an explicit point of view. We


didn't tell the model, give us the simplest formula. Actually, we could have done so because


we could have, for example, given a penalty to like the decoder when it generates too long sequences,


for example. But we didn't have to do this at all because the inductive bias comes from the fact


that simple formulas are more likely to be generated by the generator. And that's basically


the rationale behind our model is that it's always going to be biased towards the most likely formula


corresponding to the sequence. And as we were saying before, sometimes that's not good because


for the Collatz sequence, it's going to struggle to output the one minus the mask thing. But in


general, that's kind of what we want. In IQ tests, we ask for the simplest formula to explain the


observations. I'm thinking of, are there more things rather than just number sequences where


something like symbolic regression could be valuable? For example, I've always thought of


maybe reinforcement learning would be much more powerful if we didn't only...


Even if agents have a world model, what they call a world model, they usually have like


almost like a numeric world model. They just forward predict the values that are going to


happen there. I always thought, well, if I had like a symbolic representation of the world,


I could do much more powerful planning. Are you thinking of applications like these when you


develop this beyond number sequences? Or is there any interesting ones that come to your mind?


As I was saying, Pierre Aurélien, my co-author, comes from reinforcement learning. There have


already been a few papers inserting some symbolic parts into RL loops. That's definitely going to


help. Indeed, as you say, if you're a robot and you're trying to understand the world, then it's


going to be much easier if you understand Newton's law. If you want to, for example,


predict how objects are going to move, it's much easier once you understand Newton's law


than using a specific vision model to try and predict. That's going to be much more complicated.


So indeed, I think symbolic regression is going to be very useful for RL. From my point of view,


I'm more from the physics background. That's also a domain where symbolic regression would be very


useful. We have these two approaches. We have numeric regression and we have symbolic regression.


I think they're very complementary in the sense that numeric regression is very good on complex


tasks where you don't necessarily have a simple explanation for the data. Symbolic regression is


great for inferring data where you have a simple underlying rule, typically in physics, like


inferring laws from observation. I think RL and physics are definitely two huge domains of


application for symbolic regression. To make this a bit clearer, what I've done is in the appendix,


you actually have some success and failure cases of your model. I have made a little quiz out of


them and hidden a bunch of them right here. I just want to draw people's attention a little bit to


some of the... On the left, the left three columns are success cases and the right three columns are


failure cases, both of the integer model. These are integer valued sequences. Do I have this


correctly? You do consider it only a success if the formula is equivalent or do you consider it


already a success if just the predicted values are the same? You can have the two criteria. The


criteria we choose in the papers, we want the evaluations to be the same. Even if it comes up


with a different formula, it's fine as long as the ones you tested on match. That's actually one


tricky thing is that indeed, you can't really rely on the formula to check if it was correct or not


due to the degeneracy. Some papers have circumvented this by using an RL loop because


if you try to really supervise the formula, then you have to evaluate the formula, which is


non-deterministic and then you can't back propagate this. Some people have used RL loops to provide


reward signals from the evaluations. What we do is directly supervise the tokens of the formula.


Maybe we can discuss this a bit later, but that's also interesting because you could think


this is weird because our model is supervised to a formula and it's going to be penalized if it


outputs at training an equivalent formula. That turns out to not be too bad. We tried


expression simplification and it didn't help at all. It doesn't really matter. This is very


interesting what you're going to come to with the success and failure cases. The leftmost column


here is pretty simple. These are success cases. Nothing too unexpected right here.


It figures out that for example, the middle formula, this might be a bit small here even for


people to read, but this is n times the sine of gamma. Gamma is what exactly?


Euler's constant. So n times the sine of gamma squared.


The entire thing on the right hand side is a constant. It's essentially n times a constant.


The model has to somehow figure out the expression for the constant as a formula.


It cannot just predict the number.


Then it has to realize that I have to multiply this constant by n. That's why it's a straight line.


The other formulas are similar-ish. The top one for example is n minus the cosine of n.


Again, reminder, this is symbolic regression.


The next ones are weird. The top one starts off very weird, but then it continues


in the same path. You can see it's regular enough that the model could figure it out from the data points it has.


By the way, the green background, that's the input. The blue background, that's what it has to predict.


The next one I find particularly interesting. The formula is the tangent of n plus n times the last element.


This is what the output looks like. How can the model from the just the left part figure out


that this is the correct formula and then the end date just blows my mind?


Maybe the log scale would help a bit here because there is probably quite a lot of variability in the first terms.


It's just squashed by the last term, which is huge. I should have maybe put a log scale.


That's a good question. What I find really interesting with these plots,


here you're showing the success plots and on the right hand side you have the failure plots,


is that we really see how symbolic regression is different from numeric regression.


In numeric regression, you have this set of points and basically you're just trying to fit your function.


You're trying to bend the function so that it goes through the input points.


This is typically going to be very prone to overfitting.


If you can't really understand the process then you're just going to fit a function which goes through the points.


Whereas symbolic regression here isn't biased towards overfitting at all.


It's just trying to find a formula. When it fails on the right hand side, it not only fails outside the input points,


but also on the input points. It's not even able to fit the points you gave it.


This really shows a big difference.


We can see this a little bit, I think on the bottom left there's a nice case where it already fails on the inputs.


That's the best formula it can come up with. You do have a beam search in there, right?


These ones, no. Indeed the beam search does tend to pull a bit more towards overfitting.


The way we rank our beam is that we evaluate how well the formula matches the input points.


In that sense you're coming a bit closer to actually overfitting the input points.


If you use the beam size of one as using most of our experiments, then essentially you're not at all biased towards overfitting.


It seems like here it's just misjudged the formula.


On the top left is an interesting one where it looks like it's done everything correctly.


The red ones are the outputs that it's supposed to match and the black one is the line, the function it produces.


What's wrong here? Is it off by a tiny bit?


The screen is pixelated so I can't see very well.


Sorry.


Essentially we get two kinds of mistakes. We get the mistakes where it's very close.


For example, it confuses a 4 with a 5 and so it's going to be very close.


But then you have catastrophic failures where basically, for example, to confuse a cosine with an exponential or something like that.


That's just one token error but it's going to give completely wrong predictions.


That's something that you typically won't get for numerical regression. You'll always at least fit your inputs.


However, there is one thing where symbolic regression is better than numeric regression.


Once it does find the correct formula, then it's going to get perfect precision on all the subsequent numbers you're going to give it.


If you think, for example, of extrapolating the sequence, with a numerical model you're always at some point going to get wrong predictions because you're not very good at generalizing outside.


It's a typical thing that machine learning is good at interpolating but bad at extrapolating.


With symbolic regression, once you've found the correct formula, you can basically extrapolate as far as you want. You've got the right formula.


Just saying for people who probably even people in the video will not be able to read, I can confirm the formulas of these two things are completely different.


The one is the sign of something simple and the one that's predicted is a very complicated formula that just happens to almost fit or maybe even perfectly fit the input data points.


But then it is just that tiny bit off and that gets worse and worse as the output progresses.


There are a bunch of other funny ones like this one. Again, the scale here is absurd.


It's like the exponent is 224 and there's just this one output that it's supposed to match.


That's just mean to the model, honestly.


We do have horrible expressions. Our generator uses up to 10 operators. If you look at expressions here, we only chose expressions with three operators.


You can imagine how horrible the expressions are with 10 operators.


Of course, the accuracies are much lower. If you look at the ablation, our performance at 10 operators is about 10% versus 100% when you only have one operator.


I will quickly uncover the rest of these, but I encourage people to actually go and look at the success and failure cases.


Also for the floating models, I think it's really valuable and you can directly see, as you say, the differences between symbolic regression.


If you did numeric regression, even if it has a pattern like this, like a zigzag pattern or something, it would quickly degrade.


We've all seen numeric regression. Although, in your experiments, there are cases where the numeric regression is worse.


There are cases where the numeric regression is actually better than the symbolic regression.


Could you comment a little bit on the experiments, specifically in distribution, out of distribution evaluation?


Typically, in distribution, our symbolic model performs better than the numeric model because it's got the right inductive bias.


We feed in these sequences which are generated by a formula. It's much better than the numeric model at extrapolation because once it's got the correct formula,


it's going to give perfectly precise accurate predictions, extrapolate as far as it wants, etc.


However, it is slightly less good at out-of-domain generalization.


One thing you see here, I can't remember where it is in the paper, but you see that, for example, numeric regression is better when you have complex prefactors.


Here, the expressions we generate, the prefactors we have are built from integers between 1 and 10, e and pi.


That's well fitted for the symbolic model.


What happens if you replace these prefactors by prefactors which are sampled from a Gaussian distribution?


These two columns right here, the difference between those.


Exactly. What's interesting here is that, in this case, of course, the numeric regression performs better than the symbolic because numeric doesn't care at all about the fact that you're using these prefactors because it doesn't really care.


It isn't trying to approximate these complex prefactors.


What's interesting, though, is that the symbolic model still isn't that bad because it's actually able to approximate prefactors with its own vocabulary.


You've probably got a table with a few examples of this.


This was actually purely something we discovered. We weren't expecting this at all.


We suddenly plotted the predictions of the model and we realized what it was doing.


For example, here, if you use the constant 0.3333 and you feed it to our symbolic model, of course, it can't directly output 0.3333 times n because it doesn't have 0.333 in its vocabulary.


It's going to have to build somehow this constant with its own building blocks.


You can see that it does that pretty remarkably well.


This is very surprising. Basically, what happened is that during training it has seen some expressions because our expressions aren't simplified.


We don't have something that is going to evaluate the expression.


Sometimes it sees a formula which has 3 plus exponential of minus 6 and it will notice what numerical value that evaluates to in terms of the sequence.


It learns to build any constant with its own vocabulary.


It's important to say that you don't, like other, if I see this, I would first assume that you have some sort of gradient based regressor in there that approximates these constants for you.


But you don't. The model actually has learned to output these symbolic expressions for particular constants.


That's something I think which is a bit rather novel here is that we have an end-to-end transformer.


Usually in symbolic regression you have a model which predicts a skeleton, so an expression without prefactors, and then you sort of fill in the prefactors with a separate solver.


Here our model does finding the prefactors all by itself.


That's nice in a sense because it's mathematically satisfying and it also gives us some quite nice approximations.


For example, here you can see with 1.64493 it outputs pi squared over 6.


You may know that that's the sum of the inverse of squares and I think Euler in his time spent quite a lot.


He had to actually found this numerical value and he spent some time figuring out that it was pi squared over 6.


That can potentially be useful for mathematicians.


Of course the drawback of it is that this is a complex process and if you have a very complex equation with lots of complex prefactors,


then our model is going to spend a lot of its attention to building these prefactors and it's going to make the task more complex.


This is why I think our model isn't directly applicable to real-world problems like forecasting where you have very complex prefactors in front of each term of the equation.


Is there any other surprising things that you learned in the experiments?


Maybe unsurprisingly a model like this is better than Mathematica which I would have expected because I'm not a big fan of Mathematica.


Steven Wolfram is cool but I'm not too much into the way Mathematica does things except for very particular applications.


Mathematica isn't that bad actually. I was surprised at how good it was.


It has these two built-in functions, find sequence function and find linear recurrence.


Basically find sequence function is going to find non-recurrent formula it verifies.


For example if you feed it 2, 4, 8, 16 it's going to say 2 to the n.


Whereas find linear recurrence is really for when it depends on the previous terms in a linear fashion.


These are actually pretty powerful because a lot of sequences are linear and Mathematica will always basically get these right.


Because actually there's a deterministic rule to find the linear recurrence so that's fine.


Find sequence function is very limited of course and you can see it gives worse results than OEIS.


But still these functions aren't miles away from our model.


I think actually both our models and Mathematica models are struggling a bit with OEIS.


They are outside of their comfort zone.


I think mainly because...


So one thing I should say is that here we're not evaluating on random sequences from OEIS.


We selected those which have a label which says easy which means that there is a logic behind them.


There is a recurrence relation.


However, or not necessarily a recurrence relation but there is a logic.


Just to clarify the other ones, you gave some examples in the paper of the other ones would be like the number of bus stops in successive streets in New York City or something where you can't possibly know unless you consult some outside knowledge.


OEIS does have a lot of nerdy sequences which are just for the fun of it basically.


But even in the ones which are labeled as easy, a lot of the sequences don't have a recurrence relation.


For example, the sequence of primes, the sequence of divisors of n, the sequence of decimals of pi.


All these things you can't really predict and so these kind of hamper our model.


So I don't think this is like the best way to show the power of our model.


Our model is especially powerful on the sequences which are built from the generator which are very complex.


Here in Mathematica, in OEIS, our models are just only a tiny bit better than Mathematica.


I wouldn't say it's the most impressive result.


And they are specifically also worse than numeric.


You can see that the numeric models, they do outperform here and that might also be because one, of the distribution shift.


And two, if there are as well some, even though they're labeled easy, but actually you might still need some outside knowledge.


A numeric model at least will sometimes come close to the solution, right?


Close enough to count as correct.


Yeah, exactly.


A numeric model is generally going to be better indeed when there isn't a simple formula but you can still infer logic.


Yeah.


Sometimes, I mean, you give very, I mean, if you've played a bit with the demo, you'll realize that sometimes you give a very simple sequence for us.


And for some reason the model won't be able to recognize it because it uses our kind of logic which we can't really express simply as a formula.


And the numeric model will be very good at that.


So while, yeah, I'm going to quickly open the demo.


I hope I have it ready somewhere.


And maybe you can tell us, like, is there, like, in the course of this research, was there a moment where it, like, didn't work at all?


Or, I mean, you had some basis to go by, right, from the work of, let's say, Guillaume and Francois.


But was there, like, what was the biggest problem that you encountered during this research?


To be honest, this was, I was surprised at how quickly we were able to get models working in the first place, at least on the integer sequences.


It was pretty quick to get some results from that point of view.


As I was saying before, just plugged in our transformer.


We just had to build the generator basically, which isn't that hard.


I think what we struggled with a bit was basically finding a baseline to compare with.


This is why we built this numerical task, because this is such a novel kind of path in symbolic regression to look at recurrent sequences that we didn't have.


We didn't have benchmarks.


We didn't have things to compare to.


And, you know, it's a bit disappointing to show some results of indistribution accuracy if you have nothing to compare to.


So, yeah, we built this numeric model just for that purpose.


And, yeah, in terms of challenges, I really, yeah, I was surprised.


It was much easier than I thought.


Okay.


It's interesting because I think we interviewed Guillaume and co-authors on a previous paper on the machine learning street talk.


I asked them pretty much, I think, the same question.


And they already said, like, no, you know, kind of we plugged it in and it worked out and it was cool.


So, I think this is like maybe it's forbidden knowledge, but this might be like a field of deep learning where there's things actually work.


You can get like results.


It kind of it works maybe or maybe let's say you get started with something that works pretty quickly.


Yeah.


Whereas if you're in like reinforcement learning, you spend months until something actually starts working.


And the explanation is simple.


It's basically just that you have this synthetic task and so you have infinite data.


And the big problem of deep neural networks is when they don't have much data, then you really have to get clever about how you regularize, how you choose your hyperparameters, how you build your architecture.


Here you can just throw anything at it and it'll work, it'll learn as long as it's got enough parameters.


And that's one thing that you have to have a lot of compute resource for this project.


And, I mean, here the transformer is pretty big and it's trained on a huge,


every epoch we train has five million equations and trained for like three weeks or something on 16 GPU.


So it's a pretty big scale thing.


Nice.


Lastly, I just want to present this demo you built so people can try this out for themselves.


So if I input like one, two, four, eight, and that should probably already be enough.


And then I have to click away and then it will compute, it will tell me the next ones are 16, 32, 64.


That's pretty impressive.


I want to, I think I tried to challenge it a little bit.


I tried to come up with some, maybe I thought of like a music sequence like


da-da-da, da-da-da, da-da-da-da, da-da-da-da, da-da-da-da, da-da-da-da, da-da-da-da.


And it's probably too regular.


I think it'll get that one right.


So yeah, it will.


Okay.


That is fairly regular if I look at the plot.


But yeah, I invite people to go and challenge your model a little bit.


Right here, you can also choose sequences of this OEIS database and yeah, check out the model.


This is really cool.


All right.


So I think this, is there anything you want to like special that we haven't come to you


want to mention about the paper itself?


That was great for me.


Thanks for your questions.


I think that was great for me as well.


I'm always happy if I can ask like all my dumb questions to the people themselves.


In this case, Stefan, thank you very much.


Thank you and your co-authors for writing the paper and thank you so much for being here.


This was really, really fun.


Improving Intrinsic Exploration with Language Abstractions (Machine Learning Paper Explained)

Hi there, this is a comprehensive paper review on the paper Improving Intrinsic Exploration


with Language Abstractions. This is a very cool paper because it combines a language


and the information that is in language with reinforcement learning, specifically the problem


of exploration. I don't want to tell you too much more right now because we're going to


dive into the paper in just a bit. So this video will explain in detail what is in the paper,


how the method works, what they're doing. So by the end of this video, you should have a really


good idea of what's in the paper. In the next video published tomorrow, there's going to be


an interview with the authors of the paper, which is very, very cool. It's super valuable,


and I was very happy to host this interview. So I hope you draw some value out of either one of


these videos, hopefully both. As always, thank you very much for watching. Thanks to everyone


who likes and comments and supports in any way is really cool to be able to do these things,


and I'll see you around. Bye bye. Hi there. Today we're looking at improving intrinsic exploration


with language abstractions by researchers of Stanford University, University of Washington,


Meta AI and University College London. This paper on a high level uses language to facilitate


intrinsic exploration. That is when in the face of a very sparse environment, a reinforcement


learning agent has to come up with its own goals in order to make progress. So the intrinsic


exploration or intrinsic motivation refers to the fact that there's an additional reward that we


give to the agent just for attaining, let's say, new states, novel things in the environment. Now,


it turns out that that's not super duper easy because not all new things are equal,


and especially, let's say, there is a random component in the environment, then, you know,


that's going to be new every time, yet it might not be interesting. So how do you go about this


is quite a challenge. It's clear that we need something like this in sparse rewards environment,


but how exactly to do it is still challenging. This paper adds language to the mix and argues


that language descriptions could be one such source of novel, of indicators of novel states.


So we're going to go through the paper. Let me know what you think in the comments, definitely.


And yeah, let's dive in. So they say they want to solve these complex long horizon tasks with


sparse rewards. And as I already said, that is not really a picnic for reinforcement learning


agents. Usually those need very tight, very dense rewards in order to work. And that's why we give


these intrinsic rewards for exploration. And that is encouraging the agent, even in the absence of


rewards, to go out and explore things and do new things. And we hope that through the exploration,


at some point it will learn the skills or it would encounter something that will actually


give true reward. So they correctly claim there is a design choice on how to measure exploration


and an implicit, like a common answer, that the agent should be rewarded for attaining novel


states in the environment. But that is, as we already said, quite difficult to actually implement.


For example, states can look cosmetically different, but have the same underlying semantics


and thus not be truly novel. So the two fundamental challenges for intrinsic exploration they list


here is first, how can we reward true progress in the environment over meaningless exploration?


Second, how can we tell when a state is not just superficially but semantically novel? And that's


where they add in language. They say, well, if we had language describing the states, then certainly,


for example, here we have language that describes the state. Here the language description says in


what direction, indicating that you can go in a couple of directions or do something in a couple


of directions. You see here a crystal wand. That means there's something to pick up. So when you


don't have this message, that might be an indication that the state is meaningfully different, namely,


it doesn't have the crystal wand. So as you can see, these authors imagine that if we had a language


description of the environment, that could give us an indication of when something is novel and


when something is just the same but looks a little bit different. They say language obviously has


strong priors over the features and behaviors needed for meaningful interaction and skill


acquisition. And that's just a matter of fact that language has been developed to communicate things


that are useful to humans. And they also say correctly that you can describe with language


very particular things such as move left or very abstract things like acquire the amulet and defeat


the wizard. Although one of the abstraction here comes from the end, but still defeat the wizard


is a very, very abstract thing. Now, as we already said, what they're going to do here is they're


going to look at these environments, at these reinforcement learning environments. So there's


a mini grid on the left. And in mini grid, I believe the agent here, that's the red triangle,


and the agent is supposed to, I think, go to the keys, get the keys, open the doors, and eventually


get the final reward that is somewhere on the map. These are procedurally generated, so it always


kind of looks different. And that's one challenge because if you have to make sequences of actions


like go over here, get that key, go to the door, and then go further and get the reward, that is


a sequence of actions that is unlikely to happen by chance, right? To stumble over the key and to


stumble over the door and to stumble over the reward, you know, the amount of times you're


going to try randomly until that's the case is staggering. And therefore, something like Q-learning,


which just requires on random exploration, is going to almost certainly fail right here. But


this is one of the environments, which is a challenging environment that they pick up,


and that has these language descriptions, or I think in this one, they add the language


descriptions. But in any case, this is not about language models or anything like this. They assume


that they have a function, which they call L, the language annotator that takes in a state,


takes in and gives you the description. And they just assume they have an oracle that does that.


So for the environments they do test, they actually have that. And so in Minihack here,


this is even part of the game, right? In Minihack, you will always get a message like this to every


step that you do in almost most of them, most of these states have such a description available.


So again, there's this function L, which in this case is just the game engine, it takes in a state,


and it gives you back the description. So if you could guess here that we might learn this language


descriptor, right, we might even initialize it with a language model, we can use something like clip


or something like this. This is certainly in the future work, they list this, but not here. Here,


we assume we have this oracle. Now, what can we do once we have such a language description? Well,


we can use it for exploration. So there is a little bit of mathy math right here, which we're


going to skip. Essentially, this just discusses that, yeah, they have this annotator L that


produces these natural language descriptions, and they add an intrinsic reward to this. And now,


we're going to look at what the intrinsic reward is. So they're going to take two different algorithms


that are already made for intrinsic motivation, and they're going to augment them with language.


The reasoning behind it is that those two algorithms, the one is called Amigo, the other


one we'll get to in a second, they're already kind of state of the art in this domain. So what they


say is if we add language to those, and we can get a better result, then that kind of shows the


usefulness of language of the language descriptions. So we're going to look at these algorithms


briefly. Remember, these algorithms aren't by this paper, this paper is how to add language to them.


So Amigo, the adversarially motivated intrinsic goals, trains a student and a teacher. So there


is a teacher that generates goals, and then the student is just a goal conditioned policy. The


goal is, as we said, provided by the teacher. So the student is the real reinforcement learner,


but the student is simply conditioned on some goal that's provided by the teacher. It doesn't


try to solve the actual problem, it solves the goal that the teacher gives it. I mean, it probably


gets reward when it accidentally also fulfills the true reward goal. But it does get intrinsic


reward when it fulfills the goals set by the teacher. Now, the goal set by the teacher, that's


the trick, obviously, right here, the teacher policy is quite smart. The teacher policy takes


in the state of the student. So it looks at, you know, where is the student, and it needs to now


decide, what do I do? What kind of goal do I give the student? On the top left here, you see this in


in in this mini grid environment, the teacher is this network or this function right here, it gives


a coordinates that the student has to get to. And then these coordinates, as you can see there,


I'm not sure if those are the actual coordinates, but whenever the student actually reaches them,


so it provides the goal to the student, when the student reaches it, it gets reward. So that's it.


There is also a notion of a difficulty threshold. That difficulty threshold is, it increases during


training. So the idea is that at the beginning, the teacher wants to suggest kind of easy goals,


and then as time progresses, the teacher has to learn essentially how to make the goals harder


and harder. And by making the goals harder, the student essentially has a curriculum of harder


to reach skills. So the teacher should kind of learn to propose more hard goals. So I think that


the work here is definitely done mostly by this teacher network and the challenges. In any case,


there is this difficulty threshold, this difficulty threshold is increased linearly during training,


and the student, no sorry, the teacher, the teacher is given a positive reward if it proposes goals


that take the student more than T star time steps to complete, and a negative reward for goals that


are completed sooner or never completed within the finite time horizon. So you also can't go


impossible, or it can't go too hard. It needs to go exactly as hard that the student reaches the


goal, which means even if it's a possible goal, it can't go too hard for the current student. It


needs to essentially propose goals that are just outside the abilities of the current student. So


that zone of proximal development is kind of formalized in this teacher. That's Amigo. The


other, so how do we add, how do we add language to that? We saw that usually the teacher supposes


or proposes coordinates for the student to get to. Now, if we have language descriptions for every


state, so every state the student finds itself in, there is a language description. The teacher


can simply output a language description of a state. In this case, these are formulated as


as kind of instructions, right? But remember, they are just descriptions as far as I can tell


of the state. It is more evident in the mini hack environment, right? So these are just descriptions


of the state, whatever the game would output if you're in this state, and the teacher simply


proposes these. So it just says, well, here is a goal for you. Try to get to a state where the


language descriptor outputs that. So that those are the goals that the teacher can choose. Where


are we? Yeah, so we don't have X, Y goals, but we have natural language goals. The student is


rewarded if it reaches a state with a natural language description that the teacher outputs.


Easy enough. So how does the teacher do this? It selects goals from the set of possible language


descriptions in the environment. Now, initially, these are unknown. So the teacher doesn't know


yet what the environment has in store, because again, we don't assume that say extra information,


we need to get out everything of the environment. Therefore, as we go through the environment,


we collect more and more of these goals. And these are the goals that the teacher can choose.


The teacher maintains a running set of goals that is updated as the student encounters new state


descriptions. The teacher has this move to language they say creates a challenge. Not only must the


teacher choose which goal to give to the student, it must also determine which goals are achievable.


And that's why they train two different networks. There is a policy network, which produces the


distribution over goals given a student state, and a grounding network, which predicts the


probability that a goal is likely to be achieved in the first place. So remember, these environments,


they're procedurally generated. So every time the student is every new episode, I believe that's


how it works. The student is placed in some environment that it has essentially never seen


before. So now the teacher takes that in, and it produces two things. It looks at this environment,


produces two things. From the set of goals that it has, it picks one that it wants to propose.


That needs to be right. So for it cannot always do the same. And that's the interesting part right


here. So if the green door is over here, go to the green door might be very easy in one environment,


but very hard in the other environment. When I first read this, I thought, well, if you know,


if the teacher knows no goals at the beginning, and it only collects these goals that the student's


student encounters over the course of the episode, we're still kind of relying on random


exploration of the student right because any goal it hasn't achieved yet cannot be proposed.


Whereas in the original xy coordinate, I can, I believe at least I can just propose any xy


coordinate like get to that. However, since this is procedurally generated, you might imagine that


the student encounters like the green door in one environment where it's very easy, it essentially


just stumbles upon it. And then the in the next one, that's kind of a bit more challenging to


reach. So we are still good on collecting goals. The other network it does is this grounding network.


So the ground, let's call that GD, the grounding network, it gets the initial state, and it


proposes it checks which of the goals are even possible to reach. So these are two slightly


different targets, the policy or let's call that poll. Well, okay, the policy network wants to


propose goals which it finds challenging enough, right, for the student to fulfill the grounding


network wants to check which of the goals are even reachable in the first place. And the grounding


network specifically is trained as this multi class, they say a multi labeled binary cross


entropy loss, which I find to be a weird term, but okay. But essentially, it's given the initial


state of an episode, we asked the grounding network to predict the first language description


encountered along this trajectory, where t is the minimum t such that there is a description at all.


So we're training, we're training the grounding network to predict the first language description


term against all the other term in its encountered goals. This is kind of like a contrastive loss. So


the that first goal is certainly reachable from the initial state. And we simply take all the


other ones as kind of a negatives for that for that first one. And exactly, so the second one


can be seen as noisily generating negative samples of start state and unachieved description. Now,


now, yeah, based on the set of descriptions known to the teacher. Now, this seems a bit weird,


right? To train the grounding network like this. Like, what about the second text description that


was encountered? That's certainly reachable too, no? At least I would, at least I would,


I would guess so. And this is really necessary. Or maybe this here, maybe this here should be


over goals that weren't encountered in the episode at all. Right. But this seems quite weird to only


take the first encountered language description as a positive example of this grounding network.


Further, and let's go into criticism right after we conclude here. They say to summarize the teacher


training, training the teacher involves three steps. Updating the running set of descriptions


seen in the environment. That's collecting the goals, essentially. Learning the policy network


based on whether the student achieved the goals proposed by the teacher. OK, that's the same as


the original amigo. And third, learning the grounding network by predicting descriptions


encountered from initial states. OK, well, this description here I can agree with. I don't,


I just don't see why only the first is taken as the, as the positive sample. So what, what are we


doing right here and why? What I find weird is that this grounding network has to exist at all.


Right. In the original description, I don't know if these things are generated, if these, certainly


all the coordinates exist right somewhere, but they are not necessarily reachable either for the


original amigo. It seems weird that the policy network itself, with whose goal it is to propose


a goal that is just outside of the reach, essentially, of the student, couldn't itself


make the determination of whether a state is reachable at all. Because the original amigo


network seems to be perfectly capable of making that determination for a set of coordinates. Right.


So it might, you know, there is a difference in that something that go to the green door,


there might be not a green door at all in the environment. But it seems, it seems a bit weird


to split this stuff up into a different, into different networks. And it tells me maybe they


tried it first and that didn't work. So they had to throw in kind of another loss, which is, is kind


of a bit, just a bit annoying. But, you know, if it works with the extra loss, then okay. Here you


can see again, we have the amigo teacher first, that's the grounding network. What is even possible


in this environment? Then it, that is relayed to the policy network or multiplied by the output


of the policy network. Policy network predicts goals that the student in its current state could


reach, but not under the threshold. All the while we add new goals, we train the grounding network


on states that were actually reached during what language was achieved during the episodes. We take


the other ones as negatives. And then lastly, the policy network is trained like amigo. Now there is


a typo here, I believe, I believe, because here it says the reward is given if the goal is achieved


in less than T star steps. But I believe it should be more. I believe this should be more, because


that's what it says in the text. Yeah. So that's that. Yeah. I don't know why the split. So the


important difference as well is that the policy network is trained essentially with reinforcement


learning, right? It's, I guess, an actor critic framework and it's trained on the action that it


actually output, like in classic reinforcement learning fashion. Yet the grounding network seems


to be more achieved in a classic supervised sense, just as an online classifier. I'm not sure if they


have done ablations. I haven't seen the ablation of what the el amigo does without the grounding


network, but it would be interesting to see. The second, so here you can see how they add language,


right? They add language by essentially replacing that teacher student relationship where the teacher


proposes goals and coordinate. Now the teacher proposes goals in language. So that's the novelty


here. The other one, the other algorithm is this novelty algorithm. So the novelty algorithm is a


little bit different. It defines intrinsic reward to be the difference in novelty between a state


and the previous state. So there's this notion of novelty and we're not going to take that as itself.


Like we're not going to take the novelty and give the agent a reward simply for achieving whatever


we call novelty, right? And we can define novelty in whatever way we choose. What we do is we give


the reward if the agent transitions from a state of low novelty to a state of high novelty. And so


that's the, that's this thing right here. The max with zero is so that this cannot be negative. So


we don't penalize going from high novelty states to low novelty states because you know, sometimes


that is necessary. And we also only give that reward if a state is encountered for the first


time. So here the agent is encouraged to find new states because it only gets rewards when it


encounters new states. And it is especially encountered to find new states that are a


significant increase in novelty from the previous states. This is one, I guess one way. What this


avoids I guess is to get stuck in this loop. Let's say it's like you're in an environment,


right? And you're in an environment and then here is like a random, just some random thing. People


usually they say there's a TV with static on like just kind of like or there's a bunch of leaves


flowing around or something like this. And the agent that is just going for novelty would just


indefinitely stare at it. And this prevents it because whatever you call novelty, if you call


this novel like a TV with static because it's essentially a random signal, so it's super duper


novel. However, you wouldn't get a reward for consecutively looking at the TV because you would


already be in an equally novel state going to a new novel state and that will give you no reward


at all. So you're encouraged actually to go away from the TV, go somewhere else where you can


transition from a low novelty to a single high novelty state. So yeah, what they say is in the


first term the N is the novelty. This quantity describes the difference in novelty between


successive states which is clipped larger than zero. This is written a little bit weird. This


quantity here refers to the first term, not to this thing right here. This thing is just an


explanation of what's in the term. So N is the novelty and the reward is the difference in


novelty. The second term, right, only if we encounter it for the first time. And how does


this thing, how does this thing track novelty? This is an interesting concept. How do we do,


you know, like how do we know if a state is novel? Sometimes it is sufficient, they say,


to track exact state visitation counts, but obviously as soon as the environment gets


larger and a bit more complex, this is not possible anymore. So what do we do? We use


this random network distillation and I have to say I have never heard of this and that seems quite


smart. So what we do is we have a state again, so your agent is here, there is a bunch of walls


and so on. What we do is we have a random neural network. Now that's always the same, but it is


essentially random. So we take the state, we feed it through the random neural network, we get out


some vector, just some vector because it's a randomly initialized fixed neural network. It's


going to be some kind of embedding of that. Not a useful one, but just some sort of an embedding.


And then what we do is we train a, what do they call it? We train a state embedding network. So


let's call that E. We train embedding, again, this one takes this in and it tries to predict


this vector, right? It tries to predict it. Now, obviously it doesn't, it can't see the weights of


this neural network, otherwise this would be quite useless. But it tries to predict this vector and


it is trained. So the E is trained with back propagation while the blue one is fixed. Now,


the logic here is that if I encounter a new state, right? So here's my new state, agent is here,


there's just one wall here, there's like a door here. I put it through both, oops, I put it through


both of these, new color. I put it through, hey, yeah, I put it through this one and I put it through


this one. And then I get a vector here and I get a vector here. I look at the error between the two,


right? So what's the difference? If the error is small, I can safely assume that I have seen states


like this before because if the error is small, it means that this thing has learned to match this


thing for some kind of similar state, right? We know that neural networks generalize well if they


have training data in the same vicinity of the data that you want to test on. Therefore, if the


states are quite close, that means the outputs are quite close. That's a property of random neural


networks. If you don't change the states much, it depends a little bit on parameterization. But


essentially, if you change the input a little bit, the neural networks output will change a little


bit. And therefore, if you've encountered states like this before, this E would be trained on those


states, would actually learn to match the blue fixed networks output, and therefore the distance


here would be small. However, if the state is super novel, that would not have been like anything in


the training data. And therefore, this E network would make a large mistake when trying to predict


the vector and from that mistake right here, because that's you have that at inference time,


right? You can determine whether something is novel. There's a bunch of caveats, but since this


paper isn't about novelty itself, I'm not gonna I'm going to reserve that for another time. So what


do we do it to add language? That's this paper. Now, we add an additional exploration bonus based


on novelty defined according to the natural language description of states. So again, we it


this is simply a repetition of the formula, we have some sort of a notion of novelty of a linguistic


description. And we give the reward if the novelty of the new state is higher than novelty of the old


state for whatever definition, and only the first time we encounter it. So they say nL is the novelty


of the description L, as measured by a separately parameterized random network distillation network


encoding the description. So presumably, other than inputting states, now every state also has


a language description. So a language description here, a language description here, we have a


separate network that a separate random network that we can put them through. And we can, we also


have a separate embedding network, let's call that eL, the language embedding network. And we do the


exact same thing with the language as we did with the states themselves. We try to train this eL in


order to predict to match the predictions of the random network. If at inference time, the two match


closely, we assume that this is like something we've seen in the training data. And otherwise,


it's novel. So here you can see, they say, we keep the original exploration bonus as language rewards


may be sparse. So they, they add both the intrinsic reward is the original one, that is just about the


state and the new one with a hyper parameter. And here, I think it becomes clear what for me, the


biggest criticism of this paper is, and that I think, so they make the point that well, you know,


language helps. And if you if you look at the experiments, they say, linguistic exploration


outperforms non linguistic exploration. That's one of their experimental findings. You can look at


the results, although the confidence intervals like this is just reinforcement learning, I get it.


But yo, you have to work hard to make those, you know, to make these overall intervals not not


overlap. That that is, you know, good job. But still, the noise in these environments is quite


significant. And linguistic exploration excels in larger environments, which you can imagine, right,


because in larger environments, they might be also more complex environments, and therefore, just


state abstractions themselves might not be the best one. But my criticism here is that essentially,


they add extra data, right? So it's not like linguistic exploration outperforms non linguistic


exploration. It's, hey, the environment actually has this data right here. And no one without this


one, no one's used that. So people just have used the image or whatnot, and the actions and the


rewards. And there's this extra data, what if we use this extra data? Oh, we get better. Wow. And


the data is obviously very good, because it's made by humans. And the game creators have


essentially, so the game creators know which states are equal, right, they code the game. And


I, in the same vein, they produce these language descriptions. So the language descriptions are


almost like a little bit of a view into the internal state of the game code itself, right?


Even if that weren't the case, language obviously is, is quite powerful. But I get their argument


that, you know, language gives you abstraction, yada yada yada, and so on. However, I think the


gains here aren't language is better than, you know, not language, because I don't think it's a


necessarily a fair comparison. It is, you know, adding more stuff, adding more information,


especially really good, really high quality information like they have, is better than non


not adding that information. Now, obviously, it matters what they do with the information. But


yeah, I think a lot of the gains simply come from the fact that they add something on top. So not


to say like they, for example, in El Amigo, they drop the original teacher, right? But in this, in


this in this novel D, they don't even drop the original intrinsic exploration. Yeah. So, you


know, it's essentially really extra data that they add. What is interesting is that they analyze the


curricula that emerge, right? Given that it's language, you can you have a pretty good idea of


what's happening over time. And they have these nice analyses right here, where, for example,


first, the teacher proposes open the door before it proposes open the colored door. So see, here is


a variable that holds the color. So you can see that the teacher first proposes the easier goal


of opening any door, and then it proposes a lot of opening the opening colored doors, it then


discovers keys, going to the keys, picking up keys, then going next to the door with the key. And


after it goes through the door, it picks up the ball, which is the final, the final goal. So you


can see clearly that as the training progresses, the teacher gives more and more complex goals.


And that is, is kind of true, is true for El Amigo. And this novel D, it is not that true in


all the environments for the for the NetHack environment, I believe it's a little bit more,


they call it a little bit more exploratory, in that it just tries to explore a lot of stuff,


which is also good, right? That does, it doesn't need to be progressive, right? As long as the


teacher encourages the student to, you know, do this, and now, okay, now you're really good at


that. So I can't essentially propose that anymore, because you'll fulfill it in less than the


threshold time steps. Now, you know, do something else, now do something else, and do something


else. Again, these aren't the descriptions, right? It's, these are, these are meant to be


descriptions, not instructions. So this here, I guess, is a better, again, a better example.


So you want to reach a state that has the description of there is a staircase up here,


right? So you just tell the student, please reach any state with that description.


And you can see how this develops, which is pretty cool. The last thing they do is something


that I also find very, very interesting in that even though, right, even though, as far as I


understand, and I think they say this somewhere, they don't use pre trained language models or


anything like this in here. They do, obviously, output language and so on. So they need some sort


of language model, but they don't use, they don't make use of any pre training on any external data


or anything like this. Yet still, the semantics of the language seem to be captured a little bit.


For example, they do this experiment where they replace all the language goals with unique


identifiers. So go to the red door would just become token one, go to the blue door would


become token two. So now there is no shared substrings. So the model, the model is,


the model cannot generalize from this go to the door construction and sort of generalize the


skills or generalize the reachability estimate of the goal. The result is one-hut goals perform


quite competitively, which is good, right? So that lends more credence to what I say.


Like this is just, this is extra data. Then the second thing is the lAmigo is better able


to exploit semantics with a more significant improvement in aggregate performance over the


one-hut goals in contrast to lNovelD, which shows less of a difference. So at least one of the


methods is actually able to exploit these semantics in the language. And that is a promising outlook


if we now want to go ahead and use something like pre-trained language models in these,


or something like CLIP to even get the description out of the state itself. That would be really cool


or some sort of a CLIP modified for reinforcement learning. So we don't need to rely on environments


which have this language description already built in because very, very few do.


And it seems to be quite hard to get, honestly. If we want to train a good model for that,


that is challenging. If, let's say, Atari or so, very challenging, you either need to collect


labeled data for describing Atari states, which itself is really hard. And if you let three humans


do it, you're going to get three completely different descriptions. And at that point,


we're going to need these large language models because the large language models need to be able


to tell, well, these two wildly different descriptions are actually meaning the same thing.


And how much of a gain at that point is still left when all this noise comes on top of the


learned description models and of the inferring whether two language descriptions are the same or not,


whether or not there's still an actual difference there to like L-amigo and amigo remains to be seen.


This paper here uses a lot of oracles to get its data, which is fine for research,


but it's not necessarily means that this is going to be a practical thing in the future.


So, yeah, they say this, though they criticize themselves fairly well. I think they want to


alleviate the restriction on oracle language annotations, perhaps by using learned state


description models. Yeah, exciting extension would be to propose abstract goals, which is also


pretty cool. And again, something where large language models can come in and help pre-trained


ones even. You don't even have to train them. And yeah, using pre-trained, well, okay, that's


it stuck in my mind from reading it the last time. Pre-trained models to imbue semantics into the


model beforehand, they say would also be pretty interesting. Among a lot of other things,


they also criticize the noisiness and so on. So that was it for the paper overview.


Let me know what you think about this paper. I find it to be pretty interesting,


and I think it's a really cool, cool idea. And if we can extend this to not use oracles,


I would be super happy. And I think this essentially is how humans also learn a lot of times


by talking about things, by talking about goals and so on. Language does provide a really good


abstraction for these types of stuff. Yeah, let me know what you think in the comments.


Accelerating Deep Learning by Focusing on the Biggest Losers

Hi there, today we're looking at accelerating deep learning by focusing on the biggest losers by Angela Jiang et al.


This paper is pretty simple, pretty short in idea and is a pretty much an engineering paper.


So we'll go over this idea and give it a good look and discuss advantages, disadvantages and so on.


So what's the basic idea? The basic idea is the following. If you train a neural network, what do you do?


Usually you have a training data set, which I represent here. Each line is a sample.


And usually you have your network has a bunch of layers. Each line here is a layer of weights.


What you do is you group your training data set into mini batches. Let's say that's a mini batch, four samples.


And you pass it through the network. This is called the forward propagation.


You then calculate the loss. So loss of your forward propagated signal.


And then you back propagate this loss.


Now when back propagating, you want to back propagate the loss such that it reaches each of the layers and it tells each layer how to update itself.


So what you want to do is for each layer, you actually need to back prop the loss once towards the layer below it and once towards itself in order for the layer below it to continue the back prop and for the layer itself to update its weights.


So each time you back prop basically once towards the lower layer and once towards yourself.


And that's a lot of work. So you see whatever work you have passing your samples through the network here, whatever work you have, you basically double the work going back.


And that's the core idea of this paper is if you look at the following in a traditional training neural network, you'll have some overhead in each training step.


Some overhead of maybe putting the data to the GPU or something like this.


Then you have a time that you require for a forward pass. And then you have a big chunk that you require for the backward pass.


And you see it's about double the size of this forward pass. And this paper asks how can we reduce this backward pass time.


So what they propose is the following. They propose, well, if the backward pass is expensive and we do it here for each data point in these mini batches,


why don't we stop doing this and only kind of try to select examples that are important.


And once we only have selected important examples, only those examples get to do the backward pass.


Thereby, let's say if we can only select one third of the examples to do the backward pass, we can reduce the amount that's required in the backward pass,


the amount of work by one third or sorry, by two thirds. And the way they select the important examples is by looking at the loss.


So they basically say whichever examples have a high loss, this must be the important examples. These are the hard examples.


And if we only train on the hard examples or if we train on the hard examples more, then the network will learn on these hard examples faster.


Of course, there is an implication there that if your network is good on the hard examples, it's also going to be good on the easy examples.


That's like the definition of hard and easy examples. Of course, that's a kind of a simplifying assumption.


But so the idea is only select the hard examples and own by how much loss they have and only then backprop these hard examples.


And that's how they can, they can reduce this by a lot. Now there's several intricacies here.


So the setup time, of course, is the same. What they do next is they forward propagate the entire mini batch here, because they need the loss of each example.


And then therefore they need to forward propagate the entire mini batch.


At the end of this, they select the examples with the highest loss, and they only use those in training.


Now training consists actually of another forward pass, but this one is much smaller because you only forward pass the examples that you're actually training on.


And then the backward pass accordingly will also be much, much smaller because now, again, you have less samples to actually train on.


Now, the reason that you even need the second forward pass is the following.


When you do backprop, you can't simply start with a signal back here and then backprop that through the network.


That doesn't work usually with most network architectures. Namely, what you need to do is actually while you forward pass, you need to remember information at each layer.


A good example of this is the max pool operation. So in max pool, what you do is you maybe have four pixels that are next to each other, and you select one of them.


Now, you need to remember during the forward pass which one you selected. Otherwise, the backward pass won't work. You need to know which pixel to backprop through.


And that's why at each point, you need to remember information to inform the backward pass.


So that's why basically you need a second forward pass with only the examples that you want to train on.


So you forward pass once, calculate this loss, select the ones with the high loss, then forward pass these again, and then backprop only these examples.


That's the main gist of it. And this is exactly what you see here. So forward pass everything, then forward pass those again that have high loss, then backprop them.


There is actually an interesting thing in this graph again that you see that this forward pass here also is shorter than this forward pass.


I assume that's because this forward pass here actually needs to do those additional saving of information, while this forward pass here is simply a forward pass without intention of backward passing.


And you can instruct the deep learning frameworks to then not remember this information.


They have another improvement over their algorithm called stale selective backprop. So this is called selective backprop. They have stale selective backprop.


What stale selective backprop does is it says, well, we might not always need this forward pass.


What we might be able to do is actually, so first we take the entire data set. Let's use a different color here. Let's use this.


We take the entire data set, forward prop it through the network, and then save this into some database, the losses.


And then we use these losses to select the individual points here for a while.


So we perform maybe, so maybe this is training here. You start here, you do this loss calculation, and then you run your training until a couple of epochs.


And then you say, OK, now this information is really outdated. I should really update it.


And you do this entire thing again. And then you run training some more until you again stop and say, OK, now this information is stale again.


Thereby you can amortize the cost of these forward passes. So you pass your entire training set once in a while and then use those losses to select the hard examples.


And that's amortized. You can then reduce this forward pass that's used for selecting again by a lot.


And of course, the paper shows that this doesn't hurt your performance too much if you have this stale information.


Right. So this is the entire idea of the algorithm. And the algorithm is detailed here.


So very briefly, you have this buffer and you go through the data, you forward pass every example, you for each loss, you calculate the probability that you should retain it.


So it's a probabilistic framework. It's not an absolute cut off. If you decide to choose it probabilistically, you append it to this buffer.


If this buffer is of a certain size, then you do the back prop on only on this buffer. And this buffer now only has the high loss examples or the high loss examples with higher probability.


And don't forget within this backward here, there is also an implicit forward pass. And then you clear the buffer and go on.


So there's lots of, you see, there's lots of forward passes here to compute the losses. And then every now and then there is a backward pass whenever the buffer is of certain size.


Now, the probabilistic calculation of how and when to retain a sample is very simple.


You have a deck of recent losses, a history of recent losses, simply calculate the percentile that a given loss has in this history, and that percentile will then decide on the probability.


If you raise it to a power, and that looks something like this.


So what's often used in this paper is this 33% selection, that will be the blue curve. And you see the median example here, if you are in the median, then you have about a 33% chance of being retained.


If you have a higher loss than that, then your probability rises.


Alright, so the first interesting thing actually is this graphic here where they show the what the algorithm deems the hardest and easiest examples.


So examples chosen least frequently, and this is the CIFAR-10 dataset, which is images 32 by 32 in color, and you need to classify them into 10 categories.


And you see the easiest images, the ones least chosen least frequently, are almost all automobiles.


And of the automobiles, they're almost all where you see the full car with the wheels and whatnot, like this one, this one here.


So these these are what the algorithm deems easy samples.


And if you look at the hard samples, examples chosen most frequently by selective backprop, it's these here.


So it's for example, bird. And bird in this case, it's just kind of a smear. They're just kind of smears on a blue background.


It's understandably that this resolution is pretty hard to make out that this is a bird.


And airplane and automobile here you see this, it's a it's only a partial view of the thing.


It's not like the full car like we saw in the easy picture. It's only partial.


And this seems to be pretty hard and it's understandable.


This cat here, to me, it's even unclear if this is a cat or a dog.


And I think dog is also a category in CIFAR10. So the algorithm is certainly understandably confused by this example and deems it a hard example.


And here even more you see truck. And this isn't a truck as far as I can make out.


These are two humans on the picture with no truck anywhere visible.


So this seems to be a mislabeled example. And of course, mislabeled examples are going to be of high loss to the algorithm.


This is the first criticism or thing. And the authors recognize this, that if you up weigh your examples with high loss,


you are going to up weigh all the mislabeled examples as well.


And thereby, you're going to train more on the mislabeled examples and thereby you're going to possibly degrade your test performance much more than had you given every sample the same weight.


And the authors address this actually nicely by doing an experiment.


And the experiment is what if we artificially mislabel examples? How much can these algorithms tolerate?


And so they often have these graphics here where they show test error over time.


So test error and the x axis here is number of back propped images, which is kind of a time dimension in training these algorithms.


You see the blue is a traditional model and the pink is a selective back prop with a 33 percent retain rate.


So you see the selective back prop is much faster in reaching a low error.


And this first thing is simply with one percent of the labels shuffled.


So one percent of the images are mislabeled. Selective back prop is still much faster than the traditional trajectory.


If you go to 10 percent shuffled, still you can see selective back prop is faster reaching a low error.


Of course, the error now generally is higher. But if you go to 20 percent here, what you can see 20 percent shuffled labels, what you can see, it starts to become clear that these selective back prop, it retains 33 percent of the hardest examples.


So and 20 percent of the examples have a wrong label.


That means most of what it upweighs are wrongly labeled examples. Almost, let's say that there's still a lot of correctly labeled examples.


But you see, it gets to a low error, but then it gets up again as it kind of massively overfits on these wrongly labeled examples because it upweighs them so much.


It upweighs these because here still every example is hard.


Right. So these wrongly labeled examples, they'll get about the same weight as correctly labeled examples because the network isn't trained yet.


But as you go lower, it starts to massively overfit. So compared to the traditional model, it kind of just reaches this low error that, OK, is now corrupted by these wrong labels.


But it doesn't it doesn't hurt as much.


So that's kind of my first criticism. If you have a lot of noisy labels, if or if you have a lot of mislabeled examples, then this method might actually hurt more than it helps.


But the level is interesting that it can kind of tolerate 10 percent, but it gets kind of into trouble at 20 or so more percent.


So this is the first criticism, and that's how the authors address it. I really like this ablation study that they do.


Here, this is kind of the meat of the experiment.


So what they show here, these curves on the bottom and let's look at this curve is on the X axis.


You actually have wall clock time. So how much time do you need in order to reach a kind of low error?


Here is test set error. You see, the traditional model in blue has a certain trajectory.


Now, Cath 18 is a baseline. Don't worry about it.


What we're interested in is the selective back prop, which is the pink, which you can see outperforms this traditional training.


And what we're also interested in is the stale SB.


So stale meaning it has this buffer of information that reduces it's supposed to reduce the time again.


And you see that is even that even more outperforms the traditional approach.


You can also see that the staleness here apparently doesn't hurt the performance too much.


You see, the error is fairly close and it reaches this error in a much faster time.


This on CIFAR 10. They have this nice table up here where they show the speed up to reach a given error.


So what they do is they take this error of the traditional model, this test set error,


and they ask how fast are these methods in reaching this error times a constant.


So times 1.1 times 1.2 times 1.4.


Now, of course, the reaching 1.4 times the final error is much easier and thereby.


But it's also easier for the traditional model, of course. So that's the catch.


But these are kind of benchmarks they chose to how fast are these models in reaching 1.1, 1.2, 1.4 times the error of a traditionally trained model.


You can see here on CIFAR 10, for example, actually, let's go to SVHN. SVHN is the easiest of the of the data sets.


And it shows the most clear thing. So the traditional error is 1.7 percent.


And you see that the speed up is so this selective backprop is 3.4 times faster in reaching this 1.1 times the the error of this traditional model.


And it's also 3.4 times faster reaching 1.2 times. And it's 3.5 times faster in reaching 1.4 times.


The stale selective backprop is even faster. So 4.3, 4.9, 5 times faster in reaching 1.4 times this reaching 1.4 times the the error.


And so what you can what you can see here is that these methods really make it faster.


But also there's kind of two things, two important things to note in this table.


First of all, you can see as you go to the right in the table, the speed ups get higher.


And what it means is that as you need to reach as you make the problem easier.


So as you need to reach a higher error, which is as you need to reach a higher loss value, these methods are there faster.


What that means is they're really fast at reaching a somewhat decent point, which is represented here.


They're really fast, but if they need them to reach a more and more accurate performance, they themselves get slower and slower.


So this is, of course, clear because what you're doing is you're no longer treating every data point the same.


You are introducing a bias into your training by only training on the hard examples.


So you're introducing a bias, and this bias will give you a speed up, but also hurt your performance.


And thereby, if you have to get more and more accurate, you will you will lose much of that speed up because you need to reduce that bias at the end that you introduced.


So that's the first caveat. As you want to get to a higher and higher performance, these methods will help less and less because they basically introduce the bias to gain speed at the beginning of training or to reach less accurate points.


The second thing is, as you look at these problems here, so SVHN 1.7% error, CIFAR 10 is a slightly harder problem, 2.9% error, and CIFAR 100 is really a harder problem where a traditional model has 18% error.


If you look at the speed ups now, then you can see even at this rightmost end here, you have the 3.5 and 5x speed up.


Here we have a 1.5, 2x speed up. Here we have a 1.2, 1.6x speed up.


So as the problems get harder, and as the kind of models get fancier, as the classes get more, then the speed up is much lower.


And I believe that's because the bias you introduce by reweighing the samples, the bias you introduce will hurt you much more on a difficult and large problem with a large network, then it will hurt you on an easy problem, right?


Easy problem, you are fine introducing some bias, but if you have a hard, noisy problem, then this bias you introduce will hurt you much more, and thereby the speed up that these methods give you is much, much less.


And this means that the performance of these models is directly anti-correlated with the hardness of the problem, and that tells me it kind of makes it almost unusable, or it goes towards, if I look at the numbers over here, and extrapolate that to something like ImageNet,


it tells me that these methods are going to be almost useless on a dataset of the size and complexity as ImageNet, and the interesting problems nowadays are very much in the domain of more hard, more complex problems.


So the kind of usefulness of this method in practice is something that I wouldn't bet on just from reading this paper. I'm open to be convinced otherwise, but just from reading this paper, it seems like the harder you make the problem, the less these methods help, and that's exactly not what you want.


You want exactly the opposite. You want to say, oh, if I scale this up, it'll give me even more of a speed up, and that's going to be even better, but this is the opposite.


And given that they have basically no theoretical analysis of how much this bias hurts you, or how you can still make it kind of good in expectation, how you would need to correct at the end, and so on, I would first, of course, test it.


I'm very interested to see tests on larger, more complex problems, but from this, I'm a bit skeptical. I'm sorry.


Yeah, so they show that on these data sets, it clearly helps, it clearly speeds up the training, and that's, of course, that's already a good thing, and they do the required experiments, they do the ablation studies on these data sets, and so on.


So you can see here, for example, on these first graphics, on all the data sets, it clearly goes down as you introduce the more sophisticated algorithms, but again, you can see on the hard data set, it doesn't go down as much.


But they do discuss this, they're really fair to themselves, they do discuss this in their paper of how practical this is, and so on, and what else they tried and didn't work, and I think that it's a really good paper in itself, and it's a really good investigation.


Active Dendrites avoid catastrophic forgetting - Interview with the Authors

Hello, this is an interview with the authors of the paper on active dendrites. Now, if


you haven't seen it, I've made a comprehensive paper review video on this paper. And I released


that yesterday, if you watch this video as it comes out, which obviously you do. Today,


I'm going to interview the authors and we've all seen my review, so we'll be able to directly


dive in. So if you haven't seen the review yet, and you want to know what's in the paper,


maybe that is a good place to start. The authors here were really helpful and really informative,


answering all of my questions and concerns that I had and even bringing up some new interesting


insights. So I hope you learn something from this interview, or at least that it entertains


you. And if you have any comments, please let me know in the comments below the video.


I'll see you around. Bye bye. Hey there, today's sponsor is the course on introduction to graph


neural networks. This is a course by my friend Zach Joest, who is an expert in graph neural


networks, and also runs the welcome AI overlords YouTube channel has a very interesting blog


and does many other cool things. He's packed all his knowledge of graph neural networks


into one course that will educate you on both the theoretical and hands on practical aspect


on graph neural networks. Graph neural networks are really important. They're definitely one


of the most interesting areas in deep learning right now they're on the upswing, they model


data that has an underlying structure that is connected, that is not really well fit


for any of the classic formats like tables or images. They've also powered a lot of recent


advances in scientific breakthroughs, such as alpha fold protein structure predictions,


or better traffic predictions. So if you're interested in graph neural network, I'll definitely


recommend you check out that course. If you use my link, you'll get a 15% discount on


the course enrollment is open right now and lasts until April 1 or until spaces run out.


The course is a six weeks course. It's cohort based, you'll get access to a community to


discord community of other students, and you'll get all the materials and hands on experience.


Alright, let's get into the video now. See ya.


Hi everyone, today I'm here with the three joint first authors of the paper on active


dendrites, Abhi, Karan and Akash. And I'm very, very happy to have you all here. This


paper covers many areas, it covers biology, it covers neural networks, it covers kind


of different architectures of stuff. It's very cool that you all sort of are here and


are able to sort of answer my questions. Welcome, all of you.


Yeah, thanks, Janek. Thanks for having us.


It's very interesting paper. So I saw this paper and I was intrigued because it's not


often that a lot of people say they do biologically inspired things. But it's not often that someone


really goes and says, look, you know, here's what's missing, let's build it in. And then


it actually leads to something that that works. And that is, you know, the hypothesis in your


paper, the hypotheses you pose on what should happen are actually confirmed at the end.


And this is, I think, a very good story arc for a paper and a really nice thing to write


up. So is this, how did this come to be? How did you get the idea of bringing these very


two distant, not too distant, but these two distant fields together of sort of neurobiology


and deep learning?


Well, at NuMenta, one of the things we're interested in is in continual learning and


learning multiple tasks, more generally speaking. And so, you know, we're looking at a lot of


neural networks and deep learning today focuses on trying to solve a single task. So we said,


well, how is biology enabling the ability to solve multiple things in sequence or at


the same time learning different things? And so, you know, there's not a lot of work out


there on active dendrites. And it's not exactly clear what their role was. But a little while


back, we speculated that, hey, they might actually be helping at the neural level to


allow for continual learning. And so if we can build this idea into deep learning, then


there might be some prospect there for addressing problems like continual learning and multitask


learning.


So is it fair to say that it grew out of sort of a need to solve a task?


I think it grew out of the need to solve multiple tasks in sequence, either learning them together


or in sequence continuously. To add on to what Karin was saying is that we believe that


active dendrites can really aid in achieving these specialized neural circuits. And we


can apply these ideas directly to any neural network and show some competitive performance


on various benchmarks that involve continual learning setups. So I guess the purpose of


this project, if you were to just summarize it very briefly, is we just want to show a


proof of concept for a new idea that can allow deep learning to work in more dynamic environments


and scenarios.


To kind of add on to what Karin and Abhi said, so at a higher level, I think we were kind


of examining where a lot of modern deep networks fail. And that's in these streaming task settings


and multitask settings. And the kind of like, inspiration for our solution was directed


towards biology and biological neurons, which is a lot of what Numentos focuses on. And


I think quite nicely, we found these like, there are existing benchmarks and existing


tasks that show that typical deep learning networks fail in these scenarios. And we were


able to build in these like, biologically inspired neurons to improve the performance


in such dynamic settings, by using the fact that we believe active dendrites in biology


kind of do this kind of context dependent adaptation in multiple tasks.


What I found interesting is that even though you targeted a little bit towards multi-layer


perceptrons, in principle, the active dendrites architecture is sort of pluggable almost anywhere.


So you could always imagine some sort of a context dependent signal that gets routed


in and modulates the signal that exists. So I think what I'm trying to find out is there


are a number of things happening in this model. There is first of all, the modulation itself,


which is a relatively, it's not really a known concept, at least in classical deep learning,


we always have weighted sums, we rarely have the situation where two parts of the signal


are multiplied together, or one modulates the other, it happens a little bit in LSTMs


and so on. The other one is this sort of recognition of a context and, you know, being context


dependent. And then a third thing is this, this sparsity. Now, you have sort of combined


all of them. Is there one thing that you think is specifically important? Or is it sort of


the combination of things that is really what makes the difference? You have some ablations


in the paper. What can you say about this?


I think it's the combination of all these things acting together. So it's the dendrites,


which are up modulating and down modulating certain neurons to determine which ones should


become, to determine which sub network should be invoked. And then it's a sparsity on top


of that, which is ensuring that a large portion of the network is essentially not performing


or learning a certain task. And it's those two things together, which really gets at


this idea of using specialized sub networks for different things. So I wouldn't say it's


any one thing that stands out more than the others.


So when we get, let's get into the paper itself. You've seen my review of it. With respect


to just framing the problem and maybe framing the architecture as such, is there, do you


think I have captured what you've tried to say? Do you think I've left something important


out or have put emphasis on or have not put emphasis on something that you would like


to put emphasis on when it comes to like what the architecture is, what it does and how


it works?


I think your explanations for the architecture, at least were very good. I think it does definitely


does capture what we were trying to say. And the whole point to kind of reiterate is that


the same model with the same principles should work on completely separate areas. One is


the multitask reinforcement learning. The other one is continual learning with permuted


MNIST. And I think you touched upon that idea too.


I think that the kind of motivation that, I think towards the beginning of your review,


you kind of compared the typical weighted linear sum neuron with the active dendrites


one. And I think our motivation in coming up with this architecture was how can we incorporate


a lot of these properties in the active dendrites with having dendritic segments being able


to either up modulate or down modulate certain neurons in a way that didn't completely go,


completely change from normal back propagation trainable networks. So this architecture kind


of brings in that flavor of having dendrites influence certain neurons, but does so in


a way that mathematically allows for back propagation to train the networks. And I think


you touched on that pretty well as well.


Do you think it's valid to sort of bring in biological concepts, even though we train


with back propagation? Because, you know, it's very evident that at least pure, like,


correct back propagation isn't happening in the brain. Do you think it's still valid to


bring in the concepts and maybe the brain's doing something like back prop? Or do you


think we're sort of just kind of taking inspiration from biology in order to solve some of our


problems?


I think it's more so the latter. Of course, the most accurate biological neural network


would likely not use back propagation. But this is one area where I think the goal was,


can we make deep learning just a little bit more plausible? And in doing so, can we make


it a little bit more dynamic? So we're not necessarily here to remove back prop entirely


and say that that's the best way that the dendrites in this architecture can work. Although


certainly that is how it works in biology. The point was, can we just augment traditional


deep neural nets to work in more dynamic scenarios?


Now I had some criticisms with respect to just like the details of your architecture.


For example, you always or you often choose the number of dendritic segments to match


the number of tasks that you have, which obviously, if I was a researcher, I would do


the same. But can you say maybe something about how this is in the brain? Like what


numbers are we talking about? How many of these sub networks that are composed of distal


dendrites, how many are there approximately? Do you know? Do you have an idea? And what


can you say about how many we should build into a problem where we maybe don't know how


many tasks we expect?


There are, from what I recall, probably in the order of hundreds or thousands of individual


dendrite segments for each individual neuron. Actually, it might even be more than that.


The actual numbers escape me. But regarding what you said earlier about having the number


of tasks be equal to the number of segments here, we found that actually, even though


in a lot of the experiments we report here, we do set the number of dendrites to the number


of tasks, we found that we actually don't need to have that many. And we actually have


further studies which show that we can actually keep the architecture fixed and increase the


number of tasks we're doing. I'm talking about continual learning here because for multitask,


we're focused on 10 specifically. We can increase the number of tasks and the performance actually


doesn't change by much. So that shows that as we're increasing the number of dendrite


segments, we actually end up over-parameterizing the network quite a bit, which we don't need


to do.


Yeah. So this is the plot on the left right here. You just increase the number of dendritic


segments and the top line is learning 10 tasks and it doesn't get noticeably worse, which


I find to be very cool property. I don't want to have to set the parameter very specifically.


I can just set it too high and it doesn't hurt, which is cool, which leads me to the


plot on the right where you discuss the sparsity. I'm going to guess that's the sparsity parameter.


So that's the thing that ultimately controls k, right? And I find it peculiar, not that


there is an optimal setting, which I would expect because that I can't set high, that


I have to set between like zero and one, right? So there's going to be like some optimum in


between, but there's this like two bump thing going on. So what's going on there? Why is


it like really good at high sparsity and then there's like this plateau and then it just


flat like crashes down?


I think in the beginning, if you have too much, so yeah, I always think in terms of


sparsity, some converting from density to sparsity. So if it's too sparse, right, there's


not enough signal going through. And that's why as you increase the amount of signal that


you're allowing through, as you're increasing the capacity of your representation, then


you're going to get an increase in performance. But then if you're using up too many units


to create that representation, then you're going to get more interference. And as you


have more interference, you're going to forget more and more network parameters are overwritten


as you move on to subsequent tasks. And so you get a drop in accuracy. And towards the


end, you notice that it does fall drastically. Honestly, I haven't thought too much about


why that happens, although it is a pretty monotonic fall, even though I guess in that


upper curve, there's a slight bump with that. And that could just be due to seeding or something


like that.


Yeah, I was more referring to like the plateau itself, right? There's this plateau, kind


of, and I noted that there could be almost like two modes of using the sparsity. In one


mode, I have entire subnetworks that do the job. And in the other mode, I have like a


shared network. Yet I have like separate things that just kind of like track which task I'm


on, which would sort of correspond to what the baseline is doing, right? When people


say, well, the baseline has access to the task too, it can just allocate some units.


It's maybe not a perfect analogy, but I was just wondering, it was just interesting to


see that there's this kind of this type of plateau.


Yeah, that's something I guess we haven't gone too deep into, but this might just be


a property of sparse representations and how much overlap there is as you increase the


sparsity level, it could just be something to do with that.


So in your paper, you make really, which I appreciate you make really sure that you sort


of always have the same amount of, let's say, trainable parameters in your architectures,


and you show that by arranging them correctly, you can achieve a better result. You always


use this name of non-zero parameters, right? Is there a difference? Are there large swaths


of zero parameters in one of these architectures?


Yeah, so this is something that we control for in the beginning. This is why we mentioned


the idea of weight sparsity. So in the beginning, when we're actually creating the architecture,


from scratch, we decide that some layers have an X percent sparsity level applied to it.


And what that really means is that X percent of the parameters are zero throughout the


entire part of training, and even towards the end. So that's why we express everything


in non-zero parameters. So the MLPs, for instance, at least in reinforcement learning, are trained


with no weight sparsity. So it's completely dense. There are no zeros anywhere in the


layers.


And Andy, your architecture, you sort of modulate the amount of sparsity, and that is on top


of modulating the K parameter of the K winner takes all layers.


Yeah, there's two aspects to the sparsity. So one is activation sparsity, which is like


at a hidden, like when you have a hidden state vector, how many neurons remain non-zero after


the activation is applied, which is a K winner activation. And then the second aspect of


sparsity is weight sparsity, which is how connected are subsequent layers in the network.


So if a lot of the units in the weight matrix are zero, then this model is the fact that


subsequent layers in the network are not very connected, they're sparsely connected.


I guess to answer your question again on that is, it's not something with weight sparsity


at least, it's not something we modulate, it's fixed. It's a fixed percentage that we


find. And this can either be done through fine tuning or just experimentation.


Okay, because I think, yeah, I might have just over read that, but I recall that in


the introduction, you say, you know, both the weights and the activations are sparse,


but then sort of the, I think the winner takes all really focuses on the activations itself.


Have you experimented with setting, you know, something else than K to a number or a percentage,


setting maybe a threshold for sparsity or something like this, where whenever a signal


is strong enough, it is let through?


We haven't, we haven't done anything like that. But we could do that. And you know,


there's a chance that it could work out pretty well if we have a fixed threshold. But one


potential downside there is that, you know, if you have too many signals that cross the


threshold, too many units whose activation crosses the threshold, you're going to get