Deep Learning for Symbolic Mathematics

or it simply times out in the rest.


Deep Learning for Symbolic Mathematics

I would say it times out because probably Mathematica


Deep Learning for Symbolic Mathematics

never gets it wrong because it's an actual


Deep Learning for Symbolic Mathematics

symbolic manipulator with defined rules.


Deep Learning for Symbolic Mathematics

So I guess the rest 16%, it simply times out,


Deep Learning for Symbolic Mathematics

doesn't find a solution.


Deep Learning for Symbolic Mathematics

Whereas this Facebook system,


Deep Learning for Symbolic Mathematics

and they say usually finds the solution less than a second,


Deep Learning for Symbolic Mathematics

finds these solutions in 98.4% of the time


Deep Learning for Symbolic Mathematics

with a beam size of one.


Deep Learning for Symbolic Mathematics

Now, what does the beam size mean?


Deep Learning for Symbolic Mathematics

At decoding time, if you have a seek to seek model,


Deep Learning for Symbolic Mathematics

you can always opt to do what's called beam search.


Deep Learning for Symbolic Mathematics

So when you have a input sequence,


Deep Learning for Symbolic Mathematics

let's actually give an example, a cat jumps.


Deep Learning for Symbolic Mathematics

And you generate an output sequence.


Deep Learning for Symbolic Mathematics

Let's say the task is simply to continue the sentence,


Deep Learning for Symbolic Mathematics

to continue the sentence.


Deep Learning for Symbolic Mathematics

So you can generate an output sequence.


Deep Learning for Symbolic Mathematics

The output sequence could be over the dog.


Deep Learning for Symbolic Mathematics

What you can do is you can, this is beam size,


Deep Learning for Symbolic Mathematics

would be called beam size one or no beam search at all.


Deep Learning for Symbolic Mathematics

You can do what's called a beam search in that each step,


Deep Learning for Symbolic Mathematics

you actually generate multiple hypotheses


Deep Learning for Symbolic Mathematics

and then keep the best ones in memory.


Deep Learning for Symbolic Mathematics

So you, in a beam size of 10,


Deep Learning for Symbolic Mathematics

you would always consider the 10 most probable solutions.


Deep Learning for Symbolic Mathematics

And you would kind of evaluate all 10


Deep Learning for Symbolic Mathematics

and then always keep the 10 best.


Deep Learning for Symbolic Mathematics

Let's see how this goes.


Deep Learning for Symbolic Mathematics

Let's do a beam size of three in our case.


Deep Learning for Symbolic Mathematics

So a cat jumps, and then you could come up


Deep Learning for Symbolic Mathematics

with three different things.


Deep Learning for Symbolic Mathematics

This sentence could continue.


Deep Learning for Symbolic Mathematics

Cat jumps over, a cat jumps between,


Deep Learning for Symbolic Mathematics

and a cat jumps swiftly.


Deep Learning for Symbolic Mathematics

Right, so these are your three hypotheses.


Deep Learning for Symbolic Mathematics

Then we go to the next step.


Deep Learning for Symbolic Mathematics

We have to evaluate each of those, each of them.


Deep Learning for Symbolic Mathematics

So a cat jumps over the, over a, over me.


Deep Learning for Symbolic Mathematics

A cat jumps between the, between two,


Deep Learning for Symbolic Mathematics

and a cat jumps between many.


Deep Learning for Symbolic Mathematics

The cat jumps swiftly, end of sentence.


Deep Learning for Symbolic Mathematics

Cat jumps swiftly over.


Deep Learning for Symbolic Mathematics

Cat jumps swiftly, end.


Deep Learning for Symbolic Mathematics

Right, these are all valid.


Deep Learning for Symbolic Mathematics

So of these nine, you would now select again the three


Deep Learning for Symbolic Mathematics

that overall have the highest likelihood.


Deep Learning for Symbolic Mathematics

Maybe that's the following.


Deep Learning for Symbolic Mathematics

Cat jumps over the, a cat jumps over a,


Deep Learning for Symbolic Mathematics

and a cat jumps between two.


Deep Learning for Symbolic Mathematics

These three, right?


Deep Learning for Symbolic Mathematics

So you just keep these three, and then in the next step,


Deep Learning for Symbolic Mathematics

you again from these three,


Deep Learning for Symbolic Mathematics

you would want for each three hypotheses and so on.


Deep Learning for Symbolic Mathematics

So this is what's called a beam search.


Deep Learning for Symbolic Mathematics

And if you give it a beam size of 10 or 50,


Deep Learning for Symbolic Mathematics

this system tends to improve even more.


Deep Learning for Symbolic Mathematics

The way this system works is quite different


Deep Learning for Symbolic Mathematics

from Mathematica in that Mathematica, as I said,


Deep Learning for Symbolic Mathematics

is a symbolic solver that never makes mistakes,


Deep Learning for Symbolic Mathematics

but can fail to give you a solution.


Deep Learning for Symbolic Mathematics

This system simply generates an output sequence


Deep Learning for Symbolic Mathematics

that is not guaranteed to be actually a solution


Deep Learning for Symbolic Mathematics

to the problem.


Deep Learning for Symbolic Mathematics

It's just a hypothesis, but then you can quickly check


Deep Learning for Symbolic Mathematics

whether the hypothesis is correct.


Deep Learning for Symbolic Mathematics

So the nature of these math problems.


Deep Learning for Symbolic Mathematics

With integration, you can simply differentiate,


Deep Learning for Symbolic Mathematics

and with ODE, you can simply plug them in


Deep Learning for Symbolic Mathematics

to see if they're a solution.


Deep Learning for Symbolic Mathematics

It's kind of like your classic, let's say,


Deep Learning for Symbolic Mathematics

NP-hard problems or like SAT solving,


Deep Learning for Symbolic Mathematics

where you can quickly check whether something is a solution.


Deep Learning for Symbolic Mathematics

So if you have a system that generates 50 hypotheses,


Deep Learning for Symbolic Mathematics

you could quickly check which one is actually correct.


Deep Learning for Symbolic Mathematics

So these numbers here mean that one of these 50


Deep Learning for Symbolic Mathematics

that the system came up with was a correct solution.


Deep Learning for Symbolic Mathematics

And if you allow for such many hypotheses,


Deep Learning for Symbolic Mathematics

you can see it goes up quite a bit.


Deep Learning for Symbolic Mathematics

For example, the ODE solving is almost the same,


Deep Learning for Symbolic Mathematics

and here it's even worse if you take ODEs of order two.


Deep Learning for Symbolic Mathematics

It's even worse than Mathematica,


Deep Learning for Symbolic Mathematics

but if you allow for larger beam sizes,


Deep Learning for Symbolic Mathematics

you see it dramatically goes up.


Deep Learning for Symbolic Mathematics

And so it's a different approach.


Deep Learning for Symbolic Mathematics

I wouldn't be surprised if Mathematica


Deep Learning for Symbolic Mathematics

would actually implement something like this very soon


Deep Learning for Symbolic Mathematics

or just buy this off of Facebook or something,


Deep Learning for Symbolic Mathematics

or buy Facebook, buy Mathematica.


Deep Learning for Symbolic Mathematics

In whatever way, this clearly is a different approach,


Deep Learning for Symbolic Mathematics

and it appears to work better, but there's a caveat.


Deep Learning for Symbolic Mathematics

So here's the caveat that I see with this kind of thing.


Deep Learning for Symbolic Mathematics

These evaluations are done on data sets, of course,


Deep Learning for Symbolic Mathematics

and this paper goes into big detail


Deep Learning for Symbolic Mathematics

on how to generate these data sets, right?


Deep Learning for Symbolic Mathematics

So they have to pay attention to many things,


Deep Learning for Symbolic Mathematics

like many solutions are equivalent, for example, here,


Deep Learning for Symbolic Mathematics

that this solution and this solution


Deep Learning for Symbolic Mathematics

to this differential equation are the same.


Deep Learning for Symbolic Mathematics

So they have to use a symbolic framework


Deep Learning for Symbolic Mathematics

to check whether the solutions are the same and so on.


Deep Learning for Symbolic Mathematics

It is very good work, but they do evaluate on expressions


Deep Learning for Symbolic Mathematics

that fit into their data set.


Deep Learning for Symbolic Mathematics

So here in their data set, they say, okay,


Deep Learning for Symbolic Mathematics

we evaluate expressions with up to 15 internal nodes,


Deep Learning for Symbolic Mathematics

11 leave values for these four binary operators,


Deep Learning for Symbolic Mathematics

then these 15 unary operators.


Deep Learning for Symbolic Mathematics

So the expressions that they train on


Deep Learning for Symbolic Mathematics

fall into this data set, right?


Deep Learning for Symbolic Mathematics

Also just numbers from negative five to five.


Deep Learning for Symbolic Mathematics

So it is kind of to be expected that a system


Deep Learning for Symbolic Mathematics

that is trained on these things


Deep Learning for Symbolic Mathematics

would perform very well on these things


Deep Learning for Symbolic Mathematics

as opposed to Mathematica


Deep Learning for Symbolic Mathematics

that is a general purpose tool.


Deep Learning for Symbolic Mathematics

Moreover, if you look at, sorry,


Deep Learning for Symbolic Mathematics

I think this is further down.


Deep Learning for Symbolic Mathematics

For example, in integration, for the integration task,


Deep Learning for Symbolic Mathematics

they have three different ways of solving,


Deep Learning for Symbolic Mathematics

of generating data.


Deep Learning for Symbolic Mathematics

They have the forward way


Deep Learning for Symbolic Mathematics

where they simply use a symbolic integrator


Deep Learning for Symbolic Mathematics

to generate expressions.


Deep Learning for Symbolic Mathematics

They have the backward way


Deep Learning for Symbolic Mathematics

where they start from the integral


Deep Learning for Symbolic Mathematics

and then differentiate it in order to obtain


Deep Learning for Symbolic Mathematics

a training pair.


Deep Learning for Symbolic Mathematics

And they have an integration by parts method.


Deep Learning for Symbolic Mathematics

So these are three different methods


Deep Learning for Symbolic Mathematics

to come up with problems for this system to be trained on.


Deep Learning for Symbolic Mathematics

And they have very different properties


Deep Learning for Symbolic Mathematics

to the effect that if you train with one, just one,


Deep Learning for Symbolic Mathematics

it won't work well on the other.


Deep Learning for Symbolic Mathematics

So if you train with the forward method,


Deep Learning for Symbolic Mathematics

it will work very well


Deep Learning for Symbolic Mathematics

on data that has been generated with the forward method.


Deep Learning for Symbolic Mathematics

So this is down here, this is what it's trained on,


Deep Learning for Symbolic Mathematics

and this is what it's evaluated on, right?


Deep Learning for Symbolic Mathematics

You can see the diagonal is very, very strong.


Deep Learning for Symbolic Mathematics

But if you train with the backward method,


Deep Learning for Symbolic Mathematics

but you evaluate on data generated with the forward method,


Deep Learning for Symbolic Mathematics

it is actually very poor.


Deep Learning for Symbolic Mathematics

That's because in one case,


Deep Learning for Symbolic Mathematics

generally the solutions are longer than the input.


Deep Learning for Symbolic Mathematics

In the other case, the solutions are shorter.


Deep Learning for Symbolic Mathematics

So not only does this system


Deep Learning for Symbolic Mathematics

only work on the particular task here,


Deep Learning for Symbolic Mathematics

it is actually very attuned


Deep Learning for Symbolic Mathematics

to the way that this data was generated, right?


Deep Learning for Symbolic Mathematics

So in fact, I would postulate that this training data


Deep Learning for Symbolic Mathematics

is only probably a very small subset


Deep Learning for Symbolic Mathematics

of all of the things that we would like to integrate.


Deep Learning for Symbolic Mathematics

And again, the problem is made kind of worse


Deep Learning for Symbolic Mathematics

because their evaluation set


Deep Learning for Symbolic Mathematics

would also come from their distribution.


Deep Learning for Symbolic Mathematics

So what they've ultimately shown is that they can do this


Deep Learning for Symbolic Mathematics

on a very skewed, probably very biased subset


Deep Learning for Symbolic Mathematics

of this mathematical problem.


Deep Learning for Symbolic Mathematics

And on that biased subset,


Deep Learning for Symbolic Mathematics

they can outperform something like Mathematica, right?


Deep Learning for Symbolic Mathematics

They kind of defeat themselves.


Deep Learning for Symbolic Mathematics

Yeah, if you look here, even the different integration,


Deep Learning for Symbolic Mathematics

data generating methods,


Deep Learning for Symbolic Mathematics

if you only train on one of them, it doesn't generalize.


Deep Learning for Symbolic Mathematics

If you only train on forward data,


Deep Learning for Symbolic Mathematics

then if you evaluate on backward generated data,


Deep Learning for Symbolic Mathematics

it doesn't work.


Deep Learning for Symbolic Mathematics

So even the integrator can't really generalize.


Deep Learning for Symbolic Mathematics

So they have to kind of combine different method.


Deep Learning for Symbolic Mathematics

And even now we can probably easily find examples


Deep Learning for Symbolic Mathematics

that this integrator can't solve.


Deep Learning for Symbolic Mathematics

So, I mean, there is a lot of cool things here


Deep Learning for Symbolic Mathematics

and they show a number of properties that the model learns


Deep Learning for Symbolic Mathematics

just from, without them telling it to.


Deep Learning for Symbolic Mathematics

And it's cool that it works anyway.


Deep Learning for Symbolic Mathematics

As I said, this model has no programmed in notion


Deep Learning for Symbolic Mathematics

of how math works,


Deep Learning for Symbolic Mathematics

but also it kind of shows the problems


Deep Learning for Symbolic Mathematics

if you do this via a training data set


Deep Learning for Symbolic Mathematics

in that if your training data set is very skewed


Deep Learning for Symbolic Mathematics

and then your evaluation set


Deep Learning for Symbolic Mathematics

follows the same generation process,


Deep Learning for Symbolic Mathematics

the claims you can make at the end are limited.


Deep Learning for Symbolic Mathematics

And to be fair, I don't know what claims they made


Deep Learning for Symbolic Mathematics

in the press generally.


Deep Learning for Symbolic Mathematics

So I think it's a pretty cool work, check it out.


Deep Learning for Symbolic Mathematics


Save Classifier with Pickle  - Natural Language Processing With Python and NLTK p.14

What is going on everybody? Welcome to the 14th Python with NLTK for Natural Language


Processing tutorial video. In this video we're going to be talking about pickle or how to


save basically your trained algorithm. So every time you want to use your trained algorithm,


you don't have to go through and retrain it. So the training process for this algorithm,


actually this specific algorithm, the training process is not that slow. The longest part


of the process is probably just the loading in of the documents. But that said, there's


going to be times where maybe you're writing, you've got one algorithm or down the road,


we're actually going to have many algorithms and we're going to use a voting process on


those algorithms to decide what the decision is amongst all of the algos. And it adds up


pretty quickly to train each algorithm, especially if you're doing the same training set every


single time, why not save it? So pickle is a way that we can actually save Python objects


so that we can load in those objects and it's actually extremely powerful. So anyway, let's


just go ahead and hop in and show a quick example of using pickle. So first of all to


use it, we need to import it and it's just pickle like the food. So import pickle and


then what we'll do is near the bottom, just somewhere after we do this part, we can save


the classifier by doing something like, first of all we'll save save underscore classifier


is going to equal open and then we'll save it as some sort of file name, but we'll call


it naive Bayes dot pickle. And we want to open this with the intention to WB which is


right in bytes. If you're on Python two, this won't matter, but it for Python three and


beyond, which you should be on anyways, this, you have to write it as bytes. So anyways,


there's that. And then we'll do pickle dot dump. What do we want to dump? We want to


dump this classifier, right? The thing that we trained to this word classifier. What do


we want to dump? Where do we want to dump it to save underscore classifier? Okay. Then


of course as usual classifier dot cli or save classifier dot close. We'll close that file.


Good. So let's go ahead and save and run that. And while we're waiting, let's go over here.


What did we save that as? Naive Bayes pickle. Oh, the last one I did. Yes. Here it is. We'll


wait on this one to come up. Okay. So we had a 69% accuracy and we've saved it now to this


naive Bayes dot pickle file. Alrighty. So we've got that saved. Now what we could do


is instead of saving the classifier, let's just comment that part out. And then here


where we have, we're loading in this classifier, uh, instead of having it be the training,


what we can do is something like this. So we can say, whoops, where am I? Uh, there


we are. Classifier underscore F profile basically equals open. And it's the same file. Uh, naive


Bayes dot pickle. We want to open it with the intention to read in bytes again because


of Python threes, differentiation between bytes and strings. Uh, then we're going to


say classifier equals pickle dot load. What do we want to load classifier? Uh, F and then


of course, classifier dot close. And actually it's classifier F dot close. Cool. All right.


Save and run that. Um, actually, uh, sorry, let's comment this out even though it'll redefine


it anyways. We'll just comment it out to show that our, our magical trick is actually working


here with pickle. Um, so we might have a separate percentage accuracy. Ooh, this one was actually


really bad accuracy. I assure you. Um, this might actually still be back. It's the one


that we've saved. Let's see. Are we saving every time? No. Sure. Hopefully we won't get


a 56. There we go. 67. Woo. Like I said, this kind of unreliability in accuracy numbers,


you know, the 50 as you saw earlier, we got an 89% and now on the same algorithm, we got


a 56% like this is horrible. Um, I'll show you later on how we can use this voting process


to get at least a stable, uh, accuracy percent. And then as we go on, we'll kind of dig into


this accuracy and kind of see if we can figure out what's causing this much volatility in


our accuracy. Uh, but anyways, that was just a quick tutorial on saving the classifier


cause as you, you might want to tweak things and maybe you can save, like you, you might


tweak things a little bit. Maybe you want to add more words or something like that and


you want to save that classifier. This is how you can do it. You can save it. And actually


you could pickle this entire document. You could have pickled everything here. Um, you


could, you know, make a class out of this basically and pickle that. And, um, and then


you wouldn't have to run this every time cause as you add on top of this and do more analysis,


it's only going to take more and more time every time you run it. So it makes a lot of


sense to pickle it. So anyways, that's pickling. Just a real quick example with this. Although


pickle is extremely useful. This is not an NLTK thing. This is actually part of your


standard library. So, um, definitely check out pickle. Think about using it. Um, I know


that I've definitely underused pickle in my time. Uh, I didn't really think about looking


into it. It's a lot more useful than it is. So, or than I thought it was a lot more useful


than it is. Anyway, uh, that's it for this tutorial. If you have any questions or comments,


please feel free to leave them below. Otherwise, as always, thanks for watching. Thanks for


Decision Transformer: Reinforcement Learning via Sequence Modeling (Research Paper Explained)

Hello there, today we're going to look at decision transformer reinforcement learning


via sequence modeling by Lily Chen, Kevin Lu, and others of UC Berkeley, Facebook AI research


and Google Brain. On a high level this paper ditches pretty much anything and everything


of reinforcement learning in an offline RL setting, and substitutes it for simple sequence


modeling using transformers, of course. And through that, they're able to achieve some


pretty compelling results in the things they test, at least they're able to keep up and


be on par with the current best frameworks for doing offline reinforcement learning.


So we're going to look at this paper and at what it what it does in terms of sequence


modeling, and how this looks the key ingredient here, besides the transformer is going to


be the fact that we are, instead of maximizing the reward, we're going to condition on the


desired reward. And through that, we, we can sort of influence what the model is going


to do in the future, this allows more effective offline reinforcement learning, and makes


the offline RL problem pretty straightforward into a sequence modeling problem. I do have


a little bit of troubles with the paper in various aspects, but I'm sure we'll come to


that. But I'm just warning you, this might be a bit of a rant mixed with explaining the


paper, though the paper is is pretty cool. So don't get me wrong on that. That being


said, there is concurrent work also out of Berkeley, as I understand it, where it's this


is called the trajectory transformer, reinforcement learning is one big sequence modeling problem,


that uses the sequence modeling in a bit of a different way. So what they do is they use


it as sort of a world model. And then they use beam search in order to in order to find


good trajectories in that. So it's a little bit of a different approach. And I just from


skimming this paper right here, I think I, this one might be a bit more of a, of an approach


that I would subscribe to. But I guess we'll see what happens going forward. And oh, wait,


why did this show up? reinforcement learning upside down by Schmidt Huber? This must just


have gotten in here by accident. Sorry. Let's go back to this paper. They say we introduce


a framework that abstracts reinforcement learning as a sequence modeling problem. This allows


us to draw upon the simplicity and scalability of the transformer architecture and associated


advances in language modeling, such as the GPT line. And Bert, in particular, we present


the decision transformer, an architecture that casts the problem of RL as conditional


sequence modeling, unlike prior approaches that fit fit value functions or compute policy


gradients decision transformers simply outputs the optimal actions by leveraging a causally


masked transformer. Okay, so as I said, they ditch things like policy gradients or value


functions, none of that. We're simply going to do sequence modeling right here. By conditioning


on an autoregressive model on the desired return, past states and actions are decision


transformer model can get can generate future actions that achieve the desired return. So


the key concept here is going to be this desired return thing and here as well. So there are


multiple ingredients to this paper, there's a lot to to unpack right here. And lastly,


they say it achieves it matches or exceeds the performance of state of the art model


free offline RL baselines. Again, this is sort of zooming down into a problem. So we


are in the world of model free and offline reinforcement learning algorithms. There is,


as I said, there's a lot to unpack here. So first of all, what is offline reinforcement


learning, this is contrasted to online reinforcement learning. Online reinforcement learning is


where you have an agent and an environment. And the agent sort of gets to perform actions


in the environment and the environment responds with a reward and a state or the not really


a state but an observation. But we sometimes it is the state if it's not a partially observable


environment. So the agent actively gets to interact with the environment to try out things


and its goal is going to be to maximize that reward. In offline reinforcement learning,


it's a different situation. So in offline reinforcement learning, you are your agent


is here. And what you get is not an environment, but what you get is a data set. And this data


set will contain it will contain lots of experience from other agents. So you would simply get


to observe what a different agent has done. And so there's going to be a lot of like episodes


in here. So what happened in the past to this other agent, and purely by observing that


other agent, you somehow have to learn a good policy to achieve a good reward. This is different


because you cannot go out and sort of test your hypotheses in this world. You cannot


have a good idea and say, Well, I'm going to try that. You can't do sort of targeted


exploration and so on, you simply get to look at a bunch of trajectories, and then decide


what you want to do. So we need a bunch of different approaches here. And and one that


they compare to is so there are two that mainly that they compare to one is called they call


it BC, which is behavior cloning, where what you're trying to do is you simply try to mimic


the agent that you observe in the events where it has led to two good rewards, right? So


that's how you maximize the reward, you simply say, Well, that agent there, it got a good


reward. So I'm just going to try to sort of clone that behaviors as behavior cloning from


the name, I'm, I'm butchering the explanation, but roughly, that's what it's supposed to


do. The other approach is you view this as a, let's say, more traditional reinforcement


learning problem where you queue learning. So in queue learning, what you do as you are


in a state and you have maybe like three actions at your disposal. And every time you again


have three actions at your disposal. So you get this sort of tree that you could do. So


you're in the first state. And what you want is you want to ask your queue function, how


much how much is how much is this worth? Maybe the queue function is five, how much is this


worth six? And how much is this worth four? So the queue function is supposed to tell you if you


take this action, and after that action, you follow the policy. Like, after that action,


you again do ask the queue function for the queue value, which what's the total reward


you're going to get? queue learning is very, very classic reinforcement learning algorithm.


And you can actually do queue learning from a data set like this, it doesn't need to be


you yourself that makes the experience that's the thing about queue learning is that it


can be done from offline data other than policy gradients. You need sort of a you need a correction.


If you do policy gradients, and it usually doesn't work if it's complete offline, they


might work. I'm not super informed like this. But queue learning is possible from offline


data. And apparently, the current a currently good baseline is conservative queue learning,


which you're going to see in this paper, which fixes the the the bug, let's say that the


tendency for for these queue functions in the offline setting to overestimate the queue


value. So apparently, they they tend to overestimate the value that you get from certain actions.


conservative queue learning is a more like a pessimistic approach. So these are the two


baselines that we're going to compare to you'll notice behavior cloning some kind of relation


to inverse reinforcement learning, not really or Yeah, so so that's one approach queue learning


is also an approach here, we're just going to do sequence modeling. So what does this


mean? And the key concept, as I said, is going to be the condition on that reward. Sorry,


so this was offline RL. Now, there are people have pointed out problems with the approach


here, which some of those problems are simply problems of offline reinforcement learning.


So for example, which data set do you use right here, turns out in their experiments,


they use a benchmark data set, which is the data set where this agent right here is a


DQN learner, so an active reinforcement learner. So naturally, you're going to get out like


some some good episodes out of that. So it's more like learning from expert demonstration,


rather than from random random demonstrations. Okay, so it's crucially important which data


set you use, but that's, that's a fault of offline RL of the setting itself, rather than


of this particular algorithm. So I just want to want to point that out. But keep in mind


the data set they're using for their main experiments is one of, let's say, rather high


performing agent in this world. So that's that. So the second thing right here is their,


their use of the of a transformer. Now, is the use of a transformer crucial to this algorithm?


And the answer is, is no. So whenever the transformer comes to mind, this can be any


sequence modeling algorithm right here, transformers are trendy, okay. But this can be an LSD,


LSTM that does autoregressive sequence modeling, anything that does sort of autoregressive


sequence modeling is going to be good for this task right here. The the core here is


going to be this is a sequence model. It's not an RL model. In fact, transformers for


RL have been a thing, you know, usually what people do is they use LSTMs as a backbone


for reinforcement learning algorithms. Using transformers has several advantages in offline


and or online reinforcement learning algorithms. So usually you have some sort of a state right


here. So you have your history with states and actions and rewards, and so on. And an


LSTM will take in that state and, and action. Well, let's just, let's do it something like


this. So you have state action reward, state action reward, state action reward, whatever


you did in the past, right. So an LSTM will take that in, and it will propagate its hidden


state through times. I realized some of you youngsters might not actually know what an


LSTM is. This is a recurrent neural network that processes one time step at a time. And


then here, at the end, you're supposed to output whatever the next action is going to


be right, you have your history of actions, you're supposed to output whatever the next


action is going to be, and you're going to get back a state and a reward along with it.


And then you incorporate that right here into the next action. So if you train this thing


in any way, let's say Q learning policy gradient, whatnot, if it's a Q learning, you're not


going to output an action directly, you're going to output Q values, that's a minor modification


to the A. What you have to do is you have to, and that's the difficulty in reinforcement


learning general, you have to somehow make a connection between the rewards you get from


this, let's say this action gets your reward, the reward you get from the action to some


something that you predicted. So you predicted several, you predicted an action here and


an action here, right, these are these actions. Now, just because you got a reward from this


action, it doesn't actually mean that this action was the smart action or the good action,


right? If you are in a chess game, it's not the actual last move, that is the good move,


even though that move gets you the all the reward, the crucial move might have happened


20 moves before. So the the underlying reinforcement learning problem is to assign that reward


to which action was actually the smart action such that in the future, you can take that


more. So maybe this action right here was the smart action. So you need a way to figure


out that that was the smart action. And, you know, back propagation over time will do this.


But in an LSTM, you can see right here, you need to back propagate, you know, through


one, two, maybe three different computation steps in order to reach there. And now this


is three steps, but think if the good action was 50 steps ago, or 500 steps ago, this quickly


gets gets tricky. Normally, we can unroll LSTMs like this for maybe, I don't even know,


like, like, not more than a couple of dozen steps, right? So it gets tricky. So what people


do is they use what's called dynamic programming. And that is a thing that here with the sequence


modeling approach, we're going to ditch. And this, this is one of the fundamental things.


So instead of having to just learn from the reward and assign it to an action, what you're


going to do is you're also going to along with the actions right here, you're going


to output a value. And the value tells you sort of how good you are doing. The Q function


in a way is already a value. So if you're doing Q learning, you're doing this automatically.


And then the way you learn this is called temporal difference learning. So, you know,


let's say this is the this here is the final stage of the game. Okay, so you always get


a reward here, it's maybe plus one here, it's minus five, and so on. Okay, now, instead


of back propagating only that reward back, what you're going to do is at every step,


you want to predict a value, obviously, the last value is going to be equal to the reward


itself. But here, your value is sort of your expected reward in the future, if you take,


you know, the good actions that you're going to take. So here, your value might be maybe


negative 4.5. Because you know, you're actually no, you're probably going to take the action


that gives you a good reward, right? So it's maybe like plus point nine, because you're


fairly sure you're going to take that good action. And then down here, it's maybe so


you get five reward from going there. No, wait, that's the Q value, I said, that's the


Q value. So here, your value is going to be something like plus point seven. So it doesn't


really matter what the numbers are, what matters is that now you're not your learning signal


doesn't just come from the from the reward itself, your learning signal is your from


here, you're trying to predict the reward, but you're also trying to predict the output


of your own function, like one or two or three steps into the future. So if you've done an


episode, and at the end, you got a reward right here, you could your value function


right here could try to just output that reward. But that's really noisy. So what you're doing


is you're saying, Well, you know, I have predicted a value here and here and here and here and


here. So why aren't I training my value function to also predict these things? And by predict,


I basically mean, so if, if I was in this value, and this transition got me like a reward


of something, then this value here should equal to this minus this reward, because,


you know, like, that's, that's how the value is supposed to function. So you're trying


to predict the output of your own value function. This also works with the Q function. This


is the famous Bellman recurrence relation, where the Q function of a state is equal to


the root the reward you get from performing an action, according to the policy in that


state plus the Q function at the state that you're reaching. So again, with the same policy,


and the the R here is drawn from the action that the policy gives you something like this.


So the R is the result of performing the action. So this, this fundamental relation is the


basis of Q learning. And you can do as I said, right here, this is called temporal difference


learning. So what they call TD, all of this is based on concepts of dynamic programming.


We all ditch this here. And so it is important to go through so that you understand what


we're not doing. Okay, why do we need all of this? Why do we need the Q functions and


the temporal difference learning and so on? Well, because it's really hard to do that


credit assignment over long stretches of time. Now, in we can see that this is the case with


an LSTM, right, especially if we can't back propagate all the way through the LSTM in


a transformer. What does a transformer do? You have a sequence. What does a transformer


do? It uses attention in order to look at a sequence at a whole, right? It through the


attention mechanism, it can route information from any sequence element to any other sequence


element in a single step. So essentially, it technically could do this credit assignment


right here in a single step if, and that's a big if, if anything fits into its context,


okay. And that's, I think, one of the crucial criticisms of this paper right here, in that


as far as no, I don't think all it fits in all into the context. But you can see that


there's a trade off, right, you're able to do the assignment in one step, okay, but as


soon as you would like to predict correlations and do credit assignment across longer spans


than the context, you need to resort back to something like the dynamic programming


approaches right here, which they say they can ditch. Now, they don't only say that because


their context is long. But that is when they say how the transformer benefits this instead


of like an LSTM or something like this. This is the reason that you can do this credit


assignment in one step across the context. However, always think that statement has an


if, if the credit assignment needs to happen longer than one context, like if the relevant


action for the reward is more away, the transformers out of luck because doesn't fit into the context.


And we would need to go back to something like this. But there is a second reason, of


course, and that is the sequence modeling approach. And that is something I, I see at


the core of this a little bit. So the, the causal transformer, you know, cool, it's a


transformer, okay, we could use any other sequence modeling approach. Now, viewing RL


as a sequence modeling problem is a different thing. So what does this thing do? So instead


of having a neural network, that, you know, here is, here's the history, okay, this is


the history, this is the rewards you got in the past and disregard the little hat on the


R, it's the states of the past, it's the actions of the past actually extends into the past,


okay. So this is the input you get, and you would get that in any other reinforcement


learning algorithm, what you would get to is this thing right here, the current state,


right. And this goes through a little encoder, they use the DQN encoder. So this is a little


convolutional neural network, right, that encodes the state. So it's technically able


to handle very complex states and so on by simply encoding them into a latent space.


So there's no attention on the like on in the state space right here, that the attention


really happens over the over the sequence. Now, from this, right, the classic RL algorithms,


they wouldn't have this from this state would try to predict an action that maximizes the


future reward. What this does differently is they say, well, instead of giving me an


action that maximizes the future reward, I want to I want to tell the system what reward


I would like. And then it's not giving me an action to maximize the reward, it is actually


supposed to give me an action that achieves exactly the reward that I have presented.


Okay, so I asked it for a reward. And it gives me the action that corresponds to achieving


that reward in the future. This is is different, right? And I can still do reward maximization


by simply putting a high number there, right, I want to get a lot of reward. And like 21


is the maximum in Pong, which this game is right here. So you can say I want to achieve


21 reward, please give me an action that achieves 21 reward. And that will be corresponding


to getting as much reward as possible. Notice that you do need to know the maximum reward.


It doesn't actually work if you just put 1 billion billion billion, as we will like as


the their experiments kind of indicate. So that's a drawback of this. Now, just want


to go back to this paper that slipped in. Just by accident, I have this open right here,


by Schmidhuber. Don't predict rewards, it says just map them to actions. So they say,


we transform reinforcement learning into a form of supervised learning, okay, which sounds


like, you know, offline RL by turning RL on its head and the look at this. The memes are


strong in this one. Okay, upside down RL, I've actually made a video on upside down


RL, they say, or standard RL predicts rewards, while whatever this is, instead uses rewards


as task defining inputs, together with representations of time horizon and other computable functions


of historic and desired future data. RL learns to interpret these input observations as commands,


mapping them to actions through supervised learning on past, possibly accidental experience.


So this, it is actually, of course, this isn't by accident. So I knew this paper right here.


And when I read this paper, it immediately sprung into my mind. And Schmidhuber also,


as I see, it wasn't the entirely first who did anything like this, like we've known about


goal conditioned reinforcement learning for a while and so on. So this is not necessarily


a new idea. They do reference Schmidhuber's paper very briefly in in this paper, staying


stating that it's kind of a Markovian approach and, and so on, even though here you have


Markovian interfaces, and here you have non Markovian partially observable interfaces.


And the advantages that Schmidhuber names right here are very much the same. For example,


they continuously say they don't need discount factors. And here also, you have no problems


with discount factors, and so on. So I wanted to point this out. And I wanted to point out


that the paper is referenced in this paper. But essentially, here you have the three components,


the component is offline RL, plus a transformer, plus viewing the problem as a sequence modeling


problem by conditioning on the reward. So why does this make sense to condition on the


future desired reward? Well, it makes sense, first of all, because in classic reinforcement


learning, why don't we do that? Why don't we we say, I want to get this reward, please


give me the action to it. Because it's a lot more work, right? If I just want to maximize


my reward, I need a function, right? I need a neural network. Here is my state. Here is


my neural network, maybe it's a policy gradient method. Give me an action. And that action


is supposed to maximize the reward. So now I need an additional input the desired reward.


And also give me an action. Now the network doesn't only need to remember what do I need


to do to perform? Well, it needs to be able to distinguish what do I need to do to perform?


Well, what do I need to do to perform a little bit worse? What do I need to do to perform


terribly? It's a lot more stuff to remember for the network. The hope, of course, is that


with all the advances we've seen in sequence modeling, that essentially, these transformers


are capable of, of memorizing or learning all of those different things, we know that


transformers are almost unlimited in their capacity to absorb data and learn stuff. So


the hope is that these models will be capable of learning that thing. The net at doing this,


though, is this is a technique that naturally maps to offline reinforcement learning. So


offline reinforcement learning in general is a harder task than online reinforcement


learning, right for the reasons I outlined. However, this particular thing lends itself


extremely well to the task of offline reinforcement learning. So what do I mean, if you have a


history, you take one history from here, and it says, Well, I was in this state, I performed


this action, I got this reward, I was in this state, and then I came to this state, I performed


this action, I got this reward, and so on. Okay. What you can try to do, and what Q learning


tries to do is it tries to somehow learn the the Q function that takes state and action


condition on the history, and sort of predict the future rewards, and so on. So it tries


to figure out what it needed to do instead of doing what this agent did, in order to


achieve higher rewards. So it is sort of trying to look at the agent that it sees critically


and be like, you probably didn't do something well there, but it has no way to act in the


world, it has no way to go out and try it itself. Instead, this thing, it simply accepts,


it's like it accepts the history, it simply says, Oh, well, you did these things, and


you got this reward. Okay, cool. And if you know anything about these sequence models


and transformers that they can memorize stuff quite well. So going forward, maybe think


of these what these transformers do is simply memorizing the training data set. Okay, I


know it's not the case, but you memorize the training data set. Well, now, if you memorize


the training data set, and you're in this situation right here, you see a history, you


see a state, and the sort of the human tells you, I would like to get 21 reward, what the


transformer can do is it can simply say, Okay, let me go into my training data set, let me


find some, let me find some sequence where the agent was in the same kind of history,


also was in this state, and also ended up getting about 21 reward out of the future


actions. Now, what did that agent do? Well, it did this action. Okay. And it's reasonable


to assume that, you know, if you're in the same kind of history, and if you want the


same reward as that agent got, you should probably act the same as that agent did. Okay,


it is a lot like behavior cloning, though behavior cloning still focuses on sort of


getting higher reward, as I under understand it. So it simply takes what comes in as expert


demonstrations. Whereas here, you just, you accept the history as it is. And if you're


in a new situation, you the question to the sequence model is essentially, how would a


sequence that evolves like this, okay, that evolves like this, how would it continue in


the training data set? And what it will give you, it will give you the action that agents


who were in a similar situation and ended up getting that similar reward that you want


to get those, what did those agents do, just do the same thing, and you're probably going


to end up in the same place as they did. Okay, that's, that's the approach right here. You


can see how this is is useful, right? Though, again, it it only given that we ditch all


of the RL given that we ditch all of the RL mechanics right here, which they claim as


a positive and certainly it is a positive, you don't need to parse out what you needed


to do and so on, you simply accept history and say, Okay, I'm going to do the same kind


of things. Instead of that, if so, I just said, I'm going to look at agents that had


the same kind of history and were in the same kind of situation. Now, if you think about


back about this problem right here of the context length, what if the future reward


right here is crucially dependent on an action you did back here, right? You could have two


agents that have the exact same history as far as the context reaches back, but done


a different action back here. And the sequence model would have no trouble, sorry, would


have like no chance of differentiating between the two, it will they look the same, okay,


one agent ended up with a really nice reward, the other agent ended up with a really bad


reward, even worse, the data set couldn't contain an agent that ended up in the bad


reward. But had you done Q learning, you could maybe figure it out from other trajectories.


So as much as they I feel as much as they tout the ability to ditch the whole mechanic,


like the whole machinery of reinforcement learning right here, you run into the same


problem. Like, even with this, like all of this, it does not alleviate the problem. If


you want to go beyond how far you can back prop, you need to you need to use the dynamic


programming approaches. Okay, like, I don't see a way around it. Maybe I'm terribly wrong.


But yeah, so that the transformers are good for doing the credit assignment over the longer


distances than the LSTM. Yes, certainly, but that's valid for online offline RL and so


on. Whether you do sequence modeling or not, it doesn't alleviate the problem that these


approaches we're trying to solve in the first place, though, the sequence modeling approach


is different, and does bring like a different view on the problem. And again, you can do


the sequence modeling approach, because it there is hope that with these transformers,


you can actually absorb that much data and learn from that. So that is sort of the thing


we're in that that was actually already the the technique right here, we're not even past


the first page. And that is, that's already the thing you get this data. And they're like,


you can deterministically you can see that right, you can deterministically transform


this into the format they want. So this state action and desired future return or future


return, you simply look into the future, which you can do because it's a data set. And you


sort of calculate what the future reward is at this particular time step. So you can easily


generate that training data, then you can use classic sequence modeling in order to


do that. Their idea of what happens is encapsulated again, in this in this thing right here. So


this is a very, very example problem that they come up with. So they consider a task


up here of finding the shortest path in a on a directed graph, which can be posed as


an RL problem. Okay. The reward is zero when the agent is at the goal node and negative


one otherwise, we train GPT model to predict the next token in a sequence of returns to


go, which is the sum of future reward state and actions training only on random walk data


with no expert demonstrations, we can generate optimal trajectories at test time by adding


a prior to generate the highest possible returns. They also say see more details and empirical


results in the appendix. I've looked at the appendix, nothing there. I've looked at the


code, nothing there. Just just saying I mean, it is a toy example to illustrate but like,


there's nothing there of this example. So what they do is they have a graph. There is


a goal, or you're supposed to just find the shortest path. What you do is you just do


random walks, okay, some of these random walks will actually fail like this one here. So


the all the rewards are negative infinity, some of them will succeed. And then you can


generate that training data. Okay, so from here, that all the future reward is negative


four from this particular random walk you did here, okay, here, you started a different


location, also negative four, because you're going to take four steps. Now, what you do


with this sequence modeling approach is you say, I want to start from this node. However,


however, I would like to get a reward of negative three, which is a lesser reward than you got


all the way here. So what you're asking the model to do, and by the way, like, I'm pretty


sure this should say negative two to make their example compelling. Okay. But so I think


there's kind of a flaw in this toy example. But I hope you can still see what they're


doing. So you're saying I would like to get a very high reward or a low negative reward,


I guess a low magnitude negative reward going from here, which corresponds to finding a


really short path, right. And what the model is going to do is going to look at its training


data as well was I in a similar situation, in some point, like in the training data set,


and it's gonna find Yes, yes, actually, here, I was in a very much similar situation. And,


and so I wanted to get exactly exactly that reward, I was in that situation, the history


is a bit different. But you know, who cares? Now I'm here as well. And what did the agent


do that then went on and reached exactly the reward I want? Well, it did this action right


here. Okay, I'll just I'll just do that same action. This is just comes out of the sequence


model, right? So it's the sequence model simply tells you, how would a sequence that started


like this continue, and it tells you the action. And then it looks at this thing right here.


And here is a bit where it fails, right? They say each step gets you negative one reward.


So technically, at inference time, at inference time, what you would do is you would look


at here, so you get negative one from here. So here, you will put negative two. So at


the beginning, you have to specify the reward you want to get. And from there on, you can


calculate sort of the next reward, they need this to be negative one right here, actually,


because so let's just imagine that for some reason, you got a negative two here, right?


So they need this to be negative one, because that makes their example. So the sequence


model says, Well, was I in this situation? At some point, and I got out, I got a negative


one? Yes, I was here. And what did I do to achieve that? I went there, okay, I'm going


to go there. Ah, now I'm at the goal, okay, and technically find somewhat the shortest.


Now, this again, this doesn't, the example here doesn't work because he start with negative


three, you're gonna end up with negative two right here, that wouldn't match the blue one


that would actually match this one. So you would not get the shortest path. So you should


actually start out with an oracle knowing that the shortest path is negative two. That


would, of course, not match any example you have in your training data. But the sequence


model could say, Well, this is kind of close to this, right? So the most likely action


is still going to be the one right here. And then you take the one right here, and then


you're in the negative one regime. And then you match this one right here. I hope you


can see right how that that figures out a bit. So this can also handle if you don't


get the expected reward, which of course can happen, right? It's not everything is always


deterministic. So because you reassess after every step you reassess, you ask sort of your


training data set. And this is very much how we think of these big transformer language


models, what they do is they sort of interpolate the training data set. So they stitch together


different pieces of the training data set, which is you can see that happening right


here. Of course, you already saw the flaw, you need to know what reward you would like


to achieve. And so, like, by the way, lot tech is beautiful, isn't it? Maybe that's


just my thing. I don't I don't recall that being like this. So that by the way, the code


is available and also the pseudocode big props. Here you can see that the decision transformer


in blue in Atari lags a bit behind what they call TD learning. So this TD learning, that's


the the conference conservative Q learning, and the behavior cloning, which they term


BC in the open in the open AI gym, it outperforms it a little bit. And then there's these key


to door task that we're going to get into in just a bit. So I just want to quickly mention


that their primary comparison here is this CQL. And they make a big deal about sort of


not needing discount factors, and not really sure what they mean. There are usually two


different discount factors in these algorithms. So one of them is usually found right here


in the objective formulation. So here they say, what we want to do is maximize the expected


return, which is this quantity right here, okay, so what you want to do is you maximize


your expected future returns in the episode. Now, this is usually different, some people


formulate it as the expected return in the future, but discounted by a discount factor


that you raise to the power. So you essentially saying the future rewards are less valuable


than current rewards. And that gives you some sort of stability, but it also gets you shortsighted


in a sense of one. However, this is a choice, this is a choice of the problem formulation.


Now I get people train with this for maybe stability reasons, and then they still test


and actually report the undiscounted reward at the end. Okay, but I'm just saying this


is a choice. And their choice right here is different from what CQL does. So CQL explicitly


maximizes the discounted future returns, while they maximize the future returns. I just want


to point out that there is an actual difference here. The other difference is in the TD learning,


okay, so the by the way, if you don't do this, if you don't discount your returns, you get


the situation that you can you can cycle. So if you know if you if you get like positive


rewards or zero rewards for certain transitions, it can just like if someone is losing, okay,


a game. So here would be negative one, this is the only two options, either lose, or,


you know, go back here. Now chess has a built in protection against this. But other things


you can just agent will just circle forever because it doesn't cost anything. And if it


were to go here, it would actually lose. So you usually discount. No, actually, that's


not why you discount. Sorry, that that is a bad example. But there are good reasons


to discount future words here, you would actually implement some sort of a penalty like minus


point one for just any step you do. Yeah, but discounting, maybe you could you could


win if you could win, the agent could still go in circles because well, it can still win


later, right? Yeah. In any case, that's one discount. The other discount factor is in


the TD learning. So right here. And that's a different discount factor, you say, Well,


I'm going to predict this next step right here. That's probably a pretty accurate description.


And that reward here is quite a good signal, given that I am in this step right here. The


next one may be a bit more noisy, right? Because it's two steps ahead. And then I could, you


know, I could be doing different actions, maybe the transition is stochastic. So when


I learn my value function from all of these different goals, I'm going to value this target


as a learning objective. Right here, you have that recurrence relation, I'm going to value


this target, the highest, I'm going to value this one a little bit less some, I'm more


trying to match this oops, sorry, I'm more trying to match this one right here, given


that reward, then I'm going to match this one right here, giving the given the two rewards,


maybe both should be accurate. So the value should match this, their reward plus this


one, the value should also match these two rewards plus this one. But the second one


is more unsure. So the TD learning, usually you have classically called another discount


factor lambda, where you discount sort of future losses. And they say, we don't need


the discount factor right here, I don't know which one, which one they're referring to.


But what I want to point out here is that, yeah, the objective is different. So maybe


they say, we can get by with this objective. I don't see that that's a choice of the modeler.


And you run into problems with some environments, if you don't have a discount factor. In any


case, you can see right here in the experiments, for example, this is Atari. The decision transformer


outperforms CQL in some respects, it it trails it in other ones. I mean, they also look at


like these standard deviations are, are quite high. In the open AI gym, it is a bit, it


looks a bit better in that it sorry, it does outperform CQL in quite a number of things,


and also with less standard deviation right here. Yeah, also, they, they compare against


sort of behavior cloning, where you retroactively only train on the best such and such percent


of the experience. And they find that if you hit the correct percentage, which is not necessarily


the only the best trajectories, if you hit the correct percentage, sometimes behavior


cloning can actually give you a better performance. However, hitting that percentage, of course,


requires another hyper parameter search. And you as an oracle, you kind of have to, you


know, you have to go and filter and you have to try out and you don't know, you have to


have some sort of a validation set, whereas the decision transformer is just one run.


Now throughout all of this, they're sort of touting that they don't need as many like


searches and as many, you know, like here, you need to choose that percentage, you need


to figure it out. But if you look at their actual configuration of hyper parameters down


here, they do things like, well, we have one architecture for these Atari games, but then


we have a different one for pong, right? We have a context length for these Atari games,


but then a different one for pong, because pong is actually quite a sparse reward ish


game, okay, compared to these other ones. So they make the context length bigger in


order to capture a longer history, because otherwise, you couldn't differentiate the


agents and they would need to use TD or some kind of dynamic programming, right? And then


there's also this, this how the return to go conditioning, like how much reward do you


want to get? And that's a problem. Like, so here, again, they do something and this is


like they look at the baseline, they look at CQL, how much did that achieve? And then


they just choose to achieve a multiple of that one. This is like, you look at your competitor


at what you're compared to, and then you base your decisions off of the result of that.


So you know, I kind of get it. And also this multiplier they take, it is very informed


by them knowing the games right in pong, you know, you can reach at max 21. So that's the


condition on the reward of 20. In sequence, it's I think it's unbounded. So they they


do it 1.5 times the performance of that. And yeah, so I'm not I'm like, I'm not saying


this is invalid experiments. But like, this this looking at your competitor, and then


basing crucial hyper parameters off of their performance. But I'm sure I'm sure it will


work otherwise. But just know that you need to have a good idea of what reward you can


even achieve and what's possible given your data set, right? So CQL also takes into account,


like it also learns from the same data set. And that's sort of how they know what's possible


from that data set. Yeah. So is this a problem that you need to know the reward? Can't you


just put 100 billion billion billion? And the answer is no, you see right here, this


orange line is the highest reward that was observed in the data set. Now this is gamer


normalized. That's why it's not like 21. But here the experiment, it's actually a pretty


cool experiment is, since you're not only maximizing reward, you can you can ask the


model to give you any reward you want. So the green line is what you wanted. And if


the blue line is what you achieved matches the green line, exactly the model always gives


you the actions to make that reward that you requested happen. Okay. And you can see that


green line and the blue line, they match pretty accurately for a long stretch, which meaning


means that this the sequence modeling approach can really not only give you the max reward,


but it can give you sort of any reward because it remembers all the sequences. Though probably


not the lowest ones, because you're actually learning from a DQN learner that has probably


only good trajectories. Okay. But you can see as soon as you go past the highest observed


reward, it not only does it stay flat, it actually drops down again. And you can see


that pattern pretty much anywhere where you have an orange line like this. So here, you


let maybe you stay, maybe you drop down here, it's like kind of seems like you stay. It's


only that here in the sea quest, where it's a bit better, but like, this is a gamer normalized


score of three, like a gamer would achieve 100 here. But you can also see that sort of


drop compared to the green line. So that means you can't just put 100 billion, essentially,


so you need to know the reward that you're going for sometimes no problem, sometimes


actual problem, okay. And that reward is not only dependent on the game, it is also dependent


on the game, but it is also dependent on like how your data set is that you learn from is


structured, you need to know what your agent can achieve. They do some other ablations


with respect to context length, they actually find that larger context length helps. So


if you don't provide a long context, the performance drops, it makes sense in that the transformer


is able to match the history to observe trajectories better. On the other hand, technically reinforcement


learning algorithms, since these are in Atari are fully observable, if you do frame stacking,


you know, technically, an RL agent shouldn't shouldn't care about the more of the past.


But you know, RL algorithms do, they're not perfect. The last thing is that key to door


thing where they show that, okay, there, this is a an experiment, toy setting, by the way,


again, I did not find this in the appendix, I did not find code for this. So we actually


we don't know too much about this experiment. But as far as I understand, there's one room,


there's two rooms, there's three rooms. In the first room, there's a key. In the last


room, there's a door. Now, you're thrown into the first room, you get to walk around a bit,


then you're thrown into the second room, you get to walk for a variable length of time.


And then you thrown into the last room. If you have put taken the key, and you reach


the door here, then you get a good reward. Otherwise, you fail. Okay. So the middle room


is called a distractor. Because if you have something like an LSTM, or if you have something


like Q learning or something, so the problem with this, sorry, q equals r plus q, is that


this sort of looks one step ahead, okay, this recurrence relation, that means if you have


a learning signal, somewhere way down the line, you need to sort of propagate, it's


not back prop, it's actually, you need to learning step propagate the fact that there


is a signal back here, all the way through these time steps in the past, where a transformer


can just go like, okay, so this is this an experiment designed to show that this really


helps. So you can see right here, they can analyze what their system says about the expected


reward in the future. So you can always ask it, how probable is a given reward in the


future. And you can see whenever the agent doesn't pick up the key, it immediately knows,


as soon as it gets into that second room, it immediately knows it's lost, no matter


what happens in the last room. If it does pick up the key in these two situations, it


estimates a future reward of about point five. And you can see it does not degrade across


the distractor room. Okay, so no, no matter how long the distractor room is, does not


degrade. And that's the key difference between this and like, let's say TD learning, q learning


approaches, it does not, it doesn't forget, because there is no dynamic programming involved.


And then, you know, in the last thing, if it reaches the door, obviously, it says, Well,


that's a high value, if it doesn't reach the door, it changes its mind. Now, I would have


liked to see whether or not and this is why I was keen on seeing the parameters of this,


whether or not this right here is inside or outside the context length of the transformer


they used. And I'm going to guess it's still inside. Because as soon as that's outside,


or like, let's say more like this, as soon as that's outside the context length, the


system has the sequence model has no way of knowing whether that particular agent picked


up the key. So it cannot predict anything. I think what they're what they want to show


right here, sorry, that's an alarm. What they want to show right here is the fact that the


attention weighs heavily on those frames where it picks up the key or reaches the door, which


is fine, right? We can we can get that transformers learn that. However, here, I'd really, you


know, like to see what happens if you go outside of that. And again, if you go outside of that,


you're going to revert back to the old method. So ultimately, the transformer gives you a


longer context where you can do one step assignment of credit. But again, as soon as you exceed


that, as with the LSTM, as soon as you exceed these, you need the classic approaches. And


I feel the paper is a little bit is a little bit shady on the fact that they get like a


constant factor, longer context with what they're doing. But it doesn't really solve


the problem. Okay, in my mind, I might be wrong, please tell me if I'm wrong. Read the


paper for yourself. It is a good paper. I hope we can cover the trajectory transformer


Transformer Memory as a Differentiable Search Index (Machine Learning Research Paper Explained)

This is a comprehensive paper review of the paper transformer memory as a differentiable search index.


This paper is pretty crazy.


It takes an entire data set and puts it into the weights of a transformer.


Essentially, it trains a search engine not to search through documents, but just to give you the index of the document that matches your query just like that.


Boom.


So this video is a comprehensive review of the paper.


I'll explain to you what's in the paper, what it's about.


And by the end of the video, you should have a good idea of the paper itself.


The next video, which I'm going to release tomorrow will be an interview with the authors.


We'll dive right into the content and any criticisms and questions that I raised during the review.


As always, let me know what you think in the comments.


Now let's get into the video.


See you around.


Does your company have a lot of people labeling data?


Why would you leave such an important task to close source systems or self-implemented things?


Training data is your most valuable asset and human labels are really expensive.


Today's sponsor is diffgram, which is an open source platform centered around training data.


They handle everything to do with training data, especially collecting labeling serving and more and it is open source.


So you can self host all you want.


But there is one cool thing if you let them host it for you and that is unlimited pricing.


No per label annotation, no expensive servers to run.


You pay once you get as much as you want.


So thanks again to diffgram for sponsoring today's video.


Check them out using the link in the description to let them know that I sent you.


All right, let's get into the video.


Hello there.


Today, we're looking at transformer memory as a differentiable search index by researchers of Google research.


This paper on high level takes a search problem where you have to index documents and retrieve them.


And it puts all of the corpus essentially into the weights of a transformer.


So it takes the corpus and trains the transformer and then at the end, they can just give a query to the transformer


and the transformer will output the ID of the document that matches.


And it turns out for some data sets that they have for some settings and with some clever training and representation of the documents that can actually work, which is really crazy.


This kind of speaks to multiple things such as obviously our ability to overfit on stuff,


but there is some generalization here as we'll see on the other hand, also the kind of inner workings of these transformers.


And lastly, what's pretty cool is that this is completely as it says differentiable.


It's a differentiable search index, which means that this can be part of larger neural network architectures because it is fully differentiable


and it can be trained essentially end to end at once.


And that means we can potentially employ reinforcement learning agents with kind of retrieval abilities and much more things.


So we'll dive into the paper. We'll see what it's about. The idea, as I said, is pretty, pretty simple.


If you like content like this, then as always, leave a like and tell me what you think in the comments. That's always super helpful.


So as I said, they take a search problem and the search problem is essentially I have a corpus, like I have a big database of documents, right?


Here is a document. Here is a document. And I want to build an index and an index is some kind of data structure, some kind of thing.


And at the index, I can throw a query and the index will return to me an ID, a document ID that specifies which document matches my query.


Usually this is done via inverted indices. So I want to tokenize my documents, split them into little tokens, which are usually words or sub words.


And I want to stem them and lemmatize them and whatnot. Then I build a reverse index. So for every word, like in the word in, I remember which documents it appears in, like document three, document five, document 11 and so on.


And then once the query rolls in, I simply tokenize it as well. I go look into my inverted index and I look up all the documents.


And then there's also a ranking step, which means I have to now determine which of these documents is the most relevant.


And that is usually done via techniques like TF-IDF features. There is a famous technique called BM-25, which is also a baseline in this paper.


So this is the classic search kind of way, way of doing search. If you use any search engine at all, this is being done in the background for the most part.


Newer search engines are catching on. There's neural search and so on. But BM-25 is still one of the most performant things that text search has available and also other types of search.


However, there is a new push in sort of neural search and in neural search, you're trying to take your data set.


And for each document, you try to map it to some sort of a vector in vector space.


And then once the query comes in, you also map the query to a vector. And for example, you compare inner products, whichever inner product is largest.


That's the document that's relevant. This is just one way. This is what we usually call a by encoder method where the documents and the queries are mapped, both mapped individually.


So there will be an encoder here and there will be an encoder here. They all would output one vector and then the vectors are compared.


This could be the same encoder or different encoders for documents and query. This is just one method.


There's various methods such as cross encoders, re-rankers, dense retrievers, you name it. However, this method here is even more is different.


So what we want to do is we want to take the corpus as such and map that somehow into a neural network.


And we're going to talk about this somehow. But we're going to train a neural network. Essentially, how do we represent this?


Let's represent it with its layers such that when later I feed a query to the neural network, as I already said, the ID of the document is the output of the neural network.


So it doesn't output a vector that I didn't go and compare. It doesn't.


I don't have to go and feed in query document pairs and then I get out a score of how well they fit together, which I would do in a cross encoder.


No, the transformer in this case, the neural network directly gives me the ID of the document without seeing the data at inference time.


So during training, all of the data is essentially has to be mapped somehow into the weights of the neural networks.


So somewhere in these weights, that information is stored of what the documents are.


So the entire corpus is in those weights. And once I enter a query, the correct document ID can only be output, obviously, if the transformer has somehow learned what is in those documents.


So that's the setup. It's pretty simple setup once you kind of see what's going on.


It's like a meme, right?


Instead of we've been trying to neuralize search and we've still done this two step process where we train these encoders,


but then the actual search is still done using, for example, a nearest neighbor algorithm like here.


But, you know, this is just the idea of, well, why don't I just ask the neural network to output the result, right?


The resulting doc ID. Why don't I just do that? And it turns out that can work surprisingly well.


So you can do a couple of things here, but that's essentially it.


They say right here in the introduction, they use a sequence to sequence learning system to directly map a query to a relevant document ID.


They have different corpuses where they train it on the smallest corpus. This method improves the hits at one, which means that whether the top hit is the correct one,


more than 20 points from 12.4% for a dual encoder.


So the baseline here is a dual encoder, what I shown whenever there are two encoders and they each output an embedding to 33.9%.


That's a giant gain, right? That's like a 2.5x improvement.


However, on a corpus that's 30 times larger, performance is improved by nearly seven points, which is less.


It's also respectable that performance is improved at all. However, I want you to notice and that's already kind of the first indication,


a little bit of obviously what's going on here on smaller data sets.


This method does super duper well on larger data sets.


The method doesn't do that much better than a sort of cross encoder type setup, sorry, a by encoder type setup or a dual encoder type setup,


which is understandable, right? Because the smaller the data, the easier it is to absorb it all into your weights.


If the data gets larger, that obviously gets harder and harder.


There's more data to go around, which means there's more room for error, for confusion and so on.


And a classic search engine or a dual encoder is going to have an easier time in that case.


But still, it's a cool, cool paper.


It's just that it kind of gets worse with the data set scale.


It does get better with the model scaled, though.


The really exciting thing is something that I've already mentioned, and they mentioned this here.


All aspects, sorry about that, they say all aspects of retrieval are mapped into well understood machine learning tasks.


So, for example, indexing, which is building the reverted index, or even if you have the dual encoder,


you need to build the nearest neighbor index, which is a hard task in high dimensions,


is now a special case of model training. So it's just training.


And incrementally updating an index becomes just a special case of model updating.


So all the tasks are just tasks that we already understand from neural network training.


So here is a comparison of the dual encoder method,


which is the, let's say, old classic neural search method, not the BM25 retrieval,


but the neural search method, and this DSI, the differentiable search index.


So in the dual encoder method, what we do is we train this encoder.


And in this case, they train one encoder for both the queries, as well as the documents.


And what we try to do is we are going to try to use some form of contrastive loss.


If we actually have query document pairs, what we can do is we can try to get the documents,


the query and the document that go with each other to be close together,


while making the documents that are unrelated to each other be far apart.


So this is some sort of contrastive loss, obviously at inference time.


What we're going to do is we have a query, we put it through the encoder,


we get its embedding, and we do a maximum inner product search through our entire vector space of our indexed data set,


and we get a ranked list.


So it's kind of this two-step approach with building these indices in between,


and with the training objective that is not directly what we want.


It is a proxy objective because of the algorithm later needs it, the inner product search,


but it is not actually what we want.


So let's just train what we want.


In the DSI, in the differentiable search index, I simply feed my query along with,


I simply feed my query essentially to, in some form, to the system,


and the system outputs directly which document is relevant for the query.


So the way they train it, and this is one way they train it,


is where they feed in queries and documents into the system.


So this is an encoder-decoder setup.


In fact, they use, I believe, a T5 setup, if I'm not mistaken.


So it's a sequence-to-sequence task.


They feed in the queries and the documents, and they always output the document ID.


So if they feed a document, they just output the ID of the document they fed in,


and if they feed a query, they output the ID of the document that the query would hit.


So this is if you have supervised data, you can train the system already


for giving queries to output the correct document.


However, the method also works in what they call zero-shot,


which is if you do not have any queries, you simply input documents into the system,


and then you train it to output the ID of those documents.


And you hope that, because the models were pre-trained on language modeling


and on various other tasks, you hope that through that,


if you then enter a query that kind of describes the same thing as the documents,


that the system would still output the best document ID.


I mean, after all, it's constrained to output document IDs in most cases,


and therefore, it needs to give you something,


so it might as well give you the thing that is related the most.


So that's the reasoning behind it.


I've talked a lot about the different parts now of the system.


The write-up is actually pretty good.


I can recommend reading this paper from top to bottom


because it goes in a very structured form into what they investigate.


They investigate a lot of engineering choices,


which I really appreciate in the system because there are a lot of ways to do this.


And one or the other is not necessarily correct.


So they say we explore a number of variations of the DSI architecture.


They explore how do we represent documents as such.


The naive approach they say is just to index the full document.


So just input the text as such, like you can see right here,


just input the text into the encoder, output the document ID, that's it.


But maybe that's not the best thing to do.


Maybe you can throw away stop words.


Maybe you can do bag of words representation.


Maybe something is better than just inputting the first L tokens of the document.


Turns out it's not, but it's a good thing to investigate.


Then how do we represent document IDs?


The datasets, they usually just have some unique identifier per document.


In this case, it's like doc 137 and here it's doc 456.


If we do this as a sequence to sequence tasks, maybe we can do something smarter.


Maybe we can give the document IDs some sort of hierarchical notion.


They investigate that too.


And lastly, they investigate how should we index stuff?


So how should exactly should the indexing step, this training go?


They also do a lot of ablations on sort of the effect of sizes,


the effect of model size and corpus size.


And we're going to look into that as well.


So the method is called, as I said, differentiable search index.


The goal is to fully parameterize traditionally multi-stage retrieval


then rank pipelines within a single neural model.


And that encompasses two operations.


First is indexing and then the second one is retrieval.


In the DSI, we've already discussed this indexing is sequence to sequence approach


that takes a document that takes document tokens as input and generates identifiers as output.


That is indexing its training on the document collection to output their identifiers


and optionally fine tuning with labeled query sets, labeled query doc ID pairs.


The retrieval is then achieved by simply autoregressive generation.


I input something and I see what document ID comes out in the sequence to sequence model.


So it couldn't get easier than that.


Let's look a little bit into the engineering choices they consider.


First, the indexing method.


The first indexing method is what they call inputs to target.


And that is probably what I've described so far,


which is the sequence to sequence task of document tokens maps to document ID.


So they input the tokens of the document and they output the document ID.


That is the simplest method, the straightforward method from what we've heard so far.


And as far as I've read in the paper, as I understand it, this is also what works the best.


However, they proclaim that in this way, the only ever output is the document ID.


There is no sort of language learning or anything like this.


You fully rely on the pre-training for language understanding.


That is what they claim here is a potential weakness.


And other methods are targeted at, are targeted at in sort of leveraging or making that weakness go away.


They have this targets to inputs method, which they say we could also at training time,


what they call indexing time, input a document ID and then have the model decode the tokens of the document.


Now, this might seem a bit weird because it doesn't train the model to produce document IDs from tokens.


But the idea is that you could, for example, then fine tune on query document ID pairs


and that by training with this objective, you teach the model something about the document IDs


and which tokens, which document tokens are in the IDs because the model has to learn to produce the document tokens


and therefore it might make some associations or something.


I'm not exactly sure what the thing is behind, like what the reasoning is behind this, but you know, it's good to try.


It doesn't work, turns out.


There's also bi-directional, which both are done.


So during training, there is like a multitask setup where sometimes you do the doc ID to tokens


and sometimes you do the tokens to doc ID.


Also in their experiment, the bi-directional method doesn't improve much over just the plain method.


And the last one is span corruption, where you essentially input, I think, the tokens and you append the doc ID.


And then you consider this entire thing as like one piece of text that you want to predict


and you have this span corruption objective, which means that you can mark out any random spans in here between,


which also means that sometimes you mask out the document ID or maybe part of the document ID


and that kind of forces the model to learn.


It's a bit like BERT's masked language modeling, if I understand this correctly.


However, also this doesn't seem to work super well for them, even though it has actually worked well in other tasks.


So in other papers that have done things in the sort of sequence to sequence space.


OK, so now we have the indexing method of the table.


The document representation strategies are next.


The first one is direct indexing.


You say we take the first L tokens.


Again, this seems to work the best.


Just take the first L tokens of the document.


Interestingly, during the experiments, L bigger isn't necessarily better for L,


which is also might speak to a little bit of the quality and nature of the data set itself,


but also tells us again something about maybe this works in particular


because we're dealing with sizes and data set sizes and lengths of documents


that are actually possible to absorb into weights.


And it is interesting to see how as the data goes up, this becomes harder and harder.


I would question, does it become like linearly harder to put this into a set of weights?


Does it become exponentially harder if there's more data?


Not sure. It would be interesting to find out.


The other methods are, for example, set indexing that which deduplicates repeated terms


and removes stop words doesn't seem to help much.


And, you know, naturally, one might think that, you know,


if I remove stop words in my document representation, that gives me a cleaner signal.


On the other hand, these models are pre-trained on actual language,


not on cleaned up language without stop words.


They're pre-trained on actual language.


And therefore, I think they have a strong bias towards, you know, kind of correct grammar and so on


and might work with that data a lot better.


I think that might be largely behind why the direct indexing method works better over the set indexing.


And then there's what they call inverted index,


which is a bit in the spirit of how search engines classically do this.


They say we randomly subsample a single contiguous chunk of K tokens from the document.


So they're not only limited to the first L tokens,


but they always kind of take a random sub string of the document that is of that length.


Now, technically, this should work better than the direct indexing.


I like the inverted index in their experiment performs worse than the direct indexing,


and I just don't believe it.


Like, it doesn't it does not make sense, right?


Something's going on either the data set is such that for some reason,


I can find a lot of the answers that I'm looking for in the first in the beginning of the documents that are indexed,


but this is purely a property of the data set,


or it is really like the introduction of a tiny bit of noise into this,


namely that for the same document ID, I see different sub strings, I see different tokens that already kicks the method out of its comfort zone.


That seems to be like the in first instance,


it's kind of a bummer that this is the data set,


but we'll have to take it in the second instance.


It's a bit more worrisome if that were the case,


like if that fact would be already detrimental where it actually should be beneficial.


Or, yeah, maybe I'm misunderstanding something,


but it seems to me that the this last method should be superior to the first one.


So the last thing they or the next thing they investigate is how do we represent,


by the way, I'm already telling you about the experimental results.


They'll be coming up in the next section,


but I think it's easier to mention them already here than to keep everything in your head and then go to the experimental results.


But we will go into it in just a bit.


They investigate how should we represent the doc IDs.


Again, the simplest thing you can do is to have these unstructured atomic identifiers,


which essentially means that every document gets a unique identifier


and then in a sequence to sequence model, right, I have my sequence here.


This is in goes into my encoder and then it goes into a decoder and the decoder produces a sequence.


Now, every one of those tokens is in a list in a vocabulary.


The vocabulary has a certain amount of entries.


If I tokenize correctly, I have no out of vocabulary words,


and this has a some kind of a fixed size like a vocabulary size and the decoder.


It can have the same vocabulary or a different vocabulary.


In this case, I think it's the same,


but what they do in this first method is they simply extend the vocabulary for the decoder and the extra tokens here.


Every single token is represents one document ID.


This obviously only works if you know all the documents ahead of time that you're going to index,


but in their case they do so they randomly initialize those embeddings and during indexing,


they train the embeddings for those and that essentially means it's a multi-class classification problem.


At the end of the day, every sequence prediction task is,


but we're not going to predict multiple tokens.


We're going to predict exactly one token and that token comes exactly from this vocabulary,


and that means this is not a sequence to sequence task.


This is just a multi-class classification task.


Now this has advantages being multi-class classification.


It means there's one prediction.


There's no autoregressivity or anything like this.


It's essentially a classic encoder only problem.


Now this is the easy part.


The hard part is, of course, you don't leverage anything.


You introduce a lot of new classes, a lot of new embeddings,


and they claim in the experiments that these things are quite brittle,


even though in the zero shot case, apparently they work out super well.


But we'll have some comments on that too.


The next thing is naively structured string identifiers.


So they say again, like here, every document will have an arbitrary unique identifier,


which is just kind of an integer.


However, they just say, well, we'll just put the integer as a tokenizable string.


So if the integers, if the integers is like 1, 1, 2, 5,


then the model needs to predict the tokens like the strings 1, 1, 2, and 5,


or maybe it's tokenized differently,


but it will actually have to produce this thing as a string,


not as a output into an output classification bucket,


but it will have to output the string.


So this is now truly a sequence to sequence task, right?


And the last thing they consider is these semantically structured identifiers


and that's where they think, well, can't we do something better for the document IDs?


Like, can't we imbue them with some meaning?


And they come up with the following procedure.


So they have two, they have two principles they want to follow.


They say the doc ID should capture some information about the semantics of its associated document.


And second, the doc ID should be structured in a way that search space is effectively reduced after each decoding step.


This results in identifiers where semantically similar documents share identifier prefixes.


So essentially they want the documents to have multiple like the ID's


could be 255, which essentially means it's like a path, right?


It's like a folder path.


So this is super group 2 and then group 5 inside of super group 2 and then document 5 inside of that.


And the assumption is that all the documents that are in the same like group 2 slash 5,


they share some stuff such that the decoder, if it's not sure which exact document it is,


but it can already say, well, in super group 2, I find all the things that talk about,


I don't know, household items.


And then in 2 slash 5, there are all the things that talk about electric appliances in the household.


And then inside of that, there might be some documents,


but the model could consider step by step.


The model would first consider outputting sort of the super group


and then condition on that in order to output the group


and then condition on that in order to output the next level.


So that's what they do.


They do a hierarchical clustering approach, which means that they take another model.


So they take some sort of a, I think it's a BERT model, a BERT, I think, I'm not sure where they mention it,


but they take a BERT model, they put all of the documents through the BERT model,


they train and embed, I don't know if they actively train it or if they take a pre-trained one.


In any case, they have some way of embedding documents.


So they embed those documents, then they use k-means clustering to divide them into clusters.


If the clusters are still too large, they recursively subdivide them into clusters.


And here you see exactly, so this here is document 2, 3, 3, because it's in super group 2,


it's in subgroup 3, so that's 2, 3, and then it's the third document inside of that.


So that's 2, 3, 3, and presumably the 2 and the 3 prefixes,


they're kind of like the path into the hierarchy and make it easier for the model to decode.


Now, this seems like a cool idea, honestly, because it kind of makes sense.


There are, however, two conflicting things.


One is the fact that there is semantic meaning in, you know, in 2, 5, 5 or 2, 3, 3 in that case,


right, there is semantic meaning in these things and not just a random identifier.


The other one is that it is in order, so the top hierarchy is first, then the second, then the third,


which might interplay with the autoregressive way that we train these things.


So in order to separate the two things, one would need to make an experiment where you just flip it around, right?


You decode, while you decode, you do, you decode from the back, you decode like 3, 3, 2, right?


And then you essentially still retain the semantic information of the identifier,


but you drop away the autoregressivity.


So the model essentially could not condition on the supergroup while decoding the lower layers.


So you could tease that apart a little bit.


They didn't do that, but in any case, this would, I guess, be an idea of doing further ablation


and understanding into how this model works.


It is interesting, they, yeah, that's it essentially.


Okay, then how do they train? They say we try two strategies.


One is to first train the indexing step, so first feed the documents and output their IDs,


followed by a fine tuning stage where you feed queries and map them to their IDs.


Or the second strategy is to train them together in a multitask setup.


That's exactly what we saw in the diagram.


You feed documents and queries for documents, you output their document ID,


for queries, you output the corresponding document ID,


and you have some ratio of how many indexing samples and how many query samples that go in.


Turns out that second method is better, which I don't know if I would have guessed that,