And eventually, if you accumulate all that, you train on that, you get a very good policy


First Author Interview: AI & formal math (Formal Mathematics Statement Curriculum Learning)

of function.


And I think that's why we say this is that the self play that we have in two player games


is really about getting a data generation pipeline that generates good data.


And that's why we call it an unsupervised curriculum.


And in formal math, if you have a bunch of statements that you cannot prove because your


prover is just not good enough, you're just not going to get any data.


It's going to be you're going to just be stuck at that point.


And so that's kind of the main difference.


There is no way to reframe, I mean, there's no trivial or easy or obvious to me at least


ways to reframe a problem that is just too hard into a set of easier problems.


And it makes sense that you're trying to build up a curriculum.


But now also I've displayed this here with this sort of arrow of complexity that just


gets more and more complex, but it is not really the case.


It doesn't really look like this because complexity isn't just in one direction.


It's not just a statement is more complex than another one.


But there is there's also a direction I want to I think if I want to work myself up to


prove let's say the whatever general Riemann hypothesis or something like this, I can't


just prove harder and harder statements in numerics or something because I really want


to be in I don't even know what category the Riemann hypothesis number theory or complex


analysis.


Okay.


Well, you know, but the point is I can't just go about just proving any old theorems.


I have to have some sort of a direction.


So how does your and you make a little bit of a point in manual curation.


Might help here and so on.


But is what's the main force in your system driving sort of the direction that the system


becomes an expert at because there's so many directions in math, right?


It's impossible that it just becomes better.


Right.


Yeah.


So, I mean, we took the very obvious and easy way.


Actually you have in a with a formal system, you have a library of theorems that is associated


with it.


That's what what the formal community generally working on.


This is what we call math lab.


It's called math lab in lean.


And there is very few exercise or Olympiad type exercise or even exercise in math.


It's a general purpose theorems, right?


And so you should train on that data only.


You actually not that good at solving exercise because you haven't seen any very easy exercise


you'll be able to solve, but the somewhat hard ones not in whole.


And so, and we had that mini F2F benchmark, which is made of exercise, exercise that we


cared about for many reasons that we can dive into.


And so we took the easy way, which is let's just formalize a bunch of statements around


that benchmark that we care about.


And we did the most obvious thing is that we took the textbook that humans use to train


for those competitions and formalize everything out of it.


And we didn't ask ourselves much more questions than that.


And the reason why it works is because it's a textbook.


So there is a bunch of easy examples to begin with and the difficulty kind of improved nicely


for humans.


And so as we formalize the statements, we run our expectation loop on it.


And as you mentioned in that illustration, you get a few statements first, but you retrain


on them.


So you get a few more, et cetera, et cetera.


And as you do it, you really, the way I visualize it is that you're really shifting the distribution


of the model away from math leave and towards mini F2F or towards the group of statements


that you provided as a curriculum.


And so that is that curation that gives the direction.


In terms of direction, you're very right that it's a challenge.


Something that you can do as an example with formalize is you can do forward proving instead


of going backward, as you said, you take things that you know and try to compose them with


theorems that unify to the things you know.


And you keep going forward like that.


And we've tried generating some data this way.


And that data is actually, I mean, you cannot direct it easily.


And so it goes a little bit all over the place.


And we haven't found a way to make it beneficial for targeting a benchmark in particular that


we care about.


Do you see maybe a future where you mentioned the lack of self play, but there could be


some sort of an agent that comes up with these intermediate statements, these curriculum


statements that sort of tries to guess, you know, maybe here is a statement that's kind


of in between where you want to go and where you are currently.


This could be some sort of, I mean, I'm never sure because a lot of times when people propose


these agents, it's like, well, you if you have that agent, you've essentially solved


the problem, right?


But there could be some sort of thing that replaces you, the human as who has to come


up with this curriculum.


But I guess it's a bit of a future thing.


And the other avenue where I see, sorry.


So I'd like to jump on this one.


Just for a second.


It is plausible that we could build a model.


I mean, it's theoretically plausible that we could build a model that creates those


intermediate statements.


There's two challenges here.


The first one is that the number of statements that we have is actually extremely small.


When you look at the proof data in formal math, and I didn't mention it before, but


it's also a good thing to mention it.


One challenge of formal math is that data is extremely scarce.


The proof data is scarce and the statement data is even scarcer.


MathsLib is something like 60k statements, 60k context lens things.


The curriculum we use is a few hundreds.


And so to train that agents to try to simplify statements, the data that you have access


to is in existence by standards, modern language modeling standards.


So that's a really big challenge.


One thing that I think is extremely exciting that is, again, same idea, just make it simpler,


is probably actually machine translation from informal statements to formal statements.


Try the work that we've been doing, try to harvest a lot of informal statements that


there are many more out there and try to auto-formalize them.


Formalizing a statement is actually much easier than formalizing a proof.


It's still challenging, but definitely much easier.


No, no, no.


Sorry for jumping in.


So with respect to, yeah, I was also thinking, yeah, you could take all sort of the math


that's out there, but that's obviously also curated by humans a little bit.


The other point of controlling things would be the language model.


There's a lot of work in prompt engineering and things like this.


Now your language model, maybe we can go a little bit into how you train and query the


language model, which I think might need or might benefit from a bit more explanation


because I was quite vague here, right?


But essentially you have two different types of inputs that you train the language model


on.


The one you call this a proof step objective, and the other one you call this proof size


objective.


And both of them, they have a declaration and the goal.


Do you want to maybe give us a little bit, because for the declaration I was like, yeah,


it's kind of like the things you have access to.


Do you want to maybe give us a bit of insight into what these things are?


Yeah, so if we go back to, if we think about your schema about proving backwards, so the


goal is the current goal that you want to prove.


The proof step is the tactic that you want to apply.


So this is really mapping exactly the process of generating a tactic to try to simplify


the current goal.


Sorry, the goal, so if I'm here, right, the goal would be the top thing, this one right


here, and the tactic would be one node, one link to sort of the next node.


Okay.


To a new goal.


This could be the new goal, and then these could be the proof steps.


Or okay, okay.


Yes, exactly.


In here, the lines are the tactics and the circles are the goals.


And in Lean, you actually have just one goal, the tactic goes back to another goal, because


sometimes some tactic can create multiple sub goals, but because you could say, hey,


I want to introduce that cut, the cut is kind of a mini conjecture inside a proof, but Lean


kind of stacks them together.


So technically speaking, there's only one node at each end of each line.


Yeah, exactly.


The proof looks like a chain.


The final proof looks like a chain.


And the proof search looks like a tree.


And so the deacle, we condition on the deacle name, so the deacle name is the declaration


name, and it's simply the CRM name or the exercise name.


And the motivation here is to provide a proxy information for the model as to what is the


state of the formal environment at this stage, because the actual formal environment is gigantic.


There's no easy way to represent it in a compact way.


You have all the inputs, you have all the CRMs that have been defined in the same file


before that very CRM, that the CRM you're trying to prove right now.


You have a bunch of definitions, et cetera.


And so if you wanted to represent that to the model, it's technically challenging and


more importantly, it's really big.


So instead, we just give it the name of the CRM and we kind of hope that it'll provide


signal to the model as to what are the CRMs that it has access to for this one, because


it's trained on CRMs that are close to this one and the names of CRMs are somewhat similar


and related, it was in the same file, et cetera, et cetera.


So it's really kind of a trick to try to infuse a little bit of information about the environment.


Can we imagine such a name?


Is this like a human readable name or is this more like, you know, theorem 2845.8?


No, no, it's somewhat readable for the experts at least.


It's in floor smaller than floor positive, some kind of stuff like that.


It's a little bit compact, but it's still readable.


And for the exercise that we use, it's actually just the name of the competition, the girl,


the exercise number.


And the proof step that will be the tactic itself.


How is a tactic kind of described?


Is this an index into some bucket or is it also a piece of text?


Yeah, so if you're scrolling the appendix while I describe it, the tactic is really


a function call.


You're calling the tactic, which is a metaprogram.


So if you, yeah, as an example, this one, apply tactic is very trivial.


It just says try to apply that theorem to the current goal.


But you have much more advanced tactic.


And so that tactic takes an argument.


So you not only have to pick your tactic, there's only a few of those, but you actually


have to provide an argument.


So here it's a theorem name.


There's many more, but still finite.


This here is a theorem name.


And then you'll, oh yeah, here you go.


Okay.


NAT prime DVD move.


Yeah.


So that's a typical theorem.


So that's the decoration name that we condition on if we wanted to try to prove it.


And you have to apply it with, here it's applying the theorem by providing a first argument to


the theorem and then looking at one side only.


And so all of that kind of explodes the action space obviously.


And the action space is actually infinite because some tactic has arguments, mathematical


terms.


And those mathematical terms, they don't necessarily exist in the context.


If you're trying to prove an existential statements often, the easiest way is to provide a witness.


The witness is not generally in the statements.


And so you have to generate it.


And so that's the reason why the action space is actually infinite.


And that's the major difference between neural proving techniques and the kind of classical


theorem proving automated reasoning techniques.


They are extremely powerful, but there's one thing they cannot do.


It's generating exogenous mathematical terms.


And you would, in this case, your language model would directly suggest you such tactics


to apply.


So you would sample from the language model and then suggest a bunch of things.


The language model generates the full string here, apply, night, prime, HBMP.


And so we generate a number of those that gives us an approximation of a potentially


interesting action space to explore.


And on top of that, we run a proof search.


How does the proof step come into this?


Because I was a little bit, you already have some sort of a log likelihood estimation,


I would guess, for the things that you sample.


But then you also have this value, some sort of a value that you assign to how long you


think a proof is going to be.


Yeah.


So the proof size objective takes the declaration name and the current goal and try to estimate


the size of the proof for that goal.


And that's really just an instance of a value function.


That's the one that we've used here.


And it really helps guiding the proof search.


When you don't have the value function yet, so in your review, you mentioned that we bootstrap


from Theta0, which is the first model that is only trained on proof steps.


When we don't have a value function available, what we do is that we do the same proof search,


but we prioritize by log prob, as you said.


But what we use is the cumulative log prob that took for us to apply the different tactics


all the way to the current goal, which is another flavor of a value function.


A bit of a beam search type.


That is, yeah.


It's a beam tree step search.


Okay.


And okay, so I think we got a good idea of how the search itself works.


And you keep going until you prove statements.


And then you do this expert iteration steps, right, which essentially consists of you try


to prove new things, you add them back to the data set, and you train a new model on


it.


What I was kind of surprised by is that you always train from this sort of this initial


model that you have right here.


So you create your new data sets, and you always train from that.


What prevents you or what's the reasoning behind not always just continuing to train


from the most recent model?


Yeah, there's two motivations, two rationale for that.


The first one is that it makes controlling for overfeits much easier because you're really


training from scratch in a sense.


And so you control overfeits on your validation sets much more cleanly.


If you iteratively train the behavior of the validation loss, I have a tendency to be quite


erratic and unpredictable, which makes controlling for overfeits much less obvious.


So that's the one thing.


It's for basically scientific convenience in a sense.


The other thing is that it gives us an opportunity to duplicate aggressively the data.


The reason why it's important is because, to be honest, to generate those proofs we


sample proof search a lot.


There are some easy statements, we can find thousands of different proofs for it.


And so the goal is to retake all those proofs that we found so far and duplicate as much


out of it to prevent nefarious overfitting behaviors in the training.


So that's really the two main motivations for training from scratch.


Again, formal math, data is scarce.


So those data sets are not that big, even when we generate a lot of data.


And so training is not taking that much time.


So it's actually really fine to train from scratch at each iteration.


And you, so one second.


So I was, you say you have easy statements, you're able to find a lot of proofs for them,


you have hard statements, and that's difficult to reach.


But you still said at the beginning, all the statements you are attempting to prove, you


essentially already know that they're provable.


And even the ones in the curriculum, the ones you take from the textbook, I think textbooks,


they don't try to trick you with like exercises that ultimately don't really work out.


How would you, what would change here, if you were to go about proving something you


don't know if it's even provable, right?


Obviously also don't know the statements in between that might lead up to that, like how


would that look like to prove something that isn't proven yet?


Okay, so I think there's two questions there.


What would happen if you inject statements that are potentially false or even undecidable


in the mix?


And what would it take to try to prove something that we don't really know is provable yet?


I think so that's at least the way I understood the question.


If we inject statements that are not provable, that are false or undecidable, same difference


to us at least within the context of one formal system.


What happens is that nothing happens.


There's no data generated.


So you're just wasting computes.


You're really just wasting compute on those statements.


And that's going to be a challenge if we think back about automatizing the generation of


statements, that's going to be a noisy, imperfect process.


And so whether it's going to be useful for that expectation process is really a function


of the number of statements that are actually provable versus unprovable.


If your automated translation system generates one out of 20 statements that is provable


and 19 are unprovable, you're just going to be wasting a lot of computes trying to prove


something that's not going to generate any data for you.


So that's going to be a challenge there if we want to apply machine translation.


And then proving something, what do you mean by proving something that's not always provable?


Is it like trying to prove a conjecture?


You want to train or you want to solve a conjecture that exists but no one knows.


We think it's provable, which we do with most conjectures, but no one knows.


And now it's up to you and someone comes to you and say, well, let's use your system.


How would you go about that?


How would you build the curriculum?


What would change maybe in the data collection?


There are some conjectures that we can hope do not require inventing new math.


So there may be some conjecture that are eluding humans despite being very close to us.


It's just one trick away.


And so for such conjecture and imagining a system that is much more powerful than what


we have today, let's say it beats you in competitions, then you could just take your best system,


take the conjecture and search for a lot of time.


And you maybe have a hope of finding a proof that has eluded humans because it was really


tricky but you didn't need new theorems, you didn't need new definitions.


And for most of conjectures that are out there, there is good reason to believe, at least


if we look historically, that they're going to require new mathematical concepts to be


proved.


And so that exercise, which is the mathematician's exercise of defining new concepts, is something


that we are not even considering yet as a problem.


It's a whole different problem.


And to be honest, I think that it's a task that will probably more likely happen in the


future in the informal realm more than in the formal realm.


It feels like the informal realm seems to be a better space to try to come up with new


concepts and maybe then we have good at a formalization and then we can use a formal


prover to prove all the things that we conjectured, etc.


But that's something that is really far away from us.


You could sort of abuse the language models maybe to go a step, let's say further.


You always have your declaration and your goal and you generate the proof step.


Could you also maybe just input a declaration of a theorem name that you think, you know,


might conceivably exist and then let the system come up with a goal by itself even?


So like even the statement to be proven.


We've tried that.


It definitely works.


You can let the model generate goals that are valid and that can then prove.


You can even orient, we were talking about how do you orient your walk towards stuff


that interests you.


You can definitely in that case, you can definitely prompt the model where you're interested to


explore by the declaration name.


You can make up kind of funky names that look like analysis or funky names that look like


group theory or even funky names that look like math Olympiads.


The model will definitely and gladly conjecture statements.


It's actually conjecturing all the time in a way that is not leverageable, unfortunately,


when we do proof search.


When we do proof search, the way we refer to theorems that exist is by declaration name,


not by the statement themselves in Lean at least.


All the time, every proof search, the model will just invent serum by name and the name


looks really legit.


There should be math Olympiads actually because it's just a missing API because the name,


it's generally very interpretable what the model thinks should be there.


And so that kind of conjecturing behavior really exists in the model today and is probably


leverageable in interesting ways.


It's a bit crazy because that is really how I think mathematicians go about proving something.


It's like they say they're at some statement and they say, oh, well, here I need some inequality


that relates these two things to each other.


And essentially, that is exactly coming up with a name of a theorem like this.


The name would be something like in this greater than this or it's crazy.


So we actually can prove, we can extract from math lib what we call the type elaboration.


So type elaboration is to take a name of the serum and you infer the type and the type


is in type theory, the type is the statement itself.


And so we can train models on type elaboration.


So we could have them conjecture names while the proof search and then take the name and


try to type elaborate them.


That gives us a statement and then try to prove that statement.


That's something we haven't explored.


I mean, it sounds crazy.


And I'm, you know, given the directions of these systems, of these automated systems


that can essentially generate data from them for themselves, if you introduce something


like this, I'm pretty convinced this can get us a whole lot further.


I mean, how fast have these Go and Chess algorithms become?


They've become human and like one month later, they were totally superhuman.


Like it just, it happened like in an instant, which is crazy.


My question would be a little bit, this is a machine, the formal machine, you have the


humans on the other side.


Is there a good way of the two working together?


Like is there some sort of, because it seems like they have complimentary skills, one can


like search and try to prove things very quickly.


The other one maybe has more of that idea, like introducing new math and so on.


Is there a tight way where the two can work together or will it always be in the, well,


we have to translate sort of from one domain to the other?


So another definitely way, we actually released our early models.


It was almost a year ago to the lean community through a tactic that is called GPTF.


And so a formalizer could say GPTF and GPTF would answer with suggestions of things to


try.


And it's broken and clunky in many ways.


And there's a technical challenge, which is that the mass library advances every day.


It's the models you have to, are kind of easy to, they can rot quite rapidly.


For research purposes, it's very convenient for us to just say, for the next three months,


we're going to work on that commit and just not look at what's happening out there.


But yet if you want to provide value to the community, you have to stay fresh, which is


more of an engineering challenge than anything else.


But it's definitely a plan to provide our models to the community.


The most, and it's to be honest, that's the, I mean, anybody working on formal math and


ML, think about that.


That just makes sense, right?


Because formalization is so, it's not that hard, but it's time consuming.


And so if our models can speed up formalization by an order of magnitude, that would be just


tremendous.


And right there, there's already a very nice symbiosis, as you say, because if we speed


up formalization by 10X or by 2X, even by 2X, people will formalize much more stuff


and we'll get much more data and we'll get better.


And that's a loop that goes through actually people committing stuff to MATLAB and us injecting


it back eventually.


So it's kind of a long, a very long loop, but it's a loop that we plan to try to set


up for sure.


Yeah.


I mean, it would be, I think that would be sort of the best case outcome right here,


that there is like the symbiosis of just the machine helping the humans and so on before


it eventually will outperform them and make mathematicians useless.


Oh yeah.


So we're far away from that anyway.


Last, maybe last technical question from my side.


It seems like in such an iteration process, you said, for example, we can be easy statements,


we can find thousands of proofs for them and you do some deduplication to sort of reduce


the number of proofs.


If two proofs are equivalent, you take the shorter one, which is very sensible, but still,


how do you avoid that most data that you add back to the dataset is kind of useless?


Because given like three basic facts, a mathematician can probably prove 16 things, right?


And only very few of them are going to be valuable to advance towards my ultimate goals.


Like how do you make sure that what you add back to the dataset actually has some sort


of value to the expert iteration?


So the explosion of statements and proof that goes into a lot of noisy and uninteresting


stuff generally comes when you do forward proving.


If you do backward proving, you're really bounded by the statements you're trying to


prove.


So you might find thousands different proofs for something easy and all the thousands are,


they vary just because the model decided to name a variable differently and so they're


not that interesting.


And there we have much more work to do into having smarter deduplication.


But really in a sense, because, and that's the main advantage of working on formal maths,


because that data has been verified by the formal system, we know it's legit.


It's one key massive advantage that we have to do to explore interesting research ideas


compared to other domains is that we can lean on that verifier to really make sure that


we only use legit data, even if it's the model that generated it.


And I think that's key here.


And generally speaking, empirically, it's always felt like the training, basically gradient


descent is about compression and the training process is actually good at sifting through


repetitive and not necessarily repetitive, but somewhat similar data.


And so having a lot of different proofs is actually generally beneficial.


I guess the story of deep learning is that the more the better, whatever it is.


I've not gone too much into the results other than saying the expert iteration obviously


helps you to prove much harder statements compared to just the solver, whether you adjust


for compute or not.


It's also interesting that the larger models, whenever you scale up stuff, essentially you


get better.


Is there anything in the experimental results that maybe I haven't touched on that you would


like to highlight specifically?


Well I think you really covered it well.


One result that I think you almost touched on, one question and that is unanswered in


the paper is we do include the synthetic inequalities in the final experimental setup to target


mini F2F.


And actually I've run the ablation of that and they don't help that much on mini F2F.


It's not that surprising.


So if you remove them and plug the curves against mini F2F, you really get somewhat


sensibly similar stuff.


There is a few inequalities that have been solved that are challenging and it's always


a challenge because the graph tells you that it's roughly the same.


But then when you look at the proof, you feel like it's been learned through the curriculum


on synthetic inequalities.


So that's the reason why we kind of kept it here and I think it does unlock a few problems,


but it does, it's kind of a few problems at the margin.


So it's hard to make sure by just looking at averages.


And one interesting thing of course is as you say, you scale your compute, whether you


scale in model size or you scale in number of attempts and you scale in depths of search,


you always get better.


It really seems to be, and I mean it's true of most of recent deep learning, there really


seems to be performance being really a function of computes that you efficiently pour into


the model, pour into the system.


Though we've been very surprised many times that model size scaling is hard to leverage.


We know those larger models are so much smarter when you interact with them directly.


You ask questions to GPT-3, it's qualitatively better than GPT-2, right?


And here we are at the GPT-1 or GPT-2 kind of size.


And so common wisdom would say GPT-1 or 2, just dumb, right?


So why not use GPT-3 size because we're talking about math.


And really what we've seen empirically and it's probably and potentially because of bottlenecks


in our setup that we haven't yet correctly identified, is that you don't need to have


that big of a model to be efficient.


It's actually detrimental to scale the model size because then your proof search becomes


much more compute intensive.


And in terms of flops allocation, it's much more efficient to sample many more times from


a smaller models.


It tells something quite interesting.


It tells that the smaller model is basically not completely, it's not much less smart than


a larger model.


It's just that the distribution is not as crisp.


And here because we have the verifier and we can sample many times, we can juice the


good samples out of a small model by trying many times.


Yeah, maybe that becomes-


It's only because we have a verifier.


If you go to like more like really hard math statements, maybe at some point you really


need sort of the large models, but who knows.


Is there, yeah.


Was there, I'm a bit interested also in the process of the research itself.


Seeing a final paper is always really nice and cool and wow, you get to, your model does


all this thing.


Was there like particular low points during the research as well?


Like particular moments where you think, this isn't going to work out after all or things


like this?


Maybe you would like to share maybe so that other people, it helps to identify because


I think most people find themselves in spots like that.


Yes, definitely.


To be honest, I've been quite, I mean, we've been quite lucky with that project in the


sense that there's been some low points, but at any point of time, looking back three months,


in the past, we always felt like we had made good motivating progress over those three


months.


But it's obviously been a lot of struggles at many times.


I think research, at least the way I see it is a lot about struggling for quite some time


on some problems.


That's the reason why you really want to care about the problem you're working on to kind


of be able to go through that struggle.


It's actually the same as startups in a sense, you really have to care enough to be able


to go through the struggle.


To give you an idea, I started working alone.


There's no mutual people working on the project with me.


But when I started, I really took a language model and I took a data set of tactics that


I exported from, it was Metamask at the time.


Nobody had any idea whether a language model was capable of generating a tactic because


the syntax was so precise.


We were talking about interacting with the formal system.


There were no co-generation results at the time.


And so it really was an open question whether a language model is good enough to generate


formal, synthetically formal sentences, in a sense.


And so the first one was really that.


It's like not only you train your model and start sampling and you just look at your sequence


accuracy and you see that it's not zero.


And right there, it doesn't prove anything and it's far from being able to prove anything,


but it's a massive win.


You're like, yes, language models can generate kind of formal statements.


So that was really the start.


I think leading to the first paper, the first GPTF paper, the two key moments where, okay,


let's try to scale the model size and seeing that scaling is really beneficial.


It was kind of, it's not, as we discussed, kind of not as clear, but if you're just looking


at performance in terms of model size, you see that very nice scaling if you don't adjust


the compute basically.


And so that's something that is quite motivating and exciting because you know, it's kind of


the trend of the domain in many aspects.


And the key also finding of the first paper that was really a motivation to continue working


was that pre-training and you talked about that in the review and you had some questions,


but that pre-training really helps a lot and transfers very beneficially to formal math.


And it's kind of the bulk of that first paper.


And then after the first paper, you're like, oh, we have a nice result.


We found that language models can do some formal mathematics, but we were still completely


unable to prove Olympia's problems at all, even the really easy ones.


And so that's really what we started working on.


And there it's been a, it's been also a long struggle, I think, until we just decided to


bite the bullet and formalize some statements ourselves to generate that curriculum that


kind of really unlocks new capabilities and leads to the work that we've shared.


Is there anything about the paper that you want people to get away or to take away with?


Maybe you can look also a little bit beyond math, like, you know, what does this tell


us or anything you'd like people to know?


Yeah, I think so the main takeaway I think I want to share is why.


So we'll look at beyond math, but first it's why formal math is awesome.


And I think we covered that quite nicely, but to me, the main reasons is that it's reasoning


complete, right?


If you get a really impressive results in formal math, you're really confident that


you have a very impressive result in reasoning.


One of the other interesting aspects of it is that it's a inherently safe setup.


A lot of people are talking about safety and that's kind of a last harbor where we're not


yet at all at human level, yet it's safe to try to push as hard as you can because it's


like games, right?


You are embedded in a formal system.


There is no escape hatch.


And finally, the reason why I think it's so exciting is because it lets you combine a


language model with a formal verifier.


And so you're really getting the best of both worlds.


You have language models that are kind of really impressive into what they can generate,


but even GPT-3, if you give it a few deductive steps, it kind of falls off really rapidly.


And so they are capable of one step reasoning that are interesting, but not multi-step reasoning.


And so that's when you tie it with a verifier that you can basically get the value of multi-step


reasoning by interacting with the verifier that is here to verify the prediction.


And that's, I think, what is really exciting here.


The verifier kind of almost gives you the internal monologue that humans have when they


think.


It's hard to imagine a language model thinking hard during the duration of one context size,


right?


Yet here we do have that kind of property, which is exciting.


And finally, the reason why I'm super excited about it and goes beyond math in a sense,


I think, and that's the reason why it's really, I mean, OpenAI is really a great place to


work on that because it's really aligned with our mission and how we want to execute it.


The reason why is that I think if we crack formal math, we really will be providing a


blueprint on how to infuse much more reasoning in large, informal language models.


And so I really see it as kind of a small experimental lab where we can study reasoning


when we know that reasoning is kind of still lacking in those very large language models.


And so that's really that that excites me and I think it will transfer nicely.


You have formal math, you have code generation in the middle because you have unit tests,


but you can't beyond unit tests, you can't know for sure that your program is correct.


And then you have fully informal setups where you just cannot verify.


Well, I think that wraps it up pretty nicely.


Stan, thank you very much for being here.


This was really cool.


Expire-Span: Not All Memories are Created Equal: Learning to Forget by Expiring (Paper Explained)

Hello there. Today we're going to look at Not All Memories Are Created Equal, Learning


to Forget by Expiring and the system also known as ExpireSpan. It's by Sanbayar Sukbatar,


Da Ju, Spencer Poff, Stefan Roller, Arthur Slum, Jason Weston and Angela Funn of Facebook


AI Research and Luria. In this paper on a high level, the authors propose a modification


to the transformer attention mechanism that allows the systems potentially to include


much longer context spans. The way they do it is that they don't want to attend to all


of the context, but in an autoregressive way in each time step they want to decide is this


particular time step worth remembering or not? And if so, then for how long? So after


a while, these memories of the past expire, and then they are dropped and the system can


learn itself, which things are important to remember for the future, and which ones aren't.


So it has some good things, it has some limitations, it's very strong in tasks where you explicitly


have to remember individual things for a long period of time. So we'll dive into the system


right here. It's a pretty simple idea, I think, and it appears to work on the tasks that they


produce. So yeah, as always, if you like this, don't hesitate to share this out and tell


all your friends about it. I'm sure they are very, very interested. So they say the attention


mechanisms have shown promising results in sequence modeling tasks that require long


term memory, right? So the they say, however, not all content in the past is equally important


to remember, we propose expire span a method that learns to retain the most important information


and expire the irrelevant information. They say these forgetting of memories enables transformers


to scale to attend over 10s of 1000s of previous time steps efficiently, as not all states


from the previous time steps are preserved. So again, this is the core idea right here.


If you have a sequence model, like a transformer, and in this case, particular, we consider


a sort of autoregressive decoder only sequence model, which means that for the next token


to predict, like this one right here, we only care about the the past and not the future.


So this is a unidirectional sort of autoregressive style decoder. So every token can attend to


its past. Now, if you want to predict the fourth token right here in an attention mechanism,


you have to pay attention, so to say, to three things in the past, right? If you want to


predict the next token, the fifth token right here, you have to attend to this previous


one, but also all the other previous ones. So to four in the past, if you want to predict,


you see what's coming, right? If you, the more the longer your sequence gets, the more


things you need to attend to in the past, which gives us this traditional O of n squared


computation and memory requirements that attention mechanisms have. So if you get to very, very


long sequences, this can become a problem because you always need to attend to everything


in the past. So imagine this is whatever a sentence, the cat sat on the mat. Now, not


all words they say right here are equally important. So for example, it would be easy


if you wanted to predict this word right here, mat. It will be pretty easy to do so, even


if you don't remember that the word the is in front of here, right? The word, the word


sat here, sat on seems pretty important because you know, to sit on something is a good indication


that there is maybe a mat there or a chair or something like this, right? So these seem


to be worth remembering while the word the is maybe not as important. The word cat might


be semi important. And we would like a system that learns to sort of forget and remember


the correct words right here. If we only remember the more important pieces of information,


and we discard here in this case, these word the, then we also have one less thing to attend


to. And the goal is if we can get the number of important things down, then it won't be


n squared, but it will be something like O of n times m, where m is the size of the memory


that we have. This work here doesn't have an explicitly sized memory, rather, it does


the following, it goes over every element in the sequence. And every element in the


sequence, of course, gives you sort of goes through a bunch of layers gives you a prediction,


right? So here is a prediction. I misplaced this, let's go down a bit further here. So


every element in the sequence gives you first of all, a hidden state, right h here, this,


and it gives you a prediction like y, okay, so this is h one and y one, then you go to


the next element. And that with consideration, right attending this layer attends to the


last layer gives you h two. And from that it predicts y two, and so on. Let's do one


more. So in this layer, so in each layer, the sort of each layer, the future attends


to the past. And that gives you a prediction. And the attention is over these h right here


over these hidden state. Now, what this model does is it adds one component. In each time


step, it doesn't only predict the output of this particular time step, if there even is


an output, right, it also predicts this number they call E, and E is the expiration duration


of that particular memory. So E is produced every time from h. And E tells you how long


you should remember that particular h. So here, for example, h three also attends to


h one, I forgot to draw this in right here, right? Now, let's say that E one here is two,


okay, saying that this particular memory should be valid for two time steps, I'm not going


to need it longer than two time steps. Now, let's say the fourth. So the next sequence


tokens comes in h four. And h four is produced, of course, by attending to the past. But now


you want to attend to h three to h two. And because you want to attend to all of the past,


you want to attend to h one, but because this h one is already expired, you can't so the


the system would it would drop h one, you no longer can attend to h one. So this is


different from just a fixed window, right? If you have a sequence, what people previously


did was something like local attention, where you say, Okay, I have a window of like size


l, which is four. And if I predict this, this token right here, I can attend to the past


four things. If I then predict this one, I can attend to the past four things. If I predict


this one, I can attend to these past four things. So this here is different in the sense


that if you have a fixed window, again, everything is the same importance, but you just limit


how far you can look back, this works to an extent. But if there is something really important


right here, you will forget it no matter what. However, in expire span, this thing right


here can say, Well, I have an expiration date of 1 million billion, right? 1 million billion.


So for 1 million billion future time steps, things will be able to attend to that important


piece of information. However, you can say for the next thing, well, I only I expire


immediately. This is not worth remembering for the future. Okay. So I hope you got the


principle right here. They also have a drawing here, where you can see these hidden states


are produced. And these hidden states are produced naturally from forward propagating


through the model. And for each of these hidden states, one expiration date is produced. And


now in the future, when I want to produce the next hidden state, or, you know, the next


output of the next layer, I can look at the past, and I only consider the things where


the expiration date hasn't passed yet. So for anything else, like this one right here,


or this one right here, their expiration date was just too short. So this is a and only


only these go into the attention mechanism. So this is a dynamic way of saying how long


a memory should last. Now, you can immediately sort of see the weaknesses of this right here,


you have to know, at the beginning, like at the moment where you produce the signal, you


have to know for how long it's going to be valid. And that's certainly, that is certainly,


you know, the case for some things that you have to remember, like when you come across


a name in a story, that is maybe something that you know, okay, I'm going to remember


that piece of information very well, because probably it's going to be important, but not


for all right. So sometimes something big, something that you thought wasn't important,


maybe this thing right here, you you just you read it, it's in a sequence of texts,


you read that word. And, you know, it doesn't seem too important. But then all of a sudden,


because this word is something so you read on all of a sudden, that password becomes


super duper important, and you shouldn't forget it. And this is a these are effects that the


system cannot handle. The system can only decide at the moment where you consume the


token, how important is it? How for how long should I remember it? independent of what


happens in the future? You might already know a system that learns to remember things over


long pieces of time, which is the long short term memory cell or generally recurrent neural


networks that have an internal state and then at each point, they decide how to update that


state. So this here is sort of an in between between a transformer, which you cannot decide


at all how important things are and what you should remember, it's either you remember


all of it or part of it. And the LSTM on the other hand, that dynamically updates its internal


memory every single time step, right, so it can make remembering something dependent even


on the future. This Yeah, as I said, this, this is done for computational reasons, mostly,


because LSTMs you have to, you have to train one after the other, you have to back prop


through time here, you can still get away with a bit of parallelism, I think at least,


though I would argue, if I could extend this, I would argue that if you consider the point


where something expires, I would maybe build in something where the system can decide to


re retake this into memory, or, you know, like that's such that the system can revise its


own predictions about how important each of the memories are. And if you look at this


in in a, let's say a computational point, they base their work of transformer XL. So


transformer XL is sort of the baseline right here, what transformer XL does is it has long


sequences, and then it considers blocks of those sequences, and they do the same here.


So you just you chunk these sequences into different blocks. Okay. Now for each of the


elements here, you output a vector, which is that this hidden state. Now what transformer


XL does is it does the attention in block one, just as it would do regularly, and then


in block two, and then in block three, so it chunks the sequence and handles the blocks


individually. However, in block two, in order to, you know, look back, because we always


want to look back, we want to remember things, what you do is you put the hidden states that


you produced in block one, you sort of put them into like a little bit of a of a register,


I would say so you put them into. So these are the vectors, I just lay them on their


side, right? These are the vectors, and you put them just there. There is a sort of a


stop gradient right here. But you just, you just kind of put them to make them available


for the next block. So what the next block can do, when you want to predict, for example,


the hidden state of this thing, it can attend to, obviously, to the sequence elements in


its own block, right, because you consider the block as a whole, but it can also attend


to these things right here. And, again, you produce that hidden state ultimately from


it, and from it every element in that block, and those go then to be available for the


next block to attend to. And you can even remember multiple blocks like this. So you


can sort of carry forward this block as well, right, and now block three can attend to the


last two blocks. However, you can't do this infinitely, right? Otherwise, you're going


to run into the same problems. But at least this handles a bit of the the backprop issues.


And also, these things right here, they cannot attend to each other, right, there is no need


for them to attend to each other. So you don't have n squared, you have n times whatever


that here. So if this is m, and this here is n, you have O of n times n plus m. No.


Sorry. Yeah, but n is way smaller. So it's n squared, but n is way smaller, n isn't the


whole sequence length. I'm maybe b, let's call this b, the block size, right. And this


here, at maximum is n. So you have a way smaller sort of way smaller quadratic blow up only


inside the block. And you can even compress these memories here of transformer XL, you


can max pool, you can learn to compress them, and so on. So this is the system that they


base off of, right? They also consider sequences in these blocks where inside the block, it's


just regular attention. And then you can attend to the past as you would in transformer XL,


except that some of these past memories, they are forgotten. So here, these are maybe forgotten.


And maybe this one is forgotten too, until you are here, right. And then during that


time, you know, one more expired. So you can see there is a lot less stuff around. So you


get away with having a smaller memory. And you can potentially up the time that you can


look back into the past. If you only have a limited set of slots available here, you


know, you can increase that. So that's, I hope that is a bit clear how they do it, they


go block by block. And in each block, they look back. And they build this, this memory,


right here. So this, this memory here, that inside the next block, they can also attend


to. But in the memory other than transformer XL, they only consider things that have not


expired yet. And the expiration is determined at the moment where the signal where the hidden


state is produced. In fact, the expiration here is pretty simple. So you take that hidden


state that's produced by the network, and you simply perform a logistic regression on


top of it. So the logistic regression here will give you something in the range zero


to one, and you multiply that by L and L is the maximum possible length of remembering,


right. Now, these are all, you know, design choices, you know, that the, the sigmoid function


here used in logistic regression is a rather, let's say, rather steep function. So there


is a region where you sort of go up quite quickly. But there are also large regions


where it's just all or nothing, right. So I get I'm going to guess that this function


here will be either remember this or don't remember this, maybe there will be some in


the middle, but which tells me that this L setting right here might be fairly important


that you tune that for the task that you want to consider. Another thing they say is, okay,


how do we actually implement this and they implement this via a mask, like if you have


a bunch of things that you could attend to, right. The way that you don't attend to everything


is by masking out attention, attention parameters, essentially, or elements of that map. So if


I draw the same sequence twice, the attention matrix is of course constructed by outer product


of keys and queries. Right. So here is the attention matrix, every cell gets a value


of how much this x here attends to this y. And as you know, that already in these decoder


things, we need a mask because this thing here cannot attend to this thing here, this


thing here would be like this thing here, so it cannot attend. So all the upper triangular


thing right here is already dark. Well, okay, I can't draw, but we usually implement this


with a mask, right, because GPUs aren't super good at doing triagonal matrices. So we just


put a mask here and we say everything up here is off limits. Okay. Now, if we also say,


well, this, let's say this thing here has an expiration date of two, which means that


this can still attend to it, this can still attend to it, but this here cannot attend


to it. So what we need to do is, well, I might have drawn this slightly weird, but let's


say that is this, nah, it's not correct, but you go to that cell and you also mask that


out. You say you cannot attend to anything that's expired. So what you end up with is


sort of this mask where you fill in, yeah, I think after that it should all be black,


right? Where at some point the, the, the row will just be masked out from then on. So the


light squares here have a value of one and the dark squares value of zero, meaning that


you don't consider these things in the attention anymore. That's how it's implemented. If you


just do that, then you have a problem on your hand. Okay. Because this is not differentiable.


Simply putting the masking whether or not this R number, R is, is the thing still valid?


You see it's constructed from E, which is the expiration duration and the T, which is


the current time step and I, which is that I from the E. So you look back and say, is


this thing still valid? And this number, if it's positive, it's still valid. If it's negative,


it's no longer valid. If this becomes negative, it indicates the memory is expired and can


be removed from the set you attend to. So you construct a mask with just everything,


all the Rs that are positive, and use that mask in the attention like you already do


with the masking out future tokens. This is not differentiable. Okay. However, they say


with such discrete masking, the x bar span will not receive any gradient for training.


Instead we use a soft masking function that smoothly transitions from zero to one. And


this is what you can see right here. So essentially how this works is here is a memory produces


a hidden state and it says I am valid for three steps, three steps. So that means that


the mask here, how does the mask look? The mask for this particular thing looks as follows.


So here is zero and here is one. The mask, okay, well, yeah, the mask starts at one for


one, two, three, and then it drops off linearly until it's at zero. You can see this right


here. So here's the min of one, which means that it can never be higher than one, the


max of zero, which means that it cannot be lower than zero. And then in between it's


governed by this rule right here, which you can see R is a hyper parameter saying that


like a ramp drop off. Yeah, the length of a ramp that is bounded between zero and one.


And the higher this R is, if it's negative, then we're in this decreasing regime. So this


is the mask. Now you can also immediately see that talking about gradients, right? The


only place where the module that generates E, right, this is a, we generate this here,


the hidden state goes into a neural network, neural network, and that generates this expiration


date. The only place where that neural network gets a learning signal, gets a gradient is


during this drop off. No, not before, not after. The only time where this network gets


any learning signal at all is during this thing. So it is quite important, these parameters,


right? This, this here, this is upper bounded by the parameter L. And then this thing right


here is modulated by the parameter R. So these hyper parameters, I feel, have are quite important


to how this task is going to play out, if you actually want to learn anything. Because


let's say in a sequence, here is something that you need to remember, but you need to


remember it for here. If the L is too short, right, you will maximally remember it till


here, and then it's gone. Even if the L is large enough, right, then you won't get any


training signal for this, unless sort of the let's say the L, the L is large enough. So


this is your expiring span. And then it it sort of drops off the importance drops off.


And only if that drop off happens to coincide with, you know, the thing where it's important,


you do get a learning signal at a hey, maybe you should remember that thing for longer


next time, because I'm going to need it. If that is not the case, if your expiration prediction


is like this, and your drop off is done here, then you will never get a learning signal


that, hey, there might be something here, where you should remember this thing. This


is the I mean, it's the same problem you get anywhere where you're dealing with long sequences.


And it is it is it is a problem. Because ultimately, if you want to have a general training method,


where anywhere in the future, that could be something important, you have to you're going


to have sort of this quadratic, this quadratic thing, where you technically have to attend


to all the things in the past, even a little bit, because you want to make it differentiable,


because you want to learn to remember, right? If you always forget, and then there is something


here, you don't know anymore that there was something to remember, you'd somehow need


a learning signal, I guess, you could break this, maybe you could break this down into


maybe not n squared, but maybe like n log n, where you sort of build up a tree of the


past, and then you somehow realize that, okay, there is something to remember, you don't


maybe don't know what, but maybe there is something to remember. This might have been


done already. In any case, I just wanted to show you that the learning signal here is


very small, like that the window where you can learn something is very small. And that


means that kind of tasks that can be applied to, or maybe not as much as many as you would


hope. What they also do is they put an L one penalty, so an L one penalty on to these expiration


things. So they encourage the network to rather forget things. This is in order to keep the


just the predictions small, you don't want the network, you don't want the network by


default to say, well, none of this is important. And only if you get a learning signal that


something is important, then the network should predict high numbers. So ultimately, you're


going to have a sequence, right, I'm going to draw it like this, this time, and the network


will predict various spans to expire these memories. And the first thing you do is you'll


say, okay, everyone just kind of, you know, kind of go down and go down and go down, go


down. And then if, let's say, this thing right here, really profits from this thing right


here in the sequence, then and if, if this has been going down enough such that the later


one is in this ramp portion, this, this, this our portion of the former one, then you get


a learning signal saying, hey, maybe you should remember that thing for longer, right. And


then hopefully, hopefully, some next thing right here will also benefit from remembering


this thing. And now that is in this span, sorry, in this ramp region, which will give


here another boost to remember it for longer. So this is how you learn, you sort of need


a continuous reinforcing signal over different time steps in order to learn you the this


long range thing, it's it's, I don't think that generally is learnable with this system,


you need these intermediate things, or you need some kind of randomness to discover it.


And this is very close, right to reinforcement learning now. All right, and that, yeah, so


that's what they do here. They also, they have some practical considerations, where


they say, okay, because we, we cache these things, like the question is, how do you back


prop, how do you even back propagate through something like this, I said, there was a stop


gradient, right here, what you do is you cache the H, you cache these things. And then as


far as I understand, you do compute the attention, like the expiration things on the fly, like


you cache the hidden states, and then you compute the should you mask them or not, you


compute that thing on the fly. And so you can back propagate that you can back propagate


to these variables, even in the future, because you have the H's cache, I don't think the


back prop flows back to when the hidden states were produced, because we can't, right, because


you cache it, you don't have the graph available anymore. So they have a bunch of practical


considerations right here. And now they test this. So they test this in various tasks.


For example, there are these reinforcement learning tasks, there are these text instruction


tasks, there is character level language modeling, collision detection, where you have a video,


you go frame by frame. So these tasks, I guess, except the language modeling tasks, are quite


constructed such that you have to remember long things. Particularly interesting, for


example, is this one right here, where they do have this character level language model,


and then they look at what does it learn to remember. And you can see right here, if the


sentence is powerful influence in Egypt, right. And they say this, the model strongly memorizes


the two areas, Egypt, and Alexander. So if you look Egypt, right here, and this is the


visualization of the expiration time, this is strongly remembered, if you replace in


the same model, you just replace this with the word somewhere, all of a sudden, the model


doesn't remember it anymore. And if you replace it with Humpty Dumpty, again, the model remembers


it quite well. So this is an indication that the model has in fact learned that, you know,


if there is something special, and they claim if it's a name, if it's a name, or something


like this, the model remembers it well, they also say the rare words remembers those in


memory. And I'm asking myself, is this just a function of let's say, complexity, sorry,


perplexity? Like, could you just remember the things where the model perplexity is pretty


high instead of learning what to remember? Right? So you just remember sort of the things


that you would not have predicted. I'm going to guess the learned remembering is better,


just because it's learned. So it can also remember things that have a low, like that


have a big probability, but might still be important. I want to talk just a little bit


about this first task right here to show you the kind of tasks where this could be good


at. So here you have a grid world reinforcement learning approach. And you're at the start,


you were able to observe the colors of the fields you're on, right? So you're at the


start right here. And this is either a blue or red. And then what you need to do is you


need to walk all the way through this long corridor. And then you need to go to the correct


door. And the correct door is whichever one was, you know, the color was at the beginning.


And the long corridor is made such that it is too long to be in the same block, right


is too long to consider in one attention operation at the same time. And this model, they say


it learns to remember the correct thing with very little effort. So here you can see the


the comparison to transformer XL. So transformer XL also has the ability to remember that right,


it can simply attend to this thing in in the past, if given enough memory. So here you


have the memory size. And you can see it starts out by being just kind of random, because


it doesn't remember it like the memory size is too small to actually remember. And as


you give it more and more memory, it learns to attend to the correct thing in that memory.


However, expire span, it doesn't have a set memory, right? You can with the L1 penalty,


you can sort of modulate how long it forgets things. But these here are just five random


samples, I guess, of the same model. And you can see that it solves the task pretty well,


while its effective memory size, if you calculate, like if you look at, you know, what what things


you do remember, stays relatively low. So it learns to remember this correct thing right


here, which is pretty cool. However, this, there's details of how this task was constructed,


I already said, if it's just a long thing, then then we, this is like, if this was just


a long corridor, this was unlearnable. So if you look at the details here, in the appendix,


where is it? Yeah, the quarter task, the corridor length is sampled from between three and 200.


So and for the expire span, we set the maximum span to 200. So it's, it's able to remember


which, again, this L seems to be an important hyper parameter and the ramp length to 16.


So what does this mean, right? If, if you have a, let's say a, I don't even know how


many things they consider at the moment, like what's their, their block length, I'm sure


that's stated somewhere. Okay, but in this corridor task, reinforcement learning problem,


right? If you sample things that are just 200 apart, right? I guess you you can learn


because your L is 200, right? But your predictions, yeah, they, if they are too short, then you


never learn to get up there. And if they're too long, okay, you have the L1 penalty, which


makes them shorter and shorter and shorter, and eventually come into the field of learning.


But here, you sample at random, you so sometimes it's three, and sometimes it's 200. And sometimes


it's here, and sometimes it's here. So you give, you give the model a really nice training


signal where, however, wherever it currently has learned, for however long it currently


has learned to remember things, there's going to be this ramp. And there's going to be some


training runs where the length of the corridor exactly falls into this ramp. And that will


give it a training signal saying, hey, you maybe should remember that thing for longer.


Okay, for longer, and the ramp is here. And then there will be some kind of problem that


exactly falls into this ramp, right? So as in reinforcement learning, you, it is best,


I'm going to argue, if you sort of, if your loss structure guides the model to remember


things for longer, of course, this doesn't work in the character level modeling. But


there, I, I think the text is naturally structured such that if it's something important to remember,


you will find instances where that comes after 10 tokens, and you will find instances where


the need to remember comes after 20 and 50 and 100, and so on. So yeah, not for every


task, but certainly for many tasks, this might be a good solution. Again, I would advocate


to add the ability of the model to refresh these memories, not full LSTM style, so not


internally compute and update an internal state or something, but just to go there and


say, well, in the light of this new evidence, this thing right here that I want wanted to


forget now, it might still be quite important, right? So that would be my first extension.


And my second extension would be, instead of building some sort of a bank right here


that you can attend to, maybe you build some sort of a tree, like some some kind of a Merkel


tree ish thing, but not with hashes, but with with hidden latent variables, I'm sure maybe


this has already been done. Okay, that was my two cents to this paper. I think it's a


pretty cool paper. If you have problems that have super long sequences, and you have a


clear structure where it's important to remember key pieces of information, a few key pieces


of information over long distances. And if that is, if those distances are somehow distributed


a bit such that it's not only super long distances, this might work wonders. So tell me what you


Fastformer: Additive Attention Can Be All You Need (Machine Learning Research Paper Explained)

Hello there, today we'll look at fast former additive attention can be all you need by


Chu Wanwu, Fang Zhaowu, Tao Qi and Yongfen Huang. So this paper definitely wins out in the category


of most innovative paper titles of the last few months. As apparently, we've gone from is all you


need to can be all you need. So a big win on this front, as you might have guessed from this title,


the paper is introducing a new kind of attention mechanism. If you don't know what an attention


mechanism is, and you're in machine learning, you might want to find out I have a paper video on


attention is all you need. So the new attention here is additive attention, which is supposed to


be a much, much, much faster way of doing attention, thus the name fast former. This additive


attention circumvents this quadratic bottleneck that we usually have in the attention mechanism.


Instead of doing sort of multiplicative attention, they do what they call additive attention. Now,


the naming, in my opinion, is a bit confusing. And the whole concept is a bit confusing. So on


a high level, that's what they do to design a new attention mechanism. My opinion of the paper is


that it's kind of deceptively naming things to make it appear like it's an attention mechanism,


where in reality, it seems to be sort of just sort of a feed forward ish layer type of thing


that they propose, maybe not even so, you know, we'll go into that. Their promises are that,


of course, circumventing this quadratic bottleneck of attention, you can input much


longer sequences into the context of a transformer. And you can do it also much faster for the same


length of sequences, since everything is just additive and not multiplicative. We're gonna find


that out, they claim they have a lot of experimental evidence. And yeah, if you like


content like this, you know, don't hesitate to subscribe if you haven't done so already. So the


abstract reads, transformer are very powerful, okay. However, the attention make mechanism is


inefficient due to the quadratic complexity to input sequence length. They say although there


are many methods on transformer acceleration, they are still either inefficient on long sequences or


not effective enough by effective, I guess, they mean that their performance suffers too much.


So they say they propose fast former and efficient transformer model based on additive


attention. So instead of modeling the pairwise interactions between tokens, which is what


attention does, we first use additive attention mechanism to model global contexts and then


further transform each token representation based on its interaction with the global context


representations. Now, if this sounds confusing to you, it does so to me too, they go a little


bit into more detail right here, they say they have this additive attention, which is linear


complexity instead of quadratic as in usual transformers. So here is a bit more detail,


we use additive attention to summarize the input attention query matrix into a global query vector.


Then we model the interaction between the attention key and the global query vector via


element wise product to learn the global context aware key matrix, we further summarize it into a


global key vector via additive attention. Then we use element wise product to aggregate the global


key and attention value, which are further processed by a linear transformation to compute


the global context aware attention value. Finally, we add together the original attention query and


the global context aware attention value to form the final output. You know, still after this


paragraph doesn't make too much sense to me to understand. So we'll go to the diagram in just


one second. But here is essentially what they promise. Okay, they propose an additive attention


based transformer named fast former to our knowledge, fast former is the most efficient


transformer architecture. So that's one they propose the most efficient transformer architecture.


Second, we propose to model the interaction between global context and token representations


via element wise product, which can help fully model context information in an efficient way.


Okay, so they, the element wise product seems to be the second component. So there's additive


attention, there is element wise product. And then lastly, they say, you know, our experimental


data sets valid validate our approach. Alright, so here is the coded diagram of the fast former,


it's a little bit complicated. But I want to go back a little bit to the regular attention


mechanism. I know I've done this a lot. But I think in this context, it is really worth discussing.


So in a regular attention mechanism, what do you have, you have some sort of an input sequence,


each one of these things can be a vector, some sort of an embedding vector or something like this. But


it's a sequence, essentially, it's a set. But we think of it as a sequence of, let's say tokens


in natural language. And we want to transform the sequence of one layer into a sequence of equal


length of the next layer. So if we stack many of these layers together, we sort of want to improve


the representations of these tokens layer by layer by layer, such that we can at the end of the


transformer understand what each token means in the context of all other tokens. So if this is a


sentence, my house is very green, then at the at the beginning, each word is just an isolated piece


of data. At the end of these transformations, we want sort of all the tokens to be aware of all the


other tokens in the input, and sort of capture their in context meaning. Now, what we need to do


is we need to transform one set of representations into the next one. The way we do this is by the


attention mechanism. So the attention mechanism, essentially, from each of the tokens, it derives


three different things. One is called a key. So the key is a vector. So the key is a vector


for each token. And that vector describes kind of like what the content of this token is so far,


okay, so one vector is the key, which allows the token to advertise what it has to offer. The other


one is the query, which allows each token and that's also derived from the same token, but I'm


going to draw it up here. The query means what does this token want to know about the other tokens


in the sequence. So this can be different from its content. So as you see the query and the key,


they might be different, there are variants where this the same, but usually you derive two different


values from each token. And then what we do is we route by inner product. So you for every single


query, you aggregate across the entire input sequence sequence, you aggregate by inner product,


which means that this would get routed here by a lot, this one may be two, these ones, not so much,


so on. So you aggregate essentially the inner product, which for each query gives you a


histogram, a histogram across the sequence saying, okay, this information here is mildly relevant,


this one is more relevant, this one is slightly relevant, these ones aren't relevant at all for


me, this histogram, you then normalize via a softmax operation. And that gives you, I mean,


that gives you a real distribution over the input. So with the query and the key, you decide


how you want to aggregate the information in the input sequence for one particular element


in the output sequence. And you do this for every element. So for every element, you get a


distribution of how you want to aggregate. And then in the last step, every single item also


emits what's called a value. And the value is yet another vector. And the value, I guess you don't


even have to actually transform anything, the value, you can just take the information itself


of the token if you want. But essentially, the value is ultimately what you multiply together


with this distribution. And then that becomes your next layer representation for this particular


token. Right? So the whole query key attention mechanism is simply to decide how do I want to


aggregate the the different values of the input sequence for any given token in the next layer.


All right. Okay, I hope this is clear. So the the query, the key advertises what the contents are,


which is kind of like the value, the value is the actual contents. But the key is more like an


addressable representation of the content. And the query emits what do I want to know about the


others. So you won't match the queries of myself with the key of the others and that aggregates.


Now, in that context, let's look at the fast former. So we said there are two elements there


is, first of all, there is this additive attention. And that's what you can see kind of down here. So


you see there's the input, and the input gets transformed into three different things into


queries, keys and values. That is just like a regular attention mechanism. These are linear


transformations that each token independently goes through. So this token independently produces


this, this query, this key and this value. And with the same transformation, this token produces


this query, this key, and these this value. So there's no interaction, every token goes through


the same transformation, then you can see instead of now considering the interactions between each


of the queries and each of the keys, sorry, that should probably be up here. Instead of considering


this interaction, we don't do that. What we do first is we say, well, this really becomes quadratic


if we do if we consider interaction between each query and each key. Therefore, let's simply


construct one global query, okay, one global query. And then we consider the interaction of that


global query with each of the keys instead of instead of doing everything with everything. So


here is you work here, you can see how the linearness instead of the quadraticness of this


approach comes to be, instead of considering pairwise interactions, we simply construct a


single query vector. By the way, this is all this is one head. So this is one head. Usually a


transformer has multiple heads. So over here, you would have like, head number two, and so on head


number three, head number four, but in a single head, we make one query vector. Yeah, and you


immediately see what the shortcomings are here. Whereas previously, every token could sort of


dynamically decide how it wants to aggregate information and every token could do that,


you know, in a in a sort of by itself. Now, it's only the sequence as a whole that gets to decide


how it wants to aggregate information because it needs to come up with a combined query vector.


So I'm going to guess this thing here works might work quite well for tasks that have sort of a


single single minded output sort of topic classification or something like this where


you simply, you know, the global information is necessary, usually, whereas tasks that might be


more, you know, nuanced and language relevant, like considering specific interactions between


individual tokens, and so on, those might fall a lot short in this approach. Okay, but how how does


this single query vector come to be? Now, this single query vector is constructed purely, as you


can see from the queries of the individual token elements. How there's this funny construction here,


where you have you can see this is the query vector right here. And then it itself goes here


and here. So it's used twice. Okay, so we what we do is we construct this alpha value for each


query vector, and then we multiply that alpha value by the query vector itself. And then we


add this is an addition here, we add all together at the end. So essentially, this query vector here,


the global one is a weighted sum across all of the individual query vectors. Now the way that


now the question is, you know, how do we decide decide on the weight? And that's where these


alpha values come in. So let's see, here is the formulas for the alpha value. So each query vector


qi will produce the its own alpha i, how is that computed? As you can see right here,


this should be familiar to you. This is the softmax formula. So what we do is we


it's also the formula for logistic regression, if you squint a little bit. So essentially,


the alpha i's are the result of a softmax operation across the queries. So you have query one,


query two, query three, right? It's a softmax across not the queries itself, but this quantity


right here, the query multiplied by some sort of a transformation. And this now really looks


like logistic regression. This w here is a vector that is learned, this is a learned parameter


vector, right? I take the inner product with each of the queries. And that gives me like a number,


like a number, right? And then what I do is I simply normalize this by all the numbers of all


the queries, okay? So every one of these gets multiplied by this w, which gives me one number,


and then I simply normalize, I push it through the exponential function, then I normalize it.


This is essentially a logistic regression with the w being the feature vector.


Now, what does it mean? What does this mean? Okay, like we construct the final query vector


as an aggregate across all query vectors with the weightings being dependent on like a softmax or


logistic regression with respect to this learned vector w, this is always the same, right for for


every one of those queries. I can make sense of that if I think okay, this is the w here is


essentially, you know, in logistic regression, you classify so the w vector me is the sort of


the classification boundary of, you know, the one class versus the other class. So this here,


I think is essentially a little classifier that cares about one particular thing that is learned.


So this can be some intermediate feature that is useful that is learned via back propagation


in this w vector. And the the weighting of this particular head in this particular layer is then


according to that feature. So in here, there is somewhere there is a w vector. And that w vector


in this particular layer for this particular head refers to some kind of useful feature, like,


I don't know, like, is there a name of a country somewhere in the sentence? Okay. And that's what


we use as a weight to aggregate the queries. So you can immediately see that if a term if a,


you know, a token, it's if it's query sort of contains a country information, this classifier


would, you know, say, well, that particular query has a lot of the information that I am


particularly look for in this layer, therefore, the inner product will be high, therefore,


the alpha will be high, therefore, that particular query would be represented greatly in the global


query vector. So the global query vector, essentially, you can think of, I select among


all the query vectors, the ones that I care about in this particular layer in this particular head.


However, what you care about in this layer in this head is static, it's statically learned,


it's the same for every single sample. Okay. Alright, so this is sort of a weighing by


particular feature. Now, once we have the global query vector right here, how do we let it interact


with the key vector? So usually what we do is we do an inner product of the query and the key.


And then that defines sort of our aggregation distribution. However, since we only have a single


query, you know, that will not give us that will in fact not give us an, an n dimensional seek,


sorry, an n length sequence as here, that will only give us a sequence of length one in the


next layer. So we can't really do that. So what they do is they almost do an inner product,


except they don't sum, right? They do simply element wise multiplications of the queries


and the keys. Now element wise multiplication, it kind of means so it means, you know, like,


the element wise multiplication, if you think of it, if both elements are small, the result is very


small. If and if both are high, the result is very high. So there's some nonlinear dynamics going on


within the same dimension, right? There's no aggregation across dimensions. And, yeah, so


they do element wise multiplication right here in order to obtain these p vectors and the p vectors,


they are now the integration, it every p vector, p vector, so pi is equal to the element wise


multiplication of the i of key vector with the global query vector. Okay, so yeah, and the query,


the query vector itself is of course, a sum across a weighted sum across all of the queries. So if I


pull the K in, you can see that I still have k alpha j, I still have this quadratic thing here,


I still have for you know, I get I have n p vectors. And for each one, I have also n q vectors,


and I consider products of the form i j. So I still have the quadratic products in here. However,


I don't have quadratic complexity. Why? Because I don't have a softmax in between aggregating


the queries and aggregating the keys. And therefore, you know, the what is the commutative


associative rule applies, and I can simply get away with first aggregating the query and then


multiplying it as a whole by the keys. Now, of course, that are those are two linear operations


in sequence, whereas in the normal attention mechanism, I have a linear operation, then a


nonlinear one with the softmax, and then again, a linear one. And arguably, the nonlinearities


is what brings the whole power to deep learning. So, you know, this essentially, here, you can see


how it really circumvents the quadratic bottlenecks by simply saying, Well, if everything's linear,


linear, then there, you know, we can we can just add all together. Yeah, that's the trick,


essentially. Now then, you realize we're not done yet. Okay, what do we do with the p vectors? Well,


this seems familiar, right? Again, we do another one of these additive attentions. So they call


this thing additive attention, you can see from each p one, we produce a beta value, the beta


value exactly the same way as the alpha values, I suppose, at least Yes, you can see that right


here, right, the beta is exactly the same for each p, we multiply it by a learned feature vector,


which is w k right here. And then we normalize by all of them. And you know, after the exponential


function, and then we aggregate the global key via again, a weighted sum of all of these p vectors,


so this is again, additive attention in order in order to have a global key vector. And now,


exactly the same trick, we use the global key vector element wise multiplied by the value


vectors, which gives us these u vectors right here, that these apparently go through another


linear transformation to give us the r vectors. You know, you can, you can stack as many linear


transformations as you want. And then we're still not done, right? We're still not done. So


essentially, what we've done, in the end is we we take the values, which is the information we want


to forward propagate. And for each value, we element wise multiply it with this k vector.


And this k vector is a result of the keys and also this query vector. And that's a result of the


the queues. So essentially, there is no aggregation of information as is there in the


regular transformer, I don't aggregate the values from the sequence in a weighted fashion, I simply


leave each value as it is, you know, these are, as I said, these are transformations that don't


depend on the other sequence elements. So v one purely depends on e one. And the only way the only


way that token information from the other tokens can come into any token is via this aggregation


methods right here. And in that in the normalization constant, right in in the aggregation


that happens via the normalization, you know, for example, the key n could be represented more in


this global key, and then that's multiplied here to my vector one. So that's how other information


comes into any particular token. And as I said, we're still not done. After we obtain these r


vectors, we then add to them this thing right here, we add to them the query vectors again.


Now why? I don't know. But we just do. So we simply add the query vectors to the our vectors


that we have here. And that's going to be our final output. So this is stupidly complex. And


I don't think for any particular reason. So there are multiple problems right here. For example,


this transformation right here is a linear transformation. Okay, maybe it makes sense,


but it seems like you just had a linear transformation here. And this whole sum here


is sort of a linear aggregation. Ergo, yeah, okay, maybe you can justify that. But second of all,


this connection right here, right? If this is not ablated in experiment, like I don't believe squat


here. Like, I want to know how much this is clearly not something you do from the beginning,


this is clearly something you add after the other stuff don't doesn't work. So I want to see an


experiment where this connection is missing, and to decide and I want to see an experiment where


only this connection happens to decide, you know, where the actual work is going here. Then another


thing, you can see this here, the middle column is entirely useless, like, like this, this right here,


it's simply it simply the lower part is a repetition from sorry, the upper part here is a


repetition from the left. So these two things are repeating. And then the lower part is repeated


here, right? In fact, you can stack as many of these columns, they just call them query key and


value. Well, if I just call them column one, column two, and here, this, this is like the final


column fine f c f, right? I can, in fact, insert column three, column four, column five, I can


insert as many as I want, because it's just repeated, right? That there's no qualitative


difference that differentiates the queries from the keys in this model, right? Only the values are


a bit different, because at the end, they're not aggregated into this global vector with this


additive attention thing. But in essence, you know, you could do away completely with, for example,


with the key column and directly do the query multiplying them into the values completely


possible. So completely unnecessary key column. Now, you might think, okay, if the key column is


unnecessary, or if I can introduce 50 keys in between 50 key columns that always take the last


last whatever global vector and multiply it in and do additive attention? Is this really an


attention mechanism? And the answer is, kind of, but not in the way you expect. It's a bit sneaky,


honestly. See, attention is when I have, well, arguably, right, who am I to define this, but


arguably attention is when I create one of these things in a dynamic way. They and these things are


how do I aggregate information? How do I weigh information from an input sequence? Okay, that is,


in essence, an attention mechanism dynamically creating this waiting. So the only way this


actually really happens right here is where we're in this w thing, right. So this here is, in fact,


the attention mechanism, not the not the not this, this is just a weighted sum. Like, this here is


the hidden attention mechanism with it's essentially a self attention mechanism, right,


you can you can see. So the alpha i's are how do we aggregate information? And then, okay, I guess,


yeah, this belongs to the attention mechanism. But the keys and the queries, sorry, the keys and the


values are both what they call q, right? What I aggregate here, those are essentially the values,


the things to be addressed, these are essentially the keys. So the query is essentially this thing


right here, that's, that's the query. Now, the query, as you can see, is not dynamic, the query


is just statically learned, which makes this essentially into a, like a feed forward network,


or at best, an attention mechanism with a single learned query. So instead of having n queries,


now we have one query per head. And that's why I said the thing at the very beginning, if if


this is applied to a task that largely relies on, you know, single minded task, global global


information task, and so on, such as sequence classification, or something like this, it can be


that I only need a couple of intermediate really different features per layer after all, they are


vector valued. So which means that if I have eight heads, which have eight different w vectors, and


you know, there are two w vectors per layer, to be fair, there is a w here. And there's also a w


again, in this thing right here. So every column gives me essentially a new feature to extract,


right? So the number of heads times the number of these columns I have is essentially the number of


features I can have static features I can extract from such a sequence. And as I said, for global


information tasks, that might in fact be enough. And in that case, you know, good, I can I can get


around. However, I could have done the same thing, probably by Yeah, but by simply constructing less


queries than keys and reducing the sequence length or something like this. I mean, there are,


there are many ways of this. But I, I think the thing here is framed in terms of the words of an


attention mechanism, where the actual attention mechanism is simply like the thing here that


happens inside the queries, it's essentially a self attention mechanism on top of the queries


with not a dynamic, but one single fixed query. The same goes for column two, and then column


three is just kind of like, weird, like, it's kind of a weird residual connection, or something


where there's this this product here with something that's incoming. It's kind of like


a feed forward layer again, like a dynamic feed forward layer per token. Yeah. So, yes, that's,


that's why I find the name a bit deceptive right here also to formulate as query key and value


here and let and their whole talk about who we model the interaction between something,


something, something. Yeah. Okay. But what about experiments? Their experiments, I find to be


relatively lacking, they do have a lot of baseline comparisons, which is respectable. Their data sets,


however, appear to be Yeah, things like sentiment classification, topic classification tasks.


And it you know, they do perform well, I, you know, experimental results are experimental results.


And then, you know, the best numbers are achieved by ensembles, which is which is also fine, right?


But even the regular numbers right here appear to be quite competitive. So I don't exactly know.


Yeah, the complexity right here is also a bit shaky, because they sort of leave away the linear


operations and so on like Yeah. And, as I said, there are no ablations of most of the things. So


there are no ablations, for example, of this residual connection where you just randomly add


the query, like, why would you do that? Like, that doesn't even make sense. If you call this


a query, this thing, then by itself, it should carry no information to pass on by nature of being


a query. Right? So, you know, why do you why do you add it up there? You know, what's the effect


of the individual columns? How many there are? Right? So, you know, I think that's a good example


of why. You know, there are many things to ablate here to really show why this model performs well.


What they do is they compare sort of their runtime and the the runtime as the sequence


length increases. And as you can see, they're quite fast right here, which, I guess fast


transfer is this fast former, I guess fast transformer is fast former. So and the regular


transformer and they also are like a constant factor faster than others. But you know, are like,


are you a constant factor faster? Because you actually don't do any sort of attention.


I don't I don't know. So yeah, that those are my my two cents to this paper. Again, this might be


a neat model for certain tasks. It's certainly fast. It certainly doesn't make you run out of


memory as a regular transformer for a given set of tasks, it might in fact work better than a


transformer. My main problem here is with the whole framing in terms of attention. In terms of


the sort of same languages trying to pass this off as a faster transformer, which it is not.


Intro - TensorFlow Object Detection API Tutorial p.1

What is going on everybody and welcome to a TensorFlow Object Detection API overview


slash tutorial. So in this video all we're going to be doing is just doing it just the


quick setup of the TensorFlow Object Detection API. I'm going to try to I'm going to be doing


this on Windows mostly since my main focus of using the TensorFlow Object Detection API


is in games which run on Windows and not Ubuntu and not very well as often on Mac. So anyways


I'm going to be doing it on Windows. I will point out a few slight differences on Ubuntu


but it's basically the setup is the same. So we're just going to set it up and get to


the point where we can detect things in an image like this and then in the subsequent


videos we're going to show how you might customize this to your own images and then to your own


video and for my purposes the video will just will use you know streaming video from say


Grand Theft Auto. So anyway but you could change that to be your webcam or whatever


and actually maybe first I'll just do it with the webcam. Anyway enough on that let's set


this up. So they actually do have installation instructions and you could totally just follow


those. There's like a few things that will hang you up though like for example like installing


TensorFlow GPU is not that simple. It would be great if it was that simple. It's not.


So anyway so if you need instructions on installing TensorFlow GPU I've got tutorials on that.


So anyway one thing to note that's actually the reason why this is so useful is detecting


things in images has been like something you could do for a very long time but detecting